Multimodal ischemic stroke medical image segmentation method, device and equipment

Abstract

The application provides a multimodal ischemic stroke medical image segmentation method, device and equipment, which combines a mixed expert fine-tuning ViT encoder branch and a CNN encoder branch as a multimodal image double-branch encoding network to extract independent features of different modal medical images, further uses a modal missing adaptive fusion network to integrate complementary information of different modes, uses an encoder branch interaction network to interact global-local information between CNN features and ViT features, can decouple specific features of different available modes and cross-modal invariant features in the case of modal missing, fully utilizes the advantages of different types of features, deeply mines the value information of multimodal image features, and thus accurate multimodal ischemic stroke medical image segmentation and imaging are realized, and the reliability, accuracy and practicability are good.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a multi-modal ischemic stroke medical image segmentation method, device and equipment. BACKGROUND

[0002] Multi-modal computed tomography (CT) is one of the most commonly used imaging methods in clinical practice of ischemic stroke, which can comprehensively evaluate the degree of brain tissue damage and provide more reliable basis for clinical decision-making. Therefore, accurate segmentation of multi-modal CT images of ischemic stroke is of great significance in clinical practice and basic medicine. However, due to the fact that some patients cannot inject contrast agents and other reasons, multi-modal ischemic stroke medical image segmentation faces the challenge of missing modalities in actual clinical practice.

[0003] With the popularization of the basic segmentation all model (SAM) in the field of medical images, we pay attention to its great potential in the field of multi-modal ischemic stroke segmentation. Although existing medical image basic models usually use single-modal training method, the overall training process involves multiple modalities, and the model still contains rich multi-modal knowledge. Therefore, in-depth mining of multi-modal knowledge contained in the basic model has important research value for multi-modal ischemic stroke segmentation method.

[0004] However, most of the existing multi-modal ischemic stroke segmentation methods still mainly use U-shaped convolutional neural network (CNN), which simply concatenates multi-modal data at the input layer and then inputs a shared encoder to achieve segmentation. Although these methods have achieved certain segmentation effect, they ignore the heterogeneity between modalities, which may introduce irrelevant noise and fail to learn useful multi-modal complementary features. Although some methods use hierarchical fusion strategy, they still do not decouple cross-modal shared information and modality-specific information to fully utilize multi-modal complementary information, and do not consider the robustness of the method under the condition of missing modalities. Therefore, the above methods have the problems of low reliability and poor accuracy. SUMMARY

[0005] The present application proposes a multi-modal ischemic stroke medical image segmentation method, device and equipment, which can solve one of the problems in the background art.

[0006] To achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows:

[0007] In a first aspect, a multi-modal ischemic stroke medical image segmentation method is provided, comprising:

[0008] obtaining an ischemic stroke medical image to be segmented; and,

[0009] The trained multi-modal ischemic stroke medical image segmentation model is used to process the ischemic stroke medical image to be segmented, and a segmentation result is obtained.

[0010] The multi-modal ischemic stroke medical image segmentation model comprises, in sequence, an encoding part, a decoding part and a segmentation head.

[0011] The encoding part comprises:

[0012] A visual transformer (ViT) encoder branch is configured to, during training, extract global features of ischemic stroke medical images of different modalities based on a hybrid expert mechanism, wherein the hybrid expert comprises a scale expert and a multi-modal expert.

[0013] A first modality missing adaptive feature fusion module based on a transformer (Transformer) is configured to, during training, in the case of missing modalities, calculate and utilize modality-level voxel weights to adaptively fuse global features of available modalities to obtain global fusion features, wherein an input of the first modality missing adaptive feature fusion module is connected to an output of the ViT encoder branch.

[0014] A convolutional neural network (CNN) encoder branch is configured to, during training, extract local features of ischemic stroke medical images of different modalities, wherein an output of the CNN encoder branch is skip-connected to an input of the decoding part.

[0015] A second modality missing adaptive feature fusion module based on a CNN is configured to, during training, in the case of missing modalities, calculate and utilize modality-level voxel weights to adaptively fuse local features of available modalities to obtain local fusion features, wherein an input of the second modality missing adaptive feature fusion module is connected to an output of the CNN encoder branch.

[0016] An encoder branch interaction network is configured to, during training, fuse the global fusion features and the local fusion features to obtain an output of the encoding part, wherein an input of the encoder branch interaction network is connected to outputs of the first modality missing adaptive feature fusion module and the second modality missing adaptive feature fusion module, and an output of the encoder branch interaction network is connected to an input of the decoding part.

[0017] Based on the above technical scheme, the independent features of different modal medical images are extracted by combining the mixed expert fine-tuning ViT encoder branch and the CNN encoder branch as a multi-modal image double-branch encoding network, and further using a modal missing adaptive fusion network to integrate the complementary information of different modalities, using an encoder branch interaction network to interact the global-local information between the CNN features and the ViT features, and in the case of modal missing, the specific features and cross-modal invariant features of different available modalities can be decoupled, while fully utilizing the advantages of different types of features, deeply mining the value information of multi-modal image features, so as to realize accurate multi-modal ischemic stroke medical image segmentation and imaging, and the reliability, accuracy and practicability are high.

[0018] In a possible design manner of the first aspect, the ViT encoder branch includes a plurality of mixed expert fine-tuning perception encoding modules connected in sequence, and each mixed expert fine-tuning perception encoding module includes a first normalization layer, a multi-head attention layer of a convolution low-rank adapter (ConvLoRA) branch with a bypass, a second normalization layer, and a feedforward network layer with a parallel multi-modal expert branch.

[0019] In a possible design manner of the first aspect, the ConvLoRA branch includes a linear encoder, a mixed expert convolution module, and a linear decoder connected in sequence, the mixed expert convolution module includes a plurality of scale experts and a gating network for dynamically selecting the scale experts, the scale expert includes a first feature reconstruction layer, a convolution layer, and a second feature reconstruction layer connected in sequence,

[0020] The feedforward network layer includes a plurality of layers of a parallel multi-layer perception and a plurality of modal expert branches, each modal expert branch includes a first feature remodeling layer, a first point-wise convolution layer, a first deep convolution layer, a Gaussian error linear unit (GELU) activation function layer, a bilateral hemispheric disparity attention module, a second point-wise convolution layer, and a second feature remodeling layer connected in sequence, and the bilateral hemispheric disparity attention module includes a flip difference layer, a second deep convolution layer, a global average pooling layer, a Sigmoid activation function layer, and an attention residual fusion layer connected in sequence.

[0021] In a possible design manner of the first aspect, the CNN encoder branch includes a first group convolution module and a plurality of second group convolution modules connected in sequence, the first group convolution module includes a plurality of convolution blocks connected in series, and each second group convolution module includes a down-sampling module and a plurality of convolution blocks with a residual connection connected in series, and each convolution block includes a first 3D convolution layer, a group normalization layer, and a ReLU activation function layer connected in sequence.

[0022] In a possible design manner of the first aspect, the first modal missing adaptive feature fusion module comprises a first average value fusion module and a first adaptive fusion module connected in sequence, the first average value fusion module is configured to calculate a first average value representation of global features of available modalities, the first adaptive fusion module comprises a plurality of Transformer modules, a first Softmax function layer and a first fusion layer connected in sequence,

[0023] The second modal missing adaptive feature fusion module comprises a second average value fusion module and a second adaptive fusion module connected in sequence, the second average value fusion module is configured to calculate a second average value representation of local features of available modalities, and the second adaptive fusion module comprises a first convolution module, a second Softmax function layer, a second fusion layer and a second convolution module with an additional residual connection connected in sequence.

[0024] In a possible design manner of the first aspect, the encoder branch interaction network comprises a CNN feature enhancement branch, a ViT feature enhancement branch and a feature interaction branch, the CNN feature enhancement branch comprises a global max pooling layer, a first global average pooling layer, a 7×7×7 convolution layer and a first Sigmoid activation function layer connected in sequence, the ViT feature enhancement branch comprises a second global average pooling layer, a full connection layer and a second Sigmoid activation function layer connected in sequence, and the feature interaction branch comprises a 3×3×3 convolution layer, a ReLU activation function layer and a third point-wise convolution layer connected in sequence, and is configured to perform feature interaction between input CNN features and ViT features to realize complementary advantages of both.

[0025] In a possible design manner of the first aspect, the decoding part comprises a plurality of decoder modules in series, each decoder module comprises an upsampling module and a feature fusion module in series, the feature fusion module comprises a third convolution module and a fourth convolution module in series, the third convolution module comprises a first instance normalization layer, a second 3D convolution layer and a first LeakyReLU activation function layer connected in sequence, the first instance normalization layer is connected to the output of the CNN encoder branch in a skip connection manner, and the fourth convolution module is connected in a residual connection manner and comprises a second instance normalization layer, a third 3D convolution layer and a second LeakyReLU activation function layer connected in sequence.

[0026] In a possible design manner of the first aspect, the multi-modal ischemic stroke medical image segmentation model adopts a joint loss function comprising a Dice loss term and a cross-entropy loss term.

[0027] In a possible design manner of the first aspect, the multi-modal ischemic stroke medical image segmentation model adopts a joint loss function comprising a Dice loss term and a cross-entropy loss term.

[0028] An acquisition unit is configured to obtain an ischemic stroke medical image to be segmented; and

[0029] A segmentation unit is configured to process the ischemic stroke medical image to be segmented by using a trained multi-modal ischemic stroke medical image segmentation model to obtain a segmentation result,

[0030] The multi-modal ischemic stroke medical image segmentation model comprises, connected in sequence: an encoding part, a decoding part and a segmentation head,

[0031] The encoding part comprises:

[0032] A visual transformer (ViT) encoder branch is configured to, during training, extract global features of ischemic stroke medical images of different modalities based on a hybrid expert mechanism, wherein the hybrid expert comprises a scale expert and a multi-modal expert;

[0033] A first modal missing adaptive feature fusion module based on a transformer (Transformer) is configured to, during training, in the case of modal missing, adaptively fuse global features of available modalities by using voxel-level weights to obtain global fusion features, an input of the first modal missing adaptive feature fusion module being connected to an output of the ViT encoder branch;

[0034] A convolutional neural network (CNN) encoder branch is configured to, during training, extract local features of ischemic stroke medical images of different modalities, an output of the CNN encoder branch being skip-connected to an input of the decoding part;

[0035] A second modal missing adaptive feature fusion module based on a CNN is configured to, during training, in the case of modal missing, adaptively fuse local features of available modalities by using voxel-level weights to obtain local fusion features, an input of the second modal missing adaptive feature fusion module being connected to an output of the CNN encoder branch; and

[0036] An encoder branch interaction network is configured to, during training, fuse the global fusion features and the local fusion features to obtain an output of the encoding part, an input of the encoder branch interaction network being connected to outputs of the first modal missing adaptive feature fusion module and the second modal missing adaptive feature fusion module, and an output of the encoder branch interaction network being connected to an input of the decoding part.

[0037] In a third aspect, an electronic device is provided, including a processor, and a memory coupled to the processor, the memory configured to store a computer program; and the processor configured to execute the computer program stored in the memory, so that the electronic device performs the multi-modal ischemic stroke medical image segmentation method according to any possible implementation of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or related description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0039] Figure 1 is a multi-modal ischemic stroke medical image segmentation method flowchart provided by the embodiments of the present application;

[0040] Figure 2 is a whole architecture diagram of a multi-modal ischemic stroke medical image segmentation primary model provided by the embodiments of the present application;

[0041] Figure 3 is a mixed expert fine-tuning perception coding module structure diagram provided by the embodiments of the present application;

[0042] Figure 4 is a grouping convolution module structure diagram provided by the embodiments of the present application;

[0043] Figure 5 is a modal missing adaptive fusion network structure diagram provided by the embodiments of the present application;

[0044] Figure 6 is an encoder branch interaction network structure diagram provided by the embodiments of the present application;

[0045] Figure 7 is a decoder module structure diagram provided by the embodiments of the present application;

[0046] Figure 8 is an imaging method flowchart based on the multi-modal ischemic stroke medical image segmentation method provided by the embodiments of the present application. DETAILED DESCRIPTION

[0047] In order to make the purposes, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0048] It should be noted that although the functional modules are divided in the device schematic diagram, the logical order is shown in the flowchart, but in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the present application.

[0050] As Figure 1 shown, the multi-modal ischemic stroke medical image segmentation method provided by the embodiment includes the following steps:

[0051] S1. Obtain existing multi-modal CT images of ischemic stroke.

[0052] S2. Preprocess the image data obtained in step S1 to obtain a training data set; the preprocessing includes removing the skull region, multi-modal image registration, image cropping, image resampling and image standardization; specifically including the following steps:

[0053] The skull in the image is removed by using a convex optimization method;

[0054] Register other modal images to the NCCT image to eliminate the spatial differences of different modal images and make the coordinates of the same anatomical point consistent across images;

[0055] Crop the image to the brain tissue containing region to improve the segmentation efficiency;

[0056] Resample the cropped image to unify the voxel resolution of the image;

[0057] The image is standardized by using the Z-Score method;

[0058] Finally, the image preprocessing process is completed.

[0059] S3. Based on the basic model SegVoL encoder and convolutional layer, a multi-modal image double-branch encoding network and a corresponding modal missing adaptive fusion network are constructed, based on convolution and connection layer, an encoder branch interaction network is constructed, based on transposed convolution, an image segmentation network is constructed, and finally a multi-modal ischemic stroke medical image segmentation primary model including a multi-modal image double-branch encoding network, a modal missing adaptive fusion network, an encoder branch interaction network and an image segmentation network is constructed; including the following contents:

[0060] AsFigure 2 As shown, the multi-modal ischemic stroke medical image segmentation primary model is composed of a multi-modal image double-branch encoding network, a modal missing adaptive fusion network, an encoder branch interaction network and an image segmentation network; the multi-modal image double-branch encoding network includes a ViT encoder branch and a CNN encoder branch, which are respectively composed of a plurality of mixed expert fine-tuning perception encoding modules and a grouped convolution module, and are used for extracting independent ViT features and CNN features of different modalities; the modal missing adaptive fusion network includes a convolution-based modal missing adaptive feature fusion network and a Transformer-based modal missing feature adaptive fusion network, which are used for dynamically adjusting the multi-modal fusion strategy to adapt to different modal missing conditions, thereby improving the robustness of the model; the encoder branch interaction network is used for interactive fusion of ViT features and CNN features with multi-modal information, so as to utilize the local detail capturing ability of CNN and the global context modeling ability of ViT to obtain more effective infarction representation; the image segmentation network is composed of a plurality of decoder modules, which are used for processing image embedding data fused with multi-modal information and global-local information, gradually restoring the spatial resolution and outputting the final multi-modal ischemic stroke medical image segmentation result.

[0061] In specific implementation, the multi-modal image double-branch encoding network specifically includes the following contents:

[0062] The multi-modal image double-branch encoding network includes a mixed expert fine-tuning SegVoL ViT encoder branch and a five-stage CNN encoder branch; the ViT encoder branch is used for modeling global features of different modalities, and the CNN encoder branch is used for modeling local features of different modalities, so as to obtain independent image features of different modalities and different granularity levels.

[0063] The mixed expert fine-tuning ViT encoder branch specifically includes the following contents:

[0064] The ViT encoder branch is composed of 12 mixed expert fine-tuning perception encoding modules, which are used for multi-modal independent feature extraction.

[0065] As Figure 3As shown, the hybrid expert fine-tuning perceptual coding module consists of a first normalization layer, a multi-head attention layer with an introduced bypass ConvLoRA branch, a second normalization layer, and a feedforward network layer with a parallel multimodal expert branch. The reason for introducing hybrid expert fine-tuning is that the encoder parameters of the original base model SegVoL are large, and full-parameter fine-tuning suffers from high loss and overfitting. Therefore, a hybrid expert method is used to fine-tune its original Transformer block, thus constructing the hybrid expert fine-tuning perceptual coding module. Here, "hybrid expert" refers to two things: first, the scale expert in the ConvLoRA branch focusing on different receptive fields and resolutions; and second, the modality expert in the multimodal expert branch focusing on the differences in bilateral cerebral hemisphere features across different modalities. The specific calculation formula for the hybrid expert fine-tuning perceptual coding module is as follows:

[0066]

[0067]

[0068] in For the first Layered hybrid expert fine-tuning of the output of the perception coding module; For the first Layered hybrid expert fine-tuning of the output of the perception coding module For the first Attention characteristics of layers; For normalization layer, This indicates the introduction of a multi-head attention layer with a bypass ConvLoRA branch. A feedforward network layer with multimodal expert branches was introduced to enable parallelism.

[0069] In specific implementation, the multi-head attention layer that introduces the bypass ConvLoRA branch includes the following:

[0070] like Figure 3 As shown, the ConvLoRA branch consists of a pair of linear encoders and linear decoders, as well as the MoE-Conv module. It introduces the query vector and value vector of the multi-head attention mechanism to inject inductive bias into the attention mechanism, enhancing its ability to model local structures and thus reducing sample complexity. The linear encoder reduces the dimensionality of the input features, and the linear decoder restores the processed features to their original dimensions. The MoE-Conv module consists of multiple scale experts and a gating network for dynamic expert selection. Each scale expert consists of a convolutional layer and a pair of feature reconstruction layers. The feature reconstruction layers use an interpolation function to reconstruct features to a specific size, determined by a scaling factor. Decide; convolutional layers can directly capture local spatial structures, thus complementing the local inductive bias of attention mechanisms. Different scale experts have different receptive fields, focusing on feature extraction at different spatial resolutions; the gating network dynamically activates the appropriate expert based on the input image feature distribution, thus achieving adaptive processing of multi-scale information; for any scale expert, the processing process can be represented by the following formula:

[0071]

[0072] wherein represents the input feature, represents the output result of the th scale expert, is the scaling ratio of the th scale expert, represents the convolutional layer of the th scale modal expert, represents the feature reconstruction layer, which reconstructs the feature to the specified resolution size by interpolation; finally, the calculation formula of a ConvLoRA module is as follows:

[0073]

[0074] wherein represents the input feature; represents the output of the ConvLoRA module; is the lower projection weight, used to map the high-dimensional input to a low-dimensional subspace; is the upper projection weight, used to project the aggregation result of the expert back to the original dimension; is a gating network that only activates 1-bit value.

[0075] The fine-tuning of the ConvLoRA module for the multi-head attention layer can be represented as:

[0076]

[0077]

[0078] wherein represents the input feature, and represent the query vector and the value vector, respectively, and represent the frozen query vector projection weight and the frozen value vector projection weight, respectively, and represent the ConvLoRA module introduced for the query vector and the value vector, respectively; it is worth noting that for the key vector, we do not introduce the ConvLoRA module fine-tuning, and the specific calculation is , is a key vector, is a frozen key vector projection weight.

[0079] The processing procedure of the multi-head attention layer under the fine-tuning of the ConvLoRA branch can be expressed as:

[0080]

[0081] wherein is the processing result of the self-attention layer, is a linear projection function for weighted aggregation of the features of all attention heads; is a concatenation operation; is the feature of the H-th attention head; the feature of the first attention head is expressed as:

[0082]

[0083] wherein is a Softmax function, is the number of channels of the self-attention feature map, is matrix multiplication.

[0084] In a specific implementation, the feedforward network layer of the parallel multi-modal expert branch specifically includes the following contents:

[0085] The feedforward network layer is composed of an original multi-layer perceptron and a parallel multi-modal expert branch, and finally outputs the element-wise sum of the two branches:

[0086]

[0087] wherein is an input feature, denotes the original multi-layer perceptron, denotes the multi-modal expert branch.

[0088] The calculation formula of the original multi-layer perceptron is as follows:

[0089]

[0090] wherein Linear is defined as , is a Gaussian error linear unit.

[0091] The multi-modal expert branch is composed of multiple modal experts, and the specific process is: input features are distributed to experts of each modality in the batch dimension according to the corresponding order for processing, and then the results of these experts are aggregated in the batch dimension. Such design can decouple the style and content of modalities, so as to better learn and retain different modality specificity, and ensure minimal redundancy and maximum correlation of information between modalities.

[0092] As shown in Figure 3 The modal expert is a lightweight convolutional network, which is composed of a first feature reshaping layer, a first pointwise convolutional layer, a deep convolutional layer, a GELU activation function, a bilateral hemispheric difference attention module, a second pointwise convolutional layer and a second feature reshaping layer, and is used for learning specific features of different modalities in a compressed latent feature space; wherein the first feature reshaping layer is used for reshaping the original 1D sequence data into 3D image data, and the second feature reshaping layer is used for reshaping the 3D image data back to the original 1D sequence data; the first pointwise convolutional layer is used for mapping the features to a specified dimension, and correspondingly, the second pointwise convolutional layer is used for mapping the features back to the original dimension; the deep convolutional layer is used for compressing the spatial dimension and extracting structured features; the GELU activation function is used for nonlinear activation of the features; and the bilateral hemispheric difference attention module is used for capturing specific BHD features of different modalities. The specific process calculation formula is as follows:

[0093]

[0094] Wherein is the input feature reshaped into 3D image; represents pointwise convolution; represents deep convolution; is the Gaussian error linear unit activation function; represents the bilateral hemispheric difference attention module, which is used to capture the bilateral brain hemispheric difference of different modalities and highlight specific features of each modality. As shown in Figure 3As shown, the bilateral hemispheric difference attention module is composed of a flip difference layer, a 3 × 3 × 3 deep convolution layer, a global average pooling layer, a Sigmoid activation function and an attention residual fusion layer. Among them, the flip difference layer is used to obtain the preliminary bilateral brain hemispheric difference feature, specifically, it first flips the input original feature horizontally, then subtracts the input original feature from the horizontally flipped feature map and takes the absolute value; the deep convolution layer is used to calibrate the preliminary bilateral brain hemispheric difference feature; the global average pooling layer can globally aggregate the calibrated bilateral brain hemispheric difference feature in the spatial dimension, thereby generating global statistics for each channel to extract channel-level importance; the Sigmoid activation function is used to generate bilateral brain hemispheric difference attention weights; the attention residual fusion layer is used to further multiply the obtained bilateral brain hemispheric difference attention weights with the original feature for weighted fusion, highlight the specific BHD features of each modality, and accelerate network convergence through residual connection. The specific calculation formula is as follows:

[0095]

[0096] wherein represents the Sigmoid activation function; represents the global average pooling, which is defined as ; represents element-wise multiplication; represents flipping the feature map along the horizontal direction; represents the absolute value.

[0097] In specific implementation, the CNN encoder branch specifically includes the following contents:

[0098] As shown in Figure 2 , the CNN encoder branch is composed of five stages of grouped convolution modules for independent modeling of different modalities; as shown in Figure 4 , except that the first stage only contains two serial convolution blocks, each stage contains a down-sampling module realized by cross-step convolution and a serial two convolution blocks with residual connection; the convolution block is composed of 3D convolution, grouped normalization, ReLU activation function, and the calculation formula is as follows:

[0099]

[0100] wherein is the input feature, represents 3 × 3 × 3 grouped convolution, represents grouped normalization, represents ReLU activation function.

[0101] The calculation formula of the grouped convolution module of the second to fifth stage is as follows:

[0102]

[0103]

[0104] wherein is the output of the first stage, is the output of the first stage, , is the down-sampled feature of the first stage, is a down-sampling module implemented by a convolution with stride 2; in particular, the calculation formula of the grouped convolution module of the first stage is: ;

[0105] In a specific implementation, the modal missing adaptive fusion network specifically includes the following contents:

[0106] As shown in Figure 5 , the modal missing adaptive fusion network includes a convolution-based modal missing adaptive feature fusion module and a Transformer-based modal missing adaptive feature fusion module; the convolution-based modal missing adaptive feature fusion module and the Transformer-based modal missing adaptive feature fusion module are used to adaptively interact and fuse the available modal features of the CNN encoder branch and the ViT encoder branch respectively under different modal missing conditions.

[0107] The convolution-based modal missing adaptive feature fusion module specifically includes the following contents:

[0108] The convolution-based modal missing adaptive feature fusion module is composed of an average value fusion module and an adaptive fusion module, and is used to fuse the available modal features under different modal missing conditions for the output features of the CNN encoder branch.

[0109] The average value fusion module is used to average the features of all available modalities, and the specific process is represented as:

[0110]

[0111] wherein is the CNN feature of the first available modality, is the average feature of all available modalities, represents the number of available modalities.

[0112] The adaptive fusion module is used for further interaction and fusion of features of available modalities, and the specific processing process includes: concatenating the average feature with each available modality representation, and generating the correlation score of the current modality through a convolution module:

[0113]

[0114] wherein is a cascading operation, represents a 3*3*3 convolution layer, represents a Sigmoid activation function; then the correlation score is further subjected to Softmax along the modality dimension to obtain the voxel-level weight of the current modality:

[0115]

[0116] wherein is a Softmax function, when only one available modality exists, all voxel weights are 1; finally, all voxel weights are multiplied with all available modality representations element by element to obtain the fused representation, and the fused representation is fused with the average value feature through a convolution layer, and a residual connection is added to obtain the final multi-modality CNN fusion feature:

[0117]

[0118] wherein represents element by element multiplication.

[0119] In specific implementation, the modality missing adaptive feature fusion module based on the Transformer includes the following contents:

[0120] The modality missing adaptive feature fusion module based on the Transformer also consists of an average value fusion module and an adaptive fusion module, and is used for fusing the output features of the ViT encoder branch.

[0121] The average value fusion module is similar to the average value fusion module of the convolution-based modality missing adaptive feature fusion module, and is represented as:

[0122]

[0123] wherein is the CNN feature of the i-th available modality, is the average feature of all available modalities, represents the number of available modalities.

[0124] ​The adaptive fusion module consists of two Transformer modules and a Softmax function. The processing procedure is as follows: First, the ViT features of available modalities are subjected to average pooling with a stride of 2 to reduce computational complexity.

[0125]

[0126] The input ViT features. These are the ViT features after pooling. This is a pooling operation; then the pooled available modal ViT features are concatenated together:

[0127]

[0128] Indicates the characteristics after concatenation. Indicates the first One available modal feature, This is a cascaded operation; two Transformer modules are used to generate correlation scores for each modality from the concatenated features. The calculation formula for a single Transformer module is as follows:

[0129]

[0130]

[0131] in This indicates the bulls' self-attention. Indicates the output of the previous layer and This is a concatenation of all pooled available modal features along the image feature dimensions. For the attention features of the current layer, This is the output of the current layer. Indicates a feedforward network. The representation layer is normalized; subsequently, normalization yields the voxel weights for each modality. Similarly, when only one modality is available, all voxel weights are 1.

[0132]

[0133] This is the output of the last Transformer module. For the first The modal-normalized voxel weights are used; finally, the voxel weights of each modality are multiplied by the original input features voxel by voxel and summed to obtain the final fused features:

[0134]

[0135] in For the first The original input features of the available modalities, The average value of the N original available modal ViT features is obtained by passing through... This indicates element-wise multiplication.

[0136] In specific implementation, the encoder branch interaction network includes the following:

[0137] like Figure 6 As shown, the encoder branch interaction network consists of a CNN feature enhancement branch, a ViT feature enhancement branch, and a feature interaction branch. The CNN feature enhancement branch enhances the global receptive field of CNN features, improving the ability to model long-range dependencies. The ViT feature enhancement branch enhances the detail extraction capability of ViT features, improving the accuracy of lesion boundaries. The feature interaction branch utilizes the local detail features of the convolutional branch and the global context information of the ViT branch to improve feature representation capabilities, thereby improving segmentation accuracy. The calculation formula for the encoder branch interaction module is as follows:

[0138]

[0139] in These represent the outputs of the CNN feature enhancement branch, the ViT feature enhancement branch, and the feature interaction branch, respectively. This represents the output of the entire module. For cascading operations, This represents pointwise convolution.

[0140] The convolutional feature enhancement branch consists of global max pooling, global average pooling, a 7 × 7 × 7 convolutional layer, and a sigmoid activation function, calculated as follows:

[0141]

[0142] in Features of convolution branches and These represent the global max pooling and global average pooling of convolutional features, respectively. This represents element-wise multiplication. This represents a 7×7×7 convolutional layer. This represents the Sigmoid activation function.

[0143] The ViT enhancement branch consists of global average pooling, a fully connected layer, and a sigmoid activation function; the calculation formula is as follows:

[0144]

[0145] in, The features of the ViT branch, The global average pooling of the ViT features.

[0146] The feature interaction branch is obtained by concatenating the convolutional features and the ViT features in the channel, and then using a 3 × 3 × 3 convolutional layer and a pointwise convolution to capture the spatial and channel correlation, as follows:

[0147]

[0148] wherein represents a 3 × 3 × 3 convolutional layer, represents a ReLU activation function.

[0149] The above is the encoding part.

[0150] In specific implementation, the image segmentation network, also referred to as the decoding part, specifically includes the following contents:

[0151] As shown in Figure 2 The image segmentation network is composed of four-stage decoder modules;

[0152] In order to better utilize the multi-scale features in the convolutional encoder and gradually reduce the semantic gap, and produce more accurate segmentation results, as shown in Figure 7 The decoder module includes an upsampling module and a feature fusion module; the upsampling module is used to reduce the channel number while increasing the spatial resolution of the feature map, and the feature fusion module is gradually used to fuse the multi-scale features of the CNN encoder branch;

[0153] The upsampling module is implemented by transposed convolution, as follows:

[0154]

[0155] wherein is the output of the previous-stage decoder, , , represents a transposed convolutional layer with a kernel size of 2 × 2 × 2.

[0156] The feature fusion module is composed of two convolutional modules. Each convolutional module is composed of a 3D convolution, an instance normalization, and a LeakyReLU activation function; the second convolutional module is attached with a residual connection to optimize the network gradient flow and accelerate the model convergence; the specific calculation formula is as follows:

[0157]

[0158]

[0159] wherein represents a skip connection from the corresponding level CNN encoder branch, for the concatenation operation, represents a 3x3x3 convolutional layer, represents instance normalization, represents a LeakyReLU activation function, represents the intermediate features of the th decoder module, represents the output of the th decoder module.

[0160] The output of the last decoder module is passed through a segmentation head (1x1x1 convolutional layer) to map the number of channels to the number of classes, resulting in the final segmentation result.

[0161] S4. Using the training data set obtained in step S2, the multi-modal ischemic stroke medical image segmentation preliminary model constructed in step S3 is trained to obtain the final multi-modal ischemic stroke medical image segmentation model.

[0162] In specific implementation, the training process includes the following steps:

[0163] The predicted results output by the model and the real segmentation labels are input into the loss function, and the model parameters are optimized according to the results of the loss function.

[0164] The loss function is a joint loss function of Dice loss and cross-entropy loss, defined as follows:

[0165]

[0166] wherein is the weight of the loss function; is the Dice loss, and the calculation formula is as follows:

[0167]

[0168] wherein respectively represent true positive, false positive, and false negative; represents the cross-entropy loss function, and its definition is as follows:

[0169]

[0170] wherein N is the number of samples, is the predicted result output by the model, is the real label.

[0171] In the training process, the Adam optimizer is used to optimize the parameters of the model, and the poly strategy is used to adjust the learning rate.

[0172] S5. Obtain the actual target multi-modal medical image and input it into the multi-modal ischemic stroke medical image segmentation model obtained in step S4 to complete the segmentation of the actual multi-modal ischemic stroke medical image.

[0173] To illustrate the effectiveness of the segmentation method of the present embodiment, the method of the present embodiment will be compared with existing methods as follows:

[0174] nnFormer: This scheme proposes a strategy of combining Transformer and convolution. In the encoder, it uses three-dimensional SwinTransformer for feature extraction, but between adjacent SwinTransformer blocks, it uses convolution to downsample the feature map to obtain a multi-scale feature representation. In the decoder, it also uses three-dimensional SwinTransformer, and uses deconvolution to restore the resolution of the image.

[0175] PFormer: This scheme is based on UNet, which introduces a parallel hybrid module in the deep encoding stage, uses convolution blocks and modified three-dimensional SwinTransformer to learn local features and global dependencies respectively, and then unifies the output dimension to realize feature aggregation.

[0176] UNETR: This scheme uses ViT as an encoder to extract features, and then jumps the different scale features extracted by ViT to the decoder to calculate the final semantic segmentation output.

[0177] In the experiment, Dice and HD95 evaluation indexes are used to evaluate the performance of the method.

[0178] Among them, the Dice index is commonly used to evaluate the goodness of medical image segmentation, and its value is close to 1, indicating that the more accurate the segmentation result is; its calculation formula is as follows:

[0179]

[0180] Where TP, FN, and FP represent true positive, false negative, and false positive, respectively.

[0181] HD95 represents the distance between two sets, mainly used to evaluate the overlap of the segmentation boundary, and the smaller the value is, the better the segmentation performance is. The calculation formula is as follows:

[0182]

[0183] Where T and P represent the real label and the predicted segmentation result, respectively, and are the sets of curved points of T and P, respectively, represents the Euclidean distance.

[0184] The dataset used in the experiment is a multi-modal ischemic stroke dataset, covering modalities including NCCT, CTP (including four modes: CBF, CBV, MTT and TMax). The method of the embodiment and the comparative method are evaluated on the same test set, and the experimental results are shown in Table 1. The numbers in the parentheses are standard deviations.

[0185] Table 1: Comparison of segmentation performance of the method of the embodiment and the comparative method

[0186]

[0187] As can be seen from Table 1, the method proposed in the embodiment achieves the best performance in Dice and HD95 among all methods. The average Dice of all test samples obtained by model evaluation on the test set is 54.72 (24.52) %. The average Hd95 is 17.87 (13.65) mm.

[0188] At the same time, it can be seen from the table that the segmentation results of UNETR and nnFormer are not ideal, which may be due to the small size of the dataset. Although both of these two models are hybrid architectures, UNETR only uses ViT for feature extraction, which makes it difficult for the model to train when the training data is small. And nnFormer only uses convolution as the down-sampling module, so it also cannot get good segmentation results when the dataset is small. In contrast, the hybrid model PFomer models long-range dependencies and local information at the same time in the feature extraction stage, so it achieves relatively good performance, which shows that hybrid models have potential in multi-modal ischemic stroke medical image segmentation. However, since they all use input-level concatenation to obtain multi-modal information, noise and redundancy are introduced, and the complementary information between modalities cannot be effectively learned, so the segmentation performance still needs to be improved. The method of the embodiment benefits from the modal feature decoupling in hybrid expert fine-tuning, the unique CNN and ViT hybrid model design, and the effective adaptive feature fusion method, and obtains more accurate segmentation results.

[0189] In order to illustrate the effectiveness of the modal missing adaptive fusion method of the embodiment, the performance of the method of the embodiment and the existing method under different modal missing conditions is compared as follows:

[0190] mmFormer: This scheme contains three main components: a mixed modality specific encoder, which connects a convolutional encoder and an intra-modality Transformer, for local and global context modeling within each modality; an inter-modality Transformer, which builds and aligns long-range correlations across modalities to obtain modality-invariant features with global semantics corresponding to tumor regions; a decoder that progressively upsamples and fuses modality-invariant features to generate robust segmentation. In addition, auxiliary regularizers are introduced in both the encoder and the decoder, which can further enhance the model's robustness to incomplete modalities.

[0191] In the experiment, the Dice evaluation index is used to evaluate the performance of the method.

[0192] The data set used is a multi-modal ischemic stroke data set, covering modalities including NCCT, CTP (including four modes: CBF, CBV, MTT, and TMax). In the experiment, CBF and CBV are combined into a blood flow parameter map, and MTT and TMax are combined into a time parameter map. The method of this embodiment and mmFormer, which also has the ability to adapt to missing modalities, are evaluated under different missing conditions of blood flow parameter maps, time parameter maps, and NCCT modalities. The experimental results are shown in Table 2. The numbers in () are standard deviations, √ indicates that the modality image is included, and × indicates that the modality image is missing.

[0193] Table 2: Performance comparison of the method of this embodiment and the comparative method under different missing modalities

[0194]

[0195] From Table 2, it can be seen that the average Dice score of the method of this embodiment under all possible missing conditions is 48.91%. It is better than 47.21% of mmFormer, and the proposed method is better than mmFormer in all cases. This shows that the method of this embodiment learns more discriminative ischemic stroke modality features through mixed expert fine-tuning, and the designed modality missing adaptive fusion network is robust under different modality missing conditions.

[0196] As Figure 8 shown, the imaging method of the multi-modal ischemic stroke medical image segmentation method based on modality missing adaptation includes the following steps:

[0197] S1. Obtain existing multi-modal CT images of ischemic stroke;

[0198] S2. Preprocess the image data obtained in step S1 to obtain a training data set;

[0199] S3. Constructing a multi-modal image double-branch encoding network and a corresponding modal missing adaptive fusion network based on a base model SegVoL encoder and a convolutional layer, constructing an encoder branch interaction network based on a convolutional and connection layer, constructing an image segmentation network based on a transposed convolution, and finally constructing a multi-modal ischemic stroke medical image segmentation primary model including the multi-modal image double-branch encoding network, the modal missing adaptive fusion network, the encoder branch interaction network, and the image segmentation network;

[0200] S4. Training the multi-modal ischemic stroke medical image segmentation primary model constructed in step S3 by using the training data set obtained in step S2, to obtain a final multi-modal ischemic stroke medical image segmentation model.

[0201] S5. Obtaining an actual target multi-modal medical image and inputting the actual target multi-modal medical image into the multi-modal ischemic stroke medical image segmentation model obtained in step S4, to complete actual multi-modal ischemic stroke medical image segmentation.

[0202] S6. Labeling and secondary imaging the multi-modal ischemic stroke medical image segmentation result obtained in step S5 on the actual target multi-modal medical image, to obtain a medical image with the multi-modal ischemic stroke medical image segmentation result.

[0203] The imaging method provided in this embodiment can be used in an existing ischemic stroke medical image acquisition device or directly applied to a terminal. In specific use, the imaging method of this embodiment is fused into an existing medical image acquisition device or terminal, then an original multi-modal brain image is acquired by using an existing technology, then the imaging method is used to segment the ischemic stroke of the original multi-modal brain image, and the result is displayed in the image by different types of representation (color, etc.), to obtain a brain image with the ischemic stroke segmentation result, and secondary imaging and output are performed. The output brain image with the ischemic stroke medical image segmentation result can greatly facilitate subsequent work of clinicians, radiologists, and experimental personnel, etc.

[0204] The embodiment of the present application further provides a multi-modal ischemic stroke medical image segmentation device, comprising:

[0205] An acquisition unit is configured to obtain an ischemic stroke medical image to be segmented; and

[0206] A segmentation unit is configured to process the ischemic stroke medical image to be segmented by using the trained multi-modal ischemic stroke medical image segmentation model, to obtain a segmentation result.

[0207] The multi-modal ischemic stroke medical image segmentation model comprises, in sequence, an encoding part, a decoding part, and a segmentation head.

[0208] The encoding part comprises:

[0209] a visual transformer ViT encoder branch, configured to, during training, extract global features of ischemic stroke medical images of different modalities based on a hybrid expert mechanism, wherein the hybrid expert comprises a scale expert and a multi-modal expert;

[0210] a first modal missing adaptive feature fusion module based on a transformer, configured to, during training, in the case of modal missing, calculate voxel-level weights of each available modality, and perform adaptive fusion on global features of the available modalities to obtain global fusion features, an input of the first modal missing adaptive feature fusion module being connected to an output of the ViT encoder branch;

[0211] a convolutional neural network CNN encoder branch, configured to, during training, extract local features of ischemic stroke medical images of different modalities, an output of the CNN encoder branch being connected to an input of the decoding part in a skip connection manner;

[0212] a second modal missing adaptive feature fusion module based on CNN, configured to, during training, in the case of modal missing, calculate voxel-level weights of each available modality, and perform adaptive fusion on local features of the available modalities to obtain local fusion features, an input of the second modal missing adaptive feature fusion module being connected to an output of the CNN encoder branch; and,

[0213] an encoder branch interaction network, configured to, during training, fuse the global fusion features and the local fusion features to obtain an output of the encoding part, an input of the encoder branch interaction network being connected to outputs of the first modal missing adaptive feature fusion module and the second modal missing adaptive feature fusion module, and an output of the encoder branch interaction network being connected to an input of the decoding part.

[0214] The embodiment of the application further provides an electronic device, comprising a processor and a memory coupled with the processor, the memory being used to store a computer program, and the processor being used to execute the computer program stored in the memory, so that the electronic device executes the method according to any one of the above-mentioned embodiments.

[0215] The electronic device can be a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The electronic device can include, but is not limited to, a processor and a memory.

[0216] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor is a control center of the electronic device, and connects all parts of the device through various interfaces and lines.

[0217] The memory can be used to store the computer program, and the processor realizes various functions of the electronic device by running or executing the computer program stored in the memory and calling the data stored in the memory.

[0218] The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required by a function, etc. The data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0219] The embodiments of the present application also provide a storage medium, which is a computer readable storage medium, and the computer program is stored in the computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be realized. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0220] The embodiment of the present application further provides a computer program product, comprising a computer program or instructions, which, when running on a computer, enable the computer to perform the method of any possible implementation manner described above.

[0221] The above is the preferred embodiment of the present application, it should be pointed out that, for those skilled in the technology field, without departing from the principle of the present application, can make several improvements and refinements, these improvements and refinements also be considered as the protection scope of the present application.

Claims

1. A multimodal ischemic stroke medical image segmentation method, characterized in that, include: Obtain medical images of ischemic stroke to be segmented; as well as, A trained multimodal ischemic stroke medical image segmentation model is used to process the ischemic stroke medical images to be segmented, and the segmentation results are obtained. The multimodal ischemic stroke medical image segmentation model comprises, in sequence, an encoding part, a decoding part, and a segmentation head. The encoded portion includes: The ViT encoder branch of the visual transformer is used to extract global features from medical images of ischemic stroke of different modalities based on a hybrid expert mechanism during training. The hybrid experts include scale experts and multimodal experts. The first modality missing adaptive feature fusion module based on the Transformer is used to calculate and utilize the voxel-level weights of each available modality during training to adaptively fuse the global features of the available modalities in the case of modality missing, so as to obtain global fused features. The input of the first modality missing adaptive feature fusion module is connected to the output of the ViT encoder branch. A convolutional neural network (CNN) encoder branch is used to extract local features from medical images of ischemic stroke of different modalities during training. The output of the CNN encoder branch is connected to the input of the decoding part in a jump. A second modality-missing adaptive feature fusion module based on CNN is used during training, in the case of modality missing, to calculate and utilize the voxel-level weights of each available modality to adaptively fuse the local features of the available modalities, obtaining locally fused features. The input of the second modality-missing adaptive feature fusion module is connected to the output of the CNN encoder branch; and... An encoder branch interaction network is used to fuse the global fusion features and the local fusion features during training to obtain the output of the encoding part. The input of the encoder branch interaction network is connected to the output of the first modality missing adaptive feature fusion module and the second modality missing adaptive feature fusion module, and the output of the encoder branch interaction network is connected to the input of the decoding part.

2. The multimodal ischemic stroke medical image segmentation method as described in claim 1, characterized in that, The ViT encoder branch includes several hybrid expert fine-tuning perceptual coding modules connected in sequence. The hybrid expert fine-tuning perceptual coding module includes: a first normalization layer, a multi-head attention layer with a convolutional low-rank adapter ConvLoRA branch that introduces bypass, a second normalization layer, and a feedforward network layer that introduces a parallel multimodal expert branch.

3. The multimodal ischemic stroke medical image segmentation method as described in claim 2, characterized in that, The ConvLoRA branch comprises, in sequence, a linear encoder, an expert hybrid convolutional module, and a linear decoder. The expert hybrid convolutional module includes several scale experts and a gating network for dynamically selecting the scale experts. The scale experts include, in sequence, a first feature reconstruction layer, a convolutional layer, and a second feature reconstruction layer. The feedforward network layer includes a parallel multilayer perceptron and several modality expert branches. The modality expert branches include a first feature reshaping layer, a first pointwise convolutional layer, a first deep convolutional layer, a Gaussian error linear unit activation function layer, a bilateral hemisphere difference attention module, a second pointwise convolutional layer, and a second feature reshaping layer, which are connected in sequence. The bilateral hemisphere difference attention module includes a flip difference layer, a second deep convolutional layer, a global average pooling layer, a Sigmoid activation function layer, and an attention residual fusion layer, which are connected in sequence.

4. The multimodal ischemic stroke medical image segmentation method as described in claim 1, characterized in that, The CNN encoder branch includes a first grouped convolutional module and several second grouped convolutional modules connected in sequence. The first grouped convolutional module includes several convolutional blocks connected in sequence. The second grouped convolutional module includes a downsampling module connected in sequence and several convolutional blocks connected with residuals. Each convolutional block includes a first 3D convolutional layer, a grouped normalization layer and a ReLU activation function layer connected in sequence.

5. The multimodal ischemic stroke medical image segmentation method as described in claim 1, characterized in that, The first modality missing adaptive feature fusion module includes a first average fusion module and a first adaptive fusion module connected in sequence. The first average fusion module is used to calculate the first average representation of the global features of the available modalities. The first adaptive fusion module includes, in sequence, several Transformer modules, a first Softmax function layer, and a first fusion layer. The second modality missing adaptive feature fusion module includes a second average fusion module and a second adaptive fusion module connected in sequence. The second average fusion module is used to calculate the second average representation of the local features of the available modalities. The second adaptive fusion module includes a first convolution module, a second softmax function layer, a second fusion layer, and a second convolution module with additional residual connections connected in sequence.

6. The multimodal ischemic stroke medical image segmentation method as described in claim 1, characterized in that, The encoder branch interaction network includes: a CNN feature enhancement branch, a ViT feature enhancement branch, and a feature interaction branch. The CNN feature enhancement branch includes, in sequence: a global max pooling layer, a first global average pooling layer, a 7×7×7 convolutional layer, and a first sigmoid activation function layer. The ViT feature enhancement branch includes, in sequence: a second global average pooling layer, a fully connected layer, and a second sigmoid activation function layer. The feature interaction branch includes, in sequence: a 3×3×3 convolutional layer, a ReLU activation function layer, and a third pointwise convolutional layer. The feature interaction branch is used to perform feature interaction between the input CNN features and ViT features to achieve complementary advantages between the two.

7. The multimodal ischemic stroke medical image segmentation method as described in claim 1, characterized in that, The decoding section includes several decoder modules connected in series. Each decoder module includes a serial upsampling module and a feature fusion module. The feature fusion module includes a serial third convolution module and a fourth convolution module. The third convolution module includes a first instance normalization layer, a second 3D convolution layer, and a first LeakyReLU activation function layer connected in sequence. The first instance normalization layer is skipped to the output of the CNN encoder branch. The fourth convolution module includes residual connections and includes a second instance normalization layer, a third 3D convolution layer, and a second LeakyReLU activation function layer connected in sequence.

8. The multimodal ischemic stroke medical image segmentation method as described in claim 1, characterized in that, The multimodal ischemic stroke medical image segmentation model employs a joint loss function that includes a Dice loss term and a cross-entropy loss term.

9. A multimodal medical image segmentation device for ischemic stroke, characterized in that, include: The acquisition unit is used to acquire medical images of ischemic stroke to be segmented. as well as, The segmentation unit is used to process the ischemic stroke medical image to be segmented using a trained multimodal ischemic stroke medical image segmentation model to obtain the segmentation result. The multimodal ischemic stroke medical image segmentation model comprises, in sequence, an encoding part, a decoding part, and a segmentation head. The encoded portion includes: The ViT encoder branch of the visual transformer is used to extract global features from medical images of ischemic stroke of different modalities based on a hybrid expert mechanism during training. The hybrid experts include scale experts and multimodal experts. The first modality missing adaptive feature fusion module based on the Transformer is used to calculate and utilize the voxel-level weights of each available modality during training to adaptively fuse the global features of the available modalities in the case of modality missing, so as to obtain global fused features. The input of the first modality missing adaptive feature fusion module is connected to the output of the ViT encoder branch. A convolutional neural network (CNN) encoder branch is used to extract local features from medical images of ischemic stroke of different modalities during training. The output of the CNN encoder branch is connected to the input of the decoding part in a jump. A second modality-missing adaptive feature fusion module based on CNN is used during training, in the case of modality missing, to calculate and utilize the voxel-level weights of each available modality to adaptively fuse the local features of the available modalities, obtaining locally fused features. The input of the second modality-missing adaptive feature fusion module is connected to the output of the CNN encoder branch; and... An encoder branch interaction network is used to fuse the global fusion features and the local fusion features during training to obtain the output of the encoding part. The input of the encoder branch interaction network is connected to the output of the first modality missing adaptive feature fusion module and the second modality missing adaptive feature fusion module, and the output of the encoder branch interaction network is connected to the input of the decoding part.

10. An electronic device, characterized in that, The electronic device includes: a processor, and a memory coupled to the processor. The memory is used to store computer programs; and The processor is configured to execute the computer program stored in the memory, so that the electronic device performs the multimodal ischemic stroke medical image segmentation method as described in any one of claims 1-8.

Images