Multi-modal medical image segmentation system and method based on modal complementation and state space modeling

Abstract

The invention discloses a multi-modal medical image segmentation system based on modal complementation and state space modeling, and the system is constructed based on a 3D U-Net architecture, and comprises a first residual module, a second residual module, a first connection module and a segmentation head which are connected in sequence, N encoders which are formed by connecting convolution modules and down-sampling modules in sequence, each decoder is formed by sequentially connecting a residual error module, a connection module and a transposition module. One path of 3D input data is input to the depth channel convolution module, and the other path of 3D input data is input to the first residual module. And a transpose module output signal of the Nth decoder and a second residual module output signal are input into the first connection module together, are connected by the first connection module and then are output into the segmentation head, and the segmentation head outputs a segmentation signal of the multi-modal data. According to the method, the complementary characteristic between different modes in the multi-mode medical image is fully utilized, and the global context modeling and local structure sensing capabilities are considered at the same time.

Description

Technical Field

[0001] This invention relates to the field of medical image analysis and processing, and in particular to a multimodal medical image segmentation system and method based on modal complementarity and state space modeling. Background Technology

[0002] Currently, medical image segmentation plays a crucial role in clinical practice, including assisting diagnosis, developing treatment plans, and implementing treatments. Traditional medical image segmentation relies on experienced physicians, which is a labor-intensive and time-consuming process, and is easily affected by subjective interpretation and observer variability. Automated medical image segmentation methods can improve efficiency, thereby helping doctors make more accurate and faster diagnoses. Tumors are among the most common and fatal diseases of the central nervous system, and the accurate localization and segmentation of brain tumor regions are of great significance for clinical diagnosis, treatment planning, and postoperative evaluation.

[0003] In the diagnosis and treatment of brain tumors, accurate segmentation of tumor regions is crucial for both diagnosis and treatment. Multimodal MRI provides rich tissue contrast information through different modalities (such as T1, T1ce, T2, FLAIR, etc.). Different modalities emphasize different aspects in presenting tumor parenchyma, necrotic areas, and edematous structures, providing a data foundation for automated brain tumor segmentation. Currently, deep learning-based methods have become the mainstream for brain tumor segmentation. Structures such as 3D U-Net and V-Net have achieved some success in spatial context modeling, but existing technologies have the following shortcomings:

[0004] 1. Simple multimodal data fusion methods (such as direct splicing or stacking) make it difficult to explicitly model the structural complementarity of prior knowledge between modalities, which can easily lead to feature redundancy or weakening of key lesion features.

[0005] 2. Convolutional neural networks (CNNs) are limited by local receptive fields, making it difficult to capture long-range dependencies. When dealing with brain tumor regions that are structurally complex and have drastic morphological changes, they fail to transmit sufficient semantic information.

[0006] 3. Although self-attention architectures such as Transformer can model global context, they suffer from high computational cost and lack of multi-scale information in 3D medical images.

[0007] To address these issues, this invention proposes a multimodal 3D medical image segmentation method that integrates modal complementarity and state-space modeling mechanisms, focusing on the modeling and optimization of brain tumor regions with strong structural heterogeneity, blurred boundaries, and large spans. Summary of the Invention

[0008] This invention provides a multimodal medical image segmentation system and method based on modal complementarity and state space modeling to solve the technical problems existing in the prior art.

[0009] The technical solution adopted by this invention to solve the technical problems existing in the prior art is as follows:

[0010] A multimodal medical image segmentation system based on modal complementarity and state space modeling is constructed based on the 3DU-Net architecture, which includes: a first residual module, a second residual module, a first connection module and a segmentation head connected in sequence; N encoders composed of convolutional modules and downsampling modules connected in sequence; N decoders composed of residual modules, connection modules and transpose modules connected in sequence; a depth convolutional module, a third residual module and a first transpose module;

[0011] The N encoders are referred to as the first to the Nth encoders according to the signal transmission order; the N decoders are referred to as the first to the Nth decoders according to the signal transmission order.

[0012] The 3D input data is divided into two inputs: one input is sent to the depth convolution module; the other input is sent to the first residual module.

[0013] The signal output from the depth convolution module is divided into two paths: one path is input to the convolution module of the first encoder; the other path is input to the residual module of the Nth decoder.

[0014] The signal output from the i-th encoder is divided into two paths, i=1,2,…,N-1; one path is input to the convolution module of the (i+1)-th encoder; the other path is input to the residual module of the Ni-th decoder.

[0015] The signal output by the Nth encoder is input to the third residual module; the signal output by the third residual module is transposed by the first transpose module and then input together with the signal output by the residual module in the first decoder into the connection module of the first decoder. After being connected by the connection module of the first decoder, it is output to the transpose module of the first decoder.

[0016] The signal output from the transpose module of the k-th decoder and the signal output from the residual module of the (k+1)-th decoder are input together into the connection module of the (k+1)-th decoder. After being connected by the connection module of the (k+1)-th decoder, the signal is output to the transpose module of the (k+1)-th decoder, k=1,2,…,N-1;

[0017] The signal output from the transpose module of the Nth decoder and the signal output from the second residual module are input together into the first connection module. After being connected by the first connection module, they are output to the segmentation head, and the segmentation head outputs the segmented signal of the multimodal data.

[0018] Furthermore, it also includes a boundary enhancement and uncertainty evaluation mechanism module. After processing the 3D input data, the boundary enhancement and uncertainty evaluation mechanism module inputs the data to the depth convolution module and the first residual module, respectively.

[0019] The expression for the boundary reinforcement and uncertainty evaluation mechanism module is as follows:

[0020] ;

[0021] ;

[0022] In the formula:

[0023] f represents a given single-modal 3D input image;

[0024] u represents the uncertainty evaluation value, which is used to measure the model's confidence in the current prediction region;

[0025] This represents the probability after activation by the Sigmoid function, with values ​​ranging from [0,1].

[0026] This represents the output feature map of the boundary enhancement and uncertainty assessment module;

[0027] This represents the Sigmoid function, which performs a non-linear mapping on the input feature map f, giving it a probabilistic meaning.

[0028] This represents a three-dimensional convolution operation with a kernel size of 3×3×3.

[0029] This represents a function that normalizes the input feature mapping f, used to stabilize training and enhance the consistency of feature distribution.

[0030] Furthermore, a modal complementarity prior module is provided at the front end of each encoder. The modal complementarity prior module includes a modal-specific encoding module, a modal perception fusion module, and a modal residual calibration connection module connected in sequence.

[0031] set up For input data, Where C represents the total number of modes, , C; The modality-specific encoding module processes the input data as shown in the following formula:

[0032] ;

[0033] In the formula:

[0034] m represents the input data mode number;

[0035] This represents the feature representation extracted from the m-th modality through a convolution operation;

[0036] Let m represent the input feature tensor of the m-th mode; Where D represents depth; H represents height; and W represents width;

[0037] This represents a 3D convolution operation for the m-th modal input, with a kernel size of 3×3×3;

[0038] This represents the feature extraction function for the m-th mode;

[0039] To extract complementary information between modalities, all modal features are concatenated and fed into an attention mechanism for weighted fusion. The modality-aware fusion module performs weighted fusion processing on all modality feature data according to the following formula:

[0040] , in, ;

[0041] In the formula:

[0042] This represents the result of weighted fusion of multimodal features, which serves as the input for subsequent state modeling and segmentation.

[0043] The weight coefficient of the m-th modality in the fusion process is calculated through a softmax-type attention mechanism and is used to dynamically allocate the importance of different modalities.

[0044] Note the parameters indicating learnable channels;

[0045] Indicates the number of channels;

[0046] The modal residual calibration connection module is used to calibrate the residual structural deviations between modes, and it processes the feature data according to the following formula:

[0047] ;

[0048] In the formula:

[0049] This represents the alignment feature representation after processing by the modal residual calibration connection module, used to eliminate residual structural deviations between different modal features;

[0050] This represents the result of multimodal feature fusion.

[0051] This represents the feature representation of the T-th modality after convolution or feature extraction.

[0052] This represents a one-dimensional point convolution operation, used for inter-modal feature dimension mapping and residual calibration;

[0053] This indicates the input feature mode number.

[0054] Furthermore, the convolution module of each encoder includes a Mamba module and a discrete state space module connected in sequence;

[0055] The Mamba module includes: a GSC module, a first normalization layer, a Mamba layer, a second normalization layer, an MLP module, and multiple convolutional units composed of normalization layers and convolutional kernels, connected in sequence.

[0056] Each convolutional unit has a different kernel size; each convolutional unit processes the input features in parallel and the processed features are fused together, let the fused output feature be A;

[0057] The features processed by the GSC module are fused together with the features processed by the Mamba layer, and the output feature after fusion is B.

[0058] Feature A, feature B, and the feature processed by the MLP module are fused together to output feature C.

[0059] Furthermore, the GSC module includes three convolutional units, referred to as convolutional unit A, convolutional unit B, and convolutional unit C, respectively; the kernel sizes of convolutional units A and B are the same and both are larger than the kernel size of convolutional unit C.

[0060] The data to be processed is simultaneously input into convolutional unit A and convolutional unit C. The features output by convolutional unit A and convolutional unit C are fused and then input into convolutional unit B. The features output by convolutional unit B are fused with the data to be processed and then output.

[0061] Furthermore, the discrete state space module is a state space module constructed based on a multi-resolution state feature pyramid.

[0062] Furthermore, the discrete state space module is constructed based on a state feature pyramid with three levels, which are referred to as: high-level state feature pyramid, intermediate-level state feature pyramid and low-level state feature pyramid.

[0063] Advanced state feature pyramids are used to extract features with relatively high resolution and relatively small time scale.

[0064] The intermediate state feature pyramid is used to extract features with relatively centered resolution and time scale.

[0065] Low-level state feature pyramids are used to extract features with relatively low resolution and relatively large time scale.

[0066] The state equations and output equations of the discrete state-space module are shown below:

[0067] ;

[0068] ;

[0069] ;

[0070] ;

[0071] In the formula:

[0072] t represents time.

[0073] t-1 represents the time preceding time t;

[0074] Represents the state vector at time t;

[0075] Represents the state vector at time t-1;

[0076] The state at time t is represented by the high-level state feature pyramid.

[0077] The state at time t is the intermediate state feature pyramid.

[0078] The low-level state feature pyramid represents the state at time t.

[0079] Represents the state transition matrix;

[0080] This represents the input features at time t;

[0081] This represents the input mapping matrix at time t, used to map the input features x. t Mapped to a multi-resolution state space;

[0082] The autoregressive transition parameters of the high-level state feature pyramid at time t;

[0083] The autoregressive transition parameters of the intermediate state feature pyramid at time t are represented.

[0084] The autoregressive transition parameters of the low-level state feature pyramid at time t are represented.

[0085] The input mapping coefficients represent the high-level state feature pyramid at time t.

[0086] The input mapping coefficients represent the intermediate state feature pyramid at time t.

[0087] The input mapping coefficients represent the low-level state feature pyramid at time t.

[0088] Represents the coupling weights between high-resolution and medium-resolution states, used to model cross-layer information interaction;

[0089] Represents the coupling weights between low-resolution and medium-resolution states, used to model cross-layer information interaction;

[0090] Represents the coupling weights between medium-resolution and low-resolution states, used to model cross-layer information interaction;

[0091] This represents the interaction weight matrix between the high-level state and the input features.

[0092] This invention also provides a multimodal medical image segmentation method based on modal complementarity and state space modeling, utilizing the aforementioned multimodal medical image segmentation system based on modal complementarity and state space modeling. The method includes the following steps: setting a modal complementarity prior module at the front end of each encoder. The modal complementarity prior module includes a modality-specific encoding module, a modality-aware fusion module, and a modality residual calibration connection module connected in sequence, so as to combine the shared representation with the modality-specific path and realize the collaborative modeling of cross-modal shared information and difference information.

[0093] Furthermore, a boundary enhancement and uncertainty evaluation mechanism module is set up to improve the boundary sensitivity and region discrimination of the input. After processing the 3D input data using the boundary enhancement and uncertainty evaluation mechanism module, the data is then input into the depth convolution module and the first residual module, respectively. The expression of the boundary enhancement and uncertainty evaluation mechanism module is as follows:

[0094] The expression for the boundary reinforcement and uncertainty evaluation mechanism module is as follows:

[0095] ;

[0096] ;

[0097] In the formula:

[0098] f represents a given single-modal 3D input image;

[0099] u represents the uncertainty evaluation value, which is used to measure the model's confidence in the current prediction region;

[0100] This represents the probability after activation by the Sigmoid function, with values ​​ranging from [0,1].

[0101] This represents the output feature map of the boundary enhancement and uncertainty assessment module;

[0102] This represents the Sigmoid function, which performs a non-linear mapping on the input feature map f, giving it a probabilistic meaning.

[0103] This represents a three-dimensional convolution operation with a kernel size of 3×3×3.

[0104] This represents a function that normalizes the input feature mapping f, used to stabilize training and enhance the consistency of feature distribution.

[0105] Furthermore, by integrating the Mamba module and a multi-resolution state feature pyramid structure, hierarchical feature representations are constructed from different temporal and / or spatial resolutions.

[0106] The advantages and positive effects of this invention are:

[0107] 1. By introducing boundary enhancement and uncertainty evaluation mechanisms, the boundary sensitivity and regional discrimination of the input are effectively improved.

[0108] 2. In the encoder part, in order to make full use of the complementary information between modalities in terms of tissue boundaries, texture details and lesion contrast, a modal complementary prior modeling mechanism was designed, which combines shared representation with modal-specific paths to achieve collaborative modeling of cross-modal shared information and differential information.

[0109] 3. In the deep semantic modeling stage, the lightweight Mamba module and the multi-resolution state pyramid structure have been integrated into the remote dependency model, and hierarchical feature representations have been constructed from different temporal and / or spatial resolutions. Attached Figure Description

[0110] Figure 1 This is a schematic diagram of the structure of a multimodal medical image segmentation system based on modal complementarity and state space modeling according to the present invention.

[0111] Figure 2This is a schematic diagram of the structure of a boundary enhancement and uncertainty evaluation mechanism module of the present invention.

[0112] Figure 3 This is a schematic diagram of the structure of an MCPM according to the present invention.

[0113] Figure 4 This is a schematic diagram of the structure of a Mamba module according to the present invention.

[0114] Figure 5 This is a schematic diagram of the structure of a GSC module according to the present invention.

[0115] Figure 6 This is a schematic diagram of the structure of a Mamba module and a parallel correlation scan module according to the present invention.

[0116] In the picture:

[0117] C×D×H×W: Represents the four-dimensional dimensions of the input data.

[0118] C: Represents the number of modes.

[0119] D: Represents the depth of the volume data.

[0120] H: Represents the height of the volume data.

[0121] W: Represents the width of the volume data.

[0122] A: Represents the state transition matrix in the state-space model.

[0123] X: Represents the input feature vector or input sequence.

[0124] B: Represents the input mapping matrix, used to map input features to the state space.

[0125] Y: Represents the output vector of the state-space model.

[0126] E: Represents the weights of the system's state variables mapped to the output at time t;

[0127] Represents: sigmoid function.

[0128] Conv1×1×1: Represents a 3D convolution operation with a kernel size of 1×1×1, typically used for channel dimension mapping and feature compression.

[0129] Conv3×3×3: This indicates a 3D convolution operation with a kernel size of 3×3×3, used for local spatial feature extraction.

[0130] Conv5×5×5: This indicates a 3D convolution operation with a kernel size of 5×5×5, used to expand the receptive field to capture a wider range of contextual information.

[0131] Conv7×7×7: This indicates a 3D convolution operation with a kernel size of 7×7×7, used for mesoscale global feature extraction.

[0132] Conv9×9×9: This indicates a 3D convolution operation with a kernel size of 9×9×9, used for long-distance dependency feature modeling. Detailed Implementation

[0133] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0134] The Chinese definitions of the following English words, phrases, and abbreviations in this application are as follows:

[0135] BraTS: Brain Tumor Segmentation Challenge.

[0136] SAM: Segment Anything Model, a transferable general segmentation model.

[0137] MRI: Magnetic Resonance Imaging.

[0138] BEUE: Boundary Enhancement and Uncertainty Evaluation.

[0139] MCPM: Modality Complementarity Prior Modeling.

[0140] Mamba: A novel state-space and sequence modeling structure that can efficiently extract long-range dependency features.

[0141] Seg-head / SegHead: The segmentation head, used to output the final segmentation result.

[0142] SSM: State Space Model.

[0143] PCS: Parallel Correlated Scanning.

[0144] GSC: Gated Spatial Convolution.

[0145] Norm: Normalization operations, including layer normalization, batch normalization, etc.

[0146] Transformer: A neural network architecture based on attention mechanism, used for global feature modeling.

[0147] FLAIR: Fluid Attenuated Inversion Recovery Sequence MRI.

[0148] 3D-UNet: A three-dimensional U-Net network structure used for medical image segmentation.

[0149] Deep-wise Convolution: Depthwise Convolution.

[0150] Conv. Block: The Convolutional Block consists of convolutional layers, normalization layers, and activation functions.

[0151] Down-sampling: A downsampling operation used to reduce the size of feature maps and increase the receptive field.

[0152] 3D Input: Input medical image data in three dimensions.

[0153] Res Block: Residual Block, which contains convolutional units with residual connections.

[0154] Feature: Feature Map.

[0155] Transpose Conv.: Deconvolution or transpose convolution operation, used for upsampling.

[0156] Concatenation: A feature concatenation operation that merges multiple feature maps along the channel dimension.

[0157] Estimation: Represents a prediction or estimation module, which typically outputs boundary or uncertainty-related features.

[0158] Modality Aware Fusion: A modality-aware fusion module used for weighted fusion of multimodal features.

[0159] Replace: The replacement operation is used for alternative designs between modules.

[0160] Layer Normalization: Layer normalization.

[0161] Projection: A projection operation used for transforming feature dimensions.

[0162] Skip Connection: Used for fusing features from deep and shallow layers.

[0163] Please see Figures 1 to 6 A multimodal medical image segmentation system based on modal complementarity and state space modeling is proposed. The system is built on a 3D U-Net architecture and includes: a first residual module, a second residual module, a first connection module and a segmentation head connected in sequence; N encoders composed of convolution modules and downsampling modules connected in sequence; N decoders composed of residual modules, connection modules and transpose modules connected in sequence; a depth convolution module, a third residual module and a first transpose module.

[0164] The N encoders are referred to as the first to the Nth encoders according to the signal transmission order; the N decoders are referred to as the first to the Nth decoders according to the signal transmission order.

[0165] The 3D input data is divided into two inputs: one input is sent to the depth convolution module; the other input is sent to the first residual module.

[0166] The signal output from the depth convolution module is divided into two paths: one path is input to the convolution module of the first encoder; the other path is input to the residual module of the Nth decoder.

[0167] The signal output from the i-th encoder is divided into two paths, i=1,2,…,N-1; one path is input to the convolution module of the (i+1)-th encoder; the other path is input to the residual module of the Ni-th decoder.

[0168] The signal output by the Nth encoder is input to the third residual module; the signal output by the third residual module is transposed by the first transpose module and then input together with the signal output by the residual module in the first decoder into the connection module of the first decoder. After being connected by the connection module of the first decoder, it is output to the transpose module of the first decoder.

[0169] The signal output from the transpose module of the k-th decoder and the signal output from the residual module of the (k+1)-th decoder are input together into the connection module of the (k+1)-th decoder. After being connected by the connection module of the (k+1)-th decoder, the signal is output to the transpose module of the (k+1)-th decoder, k=1,2,…,N-1;

[0170] The signal output from the transpose module of the Nth decoder and the signal output from the second residual module are input together into the first connection module. After being connected by the first connection module, they are output to the segmentation head, and the segmentation head outputs the segmented signal of the multimodal data.

[0171] U-Net's architecture is U-shaped and consists of three parts: an encoder on the left, a decoder on the right, and gray skip connections in the middle.

[0172] The structure of U-Net can be divided into two parts: the shrinking path (encoder) and the expanding path (decoder).

[0173] Shrinking Path (Encoder): The encoder can consist of a series of convolutional layers, activation functions (such as ReLU), and max pooling layers. Each convolutional layer extracts image features and reduces the spatial dimensionality of the image. As the model progresses, the extracted features become more advanced and abstract.

[0174] Extended Path (Decoder): The decoder can consist of a series of upsampling, convolutional layers, and activation functions (such as ReLU). Upsampling operations increase the size of the input feature map, gradually restoring it to the spatial resolution of the original input image. In the extended path, low-level features from the encoder are also combined with high-level features from the decoder through skip connections, enabling the network to capture multi-scale information.

[0175] At the end of the decoder, a 1x1 convolutional layer is typically used to transform the feature map into an appropriate number of categories. Next, a pixel-level classification loss (such as cross-entropy loss) is used to train the U-Net model.

[0176] Preferably, a multimodal medical image segmentation system based on modal complementarity and state-space modeling may further include boundary enhancement and uncertainty evaluation mechanism modules. After processing the 3D input data, the boundary enhancement and uncertainty evaluation mechanism modules are then input into the depth convolution module and the first residual module, respectively.

[0177] The boundary enhancement and uncertainty evaluation mechanism module may include a convolution module and a prediction and evaluation module connected to it.

[0178] The prediction and evaluation module can be used to improve the perception of lesion edge structures and suppress uncertain regions.

[0179] The expression for the boundary reinforcement and uncertainty evaluation mechanism module can be as follows:

[0180] ;

[0181] ;

[0182] In the formula:

[0183] f represents a given single-modal 3D input image;

[0184] u represents the uncertainty evaluation value, which is used to measure the model's confidence in the current prediction region;

[0185] This represents the probability after activation by the Sigmoid function, with values ​​ranging from [0,1].

[0186] This represents the output feature map of the boundary enhancement and uncertainty assessment module;

[0187] This represents the Sigmoid function, which performs a non-linear mapping on the input feature map f, giving it a probabilistic meaning.

[0188] This represents a three-dimensional convolution operation with a kernel size of 3×3×3.

[0189] Indicates the multiplication sign;

[0190] This represents the normalization function.

[0191] This represents a function that normalizes the input feature mapping f, used to stabilize training and enhance the consistency of feature distribution.

[0192] Preferably, a modal complementary prior module may be provided at the front end of each encoder. The modal complementary prior module may include a modal-specific encoding module, a modal sensing fusion module, and a modal residual calibration connection module connected in sequence.

[0193] Can be set as follows: For input data, Where C represents the total number of modes, , C; The modality-specific encoding module processes the input data as shown in the following formula:

[0194] ;

[0195] In the formula:

[0196] m represents the input data mode number;

[0197] This represents the feature representation extracted from the m-th modality through a convolution operation;

[0198] Let m represent the input feature tensor of the m-th mode; Where D represents depth; H represents height; and W represents width;

[0199] This represents a 3D convolution operation for the m-th modal input, with a kernel size of 3×3×3;

[0200] This represents the feature extraction function for the m-th mode;

[0201] To extract complementary information between modalities, all modal features are concatenated and fed into an attention mechanism for weighted fusion. The modality-aware fusion module performs weighted fusion processing on all modality feature data according to the following formula:

[0202] , in, ;

[0203] In the formula:

[0204] This represents the result of weighted fusion of multimodal features, which serves as the input for subsequent state modeling and segmentation.

[0205] The weight coefficient of the m-th modality in the fusion process is calculated through a softmax-type attention mechanism and is used to dynamically allocate the importance of different modalities.

[0206] Note the parameters indicating learnable channels;

[0207] Indicates the number of channels;

[0208] The modal residual calibration connection module is used to calibrate the residual structural deviations between modes, and it processes the feature data according to the following formula:

[0209] ;

[0210] In the formula:

[0211] This represents the alignment feature representation after processing by the modal residual calibration connection module, used to eliminate residual structural deviations between different modal features;

[0212] This represents the result of multimodal feature fusion.

[0213] This represents the feature representation of the T-th modality after convolution or feature extraction.

[0214] This represents a one-dimensional point convolution operation, used for inter-modal feature dimension mapping and residual calibration;

[0215] This indicates the input feature mode number.

[0216] Preferably, the convolution module of each encoder may include a Mamba module and a discrete state space module connected in sequence.

[0217] The Mamba module may include: a GSC module, a first normalization layer, a Mamba layer, a second normalization layer, an MLP module, and multiple convolutional units composed of normalization layers and convolutional kernels, connected in sequence.

[0218] Each convolutional unit has a different kernel size; each convolutional unit processes the input features in parallel and the processed features are fused together, let the fused output feature be A.

[0219] The features processed by the GSC module are fused together with the features processed by the Mamba layer, and the output feature after fusion is B.

[0220] Feature A, feature B, and the feature processed by the MLP module are fused together to output feature C.

[0221] Preferably, the GSC module may include three convolutional units, referred to as convolutional unit A, convolutional unit B, and convolutional unit C, respectively; the kernel sizes of convolutional units A and B are the same and both are larger than the kernel size of convolutional unit C;

[0222] The data to be processed can be simultaneously input into convolutional unit A and convolutional unit C. The features output by convolutional unit A and convolutional unit C are fused and then input into convolutional unit B. The features output by convolutional unit B are fused with the data to be processed and then output.

[0223] Preferably, the calculation formula for the GSC module can be as follows:

[0224] ;

[0225] In the formula:

[0226] This indicates the data to be processed;

[0227] This indicates the characteristics output after processing by the GSC module;

[0228] This represents a 3D convolution operation with a kernel size of 3×3×3, used to extract local spatial features;

[0229] This represents a 3D convolution operation with a kernel size of 1×1×1, used to extract local spatial features;

[0230] Indicates the multiplication sign;

[0231] This represents the normalization function.

[0232] Preferably, the discrete state space module can be a state space module constructed based on a multi-resolution state feature pyramid.

[0233] Preferably, the discrete state space module can be constructed based on a state feature pyramid with three levels, which are referred to as: high-level state feature pyramid, intermediate-level state feature pyramid and low-level state feature pyramid.

[0234] The state equations of the state-space module are as follows:

[0235] ;

[0236] The output equation of the state-space module is shown below:

[0237] ;

[0238] In the discrete state-space module, the model retains the value of each received discrete signal until a new discrete signal arrives; the duration of this retention is controlled by a learnable parameter Δ, called the step size, which represents the time resolution of the input discretization. The state equations and output equations of the discrete state-space module are shown below:

[0239] ;

[0240] ;

[0241] ;

[0242] ;

[0243] In the formula:

[0244] t represents time.

[0245] t-1 represents the time preceding time t;

[0246] Represents the state vector at time t;

[0247] This represents the state vector at time t-1; that is, the state vector at the time before time t, which serves as the recursive input.

[0248] Represents the state variables of the system at time t;

[0249] This represents the system's input (excitation).

[0250] y(t) represents the system's output (response);

[0251] represent The rate of change;

[0252] Represents the input features at time t;

[0253] The output feature at time t is the result of the combined effect of the state vector and the input feature.

[0254] This represents the discretized state transition matrix;

[0255] This represents the discretized input mapping matrix;

[0256] This represents the output mapping matrix, used to map the state vector. Projected onto output features;

[0257] Indicates the step size;

[0258] This represents the state transition matrix, used to describe the dynamic relationship of state evolution over time;

[0259] This represents the input mapping matrix, used to map input features. Project onto the state space;

[0260] E indicates Weights mapped to the output;

[0261] F indicates The weights directly involved in generating the output y(t) without going through the state;

[0262] Represents the identity matrix;

[0263] This represents the discretization increment of the state transition matrix;

[0264] This represents the discretization increment of the input mapping matrix.

[0265] The discrete state space module is constructed based on a state feature pyramid with three levels, which are called the high-level state feature pyramid, the intermediate state feature pyramid, and the low-level state feature pyramid, respectively.

[0266] Advanced state feature pyramids are used to extract features with relatively high resolution and relatively small time scale.

[0267] The intermediate state feature pyramid is used to extract features with relatively centered resolution and time scale.

[0268] Low-level state feature pyramids are used to extract features with relatively low resolution and relatively large time scale.

[0269] The state-space model based on the multi-resolution state pyramid is shown below:

[0270] ;

[0271] ;

[0272] ;

[0273] ;

[0274] in, ;

[0275] ;

[0276] In the formula:

[0277] t represents time.

[0278] t-1 represents the time preceding time t;

[0279] Represents the state vector at time t;

[0280] Represents the state vector at time t-1;

[0281] The state at time t is represented by the high-level state feature pyramid.

[0282] The state at time t is the intermediate state feature pyramid.

[0283] Low-level state feature pyramid: state at time t;

[0284] The autoregressive transition parameters of the high-level state feature pyramid at time t;

[0285] The autoregressive transition parameters of the intermediate state feature pyramid at time t are represented.

[0286] The autoregressive transition parameters of the low-level state feature pyramid at time t are represented.

[0287] The input mapping coefficients represent the high-level state feature pyramid at time t.

[0288] The input mapping coefficients represent the intermediate state feature pyramid at time t.

[0289] The input mapping coefficients represent the low-level state feature pyramid at time t.

[0290] Represents the coupling weights between high-resolution and medium-resolution states, used to model cross-layer information interaction;

[0291] Represents the coupling weights between low-resolution and medium-resolution states, used to model cross-layer information interaction;

[0292] Represents the coupling weights between medium-resolution and low-resolution states, used to model cross-layer information interaction;

[0293] This represents the output weights of the high-level state feature pyramid;

[0294] This represents the output weights of the intermediate state feature pyramid;

[0295] This represents the output weights of the low-level state feature pyramid;

[0296] This represents the joint state transition matrix between the high-level state and the intermediate and low-level states after discretization.

[0297] This represents the joint input mapping matrix between the discretized high-level state and the intermediate and low-level states;

[0298] This represents the joint state transition matrix between the high-level state and the intermediate and low-level states.

[0299] This represents the joint input mapping matrix between high-level states and intermediate and low-level states;

[0300] This indicates the discretization time step.

[0301] matrix Mapping high-level states to intermediate-level states, matrix Mapping lower-level states to intermediate-level states, and the matrix Use the current advanced status and the previous low-level state To avoid circular dependencies; matrix Map intermediate states to lower states.

[0302] The state equations and output equations of the discrete state-space module are shown below:

[0303] ;

[0304] ;

[0305] ;

[0306] ;

[0307] In the formula:

[0308] t represents time.

[0309] t-1 represents the time preceding time t;

[0310] This represents the state vector at time t; here it refers to the state vector of the discrete state space module at time t, which is composed of high-level, intermediate-level and low-level states.

[0311] This represents the state vector at time t-1; here it refers to the state vector of the discrete state-space module at time t-1.

[0312] The state at time t is represented by the high-level state feature pyramid.

[0313] The state at time t is the intermediate state feature pyramid.

[0314] The low-level state feature pyramid represents the state at time t.

[0315] Represents the state transition matrix;

[0316] This represents the input features at time t;

[0317] This represents the input mapping matrix at time t, used to map the input features x. tMapped to a multi-resolution state space;

[0318] The autoregressive transition parameters of the high-level state feature pyramid at time t;

[0319] The autoregressive transition parameters of the intermediate state feature pyramid at time t are represented.

[0320] The autoregressive transition parameters of the low-level state feature pyramid at time t are represented.

[0321] The input mapping coefficients represent the high-level state feature pyramid at time t.

[0322] The input mapping coefficients represent the intermediate state feature pyramid at time t.

[0323] The input mapping coefficients represent the low-level state feature pyramid at time t.

[0324] Represents the coupling weights between high-resolution and medium-resolution states, used to model cross-layer information interaction;

[0325] Represents the coupling weights between low-resolution and medium-resolution states, used to model cross-layer information interaction;

[0326] Represents the coupling weights between medium-resolution and low-resolution states, used to model cross-layer information interaction;

[0327] This represents the interaction weight matrix between the high-level state and the input features.

[0328] This invention also provides an embodiment of a multimodal medical image segmentation method based on modal complementarity and state space modeling, utilizing the above-mentioned multimodal medical image segmentation system based on modal complementarity and state space modeling. The method includes the following steps: setting a modal complementarity prior module at the front end of each encoder. The modal complementarity prior module includes a modality-specific encoding module, a modality-aware fusion module, and a modality residual calibration connection module connected in sequence, so as to combine the shared representation with the modality-specific path and realize the collaborative modeling of cross-modal shared information and difference information.

[0329] Preferably, a boundary enhancement and uncertainty evaluation mechanism module can be set to improve the boundary sensitivity and region discrimination of the input. After processing the 3D input data using the boundary enhancement and uncertainty evaluation mechanism module, the data is then input to the depth channel convolution module and the first residual module, respectively. The expression of the boundary enhancement and uncertainty evaluation mechanism module can be as follows:

[0330] ;

[0331] ;

[0332] In the formula:

[0333] f represents a given single-modal 3D input image;

[0334] u represents the uncertainty evaluation value, which is used to measure the model's confidence in the current prediction region;

[0335] This represents the probability after activation by the Sigmoid function, with values ​​ranging from [0,1].

[0336] This represents the output feature map of the boundary enhancement and uncertainty assessment module;

[0337] This represents the Sigmoid function, which performs a non-linear mapping on the input feature map f, giving it a probabilistic meaning.

[0338] This represents a three-dimensional convolution operation with a kernel size of 3×3×3.

[0339] This represents a function that normalizes the input feature mapping f, used to stabilize training and enhance the consistency of feature distribution.

[0340] Preferably, the Mamba module and a multi-resolution state feature pyramid structure can be integrated to construct hierarchical feature representations from different temporal and / or spatial resolutions.

[0341] The integration of Mamba modules and the multi-resolution state feature pyramid structure can be referenced from the aforementioned structure.

[0342] The structure, workflow, and working principle of the present invention are further illustrated below with reference to a preferred embodiment:

[0343] To address the problems in existing technologies and fully utilize the complementary characteristics between different modalities in multimodal medical images, while taking into account both global context modeling and local structure perception capabilities, this invention provides a multimodal medical image segmentation system and method based on modal complementarity and state space modeling. This method is based on an improved 3D U-Net architecture and integrates boundary enhancement and uncertainty assessment mechanisms, modal complementarity prior modeling mechanisms, lightweight state space modeling modules, and multi-resolution state pyramid structures.

[0344] This invention provides a multimodal medical image segmentation system based on modal complementarity and state space modeling. This system is based on an improved 3D U-Net architecture and integrates boundary enhancement and uncertainty assessment mechanisms, modal complementarity prior modeling mechanisms, a lightweight state space modeling module, and a multi-resolution state pyramid structure. Figure 1 As shown.

[0345] A multimodal medical image segmentation system based on modal complementarity and state space modeling is provided. The system is built on the 3DU-Net architecture and includes: a first residual module, a second residual module, a first connection module and a segmentation head connected in sequence; N encoders composed of convolutional modules and downsampling modules connected in sequence; N decoders composed of residual modules, connection modules and transpose modules connected in sequence; a depth convolutional module, a third residual module and a first transpose module.

[0346] The N encoders are referred to as the first to the Nth encoders according to the signal transmission order; the N decoders are referred to as the first to the Nth decoders according to the signal transmission order.

[0347] The 3D input data is divided into two inputs: one input is sent to the depth convolution module; the other input is sent to the first residual module.

[0348] The signal output from the depth convolution module is divided into two paths: one path is input to the convolution module of the first encoder; the other path is input to the residual module of the Nth decoder.

[0349] The signal output from the i-th encoder is divided into two paths, i=1,2,…,N-1; one path is input to the convolution module of the (i+1)-th encoder; the other path is input to the residual module of the Ni-th decoder.

[0350] The signal output by the Nth encoder is input to the third residual module; the signal output by the third residual module is transposed by the first transpose module and then input together with the signal output by the residual module in the first decoder into the connection module of the first decoder. After being connected by the connection module of the first decoder, it is output to the transpose module of the first decoder.

[0351] The signal output from the transpose module of the k-th decoder and the signal output from the residual module of the (k+1)-th decoder are input together into the connection module of the (k+1)-th decoder. After being connected by the connection module of the (k+1)-th decoder, the signal is output to the transpose module of the (k+1)-th decoder, k=1,2,…,N-1;

[0352] The signal output from the transpose module of the Nth decoder and the signal output from the second residual module are input together into the first connection module. After being connected by the first connection module, they are output to the segmentation head, and the segmentation head outputs the segmented signal of the multimodal data.

[0353] The improved 3D-Unet encoder comprises a convolutional neural network skeleton (depth convolutional module) and a series of convolutional and downsampling modules. The convolutional neural network skeleton uses 7×7×7 convolutional kernels and 3×3×3 padding, with a convolutional stride of 2×2×2. The input 3D image size is C×D×H×W, where C represents the total number of modalities, allowing input data from multiple modalities. The depth convolutional module size is 48×D / 2×H / 2×W / 2. Feature maps are extracted through four consecutive convolutional and downsampling modules. In the 3D-Unet decoder, a transposed convolution algorithm is used to obtain the 3D-Unet image, which is then processed. At the end of the network, a Seghead module is added to output the segmentation results of the multimodal data.

[0354] This paper introduces several enhancement modules to improve the robustness and accuracy of the model in identifying complex lesion structures.

[0355] 1. By introducing boundary enhancement and uncertainty evaluation mechanisms, the boundary sensitivity and regional discrimination of the input are effectively improved.

[0356] 2. In the encoder part, in order to make full use of the complementary information between modalities in terms of tissue boundaries, texture details and lesion contrast, a modal complementary prior modeling mechanism was designed, which combines shared representation with modal-specific paths to achieve collaborative modeling of cross-modal shared information and differential information.

[0357] 3. In the deep semantic modeling stage, the lightweight Mamba module and the multi-resolution state pyramid structure have been integrated into the remote dependency model, and hierarchical feature representations have been constructed from different temporal and / or spatial resolutions.

[0358] The overall network structure not only retains the spatial modeling advantages of 3D segmentation but also achieves semantic enhancement and structural refinement guided by modal fusion, thereby improving the ability to perform multimodal fine segmentation of complex lesions. The following sections introduce several modules:

[0359] (1) Boundary Enhancement and Uncertainty Assessment (BEUE):

[0360] In multimodal 3D lesion data, to enhance the determinism of key features, suppress high-uncertainty features, and minimize the impact of background on segmentation results, such as... Figure 2 As shown, boundary enhancement and uncertainty assessment (BEUE) has been proposed.

[0361] like Figure 2 As shown, boundary enhancement is achieved using Conv3×3×3 convolution. For blank background regions and complete lesion regions, the convolution calculation result is approximately 0. The convolution kernel is activated when scanning the lesion boundary. This convolution kernel is sensitive to boundaries and can enhance the boundary features of the multimodal original data. Given a single-modal 3D input image f, the calculation process of uncertainty u is shown in formula (1).

[0362] (1)

[0363] Therefore, the output feature map fen of the boundary enhancement and uncertainty assessment module is given by equation (2).

[0364] (2)

[0365] (2) Modal Complementary Prior Model (MCPM):

[0366] The size of a 3D multimodal medical image is C×D×H×W, where C represents the number of modalities (e.g., T1, T2, FLAIR). To fully model the complementarity of different modalities in terms of structural resolution, boundary features, and lesion contrast, and to address the significant information differences between modalities and the problems of coarse fusion methods in multimodal medical images, a modality complementarity prior modeling (MCPM) mechanism has been proposed and placed in the network's encoder. Its aim is to guide the encoder to extract distinctive modal features based on modality specificity and achieve high-quality modal co-representation through attention fusion and residual compensation. The structure of MCPM is as follows: Figure 3 As shown.

[0367] The entire mechanism is integrated at the front end of the encoder, including modality-specific coding, modality-aware fusion, and modality-consistent residual alignment.

[0368] 1) Modality-specific encoding:

[0369] set up For input data, Where C represents the total number of modes, , To maintain the unique representation between modes, each mode is passed through a set of independent shallow convolutional encoders in equation (3).

[0370] The modality-specific encoding module processes the input data as shown in the following formula:

[0371] (3)

[0372] The design preserves the structural differences in edge, texture, and brightness response of various modalities, reflecting the structural prior of modal complementarity.

[0373] 2) Modal perception fusion:

[0374] In order to extract complementary information between modalities, all modal features are concatenated and fed into an attention mechanism for weighted fusion, thereby generating a joint representation as shown in equation (4).

[0375] , in, (4)

[0376] In the formula:

[0377] This represents the result of weighted fusion of multimodal features, which serves as the input for subsequent state modeling and segmentation.

[0378] The weight coefficient of the m-th modality in the fusion process is calculated through a softmax-type attention mechanism and is used to dynamically allocate the importance of different modalities.

[0379] Note the parameters indicating learnable channels;

[0380] Indicates the number of channels;

[0381] This mechanism reflects the guiding role of modal complementarity priors in the feature fusion process.

[0382] 3) Modally consistent residual alignment:

[0383] Although the attention mechanism integrates multimodal information, residual structural biases between modalities may still affect depth modeling. To further enhance the consistency between modalities, a modal residual calibration connection was designed, as shown in Equation (5).

[0384] (5)

[0385] In the formula:

[0386] This represents the alignment feature representation after processing by the modal residual calibration connection module, used to eliminate residual structural deviations between different modal features;

[0387] This represents the result of multimodal feature fusion.

[0388] This represents the feature representation of the T-th modality after convolution or feature extraction.

[0389] This represents a one-dimensional point convolution operation, used for inter-modal feature dimension mapping and residual calibration;

[0390] This indicates the input modality number.

[0391] This structure introduces a modal calibration residual branch based on fused features, which enhances the consistency of feature comparison and network stability, providing more robust multimodal semantic features for the downstream encoder backbone.

[0392] In summary, MCPM relies on prior knowledge of modality-specific structural differences, captures modality-specific representations through independent paths, and then achieves dynamic fusion of complementary information with the help of attention mechanisms. The fusion features extracted by the MCPM method maintain the advantages of modality differences while having high consistency. The joint semantic expression with consistency and discriminability provides a stable and robust feature foundation for subsequent deep encoding, state space modeling, and fine segmentation.

[0393] (3) Mamba module:

[0394] Convolutional modules extract features from the 3D input volume. Due to the inherent locality of convolutional layers, CNN-based methods struggle to model global relationships. However, modeling global and multi-scale features is crucial for 3D medical image segmentation. While the Transformer architecture can capture global information, it incurs significant computational burden when processing long feature sequences and is prone to losing multi-scale information. To balance multi-scale, global, and local feature representations simultaneously, we propose a Mamba block that maintains high efficiency during both training and inference. The structure of the Mamba block is as follows: Figure 4 As shown.

[0395] To balance the global and local features of the 3D feature matrix simultaneously, a hierarchical feature extraction module with high-dimensional 3D convolutional kernels is established in the Mamba module to extract the local spatial features of the lesion region. The original feature map is normalized using Norm, and then convolved with 5×5×5, 7×7×7, and 9×9×9 convolutional kernels respectively to extract the local features of the lesion at different scales. Finally, the features are fused together. Assume that the input of the hierarchical feature extraction module with high-dimensional 3D convolutional kernels is f, and the output is m, as shown in equation (6).

[0396] (6)

[0397] In the formula:

[0398] This represents a 3D convolution operation with a kernel size of 5×5×5, used to extract local spatial features;

[0399] This represents a 3D convolution operation with a kernel size of 7×7×7, used to extract local spatial features;

[0400] This indicates a 3D convolution operation with a kernel size of 9×9×9, used to extract local spatial features;

[0401] The Mamba layer models feature dependencies by flattening 3D features into a 1D sequence, but this operation discards spatial information. To capture the spatial context before Mamba processing, a gated spatial convolution (GSC) module is applied within the Mamba module. The architecture of the GSC module is as follows: Figure 5 As shown.

[0402] like Figure 5 As shown, the input 3D feature matrix is ​​fed into two convolutional blocks (each block includes a normalization layer and a convolutional computation layer), with kernel sizes of 3×3×3 and 1×1×1, respectively. The feature matrices output from the two convolutional blocks are multiplied pixel-by-pixel to control information transmission, similar to a gating mechanism. Finally, the convolutional blocks are used to further fuse features, while the residual connections are used to re-input the feature matrix. Let f represent the input features and m represent the output features of the GSC module, as shown in equation (7).

[0403] (7)

[0404] In the formula:

[0405] This indicates the data to be processed;

[0406] This indicates the characteristics output after processing by the GSC module;

[0407] This represents a 3D convolution operation with a kernel size of 3×3×3, used to extract local spatial features;

[0408] This represents a 3D convolution operation with a kernel size of 1×1×1, used to extract local spatial features;

[0409] This represents the normalization function.

[0410] State-space models (SSM) and Mamba are rapidly developing backbone network models in recent years. The Mamba module is used to extract global long-sequence features from the feature matrix. Parallel Correlated Scan (PCS) is a key component of the Mamba model, designed to address the computational problems caused by the selection mechanism, accelerate the training process, and reduce memory requirements. The Mamba module and Parallel Correlated Scan are as follows: Figure 6 As shown.

[0411] (1) State space module based on multi-resolution state pyramid:

[0412] The state equation and output equation of the SSM are given by equation (8).

[0413] (8)

[0414] ;

[0415] The state equation shows that the hidden state representation is achieved by interpolating matrix B with the input. Multiply them and add them to matrix A and the state. This product is obtained from the product of the two terms. This product can also be expressed as... The output equation represents matrix B and the state. The product of the input and the matrix D. The product of the four matrices A, B, C, and D. These four matrices are parameters that can be trained.

[0416] The state equations and output equations of the Discrete State Space Module (SSM) are given by equation (9).

[0417] (9)

[0418]

[0419] In discrete SSM, the model retains each received discrete signal value until a new discrete signal arrives. This zero-order hold (ZOH) operation effectively constructs a continuous signal for SSM processing. The duration of this value retention is controlled by a learnable parameter Δ, called the step size, which represents the time resolution of the input discretization, as shown in Equation (10).

[0420] (10)

[0421] Construct a multi-scale state representation so that the model can capture both short-term and long-term dependencies simultaneously; design a state pyramid with three levels: high-level, mid-level, and low-level.

[0422] Advanced status Rapidly changing, focusing on local details (high resolution, small time scale).

[0423] Intermediate state Moderate pace of change, focusing on medium-term dependencies.

[0424] low-level state : Slow-moving, focusing on global information (low resolution, large time scale).

[0425] State-space module (SSM) based on multi-resolution state pyramid is shown in formula (11).

[0426]

[0427] (11)

[0428]

[0429] in,

[0430]

[0431] matrix Mapping high-level states to intermediate-level states, matrix Mapping lower-level states to intermediate-level states, and the matrix Use the current advanced status and the previous low-level state To avoid circular dependencies. Matrix Map intermediate states to lower states.

[0432] The discrete state-space equation (SSM) is given by equation (12).

[0433] (12)

[0434] in,

[0435] , ;

[0436] ;

[0437] For definitions of expressions not listed in the above formulas, please refer to the aforementioned expression definitions.

[0438] Based on the above analysis, the multimodal medical image segmentation method of the present invention, which is based on modal complementarity and state space modeling, can be carried out according to the following steps:

[0439] Step 1: The BraTS multimodal brain tumor segmentation dataset and the SAM 3D MRI dataset were preprocessed and then fed into the network, as follows: Figure 1 As shown.

[0440] Step 2: Enhance boundaries and suppress noise using the BEUE module, such as... Figure 2 As shown.

[0441] Step 3: The MCPM module extracts and fuses modal features, such as... Figure 3 As shown.

[0442] Step 4: Modeling global and cross-scale information using Mamba blocks and multi-resolution state pyramids, such as... Figure 4 As shown.

[0443] Step 5: The decoder recovers the feature resolution through transposed convolutions and skip connections.

[0444] Finally passed Figure 1 The SeHead module outputs the segmentation results, and medical image segmentation can be achieved through the above steps. Experimental results show that the invention exhibits excellent structural preservation and prediction consistency in the tumor boundary region, verifying the effectiveness and generalizability of the method in complex brain tumor scenarios, and has important clinical application value.

[0445] The aforementioned 3D U-Net architecture, including the first residual module, the second residual module, the first connection module, the segmentation head, the convolution module, the downsampling module, the residual module, the connection module and the transpose module, the depthwise convolution module, the third residual module and the first transpose module, the boundary enhancement and uncertainty evaluation mechanism module, the prediction and evaluation module, the Sigmoid function, the normalization processing function, and other functional modules and algorithms, can all adopt existing functional modules and algorithms, or adopt existing functional modules and algorithms and construct and implement them using conventional technical means.

[0446] The embodiments described above are only used to illustrate the technical ideas and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The patent scope of the present invention should not be limited by these embodiments. That is, any equivalent changes or modifications made in accordance with the spirit disclosed in the present invention still fall within the patent scope of the present invention.

Claims

1. A multimodal medical image segmentation system based on modal complementarity and state-space modeling, characterized in that, The system is built on a 3D U-Net architecture, which includes: a first residual module, a second residual module, a first connection module and a segmentation head connected in sequence; N encoders composed of convolution modules and downsampling modules connected in sequence; N decoders composed of residual modules, connection modules and transpose modules connected in sequence; a depth convolution module, a third residual module and a first transpose module. The N encoders are referred to as the first to the Nth encoders according to the signal transmission order; the N decoders are referred to as the first to the Nth decoders according to the signal transmission order. The 3D input data is divided into two inputs: one input is sent to the depth convolution module; the other input is sent to the first residual module. The signal output from the depth convolution module is divided into two paths: one path is input to the convolution module of the first encoder; the other path is input to the residual module of the Nth decoder. The signal output from the i-th encoder is divided into two paths, i=1,2,…,N-1; one path is input to the convolution module of the (i+1)-th encoder; the other path is input to the residual module of the Ni-th decoder. The signal output by the Nth encoder is input to the third residual module; the signal output by the third residual module is transposed by the first transpose module and then input together with the signal output by the residual module in the first decoder into the connection module of the first decoder. After being connected by the connection module of the first decoder, it is output to the transpose module of the first decoder. The signal output from the transpose module of the k-th decoder and the signal output from the residual module of the (k+1)-th decoder are input together into the connection module of the (k+1)-th decoder. After being connected by the connection module of the (k+1)-th decoder, the signal is output to the transpose module of the (k+1)-th decoder, k=1,2,…,N-1; The signal output from the transpose module of the Nth decoder and the signal output from the second residual module are input together into the first connection module. After being connected by the first connection module, they are output to the segmentation head, and the segmentation head outputs the segmented signal of the multimodal data.

2. The multimodal medical image segmentation system based on modal complementarity and state-space modeling according to claim 1, characterized in that, It also includes a boundary enhancement and uncertainty evaluation mechanism module. After processing the 3D input data, the boundary enhancement and uncertainty evaluation mechanism module inputs the data into the depth convolution module and the first residual module, respectively. The expression for the boundary reinforcement and uncertainty evaluation mechanism module is as follows: ； ； In the formula: f represents a given single-modal 3D input image; u represents the uncertainty evaluation value, which is used to measure the model's confidence in the current prediction region; This represents the probability after activation by the Sigmoid function, with values ​​ranging from [0,1]. This represents the output feature map of the boundary enhancement and uncertainty assessment module; This represents the Sigmoid function, which performs a non-linear mapping on the input feature map f, giving it a probabilistic meaning. This represents a three-dimensional convolution operation with a kernel size of 3×3×3. This represents a function that normalizes the input feature mapping f, used to stabilize training and enhance the consistency of feature distribution.

3. The multimodal medical image segmentation system based on modal complementarity and state-space modeling according to claim 1, characterized in that, Each encoder also has a modal complementarity prior module at its front end. The modal complementarity prior module includes a modal-specific coding module, a modal perception fusion module, and a modal residual calibration connection module connected in sequence. set up For input data, Where C represents the total number of modes, , The modality-specific encoding module processes the input data as shown in the following formula: ； In the formula: m represents the input data mode number; This represents the feature representation extracted from the m-th modality through a convolution operation; Let m represent the input feature tensor of the m-th mode; Where D represents depth; H represents height; and W represents width; This represents a 3D convolution operation for the m-th modal input, with a kernel size of 3×3×3; This represents the feature extraction function for the m-th mode; To extract complementary information between modalities, all modal features are concatenated and fed into an attention mechanism for weighted fusion. The modality-aware fusion module performs weighted fusion processing on all modality feature data according to the following formula: , in, ; In the formula: This represents the result of weighted fusion of multimodal features, which serves as the input for subsequent state modeling and segmentation. The weight coefficient of the m-th modality in the fusion process is calculated through a softmax-type attention mechanism and is used to dynamically allocate the importance of different modalities. Note the parameters indicating learnable channels; Indicates the number of channels; The modal residual calibration connection module is used to calibrate the residual structural deviations between modes, and it processes the feature data according to the following formula: ； In the formula: This represents the alignment feature representation after processing by the modal residual calibration connection module, used to eliminate residual structural deviations between different modal features; This represents the result of multimodal feature fusion. This represents the feature representation of the T-th modality after convolution or feature extraction. This represents a one-dimensional point convolution operation, used for inter-modal feature dimension mapping and residual calibration; This indicates the input feature mode number.

4. The multimodal medical image segmentation system based on modal complementarity and state-space modeling according to claim 1, characterized in that, Each encoder's convolution module includes a Mamba module and a discrete state space module connected in sequence; The Mamba module includes: a GSC module, a first normalization layer, a Mamba layer, a second normalization layer, an MLP module, and multiple convolutional units composed of normalization layers and convolutional kernels, connected in sequence. Each convolutional unit has a different kernel size; each convolutional unit processes the input features in parallel and the processed features are fused together, let the fused output feature be A; The features processed by the GSC module are fused together with the features processed by the Mamba layer, and the output feature after fusion is B. Feature A, feature B, and the feature processed by the MLP module are fused together to output feature C.

5. The multimodal medical image segmentation system based on modal complementarity and state-space modeling according to claim 4, characterized in that, The GSC module includes three convolutional units, referred to as convolutional unit A, convolutional unit B, and convolutional unit C. The kernel sizes of convolutional units A and B are the same and are both larger than the kernel size of convolutional unit C. The data to be processed is simultaneously input into convolutional unit A and convolutional unit C. The features output by convolutional unit A and convolutional unit C are fused and then input into convolutional unit B. The features output by convolutional unit B are fused with the data to be processed and then output.

6. The multimodal medical image segmentation system based on modal complementarity and state-space modeling according to claim 4, characterized in that, The discrete state space module is a state space module constructed based on a multi-resolution state feature pyramid.

7. The multimodal medical image segmentation system based on modal complementarity and state-space modeling according to claim 6, characterized in that, The discrete state space module is constructed based on a state feature pyramid with three levels, which are called the high-level state feature pyramid, the intermediate state feature pyramid, and the low-level state feature pyramid, respectively. Advanced state feature pyramids are used to extract features with relatively high resolution and relatively small time scale. The intermediate state feature pyramid is used to extract features with relatively centered resolution and time scale. Low-level state feature pyramids are used to extract features with relatively low resolution and relatively large time scale. The state equations and output equations of the discrete state-space module are shown below: ； ； ； ； In the formula: t represents time. t-1 represents the time preceding time t; Represents the state vector at time t; Represents the state vector at time t-1; The state at time t is represented by the high-level state feature pyramid. The state at time t is the intermediate state feature pyramid. The low-level state feature pyramid represents the state at time t. Represents the state transition matrix; This represents the input features at time t; This represents the input mapping matrix at time t, used to map the input features x. t Mapped to a multi-resolution state space; The autoregressive transition parameters of the high-level state feature pyramid at time t; The autoregressive transition parameters of the intermediate state feature pyramid at time t are represented. The autoregressive transition parameters of the low-level state feature pyramid at time t are represented. The input mapping coefficients represent the high-level state feature pyramid at time t. The input mapping coefficients represent the intermediate state feature pyramid at time t. The input mapping coefficients represent the low-level state feature pyramid at time t. Represents the coupling weights between high-resolution and medium-resolution states, used to model cross-layer information interaction; Represents the coupling weights between low-resolution and medium-resolution states, used to model cross-layer information interaction; Represents the coupling weights between medium-resolution and low-resolution states, used to model cross-layer information interaction; This represents the interaction weight matrix between the high-level state and the input features.

8. A multimodal medical image segmentation method based on modal complementarity and state space modeling using the multimodal medical image segmentation system based on modal complementarity and state space modeling as described in claim 1, characterized in that, The method includes the following steps: setting a modal complementary prior module at the front end of each encoder. The modal complementary prior module includes a modal-specific encoding module, a modal-aware fusion module, and a modal residual calibration connection module connected in sequence, so as to combine the shared representation with the modal-specific path and realize the collaborative modeling of cross-modal shared information and difference information.

9. The multimodal medical image segmentation method based on modal complementarity and state space modeling according to claim 8, characterized in that, A boundary enhancement and uncertainty evaluation mechanism module is set up to improve the boundary sensitivity and region discrimination of the input. After processing the 3D input data using the boundary enhancement and uncertainty evaluation mechanism module, the data is then fed into the depth convolution module and the first residual module, respectively. The expression of the boundary enhancement and uncertainty evaluation mechanism module is as follows: The expression for the boundary reinforcement and uncertainty evaluation mechanism module is as follows: ； ； In the formula: f represents a given single-modal 3D input image; u represents the uncertainty evaluation value, which is used to measure the model's confidence in the current prediction region; This represents the probability after activation by the Sigmoid function, with values ​​ranging from [0,1]. This represents the output feature map of the boundary enhancement and uncertainty assessment module; This represents the Sigmoid function, which performs a non-linear mapping on the input feature map f, giving it a probabilistic meaning. This represents a three-dimensional convolution operation with a kernel size of 3×3×3. This represents a function that normalizes the input feature mapping f, used to stabilize training and enhance the consistency of feature distribution.

10. The multimodal medical image segmentation method based on modal complementarity and state space modeling according to claim 8, characterized in that, Integrating the Mamba module and a multi-resolution state feature pyramid structure, hierarchical feature representations are constructed from different temporal and / or spatial resolutions.

Images