A dual-channel cross-fusion left atrial segmentation method based on SAM and SSM
By constructing a dual-channel cross-fusion method based on SAM and SSM, and utilizing the Trans encoder and TS encoder combined with a cross-attention module, the problem of shape and position differences in left atrial MRI image segmentation was solved, achieving higher segmentation accuracy and robustness.
Patent Information
- Application Number
- CN202410692074.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-31
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-05-31
AI Technical Summary
Existing technologies for segmenting left atrial MRI images suffer from problems such as large differences in shape and position, complex overlap, and limited label data, which increase the difficulty of segmentation and reduce accuracy.
A dual-channel cross-fusion method based on SAM and SSM is adopted to construct a dual-channel encoder, including a Trans encoder and a TS encoder. Combined with a cross-attention module and a mask decoder, feature extraction and segmentation are performed using unsupervised learning and the EMD loss function.
It improves the accuracy and transferability of left atrial segmentation, effectively processes left atrial images from different imaging modalities, enhances the model's understanding of image structure and patterns, and improves the accuracy and robustness of segmentation.
Smart Images

Figure CN118710650B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical image segmentation technology, and in particular relates to a dual-channel cross-fusion left atrial segmentation method based on SAM (Segment Anything Model) and SSM (State Space Model). Background Technology
[0002] The left atrium (LA) is a crucial site for many heart diseases, including atrial fibrillation and valvular heart disease. Clinically, indicators such as ejection fraction (EF), left atrial volume index (LAVI), LA strain, LA strain rate, waveform conduction time, and left atrial systolic function (LAEF) are mainly used to assess left atrial function and further assist in the diagnosis of heart diseases. The accuracy of these indicators depends on the accurate segmentation of the left atrium from cardiac computed tomography (CT) and magnetic resonance imaging (MRI). Furthermore, accurate segmentation can help doctors accurately identify and locate pathological areas, leading to more accurate diagnosis and treatment.
[0003] Left atrial MRI images are complex, with significant variations in shape, size, and location among patients. Furthermore, the left atrium may overlap or contact other cardiac tissues (e.g., left ventricle, right atrium), increasing the difficulty of left atrial segmentation. Additionally, due to the limited availability of labeled data, unsupervised training is employed to overcome dependence on labeled data and avoid inaccurate segmentation results caused by annotation bias. The trained model exhibits better transferability and generalization ability. Unsupervised learning focuses on exploring the inherent structure and patterns of the image, helping the model understand its features. By learning statistical information and similarities, unsupervised learning can discover different objects or textures within the image, thereby segmenting it into regions with a certain degree of continuity or consistency. Summary of the Invention
[0004] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to provide a dual-channel cross-fusion left atrial segmentation method based on SAM and SSM.
[0005] The present invention solves the aforementioned technical problem by adopting the following technical solution:
[0006] A dual-channel cross-fusion left atrial segmentation method based on SAM and SSM, characterized by the following steps:
[0007] Step S1: Obtain several left atrial MRI images and normalize the images to obtain a dataset;
[0008] Step S2: Construct a segmentation model; the segmentation model includes a dual-channel encoder, a cross-attention module, a cue encoder, and a mask decoder. The dual-channel encoder includes a Trans encoder and a TS encoder; the input image is flattened to obtain embedding vectors; the embedding vectors are input into the Trans encoder and the TS encoder for feature extraction, respectively. The output features of the Trans encoder and the TS encoder are input into the cross-attention module for further feature extraction to obtain attention embedding vectors; the cue encoder encodes the points, masks, and bounding boxes of the region to be segmented in the input image into embedding vectors in real time to obtain cue embedding vectors; the mask decoder decodes the attention embedding vectors and cue embedding vectors to obtain the segmentation result;
[0009] The Trans encoder contains multiple serial Transformer groups, each containing multiple Transformer blocks. The TS encoder contains multiple encoding modules and one Transformer block. Each encoding module contains a Transformer block, a sub-window state space layer, and a block state space layer. The output features of the sub-window state space layer and the block state space layer of each encoding module are residually concatenated and used as the output features of the encoding module. The output features of the Transformer block of the previous encoding module are residually concatenated with the output features of the Transformer block of the next encoding module and used as the input features of the sub-window state space layer of the next encoding module. The sub-window state space layer divides the input features into multiple sub-windows, performs feature extraction on each sub-window, and obtains multiple sub-features. All sub-features are input into the block state space layer for further feature extraction.
[0010] Step S3: Train the segmentation model using the dataset, and use the trained segmentation model for left atrial segmentation.
[0011] Furthermore, in the sub-window state space layer, the input features are divided into multiple sub-windows, and then bidirectional Mamba blocks are used to extract features on each sub-window, resulting in multiple sub-features P1, P2, ..., P k k represents the number of sub-features; sub-features P1, P2, ..., P k The input is fed into a block-based state space layer. Each sub-feature undergoes pooling, bidirectional Mamba block, and unpooling operations in sequence to obtain features F1, F2, ..., F. k Features F1, F2, ..., F k After being concatenated together, each feature is then residually connected to its corresponding sub-feature to obtain the output features of the block state space layer.
[0012] Furthermore, the input features of the bidirectional Mamba block are flattened to obtain feature R. On one hand, feature R is subjected to projection, convolution, and activation operations, and then passed through a forward state space layer to obtain feature R2. On the other hand, feature R is passed through a max pooling layer and an average pooling layer respectively. The output features of the two pooling layers are added together, and then subjected to projection, unpooling, and activation operations in sequence to obtain feature R1. Feature R1 is multiplied element-wise with feature R2 to obtain feature R3. Feature R1 is passed through a reverse state space layer and then multiplied element-wise with feature R2 to obtain feature R4. Feature R4 is added to R3, then projected, and then residually connected with feature R to obtain the output features of the bidirectional Mamba block.
[0013] Furthermore, the forward state space layer and the backward state space layer have the same structure. The input feature G passes through a linear layer, a deep convolutional layer, and an activation operation in sequence to obtain feature G1. At the same time, the input feature G passes through a linear layer, a deep convolutional layer, an activation operation, an SS2D module, and a normalization layer in sequence to obtain feature G2. Feature G2 and feature G1 are multiplied element-wise and then passed through a linear layer to obtain the output feature.
[0014] Furthermore, the Transformer group contains four Transformer blocks. The output features of the first Transformer block and the output features of the third Transformer block are residually concatenated and used as the input features of the fourth Transformer block. The output features of the second Transformer block and the output features of the fourth Transformer block are residually concatenated and used as the output features of the Transformer group.
[0015] Furthermore, in the Transformer block, the input features are sequentially processed through layer normalization, multi-head attention, and adapter, and then residually connected to themselves to obtain feature M; feature M is then processed through layer normalization and multilayer perceptron, and then residually connected to itself to obtain the output features of the Transformer block.
[0016] Furthermore, the input features of the adapter are sequentially processed through downsampling, pooling, upsampling, and activation, and then residually connected with itself to obtain the output features of the adapter.
[0017] Furthermore, in the cross-attention module, the output features of the Trans encoder and the TS encoder are first concatenated to obtain the embedding vector S. Half of the embedding vector S is used as the query Q of the attention mechanism, and the other half is used as the key K and value V of the attention mechanism, respectively. After passing through a linear layer, the query Q and key K are subjected to element-wise dot product operation to obtain the similarity vector. The similarity vector is subjected to attention weight scaling operation and Softmax function to obtain the attention weight vector. After passing through a linear layer, the value V is subjected to element-wise dot product operation with the attention weight vector to obtain the attention embedding vector S′, which is the output of the cross-attention module.
[0018] Furthermore, during the training process, the total model loss L is calculated using equation (1). tra ;
[0019] L tra =L cro +L EDM (1)
[0020] In the formula, L cro It is the cross-entropy loss of the dual-channel encoder, L EMD It is the regularization loss of the dual-channel encoder;
[0021] Cross-entropy loss L of a dual-channel encoder cro Defined as:
[0022] L cro =H(v) TS ,v Trans )+H(v TS ,v TS (2)
[0023] In the formula, H(v) TS ,v Trans ) is the output feature v of the TS encoder TS With the output features v of the Trans encoder Trans The cross-entropy between them, H(v) TS ,v TS ) is the output feature v of the TS encoder TS Cross-entropy with itself;
[0024] Regularization loss L EMD Defined as:
[0025]
[0026] In the formula, It is the EMD adaptation parameter between the output characteristics of the TS encoder and the output characteristics of the Trans encoder. These are the EMD adaptation parameters between the output features of the Trans encoder and itself. It is the EMD adaptation parameter between the output characteristics of the TS encoder and itself.
[0027] Compared with existing methods, the beneficial effects of the present invention are as follows:
[0028] 1. This invention constructs a dual-channel encoder consisting of a Trans encoder and a TS encoder. The Trans encoder focuses more on global feature extraction, while the TS encoder has greater flexibility in spatial allocation, endowing the model with modeling capabilities in a large receptive space and paying more attention to modeling the relationship between global and local features. Since both the Trans encoder and the TS encoder extract features from the input image, their output features are similar. Therefore, a cross-attention module is introduced to further extract similar features between the two output features. This fully utilizes the rich prior information such as shape and position in the input image to perform similarity regularization on the output features of the Trans encoder and the TS encoder, paying more attention to the relevant information between the two output feature distributions while ignoring irrelevant information. This allows the model to fully understand the structural and pattern information of the input image itself, improving the accuracy of segmentation.
[0029] 2. Add EMD conditional constraints to the cross-attention module and introduce EMD loss so that the Trans encoder and TS encoder can focus on learning their respective features while extracting similar features.
[0030] 3. This invention utilizes a large amount of unlabeled data to regularize the model's learning, enabling the model to effectively segment the left atrial region even without labels. It can perform different levels of analysis and processing on the effective information in left atrial images from different imaging methods, thereby improving the accuracy of left atrial segmentation. Attached Figure Description
[0031] Figure 1 This is a structural diagram of the segmentation model of the present invention;
[0032] Figure 2 This is a structural diagram of the Trans encoder of the present invention;
[0033] Figure 3 This is a structural diagram of the TS encoder of the present invention;
[0034] Figure 4 This is a structural diagram of the Transformer block of the present invention;
[0035] Figure 5 This is a structural diagram of the sub-window state space layer and the block state space layer of the present invention;
[0036] Figure 6 This is a structural diagram of the forward state space layer and the reverse state space layer of the present invention;
[0037] Figure 7 This is a structural diagram of the cross-attention module of the present invention. Detailed Implementation
[0038] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of protection of this application.
[0039] This invention provides a dual-channel cross-fusion left atrial segmentation method based on SAM and SSM (hereinafter referred to as the method, see [link]). Figure 1-7 The process includes the following steps:
[0040] Step S1: Obtain several left atrial MRI images, perform normalization processing on the images including size, contrast and noise, and perform data augmentation through random masking, transformation and other methods to obtain a dataset, and divide the dataset into training set and test set.
[0041] Step S2: Construct a segmentation model; such as Figure 1 As shown, the segmentation model uses the SAM model as its basic framework, including a dual-channel encoder, a cross-attention module, a cue encoder, and a mask decoder. The dual-channel encoder includes a Trans encoder and a TS encoder. The input image is flattened to obtain an embedding vector. The embedding vector is used for feature extraction by the Trans encoder and the TS encoder, respectively. The output features of the Trans encoder and the TS encoder are further extracted by the cross-attention module to obtain an attention embedding vector. The cue encoder encodes the points, masks, and bounding boxes of the region to be segmented in the input image into embedding vectors in real time to obtain a cue embedding vector. The mask decoder makes predictions based on the attention embedding vector and the cue embedding vector to obtain the segmentation result.
[0042] like Figure 2 As shown, the Trans encoder contains multiple (three in this embodiment) serial Transformer groups, each containing four Transformer blocks. The output feature of the first Transformer block in a Transformer group is residually concatenated with the output feature of the third Transformer block, and then used as the input feature of the fourth Transformer block. The output feature of the second Transformer block is residually concatenated with the output feature of the fourth Transformer block, and then used as the output feature of the Transformer group, which is the input feature of the next Transformer group.
[0043] like Figure 3As shown, the TS encoder includes multiple encoding modules (four in this embodiment) and a Transformer block. Each encoding module includes a Transformer block, a sub-window state space layer, and a block state space layer. The output features of the sub-window state space layer and the block state space layer of each encoding module are residually concatenated and used as the output features of the encoding module. The output features of the Transformer block of the previous encoding module are residually concatenated with the output features of the Transformer block of the next encoding module and used as the input features of the sub-window state space layer of the next encoding module.
[0044] like Figure 4 As shown, in the Transformer block, the input features sequentially pass through layer normalization, multi-head attention, and an adapter, and are then residually connected to themselves to obtain feature M. Feature M then passes through layer normalization and a multilayer perceptron, and is again residually connected to itself to obtain the output features of the Transformer block. The input features of the adapter sequentially pass through downsampling, pooling, upsampling, and activation layers, and are then residually connected to itself to obtain the output features of the adapter. Downsampling is implemented using a convolution operation with a 3×3 kernel, a stride of 2, and padding of 1.
[0045] like Figure 5 As shown, since the discreteness of the input feature pixels weakens the modeling of correlation between local neighboring pixels, the sub-window state space layer divides the input features into multiple sub-windows of size m×n, resulting in a total of... Each sub-window contains H and W, representing the length and width of the input features, respectively. Bidirectional Mamba blocks are then used to extract features from each sub-window, enabling the model to better model the relationships between local neighborhood pixels and obtain multiple sub-features P1, P2, ..., P. k Here, k represents the number of sub-features. Furthermore, the receptive field of a Transformer block is typically 7×7 pixels, while feature extraction on a sub-window can extend the receptive field to 40×40 pixels. More detailed information about local pixels will be learned by the model, compensating for the Transformer block's insensitivity to local details. Since the sub-window state space layer extracts multiple non-overlapping sub-features from the input features, a communication mechanism is needed between different sub-features for long-range dependency modeling; therefore, a block-based state space layer is introduced. The sub-window state space layer extracts multiple sub-features P1, P2, ..., P... k The input is fed into the block-based state space layer. Each sub-feature undergoes pooling to summarize important information, followed by bidirectional Mamba block extraction. Finally, unpooling is used to restore the feature size, yielding features F1, F2, ..., F... k Features F1, F2, ..., F kAfter being concatenated together, each feature is then residually connected to its respective sub-features to obtain the output features of the block state space layer. The block state space layer can model the absolute position information of the sub-features and their relative position information with other sub-features.
[0046] The input features of the bidirectional Mamba block are flattened to obtain feature R. Feature R is then processed through projection, convolution, and activation operations, followed by a forward state space layer to obtain feature R2. Feature R is then processed through a max pooling layer and an average pooling layer. The output features of these two pooling layers are added together, followed by projection, unpooling, and activation operations to obtain feature R1. Feature R1 is then element-wise multiplied with feature R2 to obtain feature R3. Feature R1 is then passed through a reverse state space layer and element-wise multiplied with feature R2 to obtain feature R4. Feature R4 is then added to R3, projected, and residual-connected with feature R to obtain the output features of the bidirectional Mamba block. The forward and reverse state space layers provide a bidirectional scanning effect, allowing the model to focus more on the central region of the image rather than the edge regions.
[0047] The forward state space layer and the backward state space layer have the same structure, such as Figure 6 As shown, on one hand, the input feature G passes through a linear layer, a deep convolutional layer, and an activation operation in sequence to obtain feature G1; on the other hand, the input feature G passes through a linear layer, a deep convolutional layer, an activation operation, an SS2D module, and a normalization layer in sequence to obtain feature G2; feature G2 and feature G1 are multiplied element-wise, and then passed through a linear layer to obtain the output feature of the forward state space layer or the backward state space layer.
[0048] The Trans encoder focuses more on extracting global features, while the TS encoder's sub-window state space layer and block state space layer have linear complexity in long sequence modeling and greater flexibility in space allocation, giving the model the ability to model in a large receptive space. At the same time, it can model the relationship between global and local features, thus performing more efficient image segmentation.
[0049] The output features of the Trans encoder and the TS encoder are similar. The output features of the two encoders are used as the input of the cross attention module. Unlike the traditional attention mechanism, the cross attention block extracts object-exclusive features for paired features that share the same object concept through the cross attention mechanism. It can fully explore the similar features between the output features of the two encoders and eliminate exclusivity through the shared information of similar features, so that the model can learn more useful prior information.
[0050] like Figure 7As shown, in the cross-attention module, the output features of the Trans encoder and the TS encoder are first concatenated to obtain the embedding vector S. Half of the embedding vector S is used as the query Q of the attention mechanism, and the other half is used as the key K and value V of the attention mechanism, respectively. After the query Q and key K pass through the linear layer, they are subjected to element-wise dot product operation to obtain the similarity vector. The similarity vector is subjected to attention weight scaling operation and Softmax function to obtain the attention weight vector. After the value V passes through the linear layer, it is subjected to element-wise dot product operation with the attention weight vector to obtain the attention embedding vector S′, which is the output of the cross-attention module.
[0051] The cue encoder encodes points, masks, bounding boxes, or text from the input image's region to be segmented into embedding vectors in real time. By simulating interactive segmentation, it uses background points to provide cues for the segmented region, enabling the model to more effectively return the segmentation mask. The mask decoder updates all embedding vectors in two directions (from cue to image embedding and back) using cue self-attention and bidirectional cross-attention, based on the attention embedding vectors output by the cross-attention module and the cue embedding vectors output by the cue encoder, to obtain the image embedding vector. Then, the image embedding vector is upsampled and mapped to a dynamic linear classifier using a multilayer perceptron. The mask foreground probability at each image location is then calculated to predict the segmented region, resulting in the segmentation mask.
[0052] Step S3: Based on the interactive unsupervised training framework, train the segmentation model and use the trained segmentation model for left atrial segmentation.
[0053] During training, the total model loss L is calculated using equation (1). tra ;
[0054] L tra =L cro +L EMD (1)
[0055] In the formula, L cro It is the cross-entropy loss of the dual-channel encoder, L EMD It is the regularization loss of the dual-channel encoder;
[0056] Cross-entropy loss L of a dual-channel encoder cro Defined as:
[0057] L cro =H(v) TS ,v Trans )+H(v TS ,v TS (2)
[0058] In the formula, H(v) TS ,v Trans) is the output feature v of the TS encoder TS With the output features v of the Trans encoder Trans The cross-entropy between them, H(v) TS ,v TS ) is the output feature v of the TS encoder TS Cross-entropy with itself;
[0059] Wherein, H(v) TS ,v Trans The calculation formula for H(v) is shown in equation (3), where H(v) is a constant. TS ,v TS The calculation is similar;
[0060] H(v TS ,v Trans ) = -logp(v Trans |v TS (3)
[0061]
[0062] In the formula, π(v TS ) is the attention embedding vector output by the cross-attention module, τ is the temperature, and w Trans It refers to the embedded word vectors of the Trans encoder.
[0063] To maximize the similarity between the output features of the TS encoder and the output features of the Trans encoder, the regularization term loss L is... EMD Defined as:
[0064]
[0065] In the formula, It is the EMD adaptation parameter between the output characteristics of the TS encoder and the output characteristics of the Trans encoder. These are the EMD adaptation parameters between the output features of the Trans encoder and itself. These are the EMD adaptation parameters between the output features of the TS encoder and itself;
[0066] The EMD adaptation parameter between features is defined as follows:
[0067]
[0068] f ij ≥0, i=1,…,n s j = 1, ..., n d (8)
[0069]
[0070] In the formula, It is the minimum flow rate, f ij It's traffic, s i It is the i-th position feature in the output features of the TS encoder, n s d is the number of positional features in the output features of the TS encoder. j It is the j-th position feature in the output features of the Trans encoder, n d c is the number of positional features in the output features of the Trans encoder. ij It is the location feature s i With d j The distance between them.
[0071] Any aspects not covered in this invention are applicable to existing technologies.
Claims
1. A dual-channel cross-fusion left atrial segmentation method based on SAM and SSM, characterized in that, The method includes the following steps: Step S1: Obtain several left atrial MRI images and normalize the images to obtain a dataset; Step S2: Construct a segmentation model; the segmentation model includes a dual-channel encoder, a cross-attention module, a cue encoder, and a mask decoder. The dual-channel encoder includes a Trans encoder and a TS encoder; the input image is flattened to obtain embedding vectors; the embedding vectors are input into the Trans encoder and the TS encoder for feature extraction, respectively. The output features of the Trans encoder and the TS encoder are input into the cross-attention module for further feature extraction to obtain attention embedding vectors; the cue encoder encodes the points, masks, and bounding boxes of the region to be segmented in the input image into embedding vectors in real time to obtain cue embedding vectors; the mask decoder decodes the attention embedding vectors and cue embedding vectors to obtain the segmentation result; The Trans encoder contains multiple serial Transformer groups, each containing multiple Transformer blocks. The TS encoder contains multiple encoding modules and one Transformer block. Each encoding module contains a Transformer block, a sub-window state space layer, and a block state space layer. The output features of the sub-window state space layer and the block state space layer of each encoding module are residually concatenated and used as the output features of the encoding module. The output features of the Transformer block of the previous encoding module are residually concatenated with the output features of the Transformer block of the next encoding module and used as the input features of the sub-window state space layer of the next encoding module. The sub-window state space layer divides the input features into multiple sub-windows, performs feature extraction on each sub-window, and obtains multiple sub-features. All sub-features are input into the block state space layer for further feature extraction. Step S3: Train the segmentation model using the dataset, and use the trained segmentation model for left atrial segmentation.
2. The dual-channel cross-fusion left atrial segmentation method based on SAM and SSM according to claim 1, characterized in that, In the sub-window state space layer, the input features are divided into multiple sub-windows, and then bidirectional Mamba blocks are used to extract features on each sub-window, resulting in multiple sub-features P1, P2, ..., P k k represents the number of sub-features; sub-features P1, P2, ..., P k The input is fed into a block-based state space layer. Each sub-feature undergoes pooling, bidirectional Mamba block, and unpooling operations in sequence to obtain features F1, F2, ..., F... k Features F1, F2, ..., F k After being concatenated together, each feature is then residually connected to its corresponding sub-feature to obtain the output features of the block state space layer.
3. The dual-channel cross-fusion left atrial segmentation method based on SAM and SSM according to claim 2, characterized in that, The input features of the bidirectional Mamba block are flattened to obtain feature R. On one hand, feature R is subjected to projection, convolution, and activation operations, and then passed through a forward state space layer to obtain feature R2. On the other hand, feature R is passed through a max pooling layer and an average pooling layer respectively. The output features of the two pooling layers are added together, and then subjected to projection, unpooling, and activation operations in sequence to obtain feature R1. Feature R1 is multiplied element-wise with feature R2 to obtain feature R3. Feature R1 is passed through a backward state space layer and then multiplied element-wise with feature R2 to obtain feature R4. Feature R4 is added with R3, then projected, and then residually connected with feature R to obtain the output features of the bidirectional Mamba block.
4. The dual-channel cross-fusion left atrial segmentation method based on SAM and SSM according to claim 3, characterized in that, The forward state space layer and the backward state space layer have the same structure. The input feature G passes through a linear layer, a deep convolutional layer and an activation operation in sequence to obtain feature G1. At the same time, the input feature G passes through a linear layer, a deep convolutional layer, an activation operation, an SS2D module and a normalization layer in sequence to obtain feature G2. Feature G2 and feature G1 are multiplied element-wise and then passed through a linear layer to obtain the output feature.
5. The dual-channel cross-fusion left atrial segmentation method based on SAM and SSM according to any one of claims 1 to 4, characterized in that, The Transformer group contains four Transformer blocks. The output features of the first Transformer block and the output features of the third Transformer block are residually concatenated and used as the input features of the fourth Transformer block. The output features of the second Transformer block and the output features of the fourth Transformer block are residually concatenated and used as the output features of the Transformer group.
6. The dual-channel cross-fusion left atrial segmentation method based on SAM and SSM according to claim 5, characterized in that, In the Transformer block, the input features are passed through layer normalization, multi-head attention, and adapter in sequence, and then residually connected with themselves to obtain feature M; After feature M is normalized by layers and processed by a multilayer perceptron, it is then residually connected to itself to obtain the output features of the Transformer block.
7. The dual-channel cross-fusion left atrial segmentation method based on SAM and SSM according to claim 6, characterized in that, The input features of the adapter are sequentially processed through downsampling, pooling, upsampling, and activation, and then residually connected with itself to obtain the output features of the adapter.
8. The dual-channel cross-fusion left atrial segmentation method based on SAM and SSM according to claim 1, characterized in that, In the cross-attention module, the output features of the Trans encoder and the TS encoder are first concatenated to obtain the embedding vector S. Half of the embedding vector S is used as the query Q of the attention mechanism, and the other half is used as the key K and value V of the attention mechanism, respectively. The query Q and key K are passed through a linear layer and then subjected to element-wise dot product operation to obtain the similarity vector. The similarity vector is subjected to attention weight scaling operation and Softmax function to obtain the attention weight vector. The value V is passed through a linear layer and then subjected to element-wise dot product operation with the attention weight vector to obtain the attention embedding vector S′, which is the output of the cross-attention module.
9. The dual-channel cross-fusion left atrial segmentation method based on SAM and SSM according to claim 1, characterized in that, During training, the total model loss L is calculated using equation (1). tra ; L tra =Lc ro +L EMD (1) In the formula, L cro It is the cross-entropy loss of the dual-channel encoder, L EMD It is the regularization loss of the dual-channel encoder; Cross-entropy loss L of a dual-channel encoder cro Defined as: L cro =H(v TS ,v Trans )+H(v TS ,v TS ) (2) In the formula, H(v) TS v Trans ) is the output feature v of the TS encoder TS With the output features v of the Trans encoder Trans The cross-entropy between them, H(v) TS v TS ) is the output feature v of the TS encoder TS Cross-entropy with itself; Regularization loss L EMD Defined as: In the formula, It is the EMD adaptation parameter between the output characteristics of the TS encoder and the output characteristics of the Trans encoder. These are the EMD adaptation parameters between the output features of the Trans encoder and itself. It is the EMD adaptation parameter between the output characteristics of the TS encoder and itself.
10. The dual-channel cross-fusion left atrial segmentation method based on SAM and SSM according to claim 9, characterized in that, Cross-entropy H(v) TS v Trans The formula for calculating ) is: H(v TS ,v Trans )=-logp(v Trans |v TS ) (3) In the formula, π(v TS ) is the attention embedding vector output by the cross-attention module, τ is the temperature, and w Trans It refers to the embedded word vectors of the Trans encoder.
Citation Information
Patent Citations
Medical image segmentation method and system based on double-branch embedded attention mechanism
CN116309650A
Spine three-dimensional structure reconstruction method based on deep learning
CN116402954A
