Semantic Segmentation System for Medical Image Sequences

Abstract

The present invention belongs to the field of medical image processing technology, specifically a semantic segmentation system for medical image sequences. The semantic segmentation system of the present invention includes a cross-frame attention module and a boundary-aware Transformer; it adopts a positioning-first-refinement model, that is, first through the cross-frame attention module, the temporal relationship between consecutive frames of the input image is mined, the correlation between frames is modeled, and the image is roughly segmented, and the target area is positioned by focusing on it; then the boundary-aware Transformer is used to further refine the target object to improve the segmentation accuracy; by learning the dependency between the image blocks extracted along the boundary, these image blocks will be recombined after processing to obtain the final segmentation mask. The present invention has achieved excellent segmentation performance on both the cardiac segmentation magnetic resonance image dataset and the polyp segmentation endoscopic dataset, and can also be widely extended to other modalities of medical images, such as CT scan images, ultrasound images, etc.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a semantic segmentation system for medical image sequences. Background Art

[0002] Automatic detection and segmentation of key areas in medical images can effectively help doctors improve the accuracy and efficiency of diagnosis. In clinical practice, many modalities of medical images are collected in the form of videos. For example, colonoscopy is a key technology required for the diagnosis of colorectal cancer, and each examination generates a video. Although some methods have been used to handle automatic polyp segmentation in static images, few works have paid attention to the temporal relationship between consecutive frames. The frames in a video usually come from an independent patient and contain the same target object (such as a polyp), so their foreground and background are highly similar. This feature also exists in magnetic resonance images and ultrasound images.

[0003] In the past decade, convolutional neural networks have made great breakthroughs in medical image segmentation. Modern semantic segmentation architectures are usually based on encoder-decoder networks. The success of convolutional neural networks based on deep encoder-decoders is largely due to their skip connections, but due to the local nature of convolution, convolutional neural networks require a large receptive field to focus on the entire image, resulting in a deep network structure and more network parameters. Visual Transformer was introduced into the image segmentation task to capture global information and learn long-range dependencies. However, due to the blurred boundaries and low contrast of medical images, both convolutional neural network-based and Transformer-based medical image segmentation methods often have the problem of inaccurate boundaries between foreground and background, which limits the segmentation performance.

[0004] Since medical images of various modalities have the nature of sequences, the segmentation of medical image sequences is also closely related to video target segmentation. Depending on the settings of the test phase, video target segmentation can be divided into semi-supervised visual target segmentation and unsupervised video target segmentation. Semi-supervised video target segmentation gives the segmentation mask of the target in the first frame, while unsupervised video target segmentation needs to extract the mask of the segmented target without any prior knowledge. For semi-supervised video target segmentation, some works use convolutional layers to encode frame features, and some works use Transformer-based architectures to capture temporal and spatial relationships. For unsupervised video target segmentation, many works focus on the importance of the intrinsic correlation between video frames and use attention mechanisms to mine this information. The present invention models the relationship between video frames in the medical image segmentation task and makes full use of the time series information in medical images.

[0005] Based on the above analysis, the present invention adopts a cross-frame attention module to mine the correlation between video frames, and improves the Transformer to address the boundary blur problem, further improving the segmentation performance. Summary of the Invention

[0006] The problem addressed by the present invention is the segmentation of medical image sequences. Existing work has two main deficiencies: (1) Most work focuses on static image segmentation, ignoring the sequential relationships of images acquired by technologies such as magnetic resonance imaging, endoscopy, and ultrasound, as well as the similar appearance between consecutive frames and the correlation across frames; (2) Whether based on convolutional neural networks or Transformer-based medical image segmentation methods, segmentation performance is limited by the low contrast and fuzzy boundaries of medical images, and the dependencies between image blocks within a frame are not well explored. To address the above problems, the present invention provides a semantic segmentation system for medical image sequences.

[0007] The semantic segmentation system for medical image sequences provided by the present invention includes a cross-frame attention module for modeling inter-frame relationships and a boundary-aware Transformer for learning intra-frame dependencies. The present invention adopts a positioning-first-then-refinement mode (different from previous work using Transformer encoders as feature extractors), that is, first, through the cross-frame attention module, the temporal relationship between consecutive frames of the input image is mined, the correlation between frames is modeled, and the image is roughly segmented, and the location is achieved by focusing on the target area. Then, the boundary-aware Transformer is used to further refine the target object, and by learning the dependency between image blocks extracted along the boundary, these image blocks are recombined after processing to obtain the final segmentation mask.

[0008] In the present invention, the cross-frame attention module is used to mine the temporal relationship between consecutive frames of the input image, model the correlation between frames, and perform rough segmentation of the image, and locate the target area by focusing on it, specifically including:

[0009] In a medical image sequence, objects in consecutive frames usually have similar foreground and background. For a continuous input of T frames (e.g., T is usually 3-6, and 3 is taken in this embodiment), a convolutional neural network-based backbone network is used to extract features for each frame. The backbone network uses the first four layers of Res2Net as feature extractors, and each layer outputs a feature map for each frame, denoted as Among them, C l 、H l and W lThe dimension, height and width of the feature map output by the first layer are respectively set; the cross-frame attention module uses the feature map output by the third layer to generate a key vector and a value vector for each frame, and then sequentially uses each frame as a query frame and other frames as reference frames to provide key vectors and value vectors; the cross-frame attention module uses the timing information between consecutive frames to realize the interaction between consecutive frames and model the relationship between frames, thereby achieving rough segmentation of the image and locating the target object;

[0010] In the present invention, the boundary-aware Transformer is used to further refine the target object by learning the dependencies between the image blocks extracted along the boundary. After processing, these image blocks are recombined to obtain the final segmentation mask, which specifically includes:

[0011] The boundary-aware Transformer extracts image mask blocks along the boundaries in the coarse segmentation image, then concatenates these boundary mask blocks with the corresponding original image blocks and inputs them into the Transformer for refinement; finally, the image mask blocks are reassembled and input into the decoder to produce a segmentation prediction map.

[0012] According to the two different stages of localization and refinement, the loss function consists of two parts:

[0013]

[0014] in, and denote the loss functions of the localization stage and refinement stage respectively; λ is a hyperparameter used to balance the loss terms of the two stages.

[0015] The loss functions of both stages use a combination of weighted binary cross entropy loss and weighted intersection-over-union loss. Compared with the standard binary cross entropy loss and intersection-over-union loss, the weighted binary cross entropy loss and weighted intersection-over-union loss Focus more on pixels that are more difficult to segment, rather than assigning the same weight to all pixels. For the loss of the localization stage, M1 represents the segmentation mask output by the cross-frame attention module, and Y represents the true value label of the image. The loss of the localization stage is obtained by calculating the difference between M1 and the true value Y.

[0016]

[0017] Calculate the difference between the final predicted segmentation mask M2 and the true value Y to obtain the loss of the refinement stage

[0018] In the present invention, the cross-frame attention module uses two independent convolutional layers to learn the feature embedding of each frame for the input T-frame continuous image; each convolutional layer is attached to the third layer of the feature extractor to generate a key vector for each frame image. Sum value vector C k and C v Represent the feature dimensions of the key vector and value vector respectively. For the t-th frame, the key vector and value vector can be expressed as:

[0019]

[0020]

[0021] Here, ψ(·) and φ(·) represent two parallel 3×3 convolutional layers.

[0022] The key vector and value vector of the query frame are regarded as the feature embedding of the query frame, and the other frames provide key vectors and value vectors for reference. The cross-frame attention module highlights the location of the target object by calculating the similarity between the key vectors of the query frame and the reference frame. The key vector is used to encode robust high-level visual semantic information, and the value vector stores the detailed information used to generate the target object mask prediction. The key vector K of the query frame q and the key vector K of the reference frame r The similarity at each position can be expressed as:

[0023] S q,i =softmax(K q,i ⊙[K r ] j ), (6)

[0024] Where ⊙ represents matrix multiplication; [·] represents channel-wise concatenation; i and j represent the position indexes on the query frame and the reference frame, respectively. Taking the first frame T1 of the image sequence as an example, when T1 is used as the query frame, the other T-1 frames are used as reference frames to provide key vectors and value vectors. At this time, the similarity between the first frame T1 and the reference frame is Get the similarity S q After that, the feature map after the query frame and the reference frame interact can be calculated based on the weighted sum of the value vector:

[0025]

[0026] Where D is the normalization factor, V q and V rDenote the value vectors of the query frame and the reference frame, respectively. Through the inter-frame attention operation, the cross-frame attention module can obtain position-sensitive features to help locate the target object. The key vector and value vector of the reference frame are first spliced ​​on the channel in formulas (6) and (7) to provide information for locating the target object in the query frame.

[0027] After the interaction is completed, this system uses the channel attention operation Att c (·) and the spatial attention operation Att s (·) Further explore the information contained in these feature maps, denoted as

[0028]

[0029] Among them, Att c (·) represents the channel attention operation, which is defined as:

[0030] Att c (x)=σ(w1(P avg (x))+w2(P max (x)))⊙x, (9)

[0031] Where x is the input vector; P avg (·) and P max (·) denotes global average pooling and global maximum pooling respectively; w1(·) and w2(·) share parameters and consist of a 1×1 convolutional layer, a ReLU activation layer, and a 1×1 convolutional layer; σ(·) is the sigmoid function.

[0032] Among them, the spatial attention operation Att s (·) is defined as:

[0033]

[0034] Among them, Q avg (·) and Q max (·) represents the average and maximum values ​​in the channel dimension respectively; Represents a 7×7 convolutional layer with padding of 3.

[0035] Will Fuse with the feature maps output by the second and fourth layers of the feature extraction network to obtain the predicted segmentation mask;

[0036] First fuse it with the feature map of the second layer to get

[0037]

[0038] Where conv(·) represents the convolution operation; Represents an upsampling operation;

[0039] Then fuse it with the feature map of the fourth layer to obtain the predicted segmentation mask of the positioning stage

[0040] By fusing low-level and high-level features, the present invention can simultaneously obtain semantic information and appearance details of the target object.

[0041] In the present invention, the boundary-aware Transformer is used to further improve the segmentation accuracy at the boundary. Distinguishing the boundary between foreground and background is a long-standing challenge faced by medical image segmentation tasks. Some work has used various refinement methods to predict more accurate target object boundaries. Recently, visual Transformers have shown strong performance in capturing global context, so this system designs a boundary-aware Transformer to improve boundary prediction accuracy. The cross-frame attention module has provided segmentation maps (rough segmentation) containing the location information of the target object, which will be refined through a shared Transformer encoder.

[0042] Given a coarse mask generated by the cross-frame attention module, the segmentation results for the interior of the target object are generally robust due to geometric constraints and appearance features. Therefore, the refinement phase focuses primarily on the boundaries of the target object. This system uses a convolutional layer to detect the approximate boundaries of the segmented object. Square image mask patches are then extracted along these boundaries, with the centers of these patches falling on boundary pixels. During the extraction process, these patches are assigned confidence scores based on the center boundary pixels. These patches still contain a significant amount of overlap and redundancy, so the K highest-scoring patches are selected for further processing, ranked from high to low in confidence score. The number of selected patches can be adjusted to optimize the balance between speed and accuracy. Simultaneously, the image patches at the corresponding locations in the original image are extracted and concatenated with the K mask patches before being fed into a Transformer encoder. This Transformer encoder learns how to classify pixels near the boundaries. These concatenated patches are first transformed into a vector through a linear mapping layer, which serves as the input to a boundary-aware Transformer. The boundary-aware Transformer consists of n Transformer layers (n can range from 2 to 6, but in this example, n is 4). Each Transformer layer includes multi-head self-attention and multi-layer perceptron, with l t Represents each layer of Transformer, the lth t The output of the layer looks like this:

[0043]

[0044]

[0045] Among them, MSA(·) and MLP(·) represent multi-head attention and multi-layer perceptron respectively; LN(·) represents layer normalization operation; and Represent the output of the previous layer and the current layer respectively. The boundary-aware Transformer only accepts image patches at the target boundary instead of the image patches obtained by full image segmentation, which can focus more on learning the dependencies between boundary patches.

[0046] Finally, these image patches are reassembled based on the extraction positions to produce a predicted segmentation map of the same size as the original image. The overlapping regions between image patches are equally weighted, and the segmentation results for these overlapping regions are averaged. These reassembled segmentation maps are then fused and decoded with the multi-level feature maps output by the feature extraction network to produce the final segmentation mask.

[0047] Advantages of the present invention include:

[0048] First, a model for segmenting medical image sequences is proposed, which adopts the method of first localization and then refinement to segment the target.

[0049] Secondly, a cross-frame attention module is designed to model the correlation between frames, and a boundary-aware Transformer is introduced to learn the dependencies between boundary blocks within a frame;

[0050] Finally, the proposed method was experimented on two medical image sequence segmentation tasks. The results showed that the proposed method demonstrated superior performance across multiple datasets for heart segmentation and polyp segmentation. Furthermore, the proposed method also exhibited good generalization capabilities, outperforming other medical image segmentation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a model framework diagram in the present invention.

[0052] Figure 2 It is a diagram of the cross-frame attention module in the present invention.

[0053] Figure 3 This is a diagram of the boundary-aware Transformer in the present invention.

[0054] Figure 4 This is a comparison of the results of this system and other medical image segmentation models on the heart segmentation dataset.

[0055] Figure 5This is a comparison of the results of this system and other medical image segmentation models on the polyp segmentation dataset. DETAILED DESCRIPTION

[0056] The present invention will be further described below with reference to the accompanying drawings and examples.

[0057] like Figure 1 As shown, the present invention adopts a localization-then-refinement architecture, employs a cross-frame attention module to explore the temporal relationship between consecutive frames, models the correlation between frames, and then uses a boundary-aware Transformer to learn the dependencies between boundary image blocks within a frame. The workflow of the present invention is as follows.

[0058] (I) The first positioning and then refinement architecture. Input T frames of continuous images, T is set to 3 in this experiment. First, use the backbone network Res2Net to extract features and output four feature maps of different scales for each frame. Among them C l 、H l and W l The dimensions, height, and width of the feature map output for the lth layer are respectively. The cross-frame attention module uses the feature map output by the third layer to generate a key vector and a value vector for each frame, and then takes each frame as a query frame in turn, and the other frames as reference frames that provide key vectors and value vectors. By utilizing the temporal information between consecutive frames, the cross-frame attention module realizes the interaction between consecutive frames and models the relationship between frames. After locating the target object, the boundary-aware Transformer extracts image mask blocks along the boundaries in the coarse segmentation map, and then splices these boundary mask blocks with the corresponding original image blocks and inputs them into the Transformer for refinement. Finally, the image mask blocks are reassembled and input into the decoder to generate a segmentation prediction map.

[0059] According to the two different stages of localization and refinement, the loss function of the architecture consists of two parts:

[0060]

[0061] in, and Denote the loss functions of the localization stage and refinement stage respectively; λ is a hyperparameter used to balance the loss terms of the two stages. In this experiment, λ is set to 1. The loss functions of both stages use a combination of weighted binary cross entropy loss and weighted intersection-over-union loss. For the loss of the localization stage, M1 represents the segmentation mask output by the cross-frame attention module, and Y represents the true value label of the image. By calculating the difference between M1 and the true value Y, the loss of the localization stage can be obtained.

[0062]

[0063] For the loss of the refinement stage, M2 is used to represent the final predicted segmentation mask. By calculating the difference between M2 and the true value Y, the loss of the refinement stage can be obtained.

[0064]

[0065] (2) The calculation process of the cross-frame attention module is as follows Figure 2 As shown. For the input T consecutive frames, the cross-frame attention module uses two parallel 3×3 convolutional layers to learn the feature embedding of each frame. Each convolutional layer is attached to the third layer of the feature extractor to generate a key vector for each frame image. Sum value vector C k and C v They represent the feature dimensions of key vector and value vector respectively, which are set to 128 and 512 in the experiment.

[0066] The cross-frame attention module highlights the location of the target object by calculating the similarity between the key vectors of the query frame and the reference frame. The key vector K of the query frame q and the key vector K of the reference frame r The similarity at each position can be expressed as:

[0067] S q,i =softmax(K q,i ⊙[K r ] j ), (4)

[0068] Where ⊙ represents matrix multiplication; [·] represents channel-wise concatenation; i and j represent the position indices on the query frame and reference frame, respectively. After obtaining the similarity, the feature map of the query frame and the reference frame can be calculated based on the weighted sum of the value vector:

[0069]

[0070] Among them, D is the normalization factor, which is set to 256 in this experiment. After the interaction is completed, this system uses channel attention and spatial attention to further mine the information contained in these feature maps, and obtains

[0071]

[0072] Among them, Att c (·) represents the channel attention operation, which is defined as:

[0073] Att c (x)=σ(w1(P avg (x))+w2(P max (x)))⊙x, (7)

[0074] Where x is the input vector; P avg (·) and P max (·) represents global average pooling and global maximum pooling respectively; w1(·) and w2(·) share parameters and consist of a 1×1 convolution layer, a ReLU activation layer, and a 1×1 convolution layer. Spatial attention operation Att s (·) is defined as:

[0075]

[0076] Among them, Q avg (·) and Q max (·) represents the average and maximum values ​​in the channel dimension respectively; Represents a 7×7 convolutional layer with padding of 3.

[0077] After channel attention and spatial attention operations, Fuse with the feature maps output by the second and fourth layers of the feature extraction network. First fuse with the feature map of the second layer:

[0078]

[0079] Among them, conv(·) represents the convolution operation; Represents an upsampling operation. It is then fused with the feature map of the fourth layer to obtain the predicted segmentation mask of the positioning stage:

[0080]

[0081] (3) The computational process of the boundary-aware Transformer is as follows Figure 3 As shown in Figure 2, the boundary-aware Transformer is used to further improve the segmentation accuracy at the boundary. The cross-frame attention module has provided segmentation maps containing the location information of the target object, which will be refined by a shared Transformer encoder.

[0082] Given a coarse mask generated by the cross-frame attention module, this system uses a convolutional layer to detect the approximate boundaries of the segmented object. Square image mask patches are then extracted along these boundaries, with their centers falling on boundary pixels. During the extraction process, these patches are assigned a confidence score for the center boundary pixel. These patches still contain a significant amount of overlap and redundancy, so the K highest-scoring patches are selected for further processing, ranked by confidence score. In this experiment, K is set to 500. The number of selected patches can be adjusted to achieve a better balance between speed and accuracy. The selected patches are cropped to ensure that their coordinates do not exceed the maximum image width and length. Simultaneously, the corresponding patches in the original image are extracted and concatenated with the K mask patches before being fed into a Transformer encoder. These concatenated patches are first transformed into a vector through a linear mapping layer. Given this vector as input, the boundary-aware Transformer applies n Transformer layers, with n set to 4 in this experiment. Each Transformer layer consists of multi-head self-attention and a multi-layer perceptron, using l t Represents each layer of Transformer, the lth t The output of the layer looks like this:

[0083]

[0084]

[0085] Among them, MSA(·) and MLP(·) represent multi-head attention and multi-layer perceptron respectively; LN(·) represents layer normalization operation; and Representing the output of the previous and current layers, respectively. Finally, these image patches are reassembled based on their extraction locations to produce a predicted segmentation map of the same size as the original image. The overlapping regions between image patches are equally weighted, and the averaged segmentation result is calculated for the overlapping regions. These reassembled segmentation maps are then fused and decoded with the multi-level feature maps output by the feature extraction network to produce the final segmentation mask.

[0086] During the training phase, the system was trained for 100 epochs with a batch size of 4. Gradient descent was performed using the Adam optimizer with a learning rate of 0.0001. The Transformer encoder used a 16×16 boundary patch size and four attention heads. The input frames were resized to 352×352, with random data augmentation including translation, rotation, and flipping. During the testing phase, images were only resized to 352×352 for evaluation without any post-processing.

[0087] This system was experimented on two tasks: heart segmentation and polyp segmentation. Heart segmentation used the RVSC2012 magnetic resonance image dataset with an image resolution of 216×256. RVSC2012 contains 48 cases, which are evenly divided into a training set, a test set 1, and a test set 2. This system was trained on the training set and tested and evaluated on two test sets. Polyp segmentation used three colonoscopy datasets: CVC-300, CVC-612, and ETIS, with image resolutions of 574×500, 384×288, and 1225×966, respectively. In the experiment, 60% of the data from CVC-300 and CVC-612 was used as the training set, and the rest was used as the test set. The entire ETIS sequence was used as the test set.

[0088] On the heart segmentation dataset RVSC2012, test sets 1 and 2, our system achieved Dice scores of 83.0% and 86.2%, and Hausdorff distances of 4.187 and 4.026, respectively. On test set 2, the Dice score exceeded the previous state-of-the-art work by 1.7%, demonstrating superior segmentation performance. On the two polyp segmentation test sets, CVC-612-T and CVC-612-V, the mDice scores reached over 80%, surpassing the previous state-of-the-art polyp segmentation methods. On the ETIS dataset used for all tests, all methods achieved lower performance on ETIS because it was not visible during training. By exploiting inter- and intra-frame correlations, our system outperformed other methods on ETIS. Figure 4 and Figure 5 The following table shows the comparison results of our system with other methods. The first column shows the input of three consecutive images, the second column shows the image annotations, and the third column shows the prediction results of our system. Our system achieves better segmentation results through inter-frame interaction and intra-frame refinement.

[0089] In summary, this paper proposes a novel medical image sequence segmentation system. It uses a cross-frame attention module and a boundary-aware Transformer to model inter- and intra-frame correlations, and performs segmentation on a cardiac magnetic resonance image dataset and a polyp segmentation endoscopic image dataset.

Claims

1. A semantic segmentation system for medical image sequences, characterized by: The proposed method includes a cross-frame attention module for modeling inter-frame relationships and a boundary-aware Transformer for learning intra-frame dependencies. It adopts a localization-first-refinement model, that is, first using the cross-frame attention module to explore the temporal relationship between consecutive frames of the input image, model the correlation between frames, and perform rough image segmentation, localizing the target area by focusing on it. The boundary-aware Transformer is then used to further refine the target object by learning the dependencies between image blocks extracted along the boundaries. After processing, these image blocks are recombined to obtain the final segmentation mask. The cross-frame attention module is used to explore the temporal relationship between consecutive frames of the input image, model the correlation between frames, and perform rough image segmentation. The target area is positioned by focusing on it, specifically including: For the input T-frame continuous images, a backbone network based on convolutional neural network is used to extract features for each frame; the backbone network uses the first four layers of Res2Net as feature extractors, and each layer outputs a feature map of each frame image, denoted as Among them, C l 、H l and W l The dimension, height and width of the feature map output by the first layer are respectively; the cross-frame attention module uses the feature map output by the third layer to generate a key vector and a value vector for each frame, and then sequentially uses each frame as a query frame and other frames as reference frames to provide key vectors and value vectors; the cross-frame attention module uses the timing information between consecutive frames to interact between consecutive frames and model the relationship between frames, thereby achieving rough segmentation of the image and locating the target object; The boundary-aware Transformer is used to further refine the target object by learning the dependencies between the image blocks extracted along the boundary. After processing, these image blocks are recombined to obtain the final segmentation mask, which specifically includes: The boundary-aware Transformer extracts image mask blocks along the boundaries of the coarse segmentation image, then concatenates these boundary mask blocks with the corresponding original image blocks and inputs them into the Transformer for refinement. Finally, the image mask blocks are reassembled and input into the decoder to produce the segmentation prediction map. According to the two different stages of localization and refinement, the loss function consists of two parts: in, and Represent the loss functions of the localization stage and refinement stage respectively; λ is a hyperparameter used to balance the loss terms of the two stages; For the loss of the localization stage, M1 represents the segmentation mask output by the cross-frame attention module, and Y represents the true value label of the image; the loss of the localization stage is obtained by calculating the difference between M1 and the true value Y Calculate the difference between the final predicted segmentation mask M2 and the true value Y to obtain the loss of the refinement stage 2. The semantic segmentation system for medical image sequences according to claim 1, characterized in that The cross-frame attention module uses two independent convolutional layers to learn the feature embedding of each frame for the input T-frame continuous image; each convolutional layer is attached to the third layer of the feature extractor to generate a key vector for each frame image. Sum value vector C k and C v Represent the feature dimensions of the key vector and value vector respectively; for the t-th frame, the key vector and value vector can be expressed as: Where ψ(·) and φ(·) represent two parallel 3×3 convolutional layers; The key vector and value vector of the query frame are regarded as the feature embedding of the query frame, and other frames provide key vectors and value vectors for reference; the cross-frame attention module highlights the location of the target object by calculating the similarity between the key vectors of the query frame and the reference frame; the key vector is used to encode robust high-level visual semantic information, and the value vector stores the detailed information used to generate the target object mask prediction; the key vector K of the query frame q and the key vector K of the reference frame r The similarity at each position is expressed as: S q,i =softmax(K q,i ⊙[K r ] j ), (6) Where ⊙ represents matrix multiplication; [·] represents channel-wise concatenation; i and j represent the position indexes on the query frame and the reference frame, respectively; the similarity S is obtained. q Finally, the feature map after the query frame and the reference frame interact is calculated based on the weighted sum of the value vector: Where D is the normalization factor, V q and V r Represent the value vectors of the query frame and the reference frame respectively; through the inter-frame attention operation, the cross-frame attention module obtains position-sensitive features to help locate the target object; the key vector and value vector of the reference frame are first spliced ​​on the channel in formula (6) and formula (7) to provide information for locating the target object in the query frame; After the interaction is completed, the channel attention operation Att is used c (·) and the spatial attention operation Att s (·), further explore the information contained in these feature maps, denoted as The channel attention operation Att c (·), defined as: To c (x)=σ(w1(P avg (x))+w2(P max (x)))⊙x, (9) Where x is the input vector; P avg (·) and P max (·) denotes global average pooling and global maximum pooling respectively; w1(·) and w2(·) share parameters and consist of a 1×1 convolutional layer, a ReLU activation layer, and a 1×1 convolutional layer; σ(·) is the sigmoid function; The spatial attention operation Att s (·), defined as: Among them, Q avg (·) and Q max (·) represents the average and maximum values ​​in the channel dimension respectively; Represents a 7×7 convolutional layer with padding of 3; Will Fuse with the feature maps output by the second and fourth layers of the feature extraction network to obtain the predicted segmentation mask; First fuse it with the feature map of the second layer to get Among them, conv(·) represents the convolution operation; Represents an upsampling operation; Then fuse it with the feature map of the fourth layer to obtain the predicted segmentation mask of the positioning stage By fusing low-level and high-level features, semantic information and appearance details of the target object are obtained simultaneously.

3. The semantic segmentation system for medical image sequences according to claim 2, characterized in that The boundary-aware Transformer is used to further improve segmentation accuracy at boundaries, specifically including: Given a coarse mask generated by the cross-frame attention module, a convolutional layer is used to detect the approximate boundaries of the segmented object. Square image mask blocks are then extracted along the boundaries, with the centers of these blocks falling on the boundary pixels. During the extraction process, these blocks are also assigned confidence scores of the center boundary pixels. These blocks still contain a lot of overlap and redundancy, so the K blocks with the highest scores are selected for subsequent processing according to the confidence score from high to low. At the same time, the image blocks at the corresponding positions in the original image are extracted and concatenated with K mask blocks before being input into a Transformer encoder. These concatenated image blocks are first transformed into a vector through a linear mapping layer, and this vector is used as the input of the boundary-aware Transformer. The boundary-aware Transformer consists of n Transformer layers. Each Transformer layer includes multi-head self-attention and multi-layer perceptrons, using l t Represents each layer of Transformer, the lth t The output of the layer looks like this: Among them, MSA(·) and MLP(·) represent multi-head attention and multi-layer perceptron respectively; LN(·) represents layer normalization operation; and Represent the output of the previous layer and the current layer respectively; Finally, these image blocks are recombined according to the extraction position to obtain a predicted segmentation map of the same size as the original image; the overlapping areas between image blocks equally consider the contribution of each image block, so the average value is calculated to obtain the segmentation result of the overlapping area; these recombined segmentation maps are then fused and decoded with the multi-level feature maps output by the feature extraction network to obtain the final segmentation mask.

Images