Extremely few sample medical image segmentation method based on bidirectional attention guidance

Abstract

The invention provides an extremely few sample medical image segmentation method based on bidirectional attention guidance, and the method comprises the steps: integrating a bidirectional attention mechanism into a feature extraction and loss function of a segmented arbitrary model, and segmenting a few sample medical image; wherein the bidirectional attention mechanism comprises a spatial domain attention module, a frequency domain attention module and an attention gradient updating mechanism; the spatial domain and frequency domain attention module combines frequency domain feature processing and spatial domain feature processing; the spatial domain and frequency domain attention module is used for capturing fine and rough features; the attention gradient updating mechanism is used for dynamically adjusting the loss weight; according to the dual method provided by the invention, feature capture and correlation are improved, and the segmentation accuracy is improved. According to the method, through verification of a plurality of medical image data sets, the performance of full supervision can be achieved only through two label samples.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image segmentation methods, and specifically relates to a very small sample medical image segmentation method based on bidirectional attention guidance. Background Art

[0002] The Segmentally Arbitrary Model (SAM) has attracted much attention due to its excellent performance in natural image segmentation tasks. However, due to the lack of training on medical-specific data, its effectiveness is reduced when applied to medical image segmentation.

[0003] To address this problem, many people have fine-tuned SAM using large amounts of annotated data. However, this data is difficult to obtain due to high costs, time constraints, expert requirements, and additional complexities such as imaging equipment variations and privacy issues. These factors have created significant obstacles to the application of SAM in medical image segmentation. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a very few-sample medical image segmentation method based on bidirectional attention guidance for few-sample medical image segmentation.

[0005] The technical solution of the present invention is: a few-sample medical image segmentation method based on bidirectional attention guidance, comprising: integrating a bidirectional attention mechanism into feature extraction and loss functions of a segmented arbitrary model to segment the few-sample medical image;

[0006] The bidirectional attention mechanism includes: spatial domain and frequency domain attention modules, and attention gradient update mechanism;

[0007] The spatial and frequency domain attention module combines frequency domain feature processing and spatial domain feature processing;

[0008] The spatial domain and frequency domain attention modules are used to capture fine and coarse features; the attention gradient update mechanism is used to dynamically adjust the loss weight;

[0009] The dataset of the few-sample medical images is represented as D = {(x i ,y i )|1≤i≤N}, where N is the total number of labeled images in D, let x i ∈R H×W×3 and i ∈R H×W denote the i-th image and its corresponding label, respectively, where H and W denote the height and width of the image, respectively.

[0010] Preferably, the frequency domain feature processing includes:

[0011] S11: Image x i ∈RH×W×3 It is input into the image encoder and the key spatial features are extracted using Transformer;

[0012] S12: The key spatial features are refined through the convolutional layer to produce a more structured feature representation F t ∈R H×W×256 ,t∈{1,2,3,4};

[0013] S13: Apply Fourier transform to transform the spatial domain feature map F t ∈R H×W×256 Converted to the frequency domain, expressed as F t ' ∈ R H ×W×256 ;

[0014] S14: In the frequency domain, perform average pooling and pass the result to the fully connected layer to generate the attention map A f ;

[0015] S15: Apply a Gaussian filter to smooth the feature map F t ' ∈ R H×W×256 , generating F t ”∈R H×W×256 ;

[0016] S16: Apply inverse Fourier transform to transform F t ” ∈ R H×W×256 Converting from the frequency domain back to the spatial domain, we get F t ”' ∈ R H×W×256 ; It is then used as the subsequent spatial domain feature processing flow; as shown below:

[0017]

[0018] Among them, t represents the number of layers in the feature map, u and v correspond to the coordinates of the feature points in the frequency domain, and σ represents the standard deviation, which is used to control the width of the Gaussian kernel.

[0019] Preferably, the spatial domain feature processing utilizes a dual-path strategy to capture detailed local features and broader contextual information;

[0020] In the first path, multi-scale pooling is applied to F t ”'∈R H×W×256 , and then the pooled results are processed through a fully connected layer to generate an attention map A c ;

[0021] In the second path, F is calculated t ”'∈R H×W×256The maximum and average values ​​of F t ”'∈R H×W×256 The maximum and average values ​​of are merged along the channel dimension, and then the merged feature map is processed using a multi-scale convolution kernel to produce an attention map A. s ; A c and A s Merge by element-wise multiplication to create A I , then A I For the original feature map F t ”'∈R H×W×256 Processed to generate an enhanced feature map F t ”” ∈ R H×W×256 ;

[0022] Finally, F t ”” ∈ R H×W×256 Layer-by-layer fusion and processing are performed through 3×3 convolutional layers to merge high-level semantic information with spatial details.

[0023] Preferably, the attention gradient update mechanism comprises: in each training batch, firstly calculating the gradient of each loss function with respect to the model parameters and accumulating the absolute sum of the gradients; then performing a dynamic weight update to reflect the relative importance of each loss in the current batch; the new weight W of each loss l {l∈1,2,3} is the gradient of the current loss g i Determined inversely proportional to the sum of the gradients of all losses, the final total loss is calculated as the weighted sum of the individual losses with the new weights.

[0024] Preferably, the attention gradient update mechanism combines the Dice loss L dice , binary cross entropy loss L bce and boundary loss L boundary ;

[0025] The attention gradient update mechanism is used to dynamically adjust the loss weight. The specific formula is as follows:

[0026] L total =W1·L dice +W2·L bce +W3·L boundary (2)

[0027]

[0028] Among them, ε represents the smoothing factor; G x and G y Represents a Sobel filter for detecting horizontal and vertical edges; represents the prediction result of the model, and M is the total number of pixels.

[0029] Preferably, boundary points and boundary boxes are also introduced into the segmented arbitrary model, including: firstly using edge detection to identify the edge of the image, and then randomly sampling coordinates along the edge as embedded boundary point prompts.

[0030] The present invention also provides a few-sample medical image segmentation system based on bidirectional attention guidance, including: a spatial domain and frequency domain attention module, an attention gradient update mechanism module, and a boundary point and boundary box module;

[0031] The spatial domain and frequency domain attention module systems are used to capture fine and coarse features;

[0032] The attention gradient update mechanism module is used for dynamic loss adjustment;

[0033] The boundary point and bounding box module is used to enhance local detail feature extraction.

[0034] The present invention provides a few-sample medical image segmentation method based on bidirectional attention guidance. The method solves the problem of segmented arbitrary model (SAM) in few-sample medical image segmentation by merging spatial domain and frequency domain attention (SFA) modules. The module provided by the present invention combines frequency domain feature processing (FFP) and spatial domain feature processing (SFP). FFP uses a Gaussian filter to reduce noise and highlight key frequency components, while SFP prioritizes important channels and spatial regions in the feature map. This dual approach improves feature capture and correlation, and improves segmentation accuracy.

[0035] In addition, the present invention also proposes an attention-based gradient update (AGU) mechanism, which can dynamically adjust the loss weight according to the gradient to enhance generalization and adaptation in few-sample learning scenarios.

[0036] Finally, in order to maximize the use of each annotated sample, the present invention adopts a combination of boundary points and bounding boxes (CPB) as hints. Bounding boxes provide global information, while boundary points provide precise shape details, thus achieving more accurate segmentation. Our method is validated on multiple medical image datasets and achieves fully supervised performance with only two labeled samples, demonstrating its effectiveness and adaptability to various applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A schematic diagram of the overall framework of the few-sample medical image segmentation method based on bidirectional attention guidance provided by the present invention;

[0040] Figure 2 A schematic diagram of the SPA module structure provided by the present invention;

[0041] Figure 3 A schematic diagram of visualizing adaptive gradient updates in the attention gradient update (AGU) mechanism provided by the present invention;

[0042] Figure 4 A visualization diagram of the segmentation results of the method of the present invention on six domains provided in the disclosed embodiment of the present invention. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present invention. Instead, they are merely examples of systems consistent with some aspects of the present invention as detailed in the appended claims.

[0044] In order to solve the problem of application barriers of SAM in few-sample medical image segmentation, this embodiment provides a very few-sample medical image segmentation method based on bidirectional attention guidance, which integrates the bidirectional attention mechanism into the feature extraction and loss function of SAM for few-sample medical image segmentation, such as Figure 1 In the present invention, the few-sample dataset is represented by D = {(x i ,y i )|1≤i≤N}, where N is the total number of labeled images in D. Let x i ∈R H×W×3 and i ∈R H×W denote the i-th image and its corresponding label, respectively, where H and W denote the height and width of the image, respectively.

[0045] 1. Spatial Frequency Attention (SFA): Current medical image segmentation methods often struggle with limited data and noise interference, which hinders their ability to extract complex features and leads to poor performance. To this end, the Spatial Frequency Attention (SFA) module is introduced, which enhances feature extraction by processing information in both the spatial and frequency domains, such as Figure 2 As shown in Figure 2. Frequency feature processing (FFP) reduces noise and highlights key point frequency components, while spatial feature processing (SFP) focuses the model on important image regions.

[0046] Specifically, frequency domain feature processing (FFP) includes: first, the image x i ∈R H×W×3 It is input into the image encoder and the Transformer is used to extract key spatial features. These features are further refined through the convolutional layer to produce a more structured feature representation F t ∈R H×W×256 ,t∈{1,2,3,4}. In order to better handle noise and irrelevant details, Fourier transform is applied to transform the spatial domain feature map F t ∈R H×W×256 Converted to the frequency domain, expressed as F t ' ∈ R H×W×256 In the frequency domain, average pooling is performed and the result is passed to a fully connected layer to generate an attention map A. f This figure highlights the important frequency components while suppressing the irrelevant or noisy frequency components, thereby reducing the impact of non-essential information in the spectrum. Next, a Gaussian filter is applied to smooth the feature map F t ' ∈ R H×W×256 , generating F t ”∈R H×W×256 Finally, the inverse Fourier transform is applied to convert F t ” ∈ R H×W×256 Converting from the frequency domain back to the spatial domain, we get F t ”' ∈ R H×W×256 , which is then used as the subsequent SFP process. The following equation outlines the specific steps involved:

[0047]

[0048] Where t represents the number of layers in the feature map. u and v correspond to the coordinates of the feature points in the frequency domain. σ represents the standard deviation, which is used to control the width of the Gaussian kernel.

[0049] Specifically, the spatial feature processing (SFP) includes: To enhance feature extraction in the spatial domain, a spatial feature processing (SFP) module is introduced, which utilizes a dual-path strategy to capture detailed local features and broader contextual information. In the first path, multi-scale pooling is applied to F t ”'∈R H×W×256 , enabling the model to capture large-scale global structures and fine-scale details, thereby improving generalization and accuracy. The pooled results are then processed through a fully connected layer to generate an attention map A c In the second path, F is calculated t ”'∈R H×W×256 The maximum and average values ​​of are taken and merged along the channel dimension. Then, these merged feature maps are processed using multi-scale convolution kernels to produce an attention map A. s . A c and A s Merge by element-wise multiplication to create A I , then A I For the original feature map F t ”' ∈ R H×W×2 Processed to generate an enhanced feature map F t ”” ∈ R H×W×256 Finally, F t ””∈R H×W×2 The images are fused and processed through 3×3 convolutional layers to combine high-level semantic information with spatial details.

[0050] 2. Attention-based Gradient Update (AGU),During model training, the loss function guides the learning direction of the model. However, standard loss functions designed for data-rich scenarios usually fail after several learning processes or when adapting to new tasks. Figure 3 As shown in (a), standard loss functions may suffer from problems with domain conversion, such as conversion from REFUGE target domain to BASE1 target domain, or between different imaging modalities, such as fundus to lung images. To overcome these problems, we propose an adaptive gradient update (AGU) mechanism to improve the adaptability of the loss function to few-sample and diverse task scenarios. In each training batch, we first calculate the gradient of each loss function with respect to the model parameters and accumulate the absolute sum of these gradients. Then a dynamic weight update is performed to reflect the relative importance of each loss in the current batch. Specifically, the new weight W of each loss is l {l∈1,2,3} is the gradient of the current loss g i is determined by the inverse of the sum of the gradients of all losses. The final total loss is calculated as the weighted sum of the individual losses using these weights. Our approach combines the Dice loss L dice, binary cross entropy loss L bce and boundary loss L boundary The AGU mechanism allows the model to automatically adjust the importance of each loss function during training without manual intervention or fixed weights. In addition, AGU also improves the model's ability to generalize and adapt to unseen data and new tasks. Figure 3 As shown in (b), the model weights are dynamically adjusted at different training stages. The specific formula is as follows:

[0051] L total =W1L dice +W2·L bce +W3·L boundary (2)

[0052]

[0053]

[0054] Where ε represents the smoothing factor. x and G y Represents a Sobel filter for detecting horizontal and vertical edges. represents the prediction result of the model, and M is the total number of pixels.

[0055] 3. Boundary point and bounding box (CPB) prompts, including: The bounding box provides a rough estimate of the target location, but often misses the precise location edge details, resulting in loss of edge information and distracting attention from non-target areas. In order to solve these problems, CPB prompts are introduced to make full use of each annotation sample and improve segmentation accuracy. First, edge detection technology is used to identify the edges of the image. Then, coordinates are randomly sampled along these edges as embedded boundary point prompts. The method of the present invention not only helps the model learn the approximate location of the target within the bounding box, but also allows detailed learning of the edge morphology and features of the target. Therefore, the model can focus on and capture the interest of the area more accurately.

[0056] We evaluated our method on three cross-domain public datasets: RIGA+ and REFUGE fundus images for optic disc (OD) and optic cup (OC) and chest X-ray for lung segmentation. For RIGA+, we used BASE1, BASE2, and BASE3 as validation sets. REFUGE was split into 50 training images and 160 validation images with no overlap. The lung chest X-ray dataset contains 704 images and was split using 4-fold cross validation due to lack of evaluation. Performance is measured by the Dice score (D,%), where higher D values ​​indicate better segmentation.

[0057] 4. The method of the present invention was evaluated on three cross-domain public datasets: RIGA+ and REFUGE fundus images for optic disc (OD) and optic cup (OC) and chest X-ray for lung segmentation. For RIGA+, BASE1, BASE2 and BASE3 were used as validation sets. REFUGE was split into 50 training images and 160 validation images with no overlap. The lung chest X-ray dataset contains 704 images and was divided using 4-fold cross validation due to lack of evaluation. Performance is measured by the Dice score (D,%), where higher D values ​​indicate better segmentation.

[0058] 5. Use the Transformer architecture and the pre-trained ViT-B model, trained on an NVIDIA GeForce RTX 4090 GPU. Use AdamW optimization (β1=0.9, β2=0.999), initial learning rate 1e-4, weight decay 0.01, global batch training 100 epochs, a batch size of 2. Images and labels are pre-processed to a resolution of 1024×1024, normalized, resized, and randomly enhanced pre-processing operations, and the pre-trained model is used to locate the OD center of the REFUGE dataset.

[0059] 6. Results and Analysis: The method of the present invention is compared with the fully supervised DeepLabv3+. SAM and the method of the present invention were tested with 2, 20, and 50 labeled samples. The quantitative results are summarized in Table 1. For OD segmentation, the method of the present invention maintains high performance with minimal domain variance, achieving a Dice score of 95.85% on the REFUGE-val target domain with 2 samples. For OC segmentation, the method of the present invention, benefiting from CPB, SFA, and AGU, performs 3.52% higher than the fully supervised one on the BASE1 target domain. In lung chest X-ray segmentation, with 2 samples, the score is 92.45%, surpassing SAM 50 samples. This shows that the model of the present invention can reach the level of models trained with more data, effectively learning to adapt to new data from limited samples. Figure 4 It is shown that the proposed method outperforms DeepLabv3+ and SAM, especially in OD and lung segmentation, and emphasizes its accuracy in OC.

[0060] Table 1: Comparison of Dice scores between different methods on six target areas.

[0061]

[0062] Ablation study: Three key components are integrated: SFA, AGU, and CPB. Ablation is performed on fundus and lung image datasets, using only 2 labeled samples. The data of the evaluation components for each dataset are shown in Table 2. After removing SFA, the Dice scores of OD, OC, and lung are reduced by 0.96%, 3.69%, and 1.51%, respectively, highlighting its role in extracting fine-grained features in limited samples. After removing AGU, the Dice scores of OD are 93.09%, OC is 87.64%, and lung is 88.63%, proving the effectiveness of its balanced loss. After removing CPB, the Dice score of OC is reduced by 3.98%, which shows its importance for the processing of local details.

[0063] In summary, the method provided by the present invention is an advanced extension of SAM for few-sample medical image segmentation. It consists of three parts: the SFA module for capturing fine and coarse features, the AGU mechanism for dynamic loss adjustment, and the CPB strategy for enhancing local detail extraction. Experiments on various cross-domain datasets show that we can match the performance of fully supervised methods using only two labeled samples, demonstrating its effectiveness in a small number of scenarios. Future work should address real-time performance in clinical settings and optimize the model to respond immediately without compromising accuracy.

[0064] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These changes and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A few-shot medical image segmentation method based on bidirectional attention guidance, characterized in that: include: Integrate the bidirectional attention mechanism into the feature extraction and loss function of the segmentation arbitrary model to segment few-shot medical images; The bidirectional attention mechanism includes: spatial domain and frequency domain attention modules, and attention gradient update mechanism; The spatial and frequency domain attention module combines frequency domain feature processing and spatial domain feature processing; The spatial domain and frequency domain attention modules are used to capture fine and coarse features; the attention gradient update mechanism is used to dynamically adjust the loss weight; The dataset of the few-sample medical images is represented as D = {(x i ,y i )|1≤i≤N}, where N is the total number of labeled images in D, let x i ∈R H×W×3 and i ∈R H×W denote the i-th image and its corresponding label, respectively, where H and W denote the height and width of the image, respectively.

2. The method for few-shot medical image segmentation based on bidirectional attention guidance according to claim 1, characterized in that: The frequency domain feature processing includes: S11: Image x i ∈R H×W×3 It is input into the image encoder and the key spatial features are extracted using Transformer; S12: The key spatial features are refined through the convolutional layer to produce a more structured feature representation F t ∈R H×W×256 ,t∈{1,2,3,4}; S13: Apply Fourier transform to transform the spatial domain feature map F t ∈R H×W×256 Convert to frequency domain, expressed as F' t ∈R H×W×256 ; S14: In the frequency domain, perform average pooling and pass the result to the fully connected layer to generate the attention map A f ; S15: Apply a Gaussian filter to smooth the feature map F t ' ∈ R H×W×256 , generating F t ”∈R H×W×256 ; S16: Apply inverse Fourier transform to transform F t ” ∈ R H×W×256 Converting from the frequency domain back to the spatial domain, we get F t ”' ∈ R H×W×256 ; It is then used as the subsequent spatial domain feature processing flow; as shown below: Among them, t represents the number of layers in the feature map, u and v correspond to the coordinates of the feature points in the frequency domain, and σ represents the standard deviation, which is used to control the width of the Gaussian kernel.

3. The method for few-shot medical image segmentation based on bidirectional attention guidance according to claim 1, characterized in that: The spatial domain feature processing utilizes a dual-path strategy to capture detailed local features and broader contextual information; In the first path, multi-scale pooling is applied to F' t ∈R H×W×256 , and then the pooled results are processed through a fully connected layer to generate an attention map A c ; In the second path, calculate F' t ∈R H×W×256 The maximum and average values ​​of F' t ∈R H×W×256 The maximum and average values ​​of are merged along the channel dimension, and then the merged feature map is processed using a multi-scale convolution kernel to produce an attention map A. s ; A c and A s Combine by element-wise multiplication to generate A I , then A I For the original feature map F' t ∈R H×W×256 Processed to generate an enhanced feature map F” t ∈R H×W×256 ; Finally, F” t ∈R H×W×256 Layer-by-layer fusion and processing are performed through 3×3 convolutional layers to merge high-level semantic information with spatial details.

4. The method for few-shot medical image segmentation based on bidirectional attention guidance according to claim 1, characterized in that: The attention gradient update mechanism includes: in each training batch, firstly, calculating the gradient of each loss function with respect to the model parameters, and accumulating the absolute sum of the gradients; then performing a dynamic weight update to reflect the relative importance of each loss in the current batch; the new weight W of each loss l {l∈1,2,3} is the gradient of the current loss g i Determined inversely proportional to the sum of the gradients of all losses, the final total loss is calculated as the weighted sum of the individual losses with the new weights.

5. The method for few-sample medical image segmentation based on bidirectional attention guidance according to claim 3, characterized in that: The attention gradient update mechanism combines the Dice loss L dice , binary cross entropy loss L bce and boundary loss L boundary ; The attention gradient update mechanism is used to dynamically adjust the loss weight. The specific formula is as follows: L total =W1·L dice +W2·L bce +W3·L boundary (2) Among them, ε represents the smoothing factor; G x and G y Represents a Sobel filter for detecting horizontal and vertical edges; represents the prediction result of the model, and M is the total number of pixels.

6. The method for few-shot medical image segmentation based on bidirectional attention guidance according to claim 1, characterized in that: The segmented arbitrary model also introduces boundary points and bounding boxes, including: first using edge detection to identify the edges of the image, and then randomly sampling coordinates along the edges as embedded boundary point prompts.

7. A few-shot medical image segmentation system based on bidirectional attention guidance, characterized in that: include: Spatial and frequency domain attention modules, attention gradient update mechanism module, and boundary point and bounding box module; The spatial domain and frequency domain attention module systems are used to capture fine and coarse features; The attention gradient update mechanism module is used for dynamic loss adjustment; The boundary point and bounding box module is used to enhance local detail feature extraction.

Images