A spatially adaptive and heat map guided retinal image segmentation method

Abstract

The present application relates to the technical field of medical image processing, in particular to a kind of space adaptive and heat map guided retinal image segmentation method, comprising: obtaining the retinal image data containing three-dimensional information and at least one focus, optimizing existing SAM model, construct a segmentation model, its architecture contains with visual Transformer as backbone, and design is introduced space adaptive feature enhancement Transformer block image encoder, two-stage mask decoder and heat map guided mask self-attention module, using the combined loss function based on cross-entropy loss and Dice loss composition supervises the training process of model, the image to be segmented is input into model to obtain segmentation result;The advantages of the scheme are that the three-dimensional structure information of retinal image can be fully tapped, the segmentation accuracy of different types of lesions can be significantly improved, and the problem of class imbalance can be alleviated, and high-precision and efficient segmentation of retinal lesions is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a retinal image segmentation method guided by spatial self-adaptation and heat map. BACKGROUND

[0002] Medical image segmentation is an important application direction of image segmentation, involving CT, MRI, OCT (optical coherence tomography) and other image analysis, and the goal is to extract the anatomical structure or lesion area of interest from complex images, which is of great significance for disease diagnosis, treatment planning and surgical navigation. Deep learning has brought breakthrough development to this field: researchers improve network structure, introduce multi-scale features and combine attention mechanism to improve segmentation accuracy; transfer learning and semi-supervised learning also alleviate the problem of insufficient labeled data in medical image segmentation.

[0003] In recent years, the emergence of the SAM model has provided a new direction for image segmentation. As a general segmentation model, SAM can achieve zero-shot or few-shot segmentation after large-scale natural image pre-training. However, due to the diversity and complexity of medical images, the performance of SAM is obviously insufficient when directly applied to medical image segmentation. Therefore, researchers have proposed improved models of SAM: for example, AdaptiveSAM modifies SAM adaptively to achieve fast adaptation to new data sets and text prompt segmentation, and has achieved certain results on medical data sets such as surgical scenarios and ultrasound images; SAMed applies a fine-tuning strategy based on low rank (LoRA) to the SAM image encoder, and jointly fine-tunes the prompt encoder and the mask decoder on the medical image segmentation dataset, and performs well on the Synapse multi-organ segmentation dataset; SAMIHS incorporates a parameter reconstruction adapter into the SAM image encoder, and combines the joint loss of binary cross-entropy loss and boundary-sensitive loss to improve the accuracy of intracranial hemorrhage segmentation.

[0004] Although image segmentation technology and SAM-derived models have made progress in the medical field, the existing public technology still has obvious shortcomings:

[0005] First, the three-dimensional information extraction capability is lacking. SAM relies on a 2D image encoder, and when processing tasks that require three-dimensional information support, it cannot fully capture spatial depth and stereo information. Even if MA-SAM injects a 3D adapter into the image encoder Transformer block to compensate for the defect, it is still limited by the 2D image processing mode formed during the pre-training phase when capturing long-distance pixel correlations in three-dimensional space. While the introduction of InceptionNeXt and other branched core structures improves efficiency, it is prone to overfitting in small sample medical scenarios, and MH-SSM (Multi-Head State Space Model) also faces challenges in adapting to medical images with different imaging conditions or sampling modes.

[0006] Secondly, the class imbalance problem solving has defects. Taking H-SAM as an example, which is specially designed for class imbalance, it relieves the problem through class balanced mask guided self-attention mechanism (CMAttn) and double-stage decoder, but the design introduces additional computational overhead and memory occupation at the same time, leading to the rise of inference delay; when the lesion class is extremely rare or the boundary is blurred, the model is significantly sensitive to the quality of the prompt information and the mask, and is easy to misfocus on the non-target area; and when migrating across devices and modalities, reparameterization is needed to maintain stability, the generalization cost is higher, and the error cascading risk brought by the double-stage decoding also leads to unstable recognition rate and boundary accuracy of small voxel lesions.

[0007] Thirdly, the task-specific knowledge utilization is insufficient. For example, although 3D SAM-Adapter injects task prior knowledge in a lightweight manner, the adapter capacity is limited, it is difficult to cover the fine-grained differences of multiple organs and multiple lesions, and it is easy to cause collaborative conflicts with the prompt encoder, leading to inaccurate matching of user prompts and image features, the parameter adjustment method and insertion position have a significant impact on performance, and the portability is poor; WTConv (wavelet convolution) expands the receptive field through wavelet transform, but in low-contrast and noisy medical images, it is easy to produce false texture details and frequency information distortion problems, and the image resolution and parameter selection requirements are strict, and expanding the receptive field does not equal to realizing effective global semantic reasoning, it is still difficult to capture the correlation between different lesions and the consistency between different slices. SUMMARY

[0008] Therefore, the present application aims to solve the technical problems of insufficient segmentation accuracy of retinal lesion regions containing three-dimensional structure information, insufficient utilization of three-dimensional spatial information, and poor segmentation effect caused by unbalanced distribution of lesion classes in existing retinal optical coherence tomography (OCT) image segmentation.

[0009] To solve the above technical problems, the present application provides a spatial self-adaptive and heat map guided retinal image segmentation method, comprising:

[0010] S1: obtaining retinal image data, the retinal image data containing three-dimensional image information and at least one retinal lesion region;

[0011] S2: constructing a retinal image segmentation model, based on the retinal image data, using a joint loss function composed of cross-entropy loss and Dice loss to supervise the training process of the retinal image segmentation model, aiming to minimize the value of the joint loss function, and obtaining a trained retinal image segmentation model;

[0012] S3: inputting the retinal image to be segmented into the trained retinal image segmentation model to obtain the segmentation result of the retinal image;

[0013] The retinal image segmentation model includes an image encoder, a two-stage mask decoder, and a heatmap-guided mask self-attention module; the image encoder encodes the retinal image into an image embedding vector.

[0014] The two-stage mask decoder includes a first-stage decoder and a second-stage decoder, both of which are based on the SAM mask decoder structure; the first-stage decoder processes the image embedding vector to generate a priori mask.

[0015] Based on the retinal image data, a corresponding heatmap is generated. The heatmap, the image embedding vector, and the prior mask are input into the heatmap-guided mask self-attention module, and the enhanced image embedding is obtained after feature enhancement processing.

[0016] After the enhanced image embedding undergoes masking refinement by the second-stage decoder, the final segmentation result of the retinal image is output to distinguish different types of retinal lesion regions.

[0017] In one embodiment of the present invention, the image encoder uses a visual Transformer as its backbone network, and the backbone network includes multiple cascaded spatial adaptive feature enhancement Transformer blocks.

[0018] Each spatial adaptive feature enhancement Transformer block includes a bidirectional multi-head gating S6 module, a multi-head self-attention block, a depth convolutional feature extraction module, and a feedforward network layer connected in sequence. A wavelet convolutional adapter is set in parallel at the feedforward network layer. The input of the wavelet convolutional adapter receives the original retinal image and the output features of the depth convolutional feature extraction module. Its output features are added to the output features of the feedforward network layer and the output features of the depth convolutional feature extraction module to obtain the output features of the spatial adaptive feature enhancement Transformer block.

[0019] In one embodiment of the present invention, the bidirectional multi-head gating S6 module includes a normalization layer and two multi-head gating S6 modules; the processing procedure of the bidirectional multi-head gating S6 module includes:

[0020] The input signal is processed through the normalization layer. Normalization is performed to obtain normalized features. ;

[0021] The normalized features By inputting the first multi-head gating S6 module in the forward direction and the second multi-head gating S6 module in the reverse direction respectively, we can obtain the forward processing features and the reverse processing features.

[0022] The forward processing feature and the reverse processing feature are spliced, a residual connection is added, and the spliced feature is added to the original input signal The addition is performed to obtain the output feature of the bidirectional multi-head gated S6 module .

[0023] In an embodiment of the present application, the multi-head gated S6 module is constructed based on an S6 module and an inter-head gating module; the processing process of the multi-head gated S6 module includes:

[0024] The normalized feature is uniformly divided into H independent features in the channel dimension, and the channel dimension of each independent feature is 1 / H of the channel dimension of the normalized feature; wherein H is an even number, and H is the number of blocks of the S6 module;

[0025] Each independent feature is subjected to nonlinear transformation and local feature extraction by a randomly initialized S6 module, and the results of the last H / 2 independent features after S6 module processing are subjected to an activation function to generate a gating signal;

[0026] The inter-head gating module performs a gating operation on the results of the first H / 2 independent features after S6 module processing through the gating signal to obtain the output feature of the multi-head gated S6 module.

[0027] In an embodiment of the present application, the deep convolution feature extraction module includes a normalization layer, a down-projection layer, a 3D Inception deep separable convolution module and an up-projection layer connected in sequence, and the input end of the normalization layer and the output end of the up-projection layer are connected in residual connection;

[0028] The processing process of the deep convolution feature extraction module includes:

[0029] The normalization layer performs normalization processing on the output feature of the multi-head self-attention block to obtain a normalized feature;

[0030] The normalized feature is compressed to a low-dimensional space through the down-projection layer to obtain a low-dimensional feature;

[0031] The low-dimensional feature is input into the 3D Inception deep separable convolution module for feature extraction to obtain a multi-scale three-dimensional feature;

[0032] The multi-scale three-dimensional feature is restored to the original feature dimension through the up-projection layer to obtain a high-dimensional restored feature;

[0033] The high-dimensional restored feature and the output feature of the multi-head self-attention block are subjected to an addition operation to obtain the output feature of the deep convolution feature extraction module.

[0034] In an embodiment of the present application, the 3D Inception deep separable convolution module is composed of a plurality of deep separable 3D convolution branches in parallel and an identity branch, the plurality of deep separable 3D convolution branches have different convolution kernel sizes; the processing process of the 3D Inception deep separable convolution module comprises:

[0035] The low-dimensional features are split and input into different deep separable 3D convolution branches, at the same time, the low-dimensional features are identity mapped through the identity branch, the results after processing by the plurality of deep separable 3D convolution branches and the results after identity mapping are spliced to obtain multi-scale three-dimensional features.

[0036] In an embodiment of the present application, the wavelet convolution adapter comprises a high-pass filtering unit, a wavelet convolution unit, a convolution layer, a down-sampling linear projection layer, a multi-layer perceptron and an up-sampling linear projection layer, the high-pass filtering unit, the wavelet convolution unit and the convolution layer are connected in sequence to form a first branch, the down-sampling linear projection layer constitutes a second branch, the output ends of the first branch and the second branch are added and connected to the input end of the multi-layer perceptron, and the output end of the multi-layer perceptron is connected to the input end of the up-sampling linear projection layer.

[0037] The processing process of the wavelet convolution adapter comprises:

[0038] After performing fast Fourier transform on the retinal original image, high-frequency information is extracted through the high-pass filtering unit;

[0039] The high-frequency information is input into the wavelet convolution unit for processing, and then the first feature is obtained through the convolution layer;

[0040] At the same time, the output feature of the deep convolution feature extraction module is reduced in size through the down-sampling linear projection layer to obtain a low-dimensional reduced feature;

[0041] The first feature and the low-dimensional reduced feature are added for feature fusion to obtain a fusion feature;

[0042] The fusion feature is input into the multi-layer perceptron for nonlinear transformation, and then the original feature size is restored through the up-sampling linear projection layer to obtain the output feature of the wavelet convolution adapter.

[0043] In an embodiment of the present application, based on the retinal image data, the method for generating a corresponding heat map is as follows:

[0044] A heat map generation model is constructed, the heat map generation model is trained through the retinal image data, and the pre-training weight of the heat map generation model is obtained after training.

[0045] In the training or inference of the retinal image segmentation model, the heat map generation model with the pre-trained weight is loaded as a feature extractor to generate a heat map of the retinal image data.

[0046] In an embodiment of the present application, the heat map, the image embedding vector and the prior mask are input into the heat map guided mask self-attention module, and the method for obtaining an enhanced image embedding through feature enhancement processing is as follows:

[0047] The heat map is normalized to obtain normalized heat map data, and the normalized heat map data is multiplied by a matrix with Gaussian noise to obtain heat map modulation noise.

[0048] The heat map modulation noise is added to the prior mask to obtain enhanced prior features.

[0049] The enhanced prior features are subjected to self-attention processing to obtain global context features.

[0050] The global context features are subjected to nonlinear transformation and then compressed in channel dimension to obtain compressed features.

[0051] The compressed features are subjected to Hadamard product operation with the image embedding output by the image encoder, and the operation result is added to the image embedding to obtain the enhanced image embedding.

[0052] In an embodiment of the present application, the expression of the joint loss function is as follows:

[0053] wherein, is the value of the joint loss function, , is a weight parameter, is a cross-entropy loss, is a Dice loss;

[0054] The calculation formula of the cross-entropy loss is as follows: N is the total sum of pixels of the feature map, i is the pixel index, i = 1, 2, …, N; and C is the number of classes of the segmentation target. represents the label of the i-th pixel in the c-th class, and each pixel belongs to only one class among the C classes; represents the probability of the model predicting that the i-th pixel belongs to the c-th class;

[0055] The calculation formula of the Dice loss is as follows: , A smoothing factor for preventing a numerator or a denominator from being zero.

[0056] The above technical solutions of the present application have the following beneficial effects compared with the prior art:

[0057] The present application can fully mine the three-dimensional structure information of the retinal OCT image and strengthen the expression of lesion features by constructing an image encoder containing a space adaptive feature enhancement Transformer block (SAFE Transformer Block), combining a bidirectional multi-head gated S6 (BMHG-S6) module to strengthen three-dimensional spatial information, a deep convolution feature extraction (DCFE) module to extract multi-scale three-dimensional features, and a WCA module to enhance high-frequency details. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to make the content of the present application easier to be clearly understood, the present application will be further described in detail below according to specific embodiments of the present application and in conjunction with the accompanying drawings.

[0059] Figure 1 is a flowchart of a space adaptive and heat map guided retinal image segmentation method provided in an embodiment of the present application;

[0060] Figure 2 is a structural diagram of a space adaptive heat map guided feature enhancement SAM model provided in an embodiment of the present application;

[0061] Figure 3 is a structural diagram of a space adaptive feature enhancement Transformer block provided in an embodiment of the present application;

[0062] Figure 4 is a structural diagram of a 3D Inception deep separable convolution module provided in an embodiment of the present application;

[0063] Figure 5 is a structural diagram of a wavelet convolution adapter provided in an embodiment of the present application;

[0064] Figure 6 is a structural diagram of a heat map guided mask self-attention module provided in an embodiment of the present application;

[0065] Figure 7are segmentation results of different methods on AI Challenger 2018 dataset, (a) represents a retinal original image, (b) represents a standard segmentation result, (c) represents a segmentation result of CPFNet, (d) represents a segmentation result of nnUnet, (e) represents a segmentation result of SAMed_h, (f) represents a segmentation result of H-SAM, (g) represents a segmentation result of MA-SAM, (h) represents a segmentation result of MedSAM, (i) represents a segmentation result of Med_SA, and (j) represents a segmentation result of the method of the present application;

[0066] Figure 8 are segmentation results of different methods on RETOUCH dataset, (a) represents a retinal original image, (b) represents a standard segmentation result, (c) represents a segmentation result of CPFNet, (d) represents a segmentation result of nnUnet, (e) represents a segmentation result of SAMed_h, (f) represents a segmentation result of H-SAM, (g) represents a segmentation result of MA-SAM, (h) represents a segmentation result of MedSAM, (i) represents a segmentation result of Med_SA, and (j) represents a segmentation result of the method of the present application. DETAILED DESCRIPTION

[0067] The present application will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the present application and implement it, but the embodiments are not limiting to the present application.

[0068] To solve the problems of insufficient three-dimensional information extraction, unbalanced lesion categories, and insufficient use of task-specific knowledge of existing SAM and its derivative models in retinal OCT segmentation, the present application provides a retinal image segmentation method based on spatial adaptation and heat map guidance, as shown in Figure 1 The method specifically comprises the following steps:

[0069] S1: obtaining retinal image data, wherein the retinal image data contains three-dimensional image information and at least one retinal lesion area;

[0070] S2: constructing a retinal image segmentation model, wherein the training process of the retinal image segmentation model is supervised based on a joint loss function composed of cross-entropy loss and Dice loss based on the retinal image data, and the training process of the retinal image segmentation model is supervised based on a joint loss function composed of cross-entropy loss and Dice loss, and the value of the joint loss function is minimized as the goal to obtain a trained retinal image segmentation model;

[0071] S3: inputting a retinal image to be segmented into the trained retinal image segmentation model to obtain a segmentation result of the retinal image;

[0072] Wherein, as Figure 2As shown, the retinal image segmentation model is a spatially adaptive heat map guided feature enhancement SAM (SHF-SAM) model, which includes an image encoder, a two-stage mask decoder, and a heat map guided mask self-attention module; the image encoder encodes the retinal image into an image embedding vector;

[0073] The two-stage mask decoder includes a first-stage decoder and a second-stage decoder, both of which are based on a SAM mask decoder structure; the first-stage decoder performs preliminary processing on the image embedding vector to generate a prior mask;

[0074] Based on the retinal image data, a corresponding heat map is generated, and the heat map, the image embedding vector, and the prior mask are input into the heat map guided mask self-attention module to obtain an enhanced image embedding after feature enhancement processing;

[0075] After the enhanced image embedding is subjected to mask refinement processing by the second-stage decoder, the final segmentation result of the retinal image is output to distinguish different types of retinal lesion regions such as PED, REA, SRF, IRF, etc.

[0076] Further, in the present embodiment, the image encoder uses a visual Transformer (ViT) as the backbone network, and the backbone network includes a plurality of cascaded spatially adaptive feature enhancement Transformer blocks (SAFE Transformer Block);

[0077] As shown in Figure 3 Each spatially adaptive feature enhancement Transformer block includes a bidirectional multi-head gated S6 (BMHG-S6) module, a multi-head self-attention (MHSA) block, a deep convolutional feature extraction (DCFE) module, and a feedforward network (FFN) layer connected in sequence, and a wavelet convolution adapter (WCA) is arranged in parallel at the FFN layer. The input end of the wavelet convolution adapter receives the retinal original image and the output feature of the deep convolutional feature extraction module, and the output feature of the wavelet convolution adapter is added to the output feature of the FFN layer and the output feature of the deep convolutional feature extraction module to obtain the output feature of the spatially adaptive feature enhancement Transformer block.

[0078] To avoid invalidation of original SAM pre-training parameters and reduce training overhead, a low-rank adaptation (LoRA) fine-tuning strategy is adopted for the MHSA block:

[0079] Keep all pre-trained parameters of the original SAM image encoder frozen, and add low-rank bypass structures only to the query projection layer and value projection layer of the MHSA block. These structures consist of two small linear layers: first, the input features are reduced to a low-rank space, and then restored to the original dimension to match the output feature channels of the frozen Transformer block.

[0080] During training, only the low-rank bypass structure is updated to achieve lightweight fine-tuning of the MHSA block and adapt it to the feature distribution of the retinal image.

[0081] The bidirectional multi-head gated S6 (BMHG-S6) module is used to capture continuous changes between adjacent retinal OCT slices, enhancing the ability to model three-dimensional spatial information. It consists of a normalization layer and two multi-head gated S6 modules. The processing procedure of the bidirectional multi-head gated S6 module includes:

[0082] The input signal is processed through the normalization layer. Normalization is performed to obtain normalized features. : ;

[0083] The normalized features By inputting the first multi-head gated S6 (MHGS6) module in the forward direction and the second multi-head gated S6 module in the reverse direction, the forward processing features and the reverse processing features are obtained.

[0084] The forward and reverse processed features are concatenated along the channel dimension, and a residual connection is added. The concatenated features are then combined with the original input signal. The sums are then performed to obtain the output characteristics of the bidirectional multi-head gating S6 module. :

[0085] , This indicates a splicing operation. This indicates a reverse processing operation.

[0086] Furthermore, the multi-head gating S6 module is built upon the S6 module and the inter-head gating (IHG) module, and its data processing procedure is as follows:

[0087] Assume the number of blocks in module S6 is H, where H is an even number. The normalized features... The channel dimension is uniformly divided into H independent features, and the channel dimension of each independent feature is the normalized feature. 1 / H of the channel dimension;

[0088] Each of the independent features is respectively subjected to nonlinear transformation and local feature extraction by a randomly initialized S6 module (convolution kernel size 3*3, activation function Gelu), and the results of the last H / 2 independent features after S6 module processing are subjected to a gating signal generated by a Gelu activation function;

[0089] The inter-head gating module performs element-wise multiplication operation on the results of the first H / 2 independent features after S6 module processing through the gating signal to obtain the output features of the MHGS6 module : , wherein σ(·) is a Gelu activation function, represents the hth independent feature, .

[0090] Further, the DCFE module comprises a normalization layer, a lower projection layer, a 3D Inception deep separable convolution module and an upper projection layer connected in sequence, and the input end of the normalization layer and the output end of the upper projection layer are connected in residual manner;

[0091] The processing process of the deep convolution feature extraction module comprises:

[0092] The normalization layer performs normalization processing on the output features of the MHSA block to obtain normalized features;

[0093] The normalized features are compressed to a low-dimensional space by the lower projection layer (linear layer ) to obtain low-dimensional features ;

[0094] The low-dimensional features are input into the 3D Inception deep separable convolution module for feature extraction to obtain multi-scale three-dimensional features ;

[0095] The multi-scale three-dimensional features are restored to the original feature dimension (consistent with the channel dimension of ) by the upper projection layer (linear layer ) to obtain high-dimensional restored features ;

[0096] The high-dimensional restored features and the output features of the MHSA block are added to obtain the output features of the deep convolution feature extraction module :

[0097] , represents a Gelu activation function, Represents the layer normalization, IDConv3D represents the 3D Inception depth separable convolution module, which is specially used for extracting three-dimensional information.

[0098] Further, as shown in Figure 4 , the 3D Inception depth separable convolution module is composed of 4 parallel depth separable 3D convolution branches and an identity branch, the convolution kernel sizes of the 4 depth separable 3D convolution branches are different, which are 3x1x1, 5x1x1, 7x1x1 and 11x1x1 (directionally constrained high / width dimension is 1, focusing on the volume information extraction of channel dimension, step is 1, padding is "same"); the processing process of the 3D Inception depth separable convolution module includes:

[0099] Splitting the low-dimensional feature , inputting it into the depth separable 3D convolution branches with different convolution kernel sizes, at the same time, performing identity mapping on the low-dimensional feature through the identity branch to preserve the original feature details; performing channel splicing operation on the results processed by the 4 depth separable 3D convolution branches and the results after identity mapping, to obtain multi-scale three-dimensional feature .

[0100] Further, the wavelet convolution adapter (WCA module) is connected in parallel with the FFN layer, and the core function is to fuse the high-frequency details (such as lesion boundaries) of the retinal OCT image and the task-specific knowledge, and to expand the model receptive field, as shown in Figure 5 , the structure includes a high-pass filtering unit, a wavelet convolution (WTConv) unit, a convolution layer with a convolution kernel size of 16x16, a down-sampling linear projection layer, a multi-layer perceptron (MLP) and an up-sampling linear projection layer, the high-pass filtering unit, the wavelet convolution unit and the convolution layer are connected in sequence to form a first branch for extracting high-frequency information; the down-sampling linear projection layer constitutes a second branch for reducing low-dimensional features; the output ends of the first branch and the second branch are added and connected to the input end of the multi-layer perceptron, and the output end of the multi-layer perceptron is connected to the input end of the up-sampling linear projection layer;

[0101] The processing process of the wavelet convolution adapter includes:

[0102] The original retinal image I is converted to the frequency domain by using fast Fourier transform, and the high-frequency information corresponding to the lesion boundary and the fine structure is extracted by the high-pass filtering unit (cutoff frequency 0.3);

[0103] The high-frequency information is input into the wavelet convolution unit for processing, and then the channel dimension is compressed through the convolution layer to obtain the first feature ;

[0104] Meanwhile, the output features of the DCFE module are down-sampled by a down-sampling linear projection layer to obtain low-dimensional reduced features ; ;

[0105] The first features and the low-dimensional reduced features are element-wise added to obtain fused features ;

[0106] The fused features are input into an MLP for nonlinear transformation, and then recovered to the same size as the original features by the up-sampling linear projection layer to obtain the output features of the wavelet convolution adapter :

[0107] , ,

[0108] wherein, is an up-sampling linear projection layer, denotes a down-sampling linear projection layer, denotes a wavelet convolution unit, denotes a high-pass filter, denotes a convolution layer with a convolution kernel size of 16x16.

[0109] In order to provide the HMAttn module with lesion saliency information, a method for generating a corresponding heat map based on the retinal image data by using an offline trained network is as follows:

[0110] A heat map generation model based on ResNet101 is constructed, and the output channel number is consistent with the lesion class number; the heat map generation model is trained by using the retinal image data, and the pre-training weight of the heat map generation model is obtained after the training is completed;

[0111] During the training or inference of the retinal image segmentation model, the pre-training weight of ResNet101 is loaded as a feature extractor, and a heat map of the class-level retinal image data is output after the retinal image is input , each channel corresponds to the saliency distribution of a class of lesions, and there is no need for online fine-tuning, thereby reducing the computational overhead.

[0112] Further, as shown in Figure 6 , the heat map , the image embedding vector output by the image encoder , and the prior mask output by the first-stage decoder are input into the HMAttn module to obtain the second-stage decoder inputThe input is input to the heat map guidance mask self-attention module (HMAttn module), and the enhanced image embedding is obtained through feature enhancement processing as follows:

[0113] The heat map is normalized to obtain normalized heat map data The normalized heat map data is multiplied by Gaussian noise to obtain heat map modulation noise ;

[0114] The heat map modulation noise is added to the prior mask element by element to obtain enhanced prior features ;

[0115] The enhanced prior features are self-attended to capture global lesion correlations to obtain global context features ;

[0116] The global context features are nonlinearly transformed by a feedforward network module, and then the channel dimension is compressed to obtain compressed features ;

[0117] The compressed features are Hadamard product operated with the image embedding output by the image encoder to obtain an operation result and the image embedding The channel addition operation is performed on the operation result and the image embedding to obtain the enhanced image embedding :

[0118] , FFN represents a feedforward network, self_atten represents self-attention, and is a Hadamard product, is a matrix multiplication.

[0119] The above enhanced features not only retain the semantic information of the original features, but also highlight the saliency regions related to the task through the heat map.

[0120] To alleviate the lesion class imbalance problem of retinal OCT images, a joint loss function composed of cross-entropy loss and Dice loss is used to supervise the model training, and the expression is as follows:

[0121] wherein, is the value of the joint loss function, , ​​​​​​​​The weight parameter is used to balance the contribution of the two types of losses, and is set to 0.5; The cross-entropy loss is The Dice loss is

[0122] The cross-entropy loss is used to measure the classification error of the predicted probability and the true label, and the calculation formula is: N is the total sum of pixels of the feature map, i is the pixel index, i = 1, 2, …, N; and C is the number of classes of the segmentation target. represents the true label of the ith pixel in the cth class, and each pixel only belongs to one class in the C classes; represents the predicted probability of the model that the ith pixel belongs to the cth class;

[0123] The Dice loss is used to measure the overlap degree of the predicted region and the true region, and is more sensitive to small sample lesions, and the calculation formula is: , is a smoothing factor used to prevent the numerator or denominator from being zero, and the value is 1e-6.

[0124] To verify the effectiveness and robustness of the SHF-SAM model in the retinal optical coherence tomography (OCT) image segmentation task, the present study carried out comparative experiments, ablation experiments and visualization verification on two public segmentation datasets, AI Challenger 2018 and RETOUCH. The experiments were based on the PyTorch 2.0 deep learning framework, and the hardware platform used NVIDIA A6000 graphics card (48GB video memory) to ensure the efficiency of model training and inference. The specific experimental design and result analysis are as follows:

[0125] The present study used two types of public challenge dataset to verify the performance of SHF-SAM, namely AI Challenger 2018 fundus edema lesion segmentation dataset and MICCAI RETOUCH challenge dataset, and the core information of the two types of datasets is as follows:

[0126] The AI Challenger dataset contains 83 three-dimensional retinal OCT data, a total of 10624 B-scan images, with an image size of 256*512, containing three types of lesions, namely retinal edema (REA), pigment epithelium detachment (PED) and subretinal fluid (SRF). According to the data division of AI Challenger 2018, 40 data were used as the training set, 43 data were used as the test set, and the present invention randomly selected 8 data from the training set as the verification set.

[0127] The RETOUCH dataset includes 70 three-dimensional retinal OCT data from three different suppliers of SD-OCT equipment, a total of 6936 B-scan images, and the image size ranges from 512*496 to 512*1024 pixels. The three devices are Cirrus, Spectralis and Topcon, which include 24, 24 and 22 data respectively. There are three types of lesions, pigment epithelial detachment (PED), subretinal fluid (SRF) and intraretinal fluid (IRF), and the corresponding segmentation gold standard images are annotated by medical professionals. The RETOUCH dataset is divided into training set, validation set and test set according to the ratio of 7:1:2.

[0128] 1. Segmentation results of AI Challenger 2018 dataset

[0129] 1.1 Comparison test results: comparison of performance of existing medical image segmentation methods

[0130] To verify the superiority of the SHF-SAM model proposed in the application, it is compared with mainstream medical image segmentation methods (including CPFNet, nnUnet, SAMed_h, H-SAM, MA-SAM, MedSAM and Med_SA), and the evaluation indexes include the Dice coefficients of various lesions, the average Dice coefficient (Mean Dice), the intersection over union (IoU), the sensitivity (Sen), the specificity (Spe) and the statistical significance (p-value), and the results are shown in Table 1.

[0131] Table 1 Comparison test results on AI Challenger 2018 dataset

[0132]

[0133] As shown in Table 1, the segmentation network CPFNet and nnUnet based on convolutional neural network as a traditional CNN segmentation model perform poorly in the OCT image segmentation task, with an average Dice coefficient of only 72.85% and 71.68% respectively, especially for SRF (the most difficult lesion to segment), the Dice coefficient is less than 65%, reflecting the weak ability to capture three-dimensional spatial correlation information.

[0134] MedSAM, MA-SAM and other medical adaptation models based on SAM improve the performance of the CNN model through pre-training weight migration and task adaptation (the average Dice of MedSAM is 75.56%), but there is still a class bias problem, that is, the segmentation accuracy of SRF is generally less than 65%, which cannot effectively solve the class imbalance problem in the dataset.

[0135] In contrast, the SHF-SAM method proposed in the present application achieves balanced and excellent segmentation performance on the three lesions:

[0136] The PED segmentation Dice coefficient reaches 84.16%, which is 10.25 percentage points higher than the suboptimal method Med_SA (73.91%); the average Dice coefficient reaches 79.78%, which is 1.82 percentage points higher than Med_SA (77.96%); the sensitivity (80.57%) and IoU (73.77%) are the best among all methods.

[0137] This advantage is due to the synergistic effect of the multiple modules of the model: the bidirectional multi-head gated S6 module (BMHG-S6) enhances the interaction of three-dimensional spatial information, the heat map guided mask attention module (HMAttn) alleviates class imbalance, and the deep convolution feature extraction (DCFE) and wavelet convolution adapter (WCA) improve the feature expression ability through multi-scale fusion and frequency domain enhancement.

[0138] 1.2 Ablation experiment results: effectiveness verification of key components and structural design

[0139] To clarify the mechanism of action of each core component of SHF-SAM and the rationality of the structural design, the LoRA fine-tuned SAM model is taken as the baseline, and three types of ablation experiments are carried out: (1) core component effectiveness verification; (2) BMHG-S6 and DCFE module deployment position optimization; (3) LoRA fine-tuning rank (r) parameter selection.

[0140] 1.2.1 Core component effectiveness verification

[0141] The experiment evaluates the influence of the bidirectional multi-head gated S6 module (BMHG-S6), the heat map guided mask attention module (HMAttn), the deep convolution feature extraction module (DCFE), and the wavelet convolution adapter module (WCA) on the performance of the model. The experimental results are shown in Table 2, where "●" indicates that the module is enabled, "○" indicates that the module is disabled, and the evaluation index is the Dice coefficient.

[0142] Table 2 Ablation experiment results of the effectiveness of each component

[0143]

[0144] When all modules are not enabled, the baseline model has an average Dice coefficient of only 65.06% on the AI Challenger 2018 dataset, especially for the segmentation of SRF (Dice 42.22%). After enabling the HMAttn module alone, the model's segmentation performance for SRF improved from 42.22% to 71.57%, with an average Dice coefficient increase of 10.62% (from 65.06% to 75.68%), indicating that the heat map guided mask attention mechanism effectively alleviates the class imbalance problem and enhances the model's attention to key areas. After enabling the BMHG-S6 module, the model's segmentation performance for PED improved to 82.02%, indicating that the bidirectional multi-head gated S6 module helps identify small target structures by capturing the continuity changes between adjacent slices. The introduction of the DCFE module improved the model's segmentation Dice coefficients for PED and SRF to 82.36% and 62.87%, respectively, with an average Dice coefficient of 75.67%, verifying the effectiveness of the 3D Inception depth separable convolution module in extracting multi-scale three-dimensional features. After enabling the WCA module alone, the model's segmentation performance for REA was the best (Dice 83.00%), but for SRF it was only 55.57%, indicating that the wavelet convolution adapter has potential in specific tasks, but still relies on the synergistic effect of other modules to improve class imbalance.

[0145] Further analysis of module combinations showed that when HMAttn was combined with BMHG-S6, the model's segmentation Dice coefficient for SRF improved to 72.31%, but the improvement in PED segmentation was not significant. When HMAttn, BMHG-S6, and DCFE modules were combined, the model's segmentation Dice coefficients for PED and REA reached 82.22% and 83.01%, respectively, with an average Dice coefficient of 77.45%, verifying the effectiveness of multi-module synergy. Finally, when all modules were enabled, SHF-SAM's segmentation Dice coefficients for PED, REA, and SRF reached 84.16%, 82.80%, and 72.39%, respectively, with an average Dice coefficient of 79.78%, which was 14.72% higher than the baseline model, fully demonstrating the complementarity of each module and the superiority of the overall design.

[0146] 1.2.2 BMHG-S6 and DCFE module deployment position optimization

[0147] To explore the optimal deployment position of BMHG-S6 and DCFE in the model architecture, experiments were conducted to compare their performance when deployed before / after the multi-head self-attention (MHSA) module, as shown in Table 3.

[0148] Table 3 Ablation experiment results of different placement positions of BMHG-S6 and DCFE modules

[0149]

[0150] As shown in Table 3, when BMHG-S6 is deployed before MHSA, the average Dice is 78.46% (increased by 3.54 percentage points compared to the deployment after MHSA), because it can reduce the calculation redundancy of MHSA and focus on the key area by screening the three-dimensional spatial features through the gating mechanism first. When DCFE is deployed after MHSA, the average Dice is 78.25% (increased by 2.63 percentage points compared to the deployment before MHSA), because it can further perform multi-scale fusion on the attention features output by MHSA to improve the feature resolution.

[0151] Therefore, when BMHG-S6 is deployed before MHSA and DCFE is deployed after MHSA, the average Dice of the model is 79.78%, which is the best among all configurations, and this structure is the final architecture of SHF-SAM.

[0152] 1.2.3 Parameter selection of LoRA fine-tuning rank (r)

[0153] LoRA reduces the parameter amount of fine-tuning through low-rank matrix decomposition. The experiment compares the influence of different ranks (r=4, 8, 16, 32) on the performance of the model, and the results are shown in Table 4.

[0154] Table 4 Ablation experiment results of different rank values

[0155]

[0156] As shown in Table 4, when r=8, the average Dice of the model is 79.78% (optimal), at which time the low-rank matrix can balance between feature expression ability and parameter redundancy; when r=4, the dimension of the low-rank matrix is insufficient, and the Dice of SRF segmentation is only 59.24%, the feature expression ability is limited; when r=16, 32, the excessive increase of parameter amount leads to overfitting of the model, and the segmentation accuracy of PED decreases (decreased by 6.7 percentage points when r=32 compared to r=8). Therefore, SHF-SAM finally selects r=8 as the LoRA fine-tuning rank.

[0157] 2, Segmentation results of RETOUCH dataset

[0158] 2.1 Comparison experiment results: performance verification under multi-device heterogeneous data

[0159] RETOUCH dataset is more comprehensive in verifying the robustness of the model because the samples come from multiple manufacturers' devices, and the data heterogeneity and lesion complexity are higher. The comparison results of SHF-SAM and advanced methods are shown in Table 5.

[0160] Table 5 Comparison experiment results on RETOUCH dataset

[0161]

[0162] From Table 5, it can be seen that the segmentation performance of various SAM-based adaptation methods fluctuates on different lesion categories. For example, MedSAM performs outstandingly on SRF (Dice 81.85%), but its IRF segmentation Dice is only 47.93%. The SHF-SAM method of the present application achieves balanced and excellent performance on all categories, with an average Dice coefficient of 79.06%, which is 4.14% higher than the suboptimal method CPFNet, and especially achieves a Dice value of 87.06% in the PED segmentation task.

[0163] 2.2 Visualization verification: qualitative analysis of segmentation details

[0164] Figure 7 and Figure 8 respectively present the segmentation result visualization comparison of different methods. From Figure 7 and Figure 8 it can be seen that the segmentation results of SHF-SAM for various lesions are closest to the ground truth, especially for complex shape, fuzzy boundary or small lesion targets (such as PED and SRF), which show stronger recognition and segmentation ability. In contrast, methods such as CPFNet and nnUNet are prone to under-segmentation or mis-segmentation in the case of fuzzy edges or class imbalance, while some SAM derivative models (such as MedSAM, MA-SAM, etc.) perform well in some categories, but the overall consistency and detail retention ability are still inferior to SHF-SAM of the present application.

[0165] In summary, SHF-SAM achieves the best performance on both AI Challenger 2018 and RETOUCH public OCT segmentation datasets. The t-test results of the average Dice coefficient show that the p value of all comparison methods is less than 0.05, indicating that the segmentation performance of the SHF-SAM model proposed in the present application is significantly better than all other advanced networks, fully verifying the effectiveness and robustness of the method in the OCT image segmentation task.

[0166] Obviously, the above embodiments are only examples for clear illustration, and are not a limitation on the implementation. For ordinary skilled persons in the art, other different forms of changes or variations can be made on the basis of the above description. Here, it is not necessary and impossible to exhaust all implementation modes. The obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A method for retinal image segmentation with spatial adaptation and heat map guidance, characterized in that, The method comprises the following steps: S1: obtaining retinal image data containing three-dimensional image information and at least one retinal lesion area; S2: constructing a retinal image segmentation model, based on the retinal image data, using a joint loss function composed of cross-entropy loss and Dice loss to supervise the training process of the retinal image segmentation model, aiming to minimize the value of the joint loss function, to obtain a trained retinal image segmentation model; S3: inputting a retinal image to be segmented into the trained retinal image segmentation model to obtain a segmentation result of the retinal image; The retinal image segmentation model comprises an image encoder, a two-stage mask decoder, and a heat map guided mask self-attention module; the image encoder encodes a retinal image into an image embedding vector; The two-stage mask decoder comprises a first-stage decoder and a second-stage decoder, both based on a SAM mask decoder structure; the first-stage decoder processes the image embedding vector to generate a prior mask; Based on the retinal image data, a corresponding heat map is generated, and the heat map, the image embedding vector, and the prior mask are input into the heat map guided mask self-attention module for feature enhancement processing to obtain an enhanced image embedding; After the enhanced image embedding is subjected to mask refinement processing by the second-stage decoder, a final segmentation result of the retinal image is output to distinguish different types of retinal lesion areas; The method of inputting the heat map, the image embedding vector, and the prior mask into the heat map guided mask self-attention module for feature enhancement processing to obtain an enhanced image embedding is as follows: The heat map is normalized to obtain normalized heat map data, and the normalized heat map data is multiplied by Gaussian noise in matrix form to obtain heat map modulation noise; The heat map modulation noise is added to the prior mask to obtain enhanced prior features; The enhanced prior features are subjected to self-attention processing to obtain global context features; After nonlinear transformation of the global context features, the channel dimension is compressed to obtain compressed features; The compressed features are subjected to Hadamard product operation with the image embedding output by the image encoder, and the operation result is added to the image embedding to obtain the enhanced image embedding.

2. The spatially adaptive and heat map guided retinal image segmentation method of claim 1, wherein, The image encoder uses a visual Transformer as a backbone network, and the backbone network comprises a plurality of cascaded spatial adaptive feature enhancement Transformer blocks. Each of the spatial adaptive feature enhancement Transformer blocks comprises a bidirectional multi-head gated S6 module, a multi-head self-attention block, a deep convolution feature extraction module and a feedforward network layer connected in sequence, and a wavelet convolution adapter is arranged in parallel at the feedforward network layer, an input end of the wavelet convolution adapter receives a retinal original image and output features of the deep convolution feature extraction module, features output by the wavelet convolution adapter and output features of the feedforward network layer and the deep convolution feature extraction module are subjected to an addition operation to obtain output features of the spatial adaptive feature enhancement Transformer block.

3. The spatially adaptive and heat map guided retinal image segmentation method of claim 2, wherein, The bidirectional multi-head gated S6 module comprises a normalization layer and two multi-head gated S6 modules, and a processing procedure of the bidirectional multi-head gated S6 module comprises: normalizing the input signal by the normalization layer to obtain a normalized feature ; normalizing the features respectively, a first multi-head gated S6 module is input in a forward direction and a second multi-head gated S6 module is input in a reverse direction to obtain forward processing features and reverse processing features; The forward processing features and the reverse processing features are spliced, and a residual connection is added, so that the spliced features and the original input signal are added The output features of the bidirectional multi-head gated S6 module are obtained by adding .

4. The spatially adaptive and heat map guided retinal image segmentation method of claim 3, wherein, The multi-head gated S6 module is constructed based on an S6 module and an inter-head gating module, and a processing procedure of the multi-head gated S6 module comprises: The normalized features are uniformly divided into H independent features in a channel dimension, and a channel dimension of each independent feature is 1 / H of a channel dimension of the normalized features; wherein H is an even number, and H is a block number of the S6 module; Each independent feature is subjected to nonlinear transformation and local feature extraction by a randomly initialized S6 module, and results after the S6 module processes the last H / 2 independent features are subjected to an activation function to generate a gating signal; The inter-head gating module performs a gating operation on results after the S6 module processes the first H / 2 independent features by using the gating signal to obtain output features of the multi-head gated S6 module.

5. The spatially adaptive and heat map guided retinal image segmentation method of claim 2, wherein, The deep convolution feature extraction module comprises a normalization layer, a down-projection layer, a 3D Inception deep separable convolution module and an up-projection layer connected in sequence, and an input end of the normalization layer and an output end of the up-projection layer are connected in a residual manner; A processing procedure of the deep convolution feature extraction module comprises: The normalization layer performs normalization processing on output features of the multi-head self-attention block to obtain normalized features; The down-projection layer is used to compress the normalized features to a low-dimensional space to obtain low-dimensional features; The low-dimensional features are input into the 3D Inception deep separable convolution module for feature extraction to obtain multi-scale three-dimensional features; The up-projection layer is used to restore the multi-scale three-dimensional features to an original feature dimension to obtain high-dimensional restored features; The high-dimensional restored features and the output features of the multi-head self-attention block are subjected to an addition operation to obtain output features of the deep convolution feature extraction module.

6. The spatially adaptive and heat map guided retinal image segmentation method of claim 5, wherein, The 3D Inception deep separable convolution module is composed of a plurality of parallel deep separable 3D convolution branches and an identity branch, and convolution kernel sizes of the plurality of deep separable 3D convolution branches are different; and a processing procedure of the 3D Inception deep separable convolution module comprises: The low-dimensional features are split and input into different deep separable 3D convolution branches, and the low-dimensional features are identity mapped through the identity branch; the results of the deep separable 3D convolution branches and the identity mapping are spliced to obtain multi-scale three-dimensional features.

7. The spatially adaptive and heat map guided retinal image segmentation method of claim 2, wherein, The wavelet convolution adapter comprises a high-pass filtering unit, a wavelet convolution unit, a convolution layer, a down-sampling linear projection layer, a multi-layer perceptron and an up-sampling linear projection layer, the high-pass filtering unit, the wavelet convolution unit and the convolution layer are sequentially connected to form a first branch, the down-sampling linear projection layer constitutes a second branch, the output ends of the first branch and the second branch are added to connect the input end of the multi-layer perceptron, and the output end of the multi-layer perceptron is connected to the input end of the up-sampling linear projection layer; The processing process of the wavelet convolution adapter comprises: After performing fast Fourier transform on the retinal original image, high-frequency information is extracted through the high-pass filtering unit; The high-frequency information is input into the wavelet convolution unit for processing, and then the first feature is obtained through the convolution layer; Meanwhile, the output feature of the deep convolution feature extraction module is reduced in size through the down-sampling linear projection layer to obtain a low-dimensional reduced feature; The first feature and the low-dimensional reduced feature are added to fuse the features to obtain a fused feature; The fused feature is input into the multi-layer perceptron for nonlinear transformation, and then the original feature size is restored through the up-sampling linear projection layer to obtain the output feature of the wavelet convolution adapter.

8. The spatially adaptive and heat map guided retinal image segmentation method of claim 1, wherein, Based on the retinal image data, the method for generating a corresponding heat map is as follows: A heat map generation model is constructed, the heat map generation model is trained through the retinal image data, and the pre-training weight of the heat map generation model is obtained after the training is completed; During the training or inference of the retinal image segmentation model, the pre-training weight of the heat map generation model is loaded as a feature extractor to generate a heat map of the retinal image data.

9. The spatially adaptive and heat map guided retinal image segmentation method of claim 1, wherein, The expression of the joint loss function is as follows: wherein, is a joint loss function value, , is a weight parameter, is a cross-entropy loss, is a Dice loss; The cross-entropy loss The calculation formula is: N is the total sum of pixels of the feature map, i is the pixel index, i = 1, 2, …, N; C is the number of classes of the segmentation target; represents the label of the i-th pixel in the c-th class, and each pixel only belongs to one class in C classes; represents the probability that the model predicts that the i-th pixel belongs to the c-th class; The Dice loss The calculation formula is: , is a smoothing factor for preventing the numerator or denominator from being zero.

Images