Retinal vessel segmentation method based on cross-attention and double-branch pooling fusion

Abstract

The application discloses a retinal blood vessel segmentation method based on cross attention and double-branch pooling fusion. The retinal fundus image to be segmented is input into a trained retinal blood vessel segmentation model to obtain a blood vessel segmentation result of the retinal fundus image, wherein the retinal blood vessel segmentation model adopts a left-right symmetrical encoder-decoder architecture. The application introduces a novel coarse segmentation mask guided cross attention module in the retinal fundus image segmentation task to reduce the semantic gap between the encoder features and the decoder features in the traditional U-Net skip connection. In addition, considering the characteristics of thin and thin retinal blood vessels, a double-branch pooling fusion module is proposed to replace the original down-sampling module of U-Net to reduce the spatial information loss of blood vessels caused by multiple down-sampling operations in the traditional U-Net. The retinal blood vessel segmentation method can significantly improve the performance of retinal blood vessel segmentation.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and computer vision technology, specifically to a retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion. Background Technology

[0002] Retinal vessel segmentation is crucial for the diagnosis of many ocular diseases, including hypertensive retinopathy, diabetic retinopathy, and arteriosclerosis. The morphological characteristics of retinal vessels, including properties such as thickness, curvature, and density, are of considerable value as indicators for detecting and diagnosing these diseases. In current clinical practice, manual visual examination remains the standard for obtaining this morphological information. However, this method is labor-intensive, time-consuming, and subjective. Furthermore, due to the numerous and narrow capillaries in the entire fundus image, resulting in low local contrast with the fundus background, manual labeling is prone to errors. Therefore, exploring automated and accurate retinal segmentation methods is of great significance.

[0003] In recent years, deep learning-based technologies have received widespread attention in the field of medical image processing. Thanks to the achievements of U-Net, a series of U-shaped encoder-decoder networks for the retina have been proposed and have achieved good results. These U-Net variants typically employ two techniques to improve performance:

[0004] (1) Multi-scale based methods. Due to the variation in blood vessel width in fundus images, some methods take a multi-scale perspective, aiming to enhance the network's ability to segment blood vessels of different widths. For example, the Pyramid Squeeze and Excitement (PSE) module in the Multi-Scale Feature Interaction Network (MFI-Net) proposed by Yiwen Ye et al. uses a patch-level pyramid design to extend the squeeze and excitement (SE) operation to multiple scales. The Multi-Scale Network Follower Network (MS-NFN) proposed by Yicheng Wu et al. scales fundus images to high and low resolutions respectively, and uses two processes to extract and aggregate multi-scale feature representations.

[0005] (2) Attention-based methods. There are also methods that use attention modules to obtain better feature representations. The fully attention-based network (FANet) proposed by Kaiqi Li et al. uses lightweight bidirectional attention blocks to model global dependencies and reduce intra-class inconsistencies. The channel and spatial attention network (CS2-Net) proposed by Lei Mou et al. introduces a dual self-attention module consisting of spatial and channel attention between the encoder and decoder to learn feature representations of vascular structures.

[0006] However, despite the good performance achieved by the aforementioned methods in vessel segmentation, they fail to consider the following two issues in retinal vessel segmentation. First, a significant semantic gap exists between the features of the symmetric encoder and decoder, potentially leading to the loss of information about vascular structures. This is because the feature sets of the encoder and decoder stages are incompatible, and simple skip connections can impair segmentation performance to some extent. While some methods use attention modules to replace skip connections (e.g., Attention-UNet), their goal is to capture context to obtain enhanced representations of encoder features, rather than reducing the semantic gap between encoder and decoder features. Second, the multi-stage downsampling operations within the U-Net framework can lead to a loss of spatial information. Many small and fragile blood vessels are visible in fundus images. While downsampling can reduce feature size to increase the network's receptive field and reduce computation, it easily results in the loss of high-resolution spatial information, which is detrimental to the segmentation of small blood vessels.

[0007] Therefore, the current problems in retinal vessel segmentation can be summarized into two aspects:

[0008] First, there is a significant semantic gap between the features of the encoder and decoder, which may lead to the loss of information about vascular structures. Therefore, it is necessary to reduce the semantic gap between the two.

[0009] Second: High-resolution spatial information is easily lost during downsampling, which is very detrimental to the segmentation of thinner and finer blood vessels. Therefore, it is necessary to minimize the loss of vascular spatial information during downsampling.

[0010] In summary, it is necessary to design a module that can alleviate the semantic gap between encoder features and decoder features, as well as a downsampling module that can minimize the loss of spatial information, thereby improving the performance of retinal vessel segmentation. These are the technical problems that urgently need to be solved. Summary of the Invention

[0011] The technical problem to be solved by this invention is how to reduce the semantic gap between encoder features and decoder features in a U-shaped encoder-decoder network, and how to reduce the loss of vascular spatial information caused during the downsampling of feature maps.

[0012] The specific technical solution adopted in this invention is as follows:

[0013] In a first aspect, the present invention provides a retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion, the specific process of which is as follows: a trained retinal vessel segmentation model is obtained, and the retinal fundus image to be segmented is input into the retinal vessel segmentation model to obtain the vessel segmentation result of the retinal fundus image; the retinal vessel segmentation model adopts a left-right symmetrical encoder-decoder architecture.

[0014] In the encoder of the retinal vessel segmentation model, the input retinal fundus image to be segmented first passes through a convolutional block with residual connections to obtain the first layer encoder features; the first layer encoder features are then passed through a dual-branch pooling fusion module and the convolutional block to obtain the second layer encoder features; the second layer encoder features are then passed through the dual-branch pooling fusion module and the convolutional block to obtain the third layer encoder features; and the third layer encoder features are then passed through the dual-branch pooling fusion module and the convolutional block to obtain the deepest layer encoder features.

[0015] In the decoder of the retinal vessel segmentation model, the deepest encoder features are sequentially upsampled and processed by the convolutional block to obtain the third-layer decoder features. The third-layer decoder features and the third-layer encoder features are then input into the cross-attention module for feature fusion to obtain the third-layer fused features. The third-layer decoder features and the third-layer fused features are then concatenated by channel stacking to obtain the third-layer skip connection features. The third-layer skip connection features are sequentially upsampled and processed by the convolutional block to obtain the second-layer decoder features. The second-layer decoder features and the second-layer encoder features are then input into the cross-attention module for feature fusion to obtain... The second-layer fusion features are obtained by concatenating the second-layer decoder features and the second-layer fusion features by channel superposition. The second-layer skip connection features are then passed through upsampling and the convolution block to obtain the first-layer decoder features. The first-layer decoder features and the first-layer encoder features are then input into the cross-attention module for feature fusion to obtain the first-layer fusion features. The first-layer decoder features and the first-layer fusion features are then concatenated by channel superposition to obtain the first-layer skip connection features. The first-layer skip connection features are then segmented using a linear classification head to obtain the blood vessel segmentation result of the retinal fundus image.

[0016] As a preferred embodiment of the first aspect, in the convolutional block, the first input data sequentially passes through a first convolutional layer, a batch normalization layer, a ReLU layer, the first convolutional layer, a batch normalization layer, and a ReLU layer to obtain first intermediate data; the first input data sequentially passes through a standard convolutional layer, a batch normalization layer, and a ReLU layer to obtain second intermediate data; the first intermediate data and the second intermediate data are added element-wise to obtain the first output data.

[0017] As a preferred embodiment of the first aspect, in the linear classification head, the second input data sequentially passes through the first convolutional layer, the standard convolutional layer, and the Sigmoid function to obtain the second output data.

[0018] As a preferred embodiment of the first aspect, in the cross-attention module, the decoder features of each layer are processed by a linear classification head to obtain a coarse segmentation mask for each layer; the decoder features of each layer and the coarse segmentation mask of each layer are subjected to a mask global average pooling operation to generate a foreground prototype and a background prototype; the foreground prototype and the background prototype are concatenated to obtain the final prototype; then, the encoder features of each layer corresponding to the decoder features of each layer are projected into the query space Q as a query of the cross-attention mechanism to obtain the query vector Q. i Projecting the final prototype onto the bond space K yields the bond vector K. i Projecting the final prototype onto the value space V yields the value vector V. i Finally, the query vector Q is... i Key vector K i Value vector V i Perform cross-attention calculations to obtain the fusion feature T of each layer output by the cross-attention module. i :

[0019]

[0020] Where σ(·) represents the softmax operation; (·) T Represents matrix transpose; Reshape(·) represents the shape transformation function; C i This represents the number of feature channels in the i-th layer.

[0021] As a preferred embodiment of the first aspect, in the dual-branch pooling fusion module, the encoder features of each layer are downsampled using an average pooling layer and a max pooling layer, respectively. The encoder features processed by the average pooling layer are then concatenated to obtain pooled concatenated encoder features. These pooled concatenated encoder features are then input into the standard convolutional layer to obtain the context features of each layer. The context features of each layer are then average-pooled along the x-direction to obtain x-direction average-pooled features. Finally, the x-direction average-pooled features are sequentially passed through the standard convolutional layer and copied row by row. To ensure consistency with the original encoder feature dimension, the x-direction context features of each layer are obtained. The context features of each layer are then average-pooled along the y-direction to obtain y-direction average-pooled features. These y-direction average-pooled features are then passed sequentially through the standard convolutional layer and a column-wise copy operation to obtain the y-direction context features of each layer. Finally, the x-direction context features and y-direction context features of each layer are passed sequentially through element-wise addition and the first convolutional layer to obtain concatenated context features. The concatenated context features are then added element-wise to the context features of each layer to obtain the corresponding bi-branch pooling fusion features of each layer.

[0022] As a preferred embodiment of the first aspect, in the dual-branch pooling fusion module, the window of the average pooling layer and the window of the maximum pooling layer are both 2×2.

[0023] As a preferred embodiment of the first aspect, the kernel size of the first convolutional layer is 3×3, the stride is 2, and the padding is 1; the kernel size of the standard convolutional layer is 1×1.

[0024] As a preferred embodiment of the first aspect, the retinal vessel segmentation model is pre-trained using labeled training data before being used for actual retinal vessel segmentation.

[0025] As a preferred embodiment of the first aspect, the loss function used for training the retinal vessel segmentation model is the Dice loss, which updates the network parameters of the retinal vessel segmentation model.

[0026] It should be noted that the retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion described above in this invention can be used for non-disease diagnosis purposes, such as scientific research, testing or data processing, and can also be used to provide intermediate parameters in the field of assisted diagnosis.

[0027] In a second aspect, the present invention provides a retinal vessel segmentation system based on cross-attention and bi-branch pooling fusion, comprising:

[0028] The image acquisition module is used to acquire images of the retina and fundus to be segmented;

[0029] The retinal vessel segmentation module is used to acquire a trained retinal vessel segmentation model. The retinal fundus image to be segmented is input into the retinal vessel segmentation model to obtain the vessel segmentation result of the retinal fundus image. The retinal vessel segmentation model adopts a left-right symmetrical encoder-decoder architecture.

[0030] In the encoder of the retinal vessel segmentation model, the input retinal fundus image to be segmented first passes through a convolutional block with residual connections to obtain the first layer encoder features; the first layer encoder features are then passed through a dual-branch pooling fusion module and the convolutional block to obtain the second layer encoder features; the second layer encoder features are then passed through the dual-branch pooling fusion module and the convolutional block to obtain the third layer encoder features; and the third layer encoder features are then passed through the dual-branch pooling fusion module and the convolutional block to obtain the deepest layer encoder features.

[0031] In the decoder of the retinal vessel segmentation model, the deepest encoder features are sequentially upsampled and processed by the convolutional block to obtain the third-layer decoder features. The third-layer decoder features and the third-layer encoder features are then input into the cross-attention module for feature fusion to obtain the third-layer fused features. The third-layer decoder features and the third-layer fused features are then concatenated by channel stacking to obtain the third-layer skip connection features. The third-layer skip connection features are sequentially upsampled and processed by the convolutional block to obtain the second-layer decoder features. The second-layer decoder features and the second-layer encoder features are then input into the cross-attention module for feature fusion to obtain... The second-layer fusion features are obtained by concatenating the second-layer decoder features and the second-layer fusion features by channel superposition. The second-layer skip connection features are then passed through upsampling and the convolution block to obtain the first-layer decoder features. The first-layer decoder features and the first-layer encoder features are then input into the cross-attention module for feature fusion to obtain the first-layer fusion features. The first-layer decoder features and the first-layer fusion features are then concatenated by channel superposition to obtain the first-layer skip connection features. The first-layer skip connection features are then segmented using a linear classification head to obtain the blood vessel segmentation result of the retinal fundus image.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] This invention introduces a Co-attention Module (CoMCA) guided by a coarse segmentation mask in retinal vessel segmentation to reduce the semantic gap between encoder and decoder features in the encoder-decoder network. It also introduces a Two-Branch Pooling Fusion Module (DBPF) to reduce the loss of high-resolution spatial information during traditional downsampling. Compared to skip connections in traditional encoder-decoder networks, CoMCA effectively reduces the semantic gap between encoder and decoder features, greatly aiding in the recovery of vascular structure information during upsampling. Compared to traditional downsampling, DBPF effectively reduces the loss of high-resolution vascular spatial information caused by downsampling, providing a gain for fine-grained vessel segmentation. Therefore, in this invention, CoMCA and DBPF significantly improve the performance of the retinal vessel segmentation model. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the retinal vessel segmentation model (CoMCAN) structure of the present invention;

[0035] Figure 2 This is a structural diagram of the Cross-Attention Module (CoMCA) of the present invention;

[0036] Figure 3 This is a structural diagram of the dual-branch pooling fusion module (DBPF) of the present invention;

[0037] Figure 4 This is a flowchart of the training and testing process for the CoMCAN retinal vessel segmentation model in this invention.

[0038] Figure 5 This is a visualization of the test results in an embodiment of the present invention. Detailed Implementation

[0039] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0040] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0041] In recent years, deep learning-based technologies have garnered widespread attention in the field of medical image processing. Due to the success of U-Net, current mainstream methods for retinal vessel segmentation in fundus images are based on U-shaped network frameworks, and these methods have achieved good results. However, current mainstream methods do not consider the following two problems: First, there is a significant semantic gap between the symmetric encoder and decoder features in the U-shaped network structure; simple skipping makes it difficult to effectively fuse the two features, leading to the loss of vascular structure information. Second, current neural network models require downsampling operations to reduce feature size, thereby increasing the network's receptive field and reducing computational cost. However, due to the thin and delicate nature of retinal vessels, multi-stage downsampling operations within the U-shaped network framework result in the loss of vascular spatial information. The core of this invention is to propose a coarse-mask-guided crossattention (CoMCA) module and a dual-branch pooling fusion (DBPF) downsampling module in the retinal vessel segmentation model to address these two problems. It should be noted that these two modules can be integrated as embedded modules into the feature fusion and feature downsampling parts of any U-Net-based semantic segmentation framework, achieving plug-and-play functionality. The CoMCA module can be used in the fusion of encoder and decoder features, while the DBPF module can be used to replace the downsampling module of any model.

[0042] Therefore, the present invention provides a retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion, specifically: acquiring a trained retinal vessel segmentation model (CoMCAN), inputting the retinal fundus image to be segmented into the above-mentioned retinal vessel segmentation model, and obtaining the vessel segmentation result of the retinal fundus image.

[0043] It should be noted that the specific form of the retinal vessel segmentation model used in this invention is not limited, as long as it has an encoder-decoder structure and can combine the cross-attention module (CoMCA) of this invention in the skip connection part and the dual-branch pooling fusion module (DBPF) of this invention in the downsampling process, such as U-Net, Fast R-CNN, etc.

[0044] In a preferred embodiment of the present invention, based on the aforementioned cross-attention module (CoMCA) and dual-branch pooling fusion module (DBPF), and using the U-Net network as the baseline network model for retinal vessel segmentation, a retinal vessel segmentation model CoMCAN combining the two modules is designed. This provides a retinal vessel segmentation method based on cross-attention and dual-branch pooling fusion. In this method, the retinal vessel segmentation model (CoMCAN) uses the dual-branch pooling fusion module (DBPF) for downsampling during the downsampling process, and uses the cross-attention module (CoMCA) for information fusion of encoder and decoder features during the skip connection process. Finally, the retinal vessel segmentation model (CoMCAN) outputs the vessel segmentation result of the retinal fundus image.

[0045] The specific structure and principle of the CoMCAN retinal vessel segmentation model described above are described in detail below.

[0046] Specifically, in this embodiment, the specific structure of the retinal vessel segmentation model CoMCAN is as follows: Figure 1 As shown, similar to the traditional U-Net, it adopts a symmetrical U-shaped encoder-decoder structure, as detailed below:

[0047] In the encoder of the retinal vessel segmentation model, the input retinal fundus image to be segmented first passes through a convolutional block (RCB) with residual connections. The structure diagram of this convolutional block is shown in... Figure 1 The diagram shows how the encoder features of the first layer, E1, are obtained. These features are then passed sequentially through a dual-branch pooling fusion module (DBPF) and a convolutional block (RCB) to obtain the encoder features of the second layer, E2. The same process is repeated to obtain the encoder features of the third layer, E3. Finally, the same process is repeated to obtain the encoder features of the deepest layer, E4. The dual-branch pooling fusion module (DBPF), as described above, is used to replace the downsampling operation.

[0048] In the decoder of the retinal vessel segmentation model, the deepest encoder feature E4 is first upsampled and processed by the aforementioned convolutional block (RCB) to obtain the third-layer decoder feature D3. Then, the third-layer decoder feature and the corresponding layer's encoder feature (i.e., the third-layer encoder feature E3) are input into the cross-attention module (CoMCA) for feature fusion to obtain the third-layer fused feature T3. The third-layer decoder feature D3 and the third-layer fused feature T3 are concatenated by channel stacking to obtain the third-layer skip connection feature R3. The third-layer skip connection feature R3 is then upsampled and processed by the convolutional block (RCB) to obtain the second-layer decoder feature D2. Finally, the second-layer decoder feature and the second-layer encoder feature are input into the cross-attention module (CoMCA) for further processing. Feature fusion is performed to obtain the fused feature T2 of the second layer. The decoder feature D2 of the second layer and the fused feature T2 of the second layer are concatenated by channel stacking to obtain the skip connection feature R2 of the second layer. The skip connection feature R2 of the second layer is then passed through upsampling and convolutional blocks (RCB) to obtain the decoder feature D1 of the first layer. The decoder feature and the encoder feature of the first layer are then input into the cross-attention module (CoMCA) for feature fusion to obtain the fused feature T1 of the first layer. The decoder feature D1 of the first layer and the fused feature T1 of the first layer are concatenated by channel stacking to obtain the skip connection feature R1 of the first layer. The skip connection feature R1 of the first layer is segmented using a linear classification head (Cls_head) to obtain the blood vessel segmentation result Y of the retinal fundus image.

[0049] It should be noted that in the convolutional block with residual connections (RCB) of the present invention, the first input data is first passed through a first convolutional layer, a batch normalization layer (BN), a ReLU layer, another first convolutional layer, a batch normalization layer (BN), and a ReLU layer in sequence to obtain the first intermediate data; then the first input data is passed through a standard convolutional layer, a batch normalization layer (BN), and a ReLU layer in sequence to obtain the second intermediate data; finally, the first intermediate data and the second intermediate data are added element by element to obtain the first output data.

[0050] It should be noted that in the linear classification head (Cls_head) of the present invention, the second input data is sequentially passed through the first convolutional layer, the standard convolutional layer, and the Sigmoid function to obtain the second output data.

[0051] It should be noted that in the cross-attention module (CoMCA) of this invention, the decoder features of each layer are processed by a linear classification head (Cls_head) to obtain the coarse segmentation mask for each layer. The decoder features of each layer and the coarse segmentation mask of each layer are then processed by masked global average pooling (MGAP) to generate a foreground prototype and a background prototype. Specifically, the masked global average pooling operation involves element-wise multiplication followed by average pooling. The foreground prototype and the background prototype are then concatenated to obtain the final prototype. Finally, the encoder features of each layer corresponding to the decoder features of each layer are projected into the query space Q as the query of the cross-attention mechanism to obtain the query vector Q. i Projecting the final prototype onto the bond space K yields the bond vector K. i Projecting the final prototype onto the value space V yields the value vector V. i Finally, the query vector Q... i Key vector K i Value vector V i Cross-attention is calculated to obtain the fusion feature R of each layer output by the cross-attention module (CoMCA). i :

[0052]

[0053] Where σ(·) represents the softmax operation; (·) T Represents matrix transpose; Reshape(·) represents the shape transformation function; C i This represents the number of feature channels in the i-th layer.

[0054] The specific structure of the cross-attention module (CoMCA) of the present invention will be described in detail below. Figure 2This is a diagram illustrating the overall structure of the Cross-Attention Module (CoMCA). The cross-attention mechanism was first applied in Natural Language Processing (NLP) and has shown excellent performance in tasks such as machine translation, text generation, and cross-image caption generation. The calculation method for cross-attention is similar to self-attention, but the difference lies in that the Query, Key, and Value in cross-attention come from different sequences. This mechanism allows information interaction between different feature maps. In a typical cross-attention mechanism, the Query comes from encoder features, and the Key and Value come from decoder features. Due to the large size of the feature maps, calculating the similarity matrix becomes computationally expensive. Therefore, considering computational limitations, to more efficiently and effectively fuse encoder and decoder features and reduce semantic friction, this invention first performs a coarse segmentation on the decoder features. The coarse segmentation mask is used to guide the generation of foreground and background prototype vectors, and these two prototype vectors are concatenated as the Key and Value of the cross-attention mechanism. The encoder features serve as the Query. All three are input into the cross-attention module for cross-attention calculation, effectively achieving information fusion between encoder and decoder features.

[0055] In this embodiment, the intermediate processes and parameters of the cross-attention module (CoMCA) are further described to help those skilled in the art better understand the principle of using the CoMCA. Specifically, the CoMCA includes a coarse segmentation mask generation part and a mask-guided cross-attention part. The input to the CoMCA is two feature maps, namely the encoder features and decoder features belonging to the same layer in the retinal vessel segmentation model (CoMCAN). The decoder features of each layer are denoted as D. i Its dimensions are (B×C) i ×H i ×W i The encoder features of each layer corresponding to the decoder features of each layer are denoted as E. i Its dimensions are (B×C) i ×H i ×W i ), where C i H i W iThese represent the number of feature channels, height, and width of the i-th layer, respectively. B is the input batch size. Retinal fundus images require preprocessing, and then batch processing is needed during the training phase. The retinal vessel segmentation model is trained using training samples from each batch. Therefore, the value of B during training depends on the sample size of each batch and is not limited in this embodiment. During the prediction phase, B can be set to 1. First, the coarse segmentation mask generation part of the cross-attention module (CoMCA): After obtaining the decoder features of each layer and the corresponding encoder features of each layer, the decoder features D of each layer... i Input a linear classification head for segmentation, and obtain the coarse segmentation mask M for each layer. i Its dimensions are (B×1×H) i ×W i Next is the mask-guided cross-attention part: utilizing a coarse segmentation mask M. i and the decoder features D of each layer i The foreground prototype P is generated by global average pooling through a mask. f and background prototype P b Foreground Prototype P f and background prototype P b The dimensions are all (B×C) i ×1). Wherein, P f =F pool (M i ⊙D i ), P b =F pool ((1-M i )⊙D i ), F pool ⊙ represents global average pooling, and ⊙ represents the Hadamard product. Then, the foreground prototype P... f and background prototype P b When pieced together, the final prototype P is obtained, with dimensions (B×C). i ×2), so that the final prototype P contains rich foreground and background information from the decoder. Then, the encoder features E of each layer are... i The query, acting as a cross-attention mechanism, is projected into the query space Q to obtain the query vector Q. i Projecting the final prototype onto the bond space K yields the bond vector K. i Projecting the final prototype onto the value space V yields the value vector V. i Finally, the query vector Q... i Key vector K i Value vector V i Cross-attention is calculated to obtain the fused features T of each layer output by the cross-attention module (CoMCA).i Its dimensions are reshaped to the original feature dimensions (B×C). i ×H i ×W i ), where the query vector Q i Key vector K i Value vector V i The function forms are as follows:

[0056]

[0057]

[0058]

[0059] in, This represents the weights, and the dimensions of each weight are (B×C). i ×C i ); Query vector Q i The dimension is (B×C) i ×H i W i ); Key vector K i The dimension is (B×C) i ×2); Value vector V i The dimensions are (B×2×C) i ).

[0060] It should be noted that in the dual-branch pooling fusion module (DBPF) of the present invention, the encoder features of each layer are downsampled using an average pooling layer and a max pooling layer, respectively. The encoder features processed by the average pooling layer are concatenated to obtain pooled encoder features. These pooled encoder features are then input into the standard convolutional layer to obtain the context features F of each layer. i ; The context features F of each layer i Average pooling is performed along the x-direction to obtain the x-direction average pooling features. These x-direction average pooling features are then passed sequentially through the standard convolutional layers described above, followed by a row-by-row copying operation (Expand operation) to ensure they match the original encoder feature dimensions, thus obtaining the x-direction context features for each layer. The context features F of each layer i Average pooling is performed along the y-direction to obtain the y-direction average pooled features. These y-direction average pooled features are then passed sequentially through the standard convolutional layers described above, followed by a column-wise copying operation (Expand operation) to obtain the y-direction context features of each layer. Finally, the x-direction context features of each layer are... context features in the y-direction of each layer After element-wise addition and the first convolutional layer, the concatenated context features are obtained; these concatenated context features are then combined with the context features F of each layer. i Adding elements together, we obtain the corresponding bi-branch pooling fusion feature E′ for each layer. i .

[0061] The specific structure of the dual-branch pooling fusion module (DBPF) of the present invention will be described in detail below. Figure 3 This is a diagram showing the overall structure of the Dual Branch Pooling Fusion Module (DBPF). Downsampling is a crucial component of deep convolutional neural networks. However, as the number of downsampling iterations increases, high-resolution spatial information is easily lost, which is highly detrimental to the segmentation of thin retinal vessels. To address this issue, this invention proposes the Dual Branch Pooling Fusion Module (DBPF) to minimize the loss of vascular spatial information during downsampling.

[0062] In this embodiment, the intermediate processes and parameters of the Dual Branch Pooling Fusion Module (DBPF) are further described to help those skilled in the art better understand the principle of using the DBPF. Specifically, the input to the DBPF is the encoder feature E of each layer. i First, to leverage the advantages of different pooling functions, the encoder features of each layer are downsampled using both average pooling and max pooling layers. The encoder features processed by the average pooling layer are then concatenated to obtain the pooled concatenated encoder features. These pooled concatenated encoder features are then input into the standard convolutional layer to detect more contextual features, resulting in the contextual features F for each layer. i =F 1×1 (Concat(F Avgpool (E i ), F Maxpool (E i ))), where F Avgpool (·) represents the average pooling layer, F Maxpool (·) represents the max pooling layer, Concat(·) represents the concatenation operation, F 1×1 (·) represents a 1×1 convolutional layer, where the context features F of each layer are... i The dimension is (B×C) i ×H i / 2×W i / 2). Next, after fusing the features from the two pooling methods, in order to capture global contextual information and enable the dual-branch pooling fusion module (DBPF) to construct the contextual relationship between capillaries and main vessels, the contextual features F of each layer are processed. iAverage pooling operations are performed in the x and y directions respectively, yielding x-axis average pooling features and y-axis average pooling features. These x-axis and y-axis average pooling features are then subjected to the standard convolutional layer and expand operation described above, restoring them to the original encoder feature dimensions, thus obtaining the x-axis context features for each layer. and the y-direction context features of each layer and The dimensions are all (B×C) i ×H i / 2×W i / 2), where Conv1D(·) represents a standard convolutional layer; This indicates that average pooling is performed in the x-direction; This indicates that average pooling is performed in the y-direction. Finally, the context features in the x-direction of each layer are... context features in the y-direction of each layer After element-wise addition and the first convolutional layer mentioned above, the concatenated context features are obtained; simultaneously, the initial context features F of each layer are... i The data is also residually connected to obtain the output of the dual-branch pooling fusion module (DBPF), which represents the dual-branch pooling fusion features of each layer. Its dimensions are (B×C) i ×H i / 2×W i / 2), where, This indicates element-wise addition; Conv2D(·) represents the first convolutional layer.

[0063] It should be noted that in this invention, the window sizes of the average pooling layer and the maximum pooling layer in the above-mentioned dual-branch pooling fusion module can be selected according to the actual situation. In this embodiment, the windows of both the average pooling layer and the maximum pooling layer are set to 2×2.

[0064] It should be noted that, in this invention, the parameter sizes of the first convolutional layer and the standard convolutional layer can be set according to actual conditions. In this embodiment, the size of the first convolutional kernel is 3×3, the stride is 2, and the padding is 1; the size of the kernel of the standard convolutional layer is 1×1.

[0065] It should be noted that the aforementioned retinal vessel segmentation model (CoMCAN) was pre-trained using labeled training data before being used for actual vessel segmentation. The training labels are denoted as... To expand the training sample, data augmentation can be performed on the training data. The loss function used for training the retinal vessel segmentation model can be the Dice loss. The specific training process can refer to the existing retinal vessel segmentation model (CoMCAN) training method, and will not be elaborated further.

[0066] The above-mentioned retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion is applied to a specific embodiment to demonstrate its technical effects.

[0067] Example

[0068] The retinal vessel segmentation model (CoMCAN) used in this embodiment has the specific network structure described above and will not be repeated here. Figure 4 As shown, the overall process of segmenting blood vessels in retinal fundus images can be divided into three stages: data preprocessing, model training, and image prediction.

[0069] The overall process in this embodiment can be divided into three stages: data preprocessing, model training, and image prediction, as detailed below. Figure 4 As shown.

[0070] 1. Data Preprocessing

[0071] Step 1. For the obtained raw retinal fundus image (this embodiment uses the public dataset DRIVE as an example), image preprocessing is performed. First, the raw retinal fundus RGB image is converted into a retinal fundus grayscale image. Then, normalization, contrast-limited adaptive histogram equalization, and gamma correction are used to enhance the contrast between the foreground and the background.

[0072] Step 2. Perform a cropping operation on the preprocessed retinal fundus images. Randomly crop 1000 48×48 blocks from each preprocessed retinal fundus image, and then perform data augmentation operations such as horizontal flipping and vertical rotation.

[0073] 2. Model Training

[0074] A subset of samples is selected from the DRIVE dataset to construct the training set, which is then divided into batches of N with a fixed batch size. Training samples with index i are selected sequentially from the training set, where i ∈ {0, 1, ..., N}, and the batch size is set to 32. The retinal vessel segmentation model (CoMCAN) is trained using each batch of training samples. The specific structure of the retinal vessel segmentation model (CoMCAN) is as described above and will not be repeated here. During training, the Dice loss function is calculated for each training sample. And based on the total loss of all training samples in the batch The network parameters throughout the model are adjusted until all batches of the training dataset have participated in the model training. After reaching the specified number of iterations, the model converges, and training is complete.

[0075] 3. Image Prediction

[0076] A subset of samples was reselected from the DRIVE dataset to construct a test set. During the testing phase, overlapping patches of size 64 and stride 16 were extracted from each test image and input into the trained retinal vessel segmentation model (CoMCAN) for vessel segmentation. The final predicted probability vector for each pixel class was then used, and the class with the highest probability was selected as the final output using activation functions such as the Sigmoid function. The segmentation results of each image patch were then concatenated to form a complete segmentation result, thus achieving vessel segmentation.

[0077] In this embodiment, the test results are as follows: Figure 5 The first row of images shows three test cases, and the second row shows the vessel segmentation results for each test case. It can be seen that the retinal vessel segmentation model (CoMCAN) incorporating the Cross-Attention Module (CoMCA) and the Bi-branch Pooling Fusion Module (DBPF) can obtain excellent segmentation results from fundus images. It achieves good performance in segmenting small and fragile vessels and improving the overall integrity of vessels. This aligns with the original intention of designing these two modules, namely, to improve the segmentation ability of the retinal vessel segmentation model (CoMCAN) for thinner and finer vessels.

[0078] It should also be noted that the retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a retinal vessel segmentation system based on cross-attention and bi-branch pooling fusion, corresponding to the retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion provided in the above embodiments, comprising:

[0079] The image acquisition module is used to acquire images of the retina and fundus to be segmented;

[0080] The retinal vessel segmentation module is used to acquire a trained retinal vessel segmentation model. The retinal fundus image to be segmented is input into the retinal vessel segmentation model to obtain the vessel segmentation result of the retinal fundus image. The retinal vessel segmentation model adopts a left-right symmetrical encoder-decoder architecture.

[0081] In the encoder of the retinal vessel segmentation model, the input retinal fundus image to be segmented first passes through a convolutional block with residual connections to obtain the first layer encoder features; the first layer encoder features are then passed through a dual-branch pooling fusion module and the convolutional block to obtain the second layer encoder features; the second layer encoder features are then passed through the dual-branch pooling fusion module and the convolutional block to obtain the third layer encoder features; and the third layer encoder features are then passed through the dual-branch pooling fusion module and the convolutional block to obtain the deepest layer encoder features.

[0082] In the decoder of the retinal vessel segmentation model, the deepest encoder features are sequentially upsampled and processed by the convolutional block to obtain the third-layer decoder features. The third-layer decoder features and the third-layer encoder features are then input into the cross-attention module for feature fusion to obtain the third-layer fused features. The third-layer decoder features and the third-layer fused features are then concatenated by channel stacking to obtain the third-layer skip connection features. The third-layer skip connection features are sequentially upsampled and processed by the convolutional block to obtain the second-layer decoder features. The second-layer decoder features and the second-layer encoder features are then input into the cross-attention module for feature fusion to obtain... The second-layer fusion features are obtained by concatenating the second-layer decoder features and the second-layer fusion features by channel superposition. The second-layer skip connection features are then passed through upsampling and the convolution block to obtain the first-layer decoder features. The first-layer decoder features and the first-layer encoder features are then input into the cross-attention module for feature fusion to obtain the first-layer fusion features. The first-layer decoder features and the first-layer fusion features are then concatenated by channel superposition to obtain the first-layer skip connection features. The first-layer skip connection features are then segmented using a linear classification head to obtain the blood vessel segmentation result of the retinal fundus image.

[0083] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion, characterized in that, The specific process is as follows: a trained retinal vessel segmentation model is obtained, and the retinal fundus image to be segmented is input into the retinal vessel segmentation model to obtain the vessel segmentation result of the retinal fundus image; the retinal vessel segmentation model adopts a left-right symmetrical encoder-decoder architecture; In the encoder of the retinal vessel segmentation model, the input retinal fundus image to be segmented first passes through a convolutional block with residual connections to obtain the first layer encoder features; the first layer encoder features are then passed through a dual-branch pooling fusion module and the convolutional block to obtain the second layer encoder features; the second layer encoder features are then passed through the dual-branch pooling fusion module and the convolutional block to obtain the third layer encoder features; and the third layer encoder features are then passed through the dual-branch pooling fusion module and the convolutional block to obtain the deepest layer encoder features. In the decoder of the retinal vessel segmentation model, the deepest encoder features are sequentially upsampled and processed by the convolutional block to obtain the decoder features of the third layer. The decoder features and encoder features of the third layer are then input into the cross-attention module for feature fusion to obtain the fused features of the third layer. The decoder features and fused features of the third layer are then concatenated by channel stacking to obtain the skip connection features of the third layer. The skip connection features of the third layer are sequentially upsampled and processed by the convolutional block to obtain the decoder features of the second layer. The decoder features and encoder features of the second layer are then input into the cross-attention module for feature fusion to obtain the fused features of the second layer. The decoder features of the second layer and the fused features of the second layer are concatenated by channel superposition to obtain the skip connection features of the second layer; the skip connection features of the second layer are then passed through upsampling and the convolution block to obtain the decoder features of the first layer; the decoder features of the first layer and the encoder features of the first layer are then input together into the cross-attention module for feature fusion to obtain the fused features of the first layer. The decoder features and fusion features of the first layer are concatenated by channel superposition to obtain the skip connection features of the first layer; a linear classification head is used to segment the skip connection features of the first layer to obtain the blood vessel segmentation results of the retinal fundus image.

2. The retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion as described in claim 1, characterized in that, In the convolutional block, the first input data sequentially passes through a first convolutional layer, a batch normalization layer, a ReLU layer, the first convolutional layer, a batch normalization layer, and a ReLU layer to obtain the first intermediate data; the first input data sequentially passes through a standard convolutional layer, a batch normalization layer, and a ReLU layer to obtain the second intermediate data; the first intermediate data and the second intermediate data are added element by element to obtain the first output data.

3. The retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion as described in claim 2, characterized in that, In the linear classification head, the second input data sequentially passes through the first convolutional layer, the standard convolutional layer, and the Sigmoid function to obtain the second output data.

4. The retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion as described in claim 2, characterized in that, In the cross-attention module, the decoder features of each layer are processed by a linear classification head to obtain a coarse segmentation mask for each layer. The decoder features and the coarse segmentation mask of each layer are then subjected to a global average pooling operation to generate a foreground prototype and a background prototype. The foreground prototype and the background prototype are concatenated to obtain the final prototype. Finally, the encoder features of each layer, corresponding to the decoder features, are projected into the query space Q as a query of the cross-attention mechanism to obtain the query vector Q. i Projecting the final prototype onto the bond space K yields the bond vector K. i Projecting the final prototype onto the value space V yields the value vector V. i Finally, the query vector Q is... i Key vector K i Value vector V i Perform cross-attention calculations to obtain the fusion feature T of each layer output by the cross-attention module. i : Where σ(·) represents the softmax operation; (·) T Represents matrix transpose; Reshape(·) represents the shape transformation function; C i This represents the number of feature channels in the i-th layer.

5. The retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion as described in claim 2, characterized in that, In the dual-branch pooling fusion module, the encoder features of each layer are downsampled using an average pooling layer and a max pooling layer, respectively. The encoder features processed by the average pooling layer are then concatenated to obtain the pooled concatenated encoder features. The pooled concatenated encoder features are then input into the standard convolutional layer to obtain the context features of each layer. The context features of each layer are then average pooled along the x-direction to obtain the x-direction average pooled features. The x-axis average pooling features are then passed sequentially through the standard convolutional layers and copied row by row to make them consistent with the original encoder feature dimensions, thus obtaining the x-axis context features of each layer; the context features of each layer are then average pooled along the y-axis to obtain the y-axis average pooling features. The average pooling features in the y-direction are then passed through the standard convolutional layer and copied column by column to obtain the y-direction context features of each layer. Finally, the x-direction context features of each layer and the y-direction context features of each layer are passed through the first convolutional layer element by element to obtain the concatenated context features. The concatenated context features are added element-wise to the context features of each layer to obtain the corresponding dual-branch pooling fusion features for each layer.

6. The retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion as described in claim 5, characterized in that, In the dual-branch pooling fusion module, the window of the average pooling layer and the window of the maximum pooling layer are both 2×2.

7. The retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion as described in claim 2, characterized in that, The first convolutional layer has a kernel size of 3×3, a stride of 2, and padding of 1; the standard convolutional layer has a kernel size of 1×1.

8. The retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion as described in claim 1, characterized in that, The retinal vessel segmentation model is pre-trained using labeled training data before being used for actual retinal vessel segmentation.

9. The retinal vessel segmentation method based on cross-attention and bi-branch pooling fusion as described in claim 8, characterized in that, The loss function used in training the retinal vessel segmentation model is the Dice loss, which updates the network parameters of the retinal vessel segmentation model.

10. A retinal vessel segmentation system based on cross-attention and bi-branch pooling fusion, characterized in that, include: The image acquisition module is used to acquire images of the retina and fundus to be segmented; The retinal vessel segmentation module is used to acquire a trained retinal vessel segmentation model. The retinal fundus image to be segmented is input into the retinal vessel segmentation model to obtain the vessel segmentation result of the retinal fundus image. The retinal vessel segmentation model adopts a left-right symmetrical encoder-decoder architecture. In the encoder of the retinal vessel segmentation model, the input retinal fundus image to be segmented first passes through a convolutional block with residual connections to obtain the first layer encoder features; the first layer encoder features are then passed through a dual-branch pooling fusion module and the convolutional block to obtain the second layer encoder features; the second layer encoder features are then passed through the dual-branch pooling fusion module and the convolutional block to obtain the third layer encoder features; and the third layer encoder features are then passed through the dual-branch pooling fusion module and the convolutional block to obtain the deepest layer encoder features. In the decoder of the retinal vessel segmentation model, the deepest encoder features are sequentially upsampled and processed by the convolutional block to obtain the decoder features of the third layer. The decoder features and encoder features of the third layer are then input into the cross-attention module for feature fusion to obtain the fused features of the third layer. The decoder features and fused features of the third layer are then concatenated by channel stacking to obtain the skip connection features of the third layer. The skip connection features of the third layer are sequentially upsampled and processed by the convolutional block to obtain the decoder features of the second layer. The decoder features and encoder features of the second layer are then input into the cross-attention module for feature fusion to obtain the fused features of the second layer. The decoder features of the second layer and the fused features of the second layer are concatenated by channel superposition to obtain the skip connection features of the second layer; the skip connection features of the second layer are then passed through upsampling and the convolution block to obtain the decoder features of the first layer; the decoder features of the first layer and the encoder features of the first layer are then input together into the cross-attention module for feature fusion to obtain the fused features of the first layer. The decoder features and fusion features of the first layer are concatenated by channel superposition to obtain the skip connection features of the first layer; a linear classification head is used to segment the skip connection features of the first layer to obtain the blood vessel segmentation results of the retinal fundus image.

Images