A retinal blood vessel segmentation method based on fusion of UNet and transformer

Abstract

The present application belongs to the field of medical image segmentation of computer vision, and particularly relates to a retinal blood vessel segmentation method based on UNet and Transformer fusion, which has the following characteristics: step 1, pre-processing the image to be trained to obtain a pre-processed image; step 2, inputting the pre-processed image into a retinal blood vessel segmentation model based on UNet and Transformer fusion to obtain a weight file, the model comprising an encoder, a decoder and a fusion attention mechanism, the encoder comprising a multi-flow cascaded convolution layer, a plurality of pooling layers and a plurality of residual modules, each convolution layer using a residual module, and the pooling layer being arranged between two convolution layer units; the decoder comprising a plurality of improved residual modules based on extended convolution, a plurality of up-sampling modules and a deconvolution layer, the up-sampling module being arranged between two adjacent improved residual modules based on extended convolution; the fusion attention mechanism taking the output of the pooling layer in the encoder and the output of the adjacent flow pooling layer as low-level feature input and high-level feature input, respectively, and the high-level feature input of the fusion attention mechanism at the third layer being a feature map formed by the Transformer module corresponding to the encoder pooling layer; step 3, loading the weight file and inputting a test fundus image into the model to obtain a retinal blood vessel segmentation result. In addition, the retinal blood vessel segmentation model of the present application is more sensitive to small blood vessels, and the segmentation accuracy is improved.

Description

Technical Field

[0001] This invention belongs to the field of medical image segmentation in computer vision, specifically relating to a retinal vessel segmentation method based on the fusion of UNet and Transformer. Background Technology

[0002] With my country's rapid development and the continuous improvement of people's living standards, people spend a lot of time using electronic devices. Prolonged use of electronic devices can easily lead to eye strain, thus the development of ophthalmology is receiving increasing attention from the public. Among all human organs, the eye is the organ that receives the most information. The health of the eyes has a significant impact on people's daily lives. Aging and poor eye habits in daily life can easily lead to eye diseases. By carefully screening for diseased areas in fundus images, ophthalmologists can make diagnoses based on their experience, assess the patient's condition, and provide appropriate treatment. Manually segmenting retinal vessels in fundus images is not only time-consuming and laborious but also cannot guarantee accuracy. Therefore, using computer-aided automatic retinal vessel segmentation technology to automatically analyze fundus images and accurately segment retinal vessels is of great significance for disease analysis and diagnosis.

[0003] Existing methods for retinal vessel segmentation and disease analysis can be divided into four main categories: (1) segmentation methods based on digital image processing. This method usually requires the designer to have rich knowledge of vessel segmentation. The designer designs a template based on their professional knowledge. This template mainly uses the relevant morphology of the vessels to extract retinal vessel features, such as texture, color, and contrast. The extracted features can be directly used as the segmentation result. (2) Supervised learning methods. This type of method uses retinal vessel image data that needs to be labeled. It requires researchers to design specific templates for retinal vessel features to extract features. Then, a machine learning model is used to classify the extracted features. Machine learning models have some advantages over traditional methods. For example, researchers do not need to have rich knowledge of vessels. Moreover, the generalization performance of the designed model is better, and it has good results on different vessel segmentation datasets. (3) Segmentation methods based on deep neural networks. Unlike the previous two methods, deep learning segmentation methods do not require manual design of template operators to extract features. The neural network can automatically learn to extract retinal vessel features and classify them. Therefore, this type of method is relatively simple compared to the previous two methods, and the model has good generalization ability. Deep learning methods require a large amount of labeled data. Using deep neural network models, end-to-end classification can be directly established. Input ocular blood vessel images and output segmented result images. (4) Based on the segmentation of retinal fundus images, researchers have used deep neural network models to quantitatively and qualitatively analyze the correlation between features, texture, color, values, and morphology in fundus images, and to predict the condition of thrombosis. Doctors can then make emergency and non-emergency treatments based on the prediction results, so that medical resources can be fully utilized, thereby assisting in the clinical diagnosis, assessment and tracking management of systemic diseases. Summary of the Invention

[0004] The main technical problem addressed by this invention is to provide a retinal vessel segmentation method based on the fusion of UNet and Transformer, which addresses the issue that convolutional neural network feature extraction structures cannot directly extract global features through layer stacking, resulting in discontinuities in small blood vessels, and the problem that Transformer structures focus excessively on global context modeling at all stages, leading to a lack of detailed localization information in low-resolution features, which cannot be recovered by directly upsampling to full resolution.

[0005] This invention provides a retinal vessel segmentation method based on the fusion of UNet and Transformer, characterized by the following steps: Step 1, preprocessing the image to be trained to obtain a preprocessed image. This preprocessing method includes data processing and data augmentation. The data processing includes histogram equalization of the color fundus image to uniformly distribute pixel-level gray levels and improve contrast; image smoothing to make the image brightness change gradually, reduce abrupt gradients, and improve image quality; gray-level conversion to change the gray value of each pixel according to a certain transformation relationship to make the image display clearer; and uniform illumination processing to eliminate the influence of uneven illumination in the image. The data augmentation includes randomly selecting an appropriate number of slices from the dataset. In order to increase the number of images, make the samples more diverse, and reduce overfitting, techniques such as rotation, translation, and elastic transformation are used to expand the data of the dataset. Step 2, inputting the preprocessed image into a UNet-based Transformer fusion method. The weight file is obtained from the retinal vessel segmentation model fused with nsformer. This model includes an encoder, a decoder, and a fusion attention mechanism. The encoder includes cascaded convolutional layers, multiple pooling layers, and multiple residual modules. Each convolutional layer uses a residual module, and the pooling layer is located between two convolutional layer units. The decoder includes cascaded multiple improved extended convolution-based residual modules, multiple upsampling modules, and deconvolutional layers. The upsampling module is located between two adjacent improved extended convolution-based residual units. The fusion attention mechanism uses the output of the pooling layer in the encoder and the output of the adjacent stream pooling layer as the low-level feature input and high-level feature input, respectively. The high-level feature input of the fusion attention mechanism in the third layer is the feature map formed by the corresponding encoder pooling layer through the Transformer module. Step 3: Load the weight file and input the test fundus image into the model to obtain the retinal vessel segmentation result.

[0006] The retinal vessel segmentation method based on the fusion of UNet and Transformer provided by this invention may also have the following features: a convolutional neural network is used to provide a feature pyramid, a Transformer network is used to extract detail information, and a UNet network is used to cascade anisotropy.

[0007] The retinal vessel segmentation method based on the fusion of UNet and Transformer provided by this invention may also have the following feature: wherein the convolutional layer is used to perform local perception on each feature of the image input to the retinal vessel segmentation model based on the fusion of UNet and Transformer.

[0008] Pooling layers are used to reduce the dimensionality of features obtained after convolutional layer processing, and to compress the amount of data and parameters. Pooling layers can reduce overfitting and improve the fault tolerance of the model.

[0009] An improved residual module based on extended convolution is used to enrich the speech information contained in high-level features, while low-level detail features are compensated into higher-level semantic features to optimize the segmentation results.

[0010] Deconvolutional layers are used to increase the output matrix dimension of the improved extended convolution-based convolutional module.

[0011] Furthermore, in multi-stream cascaded convolutional operations, the output of each stream and layer is directly connected to the subsequent layer in the same stream. Any layer of a convolutional unit replaces the standard UNet coding block with a residual module. The residual module effectively prevents gradient vanishing during training. The output of the convolutional operations under multi-stream cascaded convolutions is represented as follows:

[0012] H(x)={h l (x,K,θ)|h l (x,K,θ k )=r(h l (x,k-1,θ k-1 ),l=1,2,…,L}

[0013] Where H(x) represents a set of network operations in different encoding paths; h l (x,K,θ) represents the encoding stream of the l-th layer, K represents a continuous feature block, θ represents a learnable weight; r represents the residual operation on each feature block, including the pooling layer; represents the standard normalization function on the original input image x; L represents the total number of path flows, which is set to 4 in the experiment.

[0014] Furthermore, the output of the improved extended convolution-based residual module is represented as:

[0015] y i =f BN+ReLU (f C3 (x i +DRF(x i ));

[0016] Among them, y i f represents the output of the improved extended convolution-based residual module at layer i; BN+ReLU (■) indicates that batch normalization and ReLU activation function operations are performed sequentially; f C3 (■) represents a 3×3 convolution operation; x i denoted as the input of the improved extended convolution-based residual module of layer i; DRF(■) represents the improved extended convolution-based residual module structure.

[0017] Furthermore, the improved residual module structure DRF(■) based on extended convolution is represented as follows:

[0018] DRF(x i )=f BN+ReLU (f C (f BN+ReLU (f C (x i ))×f BN+ReLU (f C3 (x i ))×IRCB(x i )));

[0019] Among them, f C Represents a 1×1 convolution operation; IRCB(■) represents an extended convolution module structure.

[0020] Furthermore, the extended convolutional module structure IRCB(■) is represented as:

[0021] IRCB(x i )=f BN+ReLU (f C (x i ))+f BN+ReLU (f C3+r3 (f BN+ReLU (f C (x i ))));

[0022] Among them, f C3+r3 (■) indicates a dilated convolution with a dilation rate of 3.

[0023] Furthermore, the output of the fusion attention mechanism is represented as:

[0024] F i =u i +u i ⊙α i ;

[0025] Among them, F i This represents the output of the attention mechanism fusion at layer i, where the attention features α are represented as residuals. i and the original feature u i They are united.

[0026] u i =C([e i ,S(F i-1 )]);

[0027] Among them, u i Represents the original features in the fusion attention mechanism of the i-th layer; C(■) represents a convolutional kernel; S(■) represents an upsampling operation; [] represents the corresponding Concat operation; e i and F i-1These represent the results from the i-th layer encoder and the previous fusion module, respectively;

[0028] s i =P avg (f ReLU (f BN (u i )));

[0029] Among them, s i P represents the intermediate features in the fusion attention mechanism of the i-th layer; avg (■) indicates a global average pooling operation; f ReLU (■) indicates the ReLU activation function operation; f BN (■) indicates batch normalization operation;

[0030] α i =σ(f C (f ReLU (f C (s i ))));

[0031] Where, α i The attention feature in the fusion attention mechanism of the i-th layer is represented by σ(■); σ(■) represents the Sigmoid activation function operation; f C (■) represents a 1×1 convolution operation;

[0032] Furthermore, the Transformer module obtains the feature map results from the downsampling of the multi-stream cascaded multi-layer convolutional neural network. It then performs fixed-size slicing on the feature map and embeds position vectors, as shown below:

[0033]

[0034] Where P represents the size of the slice; N represents the number of slices;

[0035]

[0036] Where z0 represents the result of embedding the position vector; E represents the result of the first feature map slice. pos This represents a position vector.

[0037] The Transformer module consists of an L-layer multi-head attention mechanism and a multilayer perceptron module. The multi-head attention mechanism has two different partitioning methods: WMSA and SWMSA. The WMSA module is a self-attention module based on regular windows. For example, the result of the previous module is an 8×8 feature map, which is uniformly divided into four 4×4 windows starting from the top left pixel, and then attention processing is performed. The SWMSA module is a self-attention module based on cross-windows. For example, the result of the previous module is an 8×8 feature map, which is first divided into smaller-than-the-windows partitions starting from the top left pixel, and then divided into larger-windows partitions. When the size is 2 and 4 respectively, it is divided into four 2×2 windows, two 2×4 windows, two 4×2 windows, and one 4×4 window. The feature map is then shifted, attention processing is performed, and then the self-attention is set and the map is shifted again to restore it so that it can follow the WMSA module. This introduces the connection relationship between adjacent windows and improves the modeling capability. Therefore, the WMSA and SWMSA modules are used alternately, and the output of the Transformer module for two consecutive layers is as follows:

[0038]

[0039]

[0040]

[0041]

[0042] in, and z l These represent the feature outputs of the WMSA module and the MLP module in the l-th layer, respectively. and z l+1 These represent the feature outputs of the SWMSA module and the MLP module in layer l+1, respectively; the SWMSA module represents shifting the feature map and then shifting it again to restore it after setting self-attention; LN(■) represents the LayerNorm normalization operation.

[0043] Compared with other existing models, this invention addresses the problems of discontinuities in small blood vessels caused by the inability to directly extract global features through the layer-by-layer feature extraction structure of convolutional neural networks, and the lack of detailed localization information in low-resolution features due to the excessive focus on global context modeling in all stages of the Transformer structure, which cannot be recovered by directly upsampling to full resolution. It provides a retinal vessel segmentation model based on the fusion of UNet and Transformer, which has the advantages of sensitivity to small blood vessels and good segmentation accuracy. Specifically: the multi-stream cascaded convolutional layers can learn low-level structural features and high-level semantic features at each path, passing these different features as a rich information source to the fusion module, controlling the inter-layer dependencies between low-level structural features and high-level semantic features; multiple cascaded improved residual modules based on extended convolutions preserve more useful spatial information, and each convolutional layer is followed by batch normalization and ReLU activation to alleviate overfitting; the Transformer module in the fusion attention mechanism improves global context modeling information, compensating for the shortcomings of UNet in extracting edge and global information. Attached Figure Description

[0044] Figure 1 The flowchart shows the retinal vessel segmentation method based on the fusion of UNet and Transformer constructed in this invention.

[0045] Figure 2 This is a diagram showing the overall structure of the retinal vessel segmentation model based on the fusion of UNet and Transformer constructed in this invention.

[0046] Figure 3 This is a schematic diagram of the improved residual module structure based on extended convolution in this invention;

[0047] Figure 4 This is a schematic diagram of the fusion attention mechanism structure in this invention;

[0048] Figure 5 This is a schematic diagram of the structure of two consecutive WMSA modules and SWMSA modules in this invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] This invention provides a retinal vessel segmentation method based on the fusion of UNet and Transformer, which has the following features, such as Figure 1 The process includes the following steps: Step 1, preprocessing the images to be trained to obtain preprocessed images. This preprocessing method includes data processing and data augmentation. The data processing includes histogram equalization of the color fundus image to uniformly distribute pixel-level gray levels and improve contrast; image smoothing to make the image brightness change gradually, reduce abrupt gradients, and improve image quality; gray-level conversion to change the gray value of each pixel according to a certain transformation relationship to make the image display clearer; and uniform illumination to eliminate the influence of uneven illumination in the image. The data augmentation includes randomly selecting an appropriate number of slices from the dataset. In order to increase the number of images, make the samples more diverse, and reduce overfitting, techniques such as rotation, translation, and elastic transformation are used to expand the data in the dataset. Step 2, inputting the preprocessed images into a retinal vessel segmentation model based on the fusion of UNet and Transformer. The weight file is obtained. The model includes an encoder, a decoder, and a fusion attention mechanism. The encoder includes cascaded convolutional layers, multiple pooling layers, and multiple residual modules. Each convolutional layer uses a residual module, and the pooling layer is placed between two convolutional layer units. The decoder includes cascaded multiple improved extended convolution-based residual modules, multiple upsampling modules, and deconvolutional layers. The upsampling module is placed between two adjacent improved extended convolution-based residual units. The fusion attention mechanism uses the output of the pooling layer in the encoder and the output of the adjacent stream pooling layer as the low-level feature input and high-level feature input, respectively. The high-level feature input of the fusion attention mechanism in the third layer is the feature map formed by the corresponding encoder pooling layer through the Transformer module. Step 3: Load the weight file and input the test fundus image into the model to obtain the retinal vessel segmentation result.

[0051] To address the issues of discontinuities in fine blood vessels caused by the inability to directly extract global features through the layer-by-layer feature extraction structure of convolutional neural networks, and the lack of detailed localization information in low-resolution features due to the excessive focus on global context modeling in all stages of the Transformer structure, which cannot be recovered by directly upsampling to full resolution, this invention improves upon the existing UNet segmentation network by constructing a retinal vessel segmentation model based on the fusion of UNet and Transformer. In classic encoder-decoder segmentation methods, the encoder and decoder structures are symmetrically distributed, with skip connections between encoders and decoders at the same level. This invention introduces residual networks and Transformer structures, such as... Figure 2As shown, features are first extracted from the input data using convolutional operations. The encoder uses multi-stream cascaded convolutional layers, with each convolutional layer employing a residual module. Pooling layers are placed between two convolutional layer units. The decoder includes multiple cascaded, improved, extended convolution-based residual modules, such as... Figure 3 As shown, compared to traditional methods based on initial blocks, extracting deep features from an image can preserve more useful spatial information. Each convolutional layer is followed by batch normalization and ReLU activation to alleviate overfitting. Figure 4 As shown, information from two adjacent streams of the encoder is embedded into a fusion attention mechanism, allowing the constructed network model to automatically assign different weights to different features. This enables the model to focus on vascular information features while suppressing other irrelevant features, thereby improving the model's segmentation accuracy. Figure 5 As shown, the feature maps of the multi-stream cascaded multi-layer convolutional neural network in the encoder are downsampled and sliced ​​into fixed-size slices. Position vectors are embedded and the Transformer network structure is used to better integrate global context information and eliminate redundant information.

[0052] This invention proposes a multi-stream cascaded multi-layer convolutional module for the encoder stage of a U-Net network. In this module, cascading operations allow each layer within the same stream to directly connect to the next. Any layer in the convolutional unit replaces the standard UNet coding block with a residual module. The residual module effectively prevents gradient vanishing during training. The output of the multi-stream encoder is represented as:

[0053] H(x)={h l (x,K,θ)|h l (x,K,θ k )=r(h l (x,k-1,θ k-1 ),l=1,2,…,L}

[0054] Where H(x) represents a set of network operations in different encoding paths; h l (x,K,θ) represents the encoding stream of the l-th layer, K represents a continuous feature block, θ represents a learnable weight; r represents the residual operation on each feature block, including the pooling layer; represents the standard normalization function on the original input image x; L represents the total number of path flows, which is set to 4 in the experiment.

[0055] The improved residual module based on extended convolution proposed in this invention is used in the decoder stage of a U-shaped network. The introduction of the residual concept simplifies network training and alleviates the gradient vanishing problem caused by increasing the number of network layers. Furthermore, the use of skip connections in the residual network facilitates the acquisition of more semantic information, while low-level detailed features are compensated for in higher-level semantic features, thus optimizing the segmentation results. Figure 3 As shown, the output of the fusion attention mechanism module and the output of the encoder upsampling layer are used as the inputs for low-level and high-level features, respectively, and a concat connection operation is performed. The module then enters three cascaded branches. Two parallel enhancements contain a 1×1 and a 3×3 convolutional layer, respectively. The third branch uses extended convolution, then connects the outputs of the three parallel paths, and applies another 1×1 convolution. Finally, residuals are used to unify the cascaded features with the original features. A 3×3 convolutional layer is then used to reduce the feature dimensionality. The output of the improved extended convolution-based residual module is represented as follows:

[0056] y i =f BN+ReLU (f C3 (x i +DRF(x i ));

[0057] Among them, y i f represents the output of the improved extended convolution-based residual module at layer i; BN+ReLU (■) indicates that batch normalization and ReLU activation function operations are performed sequentially; f C3 (■) represents a 3×3 convolution operation; x i denoted as the input of the improved extended convolution-based residual module of layer i; DRF(■) represents the improved extended convolution-based residual module structure.

[0058] Furthermore, the improved residual module structure DRF(■) based on extended convolution is represented as follows:

[0059] DRF(x i )=f BN+ReLU (f C (f BN+ReLU (f C (x i ))×f BN+ReLU (f C3 (x i ))×IRCB(x i )));

[0060] Among them, f C Represents a 1×1 convolution operation; IRCB(■) represents an extended convolution module structure.

[0061] Furthermore, the extended convolutional module structure IRCB(■) is represented as:

[0062] IRCB(x i )=f BN+ReLU (f C (x i ))+f BN+ReLU (f C3+r3 (f BN+ReLU (f C (x i ))));

[0063] Among them, f C3+r3 (■) indicates a dilated convolution with a dilation rate of 3.

[0064] The fusion attention mechanism module proposed in this invention is used in the skip connection stage of a U-shaped network. It uses the output of the pooling layer in the encoder and the output of the adjacent flowing pooling layer as the low-level feature input and high-level feature input, respectively. The high-level feature input of the fusion attention mechanism in the third layer is the feature map formed by the Transformer module corresponding to the encoder pooling layer. For example... Figure 4 As shown, after the first convolutional operation, Batch Normalization (BN) and ReLU functions are used, followed by global average pooling to aggregate spatial information. After the next convolutional operation, BN is used again, and after the next convolutional operation, a Sigmoid function is used. This achieves contextual focusing of vascular information in the fundus image, eliminating redundant information. Then, residuals are used to unify the attention features with the original features, adaptively guiding segmentation. Furthermore, the output of the attention fusion mechanism is represented as:

[0065] F i =u i +u i ⊙α i ;

[0066] Among them, F i This represents the output of the attention mechanism fusion at layer i, where the attention features α are represented as residuals. i and the original feature u i They are united.

[0067] u i =C([e i ,S(F i-1 )]);

[0068] Among them, u i Represents the original features in the fusion attention mechanism of the i-th layer; C(■) represents a convolutional kernel; S(■) represents an upsampling operation; [] represents the corresponding Concat operation; e i and F i-1These represent the results from the i-th layer encoder and the previous fusion module, respectively;

[0069] s i =P avg (f ReLU (f BN (u i )));

[0070] Among them, s i P represents the intermediate features in the fusion attention mechanism of the i-th layer; avg (■) indicates a global average pooling operation; f ReLU (■) indicates the ReLU activation function operation; f BN (■) indicates batch normalization operation;

[0071] α i =σ(f C (f ReLU (f C (s i ))));

[0072] Where, α i The attention feature in the fusion attention mechanism of the i-th layer is represented by σ(■); σ(■) represents the Sigmoid activation function operation; f C (■) represents a 1×1 convolution operation;

[0073] Furthermore, the Transformer module obtains the feature map results from the downsampling of the multi-stream cascaded multi-layer convolutional neural network. It then performs fixed-size slicing on the feature map and embeds position vectors, as shown below:

[0074]

[0075] Where P represents the size of the slice; N represents the number of slices;

[0076]

[0077] Where z0 represents the result of embedding the position vector; E represents the result of the first feature map slice. pos This represents a position vector.

[0078] The Transformer module consists of an L-layer multi-head attention mechanism and a multilayer perceptron module. The multi-head attention mechanism has two different partitioning methods: WMSA and SWMSA. The WMSA module is a self-attention module based on regular windows. For example, the result of the previous module is an 8×8 feature map, which is uniformly divided into four 4×4 windows starting from the top left pixel, and then attention processing is performed. The SWMSA module is a self-attention module based on cross-windows. For example, the result of the previous module is an 8×8 feature map, which is first divided into smaller-than-the-windows partitions starting from the top left pixel, and then divided into larger-windows partitions. When the size is 2 and 4 respectively, it is divided into four 2×2 windows, two 2×4 windows, two 4×2 windows, and one 4×4 window. The feature map is then shifted, attention processing is performed, and then the self-attention is set and the map is shifted again to restore it so that it can follow the WMSA module. This introduces the connection relationship between adjacent windows and improves the modeling capability. Therefore, the WMSA and SWMSA modules are used alternately, and the output of the Transformer module for two consecutive layers is as follows:

[0079]

[0080]

[0081]

[0082]

[0083] in, and z l These represent the feature outputs of the WMSA module and the MLP module in the l-th layer, respectively. and z l+1 These represent the feature outputs of the SWMSA module and the MLP module in layer l+1, respectively; the SWMSA module represents shifting the feature map and then shifting it again to restore it after setting self-attention; LN(■) represents the LayerNorm normalization operation.

[0084] The retinal vessel segmentation method based on the fusion of UNet and Transformer provided in the embodiments of the present invention adopts a different approach from traditional medical image segmentation strategies. It combines Transformer with the traditional convolutional neural network UNet, allowing the UNet framework to provide a pathway for extracting low-level visual cues, thus compensating for Transformer's inability to handle small blood vessels effectively. Simultaneously, the application of Transformer compensates for UNet's shortcomings in extracting edge and global information. Furthermore, if a pure Transformer module is used as the encoder, because the Transformer module treats the input as a sequence and excessively focuses on global context modeling information at all stages, low-resolution features lack detailed location information. Therefore, a cascaded multi-stream convolutional neural network is used as the feature extractor to generate feature maps for the input of the Transformer module and the fusion attention mechanism module. On the other hand, convolutional neural network architectures (such as UNet) provide a pathway for extracting low-level visual cues, which can effectively compensate for this lack of detailed spatial location information. Finally, according to the retinal vessel segmentation method based on the fusion of UNet and Transformer in this embodiment, a convolutional neural network is used to provide a feature pyramid, a Transformer network is used to extract detailed information, and a UNet network is used to cascade anisotropy to achieve accurate localization, thereby more densely combining low-level features and achieving better segmentation accuracy.

[0085] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A retinal vessel segmentation method based on the fusion of UNet and Transformer, characterized in that, A retinal vessel segmentation model based on the fusion of UNet and Transformer is constructed. This model includes an encoder, a decoder, and a fusion attention mechanism. The encoder comprises cascaded multi-stream convolutional layers, multiple pooling layers, and multiple residual modules. Each convolutional layer uses a residual module, and pooling layers are positioned between two convolutional layer units. The decoder comprises cascaded multiple improved extended convolution-based residual modules, multiple upsampling modules, and deconvolutional layers. Upsampling modules are positioned between two adjacent improved extended convolution-based residual units. The fusion attention mechanism uses the outputs of the pooling layers in the encoder and the outputs of adjacent stream pooling layers as low-level feature inputs and high-level feature inputs, respectively. The high-level feature input in the third layer, which is the feature map formed by the encoder pooling layer through the Transformer module, is the feature map of the third layer's fusion attention mechanism. According to the aforementioned retinal vessel segmentation method based on the fusion of UNet and Transformer, the retinal vessel segmentation model based on the fusion of UNet and Transformer features multi-stream cascaded multi-layer convolutional operations, with each stream and layer's output directly connected to subsequent layers in the same stream. Any layer of the convolutional unit uses a residual module to replace the standard UNet encoding block. The residual module effectively prevents gradient vanishing during training. The output of the multi-stream encoder is represented as: H(x)={h ι (x,K,θ)|h ι (x,K,θ k )=r(h ι (x,k-1,θ k-1 )),l=1,2,…,L} Where H(x) represents a set of network operations in different encoding paths; h ι (x,K,θ) represents the encoding stream of the l-th layer, K represents a continuous feature block, θ represents a learnable weight; r represents the residual operation on each feature block, including the pooling layer; Let represent the standard normalization function on the original input image x; L represents the total number of path flows, which is set to 4 in the experiment; according to the retinal vessel segmentation method based on the fusion of UNet and Transformer, the output of the fusion attention mechanism in the retinal vessel segmentation model based on the fusion of UNet and Transformer is represented as: F i =in i +in i ⊙α i ; Among them, F i This represents the output of the attention mechanism fusion at layer i, where the attention features α are represented as residuals. i and the original feature u i Unite together; u i =C([e i ,S(F i-1 )]); Among them, u i Represents the original features in the fusion attention mechanism of the i-th layer; C(Z) represents a convolutional kernel; S(Z) represents an upsampling operation; [] represents the corresponding Concat operation; e i and F i-1 These represent the results from the i-th layer encoder and the previous fusion module, respectively; s i =P avg (f ReLU (f BN (u i ))); Among them, s i P represents the intermediate features in the fusion attention mechanism of the i-th layer; avg (Z) represents the global average pooling operation; f ReLU (Z) represents the ReLU activation function operation; f BN (Z) indicates batch normalization operation; a i =σ(f C (f ReLU (f C (s i )))); Where, α i σ(Z) represents the attention feature in the fusion attention mechanism of the i-th layer; f represents the Sigmoid activation function operation; C (Z) represents a 1×1 convolution operation.

2. The retinal vessel segmentation method based on the fusion of UNet and Transformer as described in claim 1, characterized in that, The output of the improved extended convolution-based residual module in the i-th layer of the retinal vessel segmentation model based on the fusion of UNet and Transformer is represented as follows: y i =f BN+ReLU (f C3 (x i +DRF(x i )); Among them, y i f represents the output of the improved extended convolution-based residual module at layer i; BN+ReLU (Z) indicates that batch normalization and ReLU activation function operations are performed sequentially; f C3 (Z) represents a 3×3 convolution operation; x i Let represent the input of the improved extended convolution-based residual module at layer i; DRF(Z) represents the structure of the improved extended convolution-based residual module.

3. The retinal vessel segmentation method based on the fusion of UNet and Transformer as described in claim 1, characterized in that, The improved residual module structure DRF(Z) based on extended convolution is represented as follows: DRF(x i )=f BN+ReLU (f C (f BN+ReLU (f C (x i ))×f BN+ReLU ( fC3 (x i ))×IRCB(x i ))); Among them, f C Represents a 1×1 convolution operation; IRCB(Z) represents the extended convolution module structure.

4. The retinal vessel segmentation method based on the fusion of UNet and Transformer as described in claim 1, characterized in that, The extended convolutional module structure IRCB(Z) is represented as: IRCB(x i )=f BN+ReLU (f C (x i ))+f BN+ReLU (f C3+r3 (f BN+ReLU (f C (x i )))); Among them, f C3+r3 (Z) represents a dilated convolution with a dilation rate of 3.

5. The retinal vessel segmentation method based on the fusion of UNet and Transformer as described in claim 1, Its features are: in, The Transformer module obtains the feature map results from the downsampling of a multi-stream cascaded multi-layer convolutional neural network. It then performs fixed-size slicing on the feature map and embeds position vectors, as shown below: Where P represents the size of the slice; N represents the number of slices; Where z0 represents the result of embedding the position vector; E represents the result of the first feature map slice. pos Represents a position vector; The Transformer module consists of an L-layer multi-head attention mechanism and a multi-layer perceptron module. The multi-head attention mechanism employs two different partitioning methods: WMSA and SWMSA. The WMSA module is a self-attention module based on regular windows. The result of the first module is an 8×8 feature map, which is uniformly divided into four 4×4 windows starting from the top-left pixel, and then attention processing is performed. The SWMSA module is a self-attention module based on cross-windows. The result of the first module is an 8×8 feature map. Starting from the top-left pixel, it first performs a partitioning operation smaller than the window size, and then performs a window size partitioning operation. When the values ​​are 2 and 4 respectively, the partitioning results in four 2×2 windows, two 2×4 windows, two 4×2 windows, and one 4×4 window. The feature map is then shifted, attention processing is performed, and after setting self-attention, it is shifted again to restore its original state so that it can immediately follow the WMSA module. The output of two consecutive layers of the Transformer module is as follows: in, and z l These represent the feature outputs of the WMSA module and the MLP module in the l-th layer, respectively. and z l+1 These represent the feature outputs of the SWMSA module and the MLP module in layer l+1, respectively; the SWMSA module represents shifting the feature map and then shifting it again to restore it after setting self-attention; LN(Z) represents the LayerNorm normalization operation.

Images