Fundus vessel segmentation method based on double skip connection and deep supervised learning

Abstract

The present invention discloses a fundus vascular segmentation method based on double skip connection and deep supervised learning, and specifically relates to the technical field of fundus vascular segmentation. A fundus data set, clinical fundus color photograph data and OCTA images are obtained, and an ophthalmologist marks the vascular information in the image on the image, and randomly selects image data to make a training set and a test set, and performs corresponding preprocessing operations on the image data to construct a deep neural network for multimodal vascular segmentation. A vascular segmentation network is constructed through a ResNetV2 sampling module, a void convolution sampling module, a Transformer encoder, a double skip connection, a decoder and deep supervision, which is used for training and prediction of vascular segmentation of fundus color photographs and OCTA images.

Description

Technical Field

[0001] The present invention relates to the technical field of fundus vascular segmentation, and in particular to a fundus vascular segmentation method based on double jump connection and deep supervised learning. Background Art

[0002] Fundus image blood vessels are the only part of the human body that can be directly observed, and changes in their characteristics can reflect the presence of a variety of diseases. For example, by observing changes in the diameter, thickness and other characteristics of blood vessels in fundus color photography (CFI, Color Fundus Image), early lesions of diseases such as diabetes and hypertension can be diagnosed; microvascular morphological changes shown by optical coherence tomography angiography (OCTA, Optical Coherence Tomography Angiography) are related to Alzheimer's disease and mild cognitive impairment. Therefore, based on CFI and OCTA two modality images, studying changes in vascular morphology has important diagnostic value, especially for the observation and detection of deeper vascular branches and microvascular parts, which is more helpful for the early diagnosis of diseases.

[0003] The exploration of vascular segmentation methods has always been a research focus, and its automatic segmentation can effectively assist doctors in analyzing and diagnosing diseases. For example, vascular segmentation can effectively assist in preoperative evaluation and planning, intraoperative navigation, and postoperative evaluation of fundus optic laser surgery, thereby achieving efficient, accurate, and minimally invasive surgical treatment. However, due to the complex vascular network structure of fundus images, uneven grayscale distribution, low contrast between blood vessels and background, and the influence of image noise, retinal vascular segmentation faces huge challenges. In addition, when doctors observe fundus vascular images with the naked eye, there are problems such as large workload, many subjective judgment factors, and poor reproducibility, which makes it impossible to obtain objective and accurate conclusions.

[0004] At present, the research on vascular segmentation algorithms based on fundus color photos is mainly based on deep learning-based vascular segmentation methods. Traditional fundus vascular segmentation algorithms mainly rely on manually selected features and segmentation through appropriate classifiers. Such methods use Gaussian filters, multi-scale matching filters, and unsupervised iterations to solve the problem of low contrast between blood vessels and background to a certain extent, but feature extraction often relies on experience and cannot fully obtain the rich information in the data. In addition, manual detection is difficult, time-consuming and laborious. It is difficult for each doctor to obtain completely consistent vascular detection results based on experience.

[0005] Deep learning methods improve segmentation accuracy by automatically extracting vascular features. Methods such as deep neural networks (DNN) and convolutional neural networks (CNN) effectively segment thin blood vessels and improve segmentation accuracy through a variety of data preprocessing and feature fusion techniques. In particular, U-Net and its improved models, such as Vessel-Net, DU-Net, and MSCNN-AM, combine residual modules, attention mechanisms, and multi-scale feature extraction to significantly improve segmentation performance.

[0006] Traditional OCTA vascular segmentation methods mainly include threshold-based, clustering-based, and model-based methods. Although threshold-based methods, such as the Ostu method and local adaptive threshold technology, are simple to implement, their segmentation effects are limited by low-contrast and high-noise OCTA images. Clustering-based methods, such as k-means clustering, group by pixel intensity, but are sensitive to noise and have unstable segmentation effects. Model-based methods, such as Markov-Gibbs random fields, use energy balance and spatial information, but still lack accuracy in microvascular segmentation.

[0007] The application of deep learning methods in OCTA vascular segmentation is gradually increasing, which can automatically extract features and improve segmentation accuracy. However, due to the dense blood vessels and low contrast in OCTA images, labeling is difficult, which limits the development of deep learning methods. For example, some methods transform the segmentation of OCTA vascular network into the segmentation problem of vascular missing area, some methods propose U-Net-based attention mechanism networks, and some methods propose image projection networks for 3D to 2D segmentation. These methods have improved the segmentation effect to a certain extent, but most of them rely on fully supervised learning, require a large amount of labeled data, have high labeling costs, and can only segment large blood vessels, and the microvascular segmentation effect is insufficient.

[0008] In summary, multimodal fundus vessel segmentation faces multiple challenges, including difficulties in data acquisition and annotation, limited algorithm processing capabilities for low-contrast and high-noise images, poor segmentation of thin vessels, insufficient model generalization, high computational cost, poor vascular connectivity, and the complexity of multimodal information fusion. In addition, the practicality and interpretability of the algorithm need to be verified in actual clinical applications, and the requirements of real-time processing need to be met. Effectively solving these problems is crucial to improving the practical application value of the vessel segmentation algorithm. Summary of the invention

[0009] The purpose of the present invention is to improve the accuracy and efficiency of segmentation and solve the problems of low segmentation accuracy and low contrast in traditional methods. Double jump connection can enhance feature extraction capability, and deep supervision helps to guide model learning at multiple levels. By combining these two technologies, it is expected that the complex vascular structures in fundus images, including large vessels and microvessels, can be better processed, thereby achieving more accurate vascular segmentation in multimodal fundus images (i.e., CFI and OCTA fundus images).

[0010] The technical scheme of the present invention is as follows: a fundus vascular segmentation method based on double jump connection and deep supervised learning, obtaining a fundus data set, clinical fundus color photo data and OCTA images, an ophthalmologist marking the vascular information in the image, and randomly selecting image data to make a training set and a test set, and performing corresponding preprocessing operations on the image data to construct a deep neural network for multimodal vascular segmentation, and constructing a vascular segmentation network through a ResNetV2 sampling module, a hole convolution sampling module, a Transformer encoder, a double jump connection, a decoder and deep supervision, which is used for training and prediction of vascular segmentation of fundus color photos and OCTA images, including the following steps:

[0011] a) Construct ResNetV2 sampling module: The ResNetV2 sampling module consists of a root convolution layer and 3 basic blocks; the root convolution layer consists of a 7×7Conv+Group-Normalization+ReLU+MaxPool; the basic block consists of multiple pre-activated bottleneck units (PreActBottleneck), and the number of pre-activated bottleneck units corresponding to each basic block is 3, 4, and 9 respectively; each pre-activated bottleneck unit structure includes 3 Group-Normalization+ReLU+Conv combination operations, and the 3 Convs are 1×1, 3×3 and 1×1 convolution layers respectively, followed by residual connection (Residual Connection), and downsampling operations are provided as needed; the input data passes through the root convolution layer and 3 basic blocks in sequence, and the intermediate features output by the root convolution layer and the 1st and 2nd basic blocks are retained as the first group of intermediate features, and the 3rd basic block outputs hidden features to the Transformer encoder;

[0012] b) Construct a dilated convolution sampling module: The basic structure of this module is similar to that of the ResNetV2 sampling module, consisting of a root convolution layer and two basic blocks. The difference is that the middle layer of the 3rd Conv is changed to a 3×3 dilated convolution with an expansion rate of 5, so that the receptive field of the convolution kernel is 11×11, thereby capturing a wider range of spatial information; the intermediate features output by the root convolution layer and the 1st and 2nd basic blocks are retained as the second set of intermediate features; its structural relationship with the ResNetV2 sampling module in the network is parallel, that is, the network input data is simultaneously input into the ResNetV2 sampling module and the dilated convolution sampling module to obtain two sets of downsampled intermediate features;

[0013] c) Constructing Transformer encoder: Transformer encoder is composed of 12 identical layers stacked together, each of which contains two sub-layers: Self-Attention Layer and Feedforward Neural Network Layer; Self-Attention Layer uses Multi-Head Self-Attention Mechanism, and simultaneously calculates Attention scores for multiple positions i in the input sequence to obtain weighted sum outputs, thereby modeling the dependency between different positions in the input; Feedforward Neural Network Layer: This layer uses a multi-layer perceptron to perform more complex nonlinear transformations on the features of each position, thereby enhancing the representation of the features. Each sub-layer uses residual connections and layer normalization;

[0014] d) Construct the decoder and double jump connection: The decoder consists of 4 upsampling blocks. Each upsampling block performs bilinear interpolation on the input features of the previous layer, doubles the size of the input feature map, and then concatenates the expanded features with the features of the corresponding sizes in the two sets of downsampled intermediate features obtained by the double downsampling in step a) and step b) to achieve feature superposition and fusion in the channel dimension. The last upsampling block does not perform double jump connection; then perform two 3×3Conv+BatchNormalization+ReLU to output two feature maps, one for further upsampling and the other for signal input for deep supervision. Finally, a complete decoder module is formed;

[0015] e) Construct deep supervision: First, the features of the 4 upsampled outputs are respectively processed by 1×1Conv to adjust the number of channels, and then the feature map size is adjusted to 512×512 through bilinear interpolation. The 4 feature maps after processing are sent to the segmentation head, and the corresponding cross entropy loss is calculated to obtain 3 auxiliary losses and 1 deep main loss. The weighted sum of the 4 losses is obtained to obtain the overall loss, and the weight ratios are 0.1, 0.3, 0.5, and 1.0 respectively. Finally, the overall loss function is used for back propagation to update all network parameters;

[0016] The multimodal fundus image data is input into the neural network for training. The overall training loss value is obtained through forward propagation layer by layer calculation and deep supervision, and then the network weight parameters are updated through back propagation. After a certain round of network training, the training is stopped until the training loss and various evaluation indicators are stable. During the training process, the optimal network model is saved in real time according to the evaluation indicators.

[0017] Finally, the test data is input into the segmentation network with the optimal weight to obtain the prediction result.

[0018] Preferably, the preprocessing operation specifically includes: reducing the image resolution, random horizontal rotation, random vertical rotation, random translation and scaling rotation, random Gaussian noise, random Gaussian blur and random height contrast of the fundus image to prevent overfitting of the training and improve the generalization of the model.

[0019] Preferably, the downsampling of the ResNetV2 sampling module in step a) has four stages: the step size of the 7×7Conv in the root convolutional layer is 2, the padding is 3, and the output channel is 64; the default step size of the 1×1Conv in the pre-activated bottleneck unit 3 times Conv is 1, the padding is 0, and the step size of the 3×3Conv is 1, and the padding is 1; the first basic block contains 3 pre-activated bottleneck units, and the number of output channels is 256; the second and third basic blocks adjust the 3×3Conv step size of the first pre-activated bottleneck unit to 2, and the number of output channels of the downsampling operation is 512 and 1024, respectively.

[0020] Preferably, in step b), the atrous convolution sampling module further obtains multi-scale feature information of the fundus image with a large receptive field, which is similar to the structure of the ResNetV2 sampling module. The 3×3Conv expansion rate of each pre-activated bottleneck unit in basic blocks 1 and 2 is adjusted to 5 to obtain a large receptive field of 11×11, capturing a wider range of contextual information without increasing the computational complexity.

[0021] Preferably, in step c), the Transformer encoder module converts the hidden features with 1024 channels obtained in the upper layer into a serialized vector representation through linear projection, and the Transformer encoder effectively processes the global and local information of the features through a multi-head self-attention mechanism and a multi-layer perceptron; the single-head self-attention mechanism is calculated as follows:

[0022] The input sequence length is n and the dimension is d. The input representation can be written as a matrix X, where the shape of X is n×d;

[0023] Perform three different linear transformations on the input X to generate query Q (Query), K key (Key) and V value (Value) matrices;

[0024] Q=XW Q ;

[0025] K=XW K ;

[0026] V=XW V ;

[0027] Among them, W Q , W K and W V is the weight matrix to be learned, with shapes of d×d k ,d×d k ,d×d v ;

[0028] Calculate the dot product of query and key to get the attention score matrix;

[0029]

[0030] The multi-head self-attention mechanism repeats the above process h times, using a different weight set W each time i Q , W i K , W i V , each group is called a head to enhance the expressiveness of the model. The specific calculation steps are as follows:

[0031]

[0032] MultiHead(X)=Concat(head1,...head h )W O ;

[0033] W O is the final weight matrix of the output;

[0034] The final Transformer encoder output feature channel number is 512.

[0035] Preferably, in step d), the decoder has a total of four upsamplings. The first three upsamplings first perform a double jump connection, that is, the upper-layer input features and the corresponding-size intermediate features obtained by the ResNetV2 sampling module and the hole convolution sampling module perform a concatenate operation in the channel dimension, and then perform two 3×3Convs with a step size of 1 and a padding of 1. The fourth upsampling does not perform a double jump operation, and the final output feature channel numbers are 256, 128, 64, and 16, respectively.

[0036] Preferably, in step e), the deep supervision first performs 1×1Conv with a step size of 1 and a padding of 0 on the intermediate features obtained by upsampling each layer, adjusts the number of channels to 16, and then performs bilinear interpolation to adjust the feature map size to 512×512, and then transmits it to a segmentation head of 3×3Conv+softmax with a step size of 1 and a padding of 1 to obtain segmentation maps of different levels and calculate the cross entropy loss with the gold standard to obtain 3 auxiliary losses and the main loss; the total loss function is:

[0037] Total_Loss=Main_Loss+λ1·Auxiliary_Loss1+λ2·Auxiliary_Loss2+λ3·Auxiliary_Loss1;

[0038] where λ i , i=1, 2, 3 are weighted coefficients, i is the upsampling level, λ1 is 0.1, λ2 is 0.3, λ3 is 0.5; Auxiliary_Loss i , i=1, 2, 3 are auxiliary losses, i is the upsampling level, Auxiliary_Loss is the auxiliary loss; finally, back propagation is performed with the overall loss function to update all network parameters;

[0039] This ensures that the intermediate layers can better learn features that are helpful for the final task and improve the convergence speed and performance of the entire network.

[0040] The present invention has the following advantages:

[0041] The present invention introduces a dilated convolution sampling module, expands the receptive field of the convolution kernel, combines double skip connections with deep supervision technology, and aims to improve the accuracy and efficiency of blood vessel segmentation in multimodal fundus images, thereby solving the problems of low segmentation accuracy and low contrast in traditional methods.

[0042] 1. For CFI modal samples

[0043] The present invention uses a strategy combining double jump connection and deep supervision to effectively extract and fuse multi-level and multi-scale feature information. Double jump connection enhances the feature extraction capability by fusing features of different scales, making the model more accurate when processing complex vascular structures. Deep supervision guides the network middle layer to learn more useful features through multi-level supervision signals during the training process, thereby improving the overall segmentation accuracy.

[0044] 2. For OCTA modality samples

[0045] Microvascular morphological changes in OCTA are related to Alzheimer's disease and mild cognitive impairment. The present invention linearly projects hidden features into serialized vector representations through the Transformer encoder module, and then uses the multi-head self-attention mechanism and multi-layer perceptron to process the global and local information of the features, thereby more comprehensively understanding and processing the complex vascular information in OCTA images. In this way, the model can more effectively identify and segment the macrovascular and microvascular structures in the image.

[0046] The present invention has good multimodal image processing capabilities. The network structure of the present invention can more comprehensively process multimodal fundus images with complex vascular information, including CFI and OCTA images. The model can more accurately segment the vascular structure in the fundus image to meet the diverse needs in clinical and research. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A framework diagram of a multimodal fundus vessel segmentation method based on double jump connection and deep supervision provided by the present invention;

[0048] Figure 2 This is the result diagram of blood vessel segmentation of the multimodal fundus image provided by the present invention. DETAILED DESCRIPTION

[0049] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Embodiment 1: The fundus vascular segmentation method based on double jump connection and deep supervised learning provided by the present invention first pre-processes the acquired public fundus data set and clinical fundus color photos and OCTA images, and then performs grayscale, normalization, contrast-limited adaptive histogram equalization, and gamma correction to highlight the image vascular information. After that, the training set data is input into the constructed neural network for training to obtain the optimal vascular segmentation weight. Finally, the test data is input into the segmentation network that calls the optimal vascular segmentation weight to obtain the final segmentation result, such as Figure 1 shown.

[0051] The fundus vessel segmentation method based on double skip connection and deep supervised learning mainly includes the following steps:

[0052] Step 1: Data preparation. The color photos are 40 randomly selected from the DRIVE public database, of which 33 do not show any signs of diabetic retinopathy and 7 show signs of mild early diabetic retinopathy. The resolution of each image is 584×565 pixels, 3-channel color image. The training set has 30 images and the test set has 10 images.

[0053] The OCTA image takes the ROSE-1 dataset, which is open sourced by the Intelligent Medical Imaging Team of Ningbo Institute of Materials, as an example. ROSE-1 includes 117 OCTA images from 39 subjects (including 26 Alzheimer's disease (AD) patients and 13 healthy controls). The resolution of each image is 304×304 pixels, a single-channel image. There are 90 images in the training set and 27 images in the test set.

[0054] In order to prevent overfitting caused by less data and to further improve the generalization performance of the model, random horizontal rotation, random vertical rotation, random translation and scaling rotation, random Gaussian noise, random Gaussian blur and random height contrast are used for data enhancement.

[0055] Step 2: Model building. This method integrates key technologies such as ResNetV2 sampling module, hole convolution sampling module, Transformer encoder, double jump connection, decoder and deep supervision to improve the accuracy and efficiency of fundus blood vessel segmentation.

[0056] First, the ResNetV2 sampling module and the dilated convolution sampling module obtain feature information of different scales through two-way downsampling operations, and use ordinary convolution and dilated convolution for feature extraction respectively. The number of channels is gradually increased in four stages, and finally 64, 256, 512 and 1024 channels of feature output are generated, and the features of the first three stages are saved for subsequent double jump connections. Secondly, the Transformer encoder converts the upper layer hidden features into serialized vector representations through linear projection, and uses multi-head self-attention mechanism and multi-layer perceptron to process global and local information. The double jump connection fuses the features of different scales obtained in the encoding stage and transmits these features to the decoding layer of the corresponding scale. Afterwards, in the decoding stage, the upsampled feature map is fused with the feature map output by the two-way downsampling and input into the decoder module to accurately capture the spatial details and enhance the recognition ability of vascular information. This method can not only selectively aggregate features at different levels and enhance the extraction of segmentation target information; it can also effectively suppress the feature redundancy problem caused by directly fusing low-level features and high-level features. Then, deep supervision adds supervision signals to the four levels of the decoder, performs weighted supervision on the output features of the four-layer decoder, helps the model better learn features that are helpful for the task of segmenting blood vessels, and improves the convergence speed and overall performance of the network. Finally, the feature information processed by deep supervision is transmitted to the segmentation head composed of a two-dimensional convolutional layer and an upsampling layer, and the feature map of the model is converted into the final segmentation result output.

[0057] Step three: model training. The present invention is carried out on a workstation equipped with an NVIDIA-RTX4080 GPU. The programming language is Python, the PyTorch deep learning framework is used, and Adam is used as the optimizer for model training. The batch size is 4, the weight decay is 0.0001, and the basic learning rate is 0.005. The fundus color photographs and OCTA images of the training data are input into the segmentation network for training. After 300 rounds of training, the model has been basically fitted. The model parameters with the best training effect are saved for each iteration during training. In order to verify the performance of the model, it is tested on the test set. The fundus color photographs and OCTA images of the test set are also pre-processed and sent to the optimal segmentation weight network to output the predicted segmentation results. The segmentation results of the fundus color photographs and OCTA images are as follows: Figure 2 shown.

[0058] Step 4: Segmentation results. Input the test data into the segmentation network with the optimal weight to obtain the prediction result. Calculate the segmentation result and the gold standard image to obtain the specific values ​​of the segmentation effect evaluation indicators F1, ACC, SE, and SP. Among them, F1 is the F1-score, which is the harmonic mean of precision and recall; ACC is the accuracy, which indicates the proportion of correctly classified pixels to the total pixels; SE is the sensitivity, which indicates the proportion of pixels correctly identified as blood vessels to all pixels that are actually blood vessels; SP is specificity, which indicates the proportion of pixels correctly identified as background to all pixels that are actually background. The specific calculation formula is as follows:

[0059]

[0060]

[0061] Among them, TP (true positive) represents pixels correctly predicted as blood vessels, TN (true negative) represents pixels correctly predicted as non-vessels, FP (false positive) represents pixels incorrectly predicted as blood vessels, and FN (false negative) represents pixels incorrectly predicted as non-vessels.

[0062] The effectiveness and performance of the algorithm are verified by comparing it with other algorithms, such as U-Net, R2U-Net, Attention U-Net, Ladder-Net, and TransUNet. The specific results are shown in Table 1 (the maximum value is shown in Table 1). From the data in the table, it can be seen that the improved method has advantages in indicators such as ACC and SE, and has certain competitiveness compared with existing algorithms.

[0063] Table 1 Performance indicators of different algorithms on the DRIVE dataset

[0064]

[0065] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A fundus vascular segmentation method based on double jump connection and deep supervised learning is developed. The fundus data set, clinical fundus color photo data and OCTA images are obtained. Ophthalmologists mark the vascular information in the images, randomly select image data to make training sets and test sets, and perform corresponding preprocessing operations on the image data. The method is characterized by: Construct a deep neural network for multimodal vascular segmentation. Use the ResNetV2 sampling module, the hole convolution sampling module, the Transformer encoder, the double jump connection, the decoder, and deep supervision to build a vascular segmentation network for training and prediction of vascular segmentation of fundus color photos and OCTA images. The steps include: a) Construct ResNetV2 sampling module: The ResNetV2 sampling module consists of a root convolution layer and 3 basic blocks; the root convolution layer consists of a 7×7Conv+Group-Normalization+ReLU+MaxPool; the basic block consists of multiple pre-activated bottleneck units, and the number of pre-activated bottleneck units corresponding to each basic block is 3, 4, and 9 respectively; each pre-activated bottleneck unit structure includes 3 Group-Normalization+ReLU+Conv combination operations, and the 3 Convs are 1×1, 3×3 and 1×1 convolution layers respectively, followed by residual connections, and downsampling operations are provided as needed; the input data passes through the root convolution layer and 3 basic blocks in sequence, retaining the intermediate features output by the root convolution layer and the 1st and 2nd basic blocks as the first set of intermediate features, and the 3rd basic block outputs hidden features to the Transformer encoder; b) Construct a dilated convolution sampling module: The basic structure of this module is similar to that of the ResNetV2 sampling module, consisting of a root convolution layer and two basic blocks. The difference is that the middle layer of the 3rd Conv is changed to a 3×3 dilated convolution with an expansion rate of 5, so that the receptive field of the convolution kernel is 11×11; the intermediate features output by the root convolution layer and the 1st and 2nd basic blocks are retained as the second set of intermediate features; its structural relationship with the ResNetV2 sampling module in the network is parallel, that is, the network input data is simultaneously input into the ResNetV2 sampling module and the dilated convolution sampling module to obtain two sets of downsampled intermediate features; c) Constructing the Transformer encoder: The Transformer encoder is composed of 12 identical layers stacked together, each of which contains two sublayers: a self-attention layer and a feedforward neural network layer. The self-attention layer uses a multi-head self-attention mechanism to simultaneously calculate the attention scores for multiple positions i in the input sequence to obtain the weighted sum output, thereby modeling the dependency between different positions in the input. Feedforward neural network layer: This layer uses a multi-layer perceptron to perform a more complex nonlinear transformation on the features of each position, and each sublayer uses residual connections and layer normalization. d) Constructing a decoder and double jump connections: The decoder consists of 4 upsampling blocks. Each upsampling block performs bilinear interpolation on the input features of the previous layer, doubles the size of the input feature map, and then concatenates the expanded features with the features of the corresponding sizes in the two sets of downsampled intermediate features obtained by the two-way downsampling in step a) and step b) to achieve feature superposition and fusion in the channel dimension. The last upsampling block does not perform double jump connections. After that, two 3×3Conv+BatchNormalization+ReLU blocks are performed to output two feature maps, one for further upsampling and the other for signal input for deep supervision, finally forming a complete decoder module. e) Construct deep supervision: First, the features of the 4 upsampled outputs are respectively processed by 1×1Conv to adjust the number of channels, and then the feature map size is adjusted to 512×512 through bilinear interpolation. The 4 feature maps after processing are sent to the segmentation head, and the corresponding cross entropy loss is calculated to obtain 3 auxiliary losses and 1 deep main loss. The weighted sum of the 4 losses is obtained to obtain the overall loss, and the weight ratios are 0.1, 0.3, 0.5, and 1.0 respectively. Finally, the overall loss function is used for back propagation to update all network parameters; The multimodal fundus image data is input into the neural network for training. The overall training loss value is obtained through forward propagation layer by layer calculation and deep supervision, and then the network weight parameters are updated through back propagation. After a certain round of network training, the training is stopped until the training loss and various evaluation indicators are stable. During the training process, the optimal network model is saved in real time according to the evaluation indicators. Finally, the test data is input into the segmentation network with the optimal weight to obtain the prediction result.

2. The method for segmenting fundus blood vessels based on double skip connection and deep supervised learning according to claim 1, characterized in that: The preprocessing operation specifically includes: reducing the image resolution, performing random horizontal rotation, random vertical rotation, random translation, scaling and rotation, random Gaussian noise, random Gaussian blur and random height contrast on the fundus image.

3. The method for retinal blood vessel segmentation based on double skip connection and deep supervised learning according to claim 1, characterized in that: In step a), the downsampling of the ResNetV2 sampling module has four stages: the step size of the 7×7Conv in the root convolutional layer is 2, the padding is 3, and the output channels are 64; in the three Convs of the pre-activated bottleneck unit, the default step size of the 1×1Conv is 1, the padding is 0, and the step size of the 3×3Conv is 1, and the padding is 1; the first basic block contains 3 pre-activated bottleneck units, and the number of output channels is 256; the second and third basic blocks adjust the step size of the 3×3Conv of the first pre-activated bottleneck unit to 2, and the number of output channels of the downsampling operation is 512 and 1024 respectively.

4. The method for segmenting fundus blood vessels based on double skip connection and deep supervised learning according to claim 1, characterized in that: In step b), the structure of the atrous convolution sampling module is similar to that of the ResNetV2 sampling module. The 3×3Conv expansion rate of each pre-activated bottleneck unit in basic blocks 1 and 2 is adjusted to 5 to obtain a large receptive field of 11×11.

5. The method for segmenting fundus blood vessels based on double skip connection and deep supervised learning according to claim 1, characterized in that: In step c), the Transformer encoder module converts the hidden features with 1024 channels obtained from the upper layer into a serialized vector representation through linear projection. The Transformer encoder effectively processes the global and local information of the features through the multi-head self-attention mechanism and the multi-layer perceptron; the single-head self-attention mechanism is calculated as follows: The input sequence length is n and the dimension is d. The input representation can be written as a matrix X, where the shape of X is n×d; Perform three different sets of linear transformations on the input X to generate query Q, K key and V value matrices; Q=XW Q ; K=XW K ; V=XW V ; Among them, W Q , W K and W V is the weight matrix to be learned, with shapes of d×d k ,d×d k ,d×d v ; Calculate the dot product of query and key to get the attention score matrix; The multi-head self-attention mechanism repeats the above process h times, using a different weight set W each time i Q , W i K , W i V , each group is called a head, and the specific calculation steps are as follows: MultiHead(X)=Concat(head1,……head h )W O ; W O is the final weight matrix of the output; The final Transformer encoder output feature channel number is 512.

6. The method for segmenting fundus blood vessels based on double skip connection and deep supervised learning according to claim 1, characterized in that: In step d), the decoder has four upsamplings. The first three upsamplings are connected by double jump connection, that is, the upper layer input features and the corresponding size intermediate features obtained by the ResNetV2 sampling module and the hole convolution sampling module are concatenated in the channel dimension, and then two 3×3Convs are performed with a step size of 1 and a padding of 1. The fourth upsampling does not perform a double jump operation, and the number of output feature channels is 256, 128, 64, and 16 respectively.

7. The method for segmenting fundus blood vessels based on double skip connection and deep supervised learning according to claim 1, characterized in that: In step e), the deep supervision first performs 1×1Conv with a step size of 1 and padding of 0 on the intermediate features obtained by upsampling each layer, adjusts the number of channels to 16, and then performs bilinear interpolation to adjust the feature map size to 512×512. Then, it is sent to the segmentation head of 3×3Conv+softmax with a step size of 1 and padding of 1 to obtain segmentation maps of different levels and calculate the cross entropy loss with the gold standard to obtain 3 auxiliary losses and the main loss; the total loss function is: Total_Loss=Main_Loss+λ1·Auxiliary_Loss1+ λ2·Auxiliary_Loss2+λ3·Auxiliary_Loss1; where λ i , i=1, 2, 3 are weighted coefficients, i is the upsampling level, λ1 is 0.1, λ2 is 0.3, λ3 is 0.5; Auxiliary_Loss i , i=1, 2, 3 are auxiliary losses, i is the upsampling level, Auxiliary_Loss is the auxiliary loss; finally, back propagation is performed with the overall loss function to update all network parameters.

Images