An OCTA image retinal blood vessel segmentation method based on an attention mechanism

Abstract

The application provides an OCTA image retinal blood vessel segmentation method based on an attention mechanism, and comprises the following steps: a convolutional neural network segmentation model with an attention mechanism is built by a backbone feature extractor, a structure feature extractor, a reinforced feature extractor and a classifier, model training and parameter optimization, and rapid positioning and accurate segmentation of a retinal blood vessel structure based on OCTA. In the segmentation process, the method uses a deep separable convolution layer with a large convolution kernel to extract features, adopts a new downsampling method to reduce the loss of important information of blood vessels at the image frame, and introduces a spatial channel attention module (STAM) to better learn the spatial channel information of feature maps of different scales, capture complete blood vessel structures, and realize rapid and accurate segmentation of OCTA retinal blood vessel images.

Description

Technical Field

[0001] This invention relates to the field of fully automated semantic segmentation technology for medical images, specifically to a method for retinal vessel segmentation in OCTA images based on an attention mechanism. Background Technology

[0002] The eye is a vital organ for human perception, and various retinal diseases pose a significant threat to human health. Medical image segmentation is a crucial step in intelligent assisted diagnosis, enabling doctors to conduct image-guided medical interventions and more effective radiological diagnoses. Many fundus lesions occur around blood vessels, and retinal fundus images contain rich retinal vascular features. Analyzing the structural characteristics of retinal vessels, such as length, width, tortuosity, and bifurcation patterns, can reveal clinical features of fundus diseases, which is of great significance for the prevention and treatment of certain related diseases. In clinical applications, retinal vessel extraction technology, through a series of analyses of retinal vessels, can assist doctors in diagnosing whether patients have eye-related diseases. Traditional retinal vessel images are ordinary color fundus images, while Optical Coherence Tomography Angiography (OCTA) is a novel imaging modality built upon Optical Coherence Tomography (OCT). It is an emerging non-invasive imaging technology that can observe vascular information in different retinal layers. Therefore, OCTA is gradually becoming one of the important tools for observing fundus-related diseases.

[0003] In recent years, convolutional neural networks (CNNs) have been able to better perform various visual tasks by combining low-level features to form abstract deep features. Deep learning solves various problems by learning from data. Early retinal vessel segmentation algorithms were based on fully convolutional neural networks, aiming to better recover the information lost during convolutional downsampling. Later, they gradually developed into three categories: first, the U-shaped "encoder-decoder symmetry" structure design represented by U-net; second, the design that introduces dilated convolutions, represented by DeepLab; and finally, the design based on the self-attention structure in the Transformer encoder. Retinal vessel datasets generally have a small number of samples, and segmentation networks based on self-attention computation and the DeepLab series require a large amount of data for training to achieve good segmentation results. Therefore, only networks based on improvements to U-net can achieve good segmentation results even with a small number of samples in the dataset.

[0004] Although the improved U-net method has been applied to blood vessel segmentation in ordinary color fundus images and achieved good results, the inventors of this application have found through research that there are relatively few studies on extracting blood vessels from OCTA fundus images, and the current deep learning-based OCTA retinal blood vessel segmentation still has the following shortcomings: (1) The size of retinal blood vessels varies greatly, including very small capillaries with a minimum diameter of only 1-2 pixels wide. The ends of blood vessels are easily confused with the background, leading to misclassification of the ends of blood vessels as background; (2) Retinal blood vessels have a complex structure similar to a tree, such as bifurcations and intersections. The shape is irregular and the distribution is uneven, resulting in discontinuity and breakage of blood vessels during the segmentation process; (3) Some retinal blood vessel edges also have lesions such as microaneurysms and exudates, which affect the segmentation results; (4) During the process of extracting semantic information of different scales by the network, the change in the size of the feature map can lead to misclassification and missed detection of blood vessels at the image border; (5) The network depth and width are relatively deep, resulting in a slow segmentation speed. Summary of the Invention

[0005] To address the technical problems in existing deep learning-based OCTA retinal vessel segmentation, such as loss of important vascular information at image borders, slow segmentation speed, inaccurate segmentation of vascular terminal regions, and incomplete capture of vascular structures resulting in discontinuous and broken vessels in the segmentation results, this invention provides an attention-based OCTA image retinal vessel segmentation method. This method constructs a deep learning hybrid model, using depthwise separable convolutional layers with large kernels for feature extraction during the segmentation process, while employing a novel downsampling method to reduce the loss of important vascular information at image borders, and adding an attention module based on a self-attention structure to achieve fast and accurate segmentation of OCTA retinal vessel images.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A method for retinal vessel segmentation in OCTA images based on an attention mechanism includes the following steps:

[0008] S1. Building a convolutional neural network segmentation model with attention mechanism:

[0009] S11. The convolutional neural network segmentation model with attention mechanism consists of a backbone feature extractor, a structural feature extractor, an enhancement feature extractor, and a classifier at different scales. The backbone feature extractor at different scales is used to downsample the retinal vascular feature map based on OCTA fundus images four times, obtaining vascular feature maps of four different scales with input feature map size ratios of 1 / 4, 1 / 16, 1 / 64, and 1 / 256, respectively. Vascular detail features are extracted from the four different scale vascular feature maps, and they are sequentially connected in descending order of feature map size. The structural feature extractor is used to further refine the 1 / 4 and 1 / 16 vascular feature maps obtained by the backbone feature extractor. Before channel stitching of the corresponding scale feature maps in the strong feature extractor, the structural features of blood vessels are accurately extracted. The enhanced feature extractor is used to upsample the 1 / 256 high-order feature map obtained by the backbone feature extractor, gradually restoring the feature map size ratio to 1 / 64, 1 / 16, 1 / 4 and 1 / 1, and extracting blood vessel detail features for feature maps of different scales at the same time. The classifier is used to classify the pixel labels according to the blood vessel features extracted from feature maps of different scales. The segmentation network has three input channels and two output channels. The input image size and the output image size are both 512×512, which can achieve end-to-end semantic segmentation.

[0010] S12. The backbone feature extractor at different scales includes eight convolutional layer groups, four neighboring block merging layers, and four neck convolutional layers. A neighboring block merging layer is placed after every two convolutional layer groups. The four neck convolutional layers are located after the last neighboring block merging layer. Each convolutional layer group consists of a channel-wise convolutional layer with a kernel size of 7×7 and a stride of 1, and a pointwise convolutional layer with a kernel size of 1×1 and a stride of 1. The structural feature extractor includes two attention layers, each including a channel attention submodule and a spatial attention submodule. The enhancement feature extractor includes eight convolutional layer groups and four upsampling layers. Two convolutional layer groups are placed after each upsampling layer. Each convolutional layer group consists of a channel-wise convolutional layer with a kernel size of 7×7 and a stride of 1, and a pointwise convolutional layer with a kernel size of 1×1 and a stride of 1. The classifier consists of a class prediction layer and a... The composition of the regression layer, The regression layer transforms the predicted category scores into a probability distribution;

[0011] S2. Model Training and Parameter Optimization:

[0012] S21. The parameters of the convolutional neural network segmentation model with attention mechanism built in step S1 are initialized using the Xavier method.

[0013] S22. The retinal vessel dataset with retinal vessel segmentation labels after data preprocessing and online enhancement is divided into training set, validation set and test set in a ratio of 7:2:1. 10-fold cross-validation is used to pre-train the network segmentation model.

[0014] S23. Input the OCTA image of the same retinal vessel section into the network, and generate the retinal vessel segmentation result through the network forward calculation. The network forward calculation includes convolution operation, nonlinear excitation, probability value transformation and multi-head attention calculation.

[0015] S24. The classification cross-entropy loss function is used as the optimization objective of the segmentation network. The objective function is defined as follows:

[0016]

[0017] in, These are classification network parameters. It is a separator tag. It is the predicted probability. The number of pixels in the image. Number of pixel categories For regularization terms, As a regularization factor, The number of model parameters;

[0018] S25, Optimization Function:

[0019] The objective function is optimized using stochastic gradient descent, and the parameters of the retinal vessel segmentation network model are updated using error backpropagation. The specific optimization process is as follows:

[0020]

[0021]

[0022]

[0023] in, Indicates the number of iterations. This indicates gradient calculation. Corresponding to the objective function in step S23 , For use The loss function when the network parameters are... , and These are gradient, momentum, and momentum coefficient, respectively. It is the learning rate;

[0024] S3. Rapid localization and precise segmentation of retinal vascular structures based on OCTA:

[0025] S31. Perform sequential data preprocessing and online enhancement processing on the OCTA-based fundus images to obtain the processed images;

[0026] S32. The image after online enhancement is fed as a three-channel input into a feature extractor consisting of a trunk feature extractor, a structural feature extractor, and an enhancement feature extractor for feature extraction. The feature map of the reconstructed retinal blood vessel image is automatically located and output.

[0027] S33. Input the reconstructed retinal blood vessel image feature map into the classifier, and use a sliding window method to predict each pixel of the feature map one by one to generate two pixel label prediction score maps with the same size as the original image.

[0028] S34, Use The function transforms the predicted scores into a probability distribution;

[0029] S35. Take the index component with the highest probability of each pixel as the pixel category label to achieve rapid localization of blood vessel structure and obtain the binary image of retinal blood vessel segmentation result.

[0030] Furthermore, in the backbone feature extractor of different scales in step S12, the kernel size of each of the two convolutional layers is 1×1 and the stride is 1, and the number of convolutional kernels is 1024 and 512 respectively. Each neighboring block merging layer consists of a convolutional layer with a kernel size of 2×2 and a stride of 2 and a convolutional layer with a kernel size of 1×1 and a stride of 1.

[0031] Furthermore, in the structural feature extractor of step S12, the channel attention submodule includes a globally max pooling layer, a globally average pooling layer, a depthwise separable convolutional layer with a kernel size of 1×2, and two convolutional layers with a kernel size of 1×1, arranged in a sequential manner; the spatial attention submodule includes a convolutional layer with a kernel size of 1×1, a multi-head self-attention computation layer, a fully connected layer, and a convolutional layer with a kernel size of 1×1, arranged in a sequential manner.

[0032] Furthermore, in step S12, each upsampling layer in the enhanced feature extractor is a deconvolutional layer with a kernel size of 3×3 and a stride of 2.

[0033] Furthermore, in step S12, the class prediction layer of the classifier has a convolution kernel size of 1×1, a number of convolution kernels of 2, and a stride of 1.

[0034] Furthermore, in step S22, the data preprocessing involves uniformly adjusting the image pixel size to 512×512 before sending it into the network, and the online enhancement involves using online data augmentation techniques such as horizontal flipping, vertical flipping, cropping, and rotation of 45°, 90°, 135°, 180°, 225°, 270°, and 315° to increase the number of training data samples to 10 times the initial size.

[0035] Furthermore, the convolution operation in step S23 is: the output feature map corresponding to any convolution kernel in the network. The following formula is used for calculation:

[0036]

[0037] in, Represents a nonlinear excitation function. Indicates the input channel index number. Indicates the number of input channels. Indicates the first The first convolutional kernel Channel weight matrix, It is a convolution loss operation. Indicates the first One input channel image.

[0038] Furthermore, the nonlinear excitation in step S23 is: using a rectified linear unit. As a nonlinear activation function, it is used to convert the output feature map generated by the convolution kernel. Each value in the rectifier linear unit undergoes a non-linear transformation. The definition is as follows:

[0039]

[0040] in, Represents the rectified linear unit function. This indicates finding the maximum value. It is an input value.

[0041] Furthermore, in step S23, the probability value is converted to: using The function transforms the predicted scores output by the network into a probability distribution. The function is defined as follows:

[0042]

[0043] in, Is the pixel belonging to the first The probability of a class Is a certain pixel in the 1st... Predicted scores for each class Represents the number of categories.

[0044] Furthermore, during the multi-head attention calculation in step S23, the multi-head self-attention mechanism performs the calculation only once, and finally merges the results. The multi-head self-attention mechanism is implemented based on the scaled dot product attention operation, which is defined as follows:

[0045]

[0046] in, It is an attention function. The input matrix represents the scaled dot product attention. express Dimensions of a matrix and Matrix transformation through multiplication and through function after and The matrices are multiplied to obtain the final output of the self-attention function.

[0047] Compared with existing technologies, the attention-based OCTA image retinal vessel segmentation method provided by this invention has the following advantages:

[0048] 1. Each convolutional layer group in the backbone feature extractor and the enhanced feature extractor of this invention consists of a channel-wise convolutional layer and a point-wise convolutional layer, thereby enabling the use of depth-separable convolutional operations based on large convolutional kernels in the retinal vessel feature extraction stage. This allows for better extraction of detailed vascular features, reduces the number of pixels that misclassify the ends of vessels as background, reduces the computational load of convolutional operations, and improves segmentation speed.

[0049] 2. In the downsampling stage, the backbone feature extractor of this invention uses a neighboring block merging operation to reduce the feature map while capturing detailed information around the feature map border, reducing the loss of important information such as blood vessels at the border in feature maps of different scales, and improving segmentation accuracy.

[0050] 3. The structural feature extractor of this invention introduces a convolutional self-attention module, namely the STAM module, to better learn the spatial channel information of feature maps at different scales, capture the complete vascular structure, avoid discontinuities and breaks in retinal vascular segmentation results, and improve the segmentation accuracy. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the retinal vessel segmentation network structure for OCTA images based on the attention mechanism provided by the present invention.

[0052] Figure 2 This is a schematic diagram of the depth-separable convolution operation structure based on a large convolution kernel provided by the present invention.

[0053] Figure 3 This is a schematic diagram of the adjacent block merging operation structure provided by the present invention.

[0054] Figure 4a This is a schematic diagram of the overall structure of the STAM module provided by the present invention.

[0055] Figure 4b This is a schematic diagram of the channel attention submodule structure in the STAM module provided by the present invention.

[0056] Figure 4c This is a schematic diagram of the spatial attention submodule structure in the STAM module provided by the present invention.

[0057] Figure 5 This is a schematic diagram of the STAM module embedded skip connection structure provided by the present invention.

[0058] Figure 6 This is a schematic diagram of the original image and manually segmented real labels provided by the present invention. Detailed Implementation

[0059] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.

[0060] Please refer to Figure 1 As shown, this invention provides a method for retinal vessel segmentation in OCTA images based on an attention mechanism, comprising the following steps:

[0061] S1. Building a convolutional neural network segmentation model with attention mechanism:

[0062] S11. A convolutional neural network segmentation model with attention mechanism is used to generate the input OCTA fundus image. This model consists of backbone feature extractors at different scales (numbers 1-16 in Table 1), structural feature extractors (numbers 29-30 in Table 1), enhancement feature extractors (numbers 17-28 in Table 1), and a classifier (number 31 in Table 1). The backbone feature extractors at different scales extract different contextual features of retinal vessels from feature maps of multiple different scales. Here, they are used to perform four downsampling operations on the retinal vessel feature map based on the OCTA fundus image. The system obtains vascular feature maps at four different scales: 1 / 4, 1 / 16, 1 / 64, and 1 / 256 of the input feature map size. Vascular detail features are extracted from these four scale maps, and they are then sequentially connected in descending order of feature map size. The structural feature extractor is used to accurately extract retinal vascular structural features. Specifically, it is used to accurately extract vascular structural features from the 1 / 4 and 1 / 16 scale vascular feature maps obtained by the main feature extractor before channel splicing with the corresponding scale feature maps in the enhancement feature extractor. The enhancement feature extractor is used to realize the retinal... While restoring the membrane vascular feature map size from a 1 / 256 scale to a 1 / 1 scale, further vascular feature extraction at different scales is performed. This involves upsampling the 1 / 256 high-order feature map obtained from the main feature extractor, progressively restoring the feature map size to 1 / 64, 1 / 16, 1 / 4, and 1 / 1. Simultaneously, vascular detail features are extracted for feature maps at different scales. After each upsampling, a skip connection is made in the channel dimension with the feature maps obtained from the main feature extractor and the structural feature extractor. Therefore, the layers corresponding to indices 18, 21, 24, and 27 in Table 1 are input... The input channels are 1024, 512, 256, and 128, respectively. While extracting detailed features, the extractor is used to restore the 1 / 1 feature map for accurate prediction of pixel category. The classifier is used to classify pixel labels based on vascular features extracted from feature maps of different scales. The input of the segmentation network is three channels, representing the RGB color three channels of the OCTA fundus image, and the output is two channels, representing the probability that a pixel belongs to a vascular region and a non-vascular region (background), respectively. The input image size and the output image size are the same, both 512×512, which can achieve end-to-end semantic segmentation.

[0063] S12. The backbone feature extractor of different scales includes eight convolutional layer groups, four neighboring block merging layers, and four neck convolutional layers. A neighboring block merging layer is placed after every two convolutional layer groups, and the four neck convolutional layers are located after the last neighboring block merging layer. Each convolutional layer group consists of a channel-wise convolutional layer with a kernel size of 7×7 and a stride of 1, and a pointwise convolutional layer with a kernel size of 1×1 and a stride of 1. The first and last neck convolutional layers consist of a channel-wise convolutional layer with a kernel size of 7×7 and a stride of 1, and a pointwise convolutional layer with a kernel size of 1×1 and a stride of 1. The two middle neck convolutional layers have a kernel size of 1×1 and a stride of 1. The depthwise separable convolution operation based on large kernels is shown in the appendix. Figure 2 As shown, the specific steps are as follows: Figure 2 The number of channels before the operation is n. First, n 7×7 convolutional kernels perform channel-by-channel convolution on each channel, then concatenate them according to the channel dimension. Finally, m 1×1 convolutional kernels are used for point-by-point convolution to obtain the final output channel m. The specific values ​​of m and n are the number of convolutional kernels in each convolutional layer group in Table 1. Each neighboring block merging layer consists of a convolutional layer with a kernel size of 2×2 and a stride of 2, and a convolutional layer with a kernel size of 1×1 and a stride of 1. The merging operation of the neighboring block merging layer is shown in the appendix. Figure 3 shown, specific Figure 3 In the diagram, red, yellow, blue, and green represent four neighboring blocks, and the numbers 1, 2, 3, and 4 within each block are their corresponding index values. The structural feature extractor includes two attention layers (STAM modules). Each attention layer, or STAM module, includes a channel attention submodule and a spatial attention submodule, as shown in the attached diagram. Figure 4a As shown; the channel attention submodule includes a globally max pooling layer, a globally average pooling layer, a depthwise separable convolutional layer with a kernel size of 1×2, and two convolutional layers with a kernel size of 1×1, as shown in the attached figure. Figure 4b As shown; the spatial attention submodule includes a sequentially connected convolutional layer with a kernel size of 1×1, a multi-head self-attention computation layer, a fully connected layer, and another convolutional layer with a kernel size of 1×1, as shown in the attached figure. Figure 4c As shown in the attached diagram, the specific embedding of the structural feature extractor is illustrated. Figure 5 As shown; the enhanced feature extractor includes eight convolutional layer groups and four upsampling layers. Each upsampling layer is followed by two convolutional layer groups. Each convolutional layer group consists of a channel-wise convolutional layer with a kernel size of 7×7 and a stride of 1, and a pointwise convolutional layer with a kernel size of 1×1 and a stride of 1. Each upsampling layer is a deconvolutional layer with a kernel size of 3×3 and a stride of 2. The classifier consists of a class prediction layer and a... The composition of the regression layer, The regression layer converts the category prediction score into a probability distribution. The convolution kernel size of the category prediction layer is 1×1, the number of convolution kernels is 2, and the stride is 1. The specific parameters of the convolutional neural network segmentation model with attention mechanism are shown in Table 1 below.

[0064] Table 1. Parameter table of convolutional neural network segmentation model with attention mechanism

[0065] Serial Number Layer name kernel size Step length Number of convolution kernels Size ratio enter Output 1 Convolutional group 1_1 7×7，1×1 1，1 3,64 1 512×512×3 512×512×64 2 Convolutional group 1_2+ReLU 7×7，1×1 1，1 64,64 1 512×512×64 512×512×64 3 Neighboring block merging layer 1 2×2，1×1 2，1 64×4,64 1 / 4 512×512×64 256×256×64 4 Convolutional group 2_1 7×7，1×1 1，1 64,128 1 / 4 256×256×64 256×256×128 5 Convolutional group 2_2+ReLU 7×7，1×1 1，1 128128 1 / 4 256×256×128 256×256×128 6 Neighboring block merging layer 2 2×2，1×1 2，1 128×4,128 1 / 16 256×256×128 128×128×128 7 Convolutional group 3_1 7×7，1×1 1，1 128,256 1 / 16 128×128×128 128×128×256 8 Convolutional group 3_2+ReLU 7×7，1×1 1，1 256,256 1 / 16 128×128×256 128×128×256 9 Neighboring block merging layer 3 2×2，1×1 2，1 256×4,256 1 / 64 128×128×256 64×64×256 10 Convolutional group 4_1 7×7，1×1 1，1 256,512 1 / 64 64×64×256 64×64×512 11 Convolutional group 4_2+ReLU 7×7，1×1 1，1 512,512 1 / 64 64×64×512 64×64×512 12 Neighboring block merging layer 4 2×2，1×1 2，1 512×4,512 1 / 64 64×64×512 32×32×512 13 Neck convolution layer 1_1 7×7，1×1 1，1 512,512 1 / 64 32×32×512 32×32×512 14 Neck convolution layer 1_2 1×1 1 1024 1 / 256 32×32×512 32×32×1024 15 Neck convolution layer 1_3 1×1 1 512 1 / 256 32×32×1024 32×32×512 16 Neck convolution layer 1_4 7×7，1×1 1，1 512,1024 1 / 256 32×32×512 32×32×1024 17 Upsampling layer 1 3×3 2 512 1 / 64 32×32×1024 64×64×512 18 Convolutional group 5_1 7×7，1×1 1，1 1024,512 1 / 64 64×64×1024 64×64×512 19 Convolutional group 5_2+ReLU 7×7，1×1 1，1 512,512 1 / 64 64×64×512 64×64×512 20 Upsampling layer 2 3×3 2 256 1 / 16 64×64×512 128×128×256 21 Convolutional group 6_1 7×7，1×1 1，1 512,256 1 / 16 128×128×512 128×128×256 22 Convolutional group 6_2+ReLU 7×7，1×1 1，1 256,256 1 / 16 128×128×256 128×128×256 23 Upsampling layer 3 3×3 2 128 1 / 4 128×128×256 256×256×128 24 Convolutional Group 7_1 7×7，1×1 1，1 256,128 1 / 4 256×256×256 256×256×128 25 Convolutional group 7_2+ReLU 7×7，1×1 1，1 128,128 1 / 4 256×256×128 256×256×128 26 Upsampling layer 4 3×3 2 64 1 256×256×128 512×512×64 27 Convolutional group 8_1 7×7，1×1 1，1 128,64 1 512×512×128 512×512×64 28 Convolutional groups 8_2+ReLU 7×7，1×1 1，1 64,64 1 512×512×64 512×512×64 29 Attention layer 2 - - - - 128×128×128 256×256×128 30 Attention layer 1 - - - - 256×256×64 512×512×64 31 Classifier + SoftMax 1×1 1 2 1 512×512×64 512×512×2

[0066] In the table above, the padding for a kernel size of 7×7 is 3, the padding for a kernel size of 3×3 is 1, and the padding for a kernel size of 1×1 is 0.

[0067] S2. Model Training and Parameter Optimization:

[0068] S21. The parameters of the convolutional neural network segmentation model with attention mechanism built in step S1 are initialized using the Xavier method.

[0069] S22. The retinal vessel dataset with retinal vessel segmentation labels after data preprocessing and online enhancement is divided into training, validation, and test sets in a 7:2:1 ratio, and 10-fold cross-validation is used to pre-train the network segmentation model. As a specific implementation, the inventors of this application obtained data from 300 patients with pixel-level segmentation labels. The dataset used was the OCTA_6M subset of the 20th version of the retinal vessel dataset OCTA-500 published by Professor Chen Qiang and his team on IEEE-DataPort. The original images used were OCTA images of the entire eye, and the labels used were retinal vessel segmentation labels (manually segmented real labels). There were 300 original images and 300 manually segmented real labels. Examples of the original images and real labels are attached. Figure 6 As shown, the dataset is to be randomly divided into training, validation, and test sets in a 7:2:1 ratio, containing 210 training images, 60 validation images, and 30 test images, respectively. To standardize the input size of different networks in subsequent comparative experiments, the pixel size is uniformly adjusted to 512×512 before being fed into the network, as shown in Table 2 below:

[0070] Table 2 Dataset Distribution

[0071] Dataset Sample size Sample dimensionality training set 210 512 ×512×3 Validation set 60 512×512×3 test set 30 512×512×3

[0072] During training, the limited amount of training data is one of the important factors causing model underfitting. Since OCTA images are relatively scarce and the OCTA-500 retinal vessel segmentation dataset has a small amount of data, in order to reduce the impact of underfitting, online data augmentation techniques such as horizontal flipping, vertical flipping, cropping, and rotation of 45°, 90°, 135°, 180°, 225°, 270°, and 315° were used to increase the number of training data samples to 10 times the initial amount.

[0073] S23. Input OCTA images of the same retinal vessel section into the network, and generate retinal vessel segmentation results through network forward computation, wherein the network forward computation includes:

[0074] Convolution operation: The output feature map corresponding to any convolution kernel in the network. The following formula is used for calculation:

[0075]

[0076] in, Represents a nonlinear excitation function. Indicates the input channel index number. Indicates the number of input channels. Indicates the first The first convolutional kernel Channel weight matrix, It is a convolution loss operation. Indicates the first One input channel image.

[0077] Nonlinear excitation: using rectified linear units As a nonlinear excitation function , It is the activation function of the network, used to convert the output feature map generated by the convolution kernel. Each value in the rectifier linear unit undergoes a non-linear transformation. The definition is as follows:

[0078]

[0079] in, Represents the rectified linear unit function. This indicates finding the maximum value. It is an input value.

[0080] Probability value conversion: using The function transforms the predicted scores output by the network into a probability distribution. The function is defined as follows:

[0081]

[0082] in, Is the pixel belonging to the first The probability of a class Is a certain pixel in the 1st... Predicted scores for each class Represents the number of categories.

[0083] Multi-head self-attention calculation: The function is a process of weighted transformation of the output feature map. Compared with the ordinary self-attention mechanism, the multi-head self-attention mechanism can obtain features in multiple dimensions. During multi-head attention computation, the multi-head self-attention mechanism only performs the calculation once and finally merges the results. The multi-head self-attention mechanism is based on the scaled dot product attention operation, which is defined as follows:

[0084]

[0085] in, It is an attention function. The input matrix represents the scaled dot product attention. express Dimensions of a matrix and Matrix transformation through multiplication and through function after and The matrices are multiplied to obtain the final output of the self-attention function.

[0086] The multi-head self-attention mechanism involves performing h different linear mappings, applying scaling-dot-product attention operations in parallel to the different mapping results, concatenating the results, and then inputting them into the linear mapping layer to obtain the output of the multi-head self-attention mechanism. After h linear mappings, the model can learn relevant information in different representation subspaces.

[0087] S24. The classification cross-entropy loss function is used as the optimization objective of the segmentation network. The loss function is designed as follows:

[0088]

[0089] in, These are classification network parameters. It is a separator tag. It is the predicted probability. The number of pixels in the image. The number of pixel categories, in the experiment , ;

[0090] To prevent overfitting, an L2 regularization term is added to the loss function, resulting in the final objective function, which is defined as follows:

[0091]

[0092] in, For regularization terms, As a regularization factor, This represents the number of model parameters.

[0093] S25, Optimization Function:

[0094] The objective function is optimized using stochastic gradient descent, and the parameters of the retinal vessel segmentation network model are updated using error backpropagation. The specific optimization process is as follows:

[0095]

[0096]

[0097]

[0098] in, Indicates the number of iterations. This indicates gradient calculation. Corresponding to the objective function in step S23 , For use The loss function when the network parameters are... , and These are the gradient, momentum, and momentum coefficient, respectively. Let... ; This is the learning rate, initially set to 1e. -3 The number of iterations decreases by 1 / 10 every 50 iterations until it reaches 1e. -5 until.

[0099] S3. Rapid localization and precise segmentation of retinal vascular structures based on OCTA:

[0100] S31. Perform data preprocessing and online enhancement processing on the OCTA-based fundus images sequentially to obtain the processed images. For details, please refer to the data preprocessing and online enhancement techniques in step S22.

[0101] S32. The image after online enhancement is fed as a three-channel input into a feature extractor composed of a backbone feature extractor, a structural feature extractor, and an enhancement feature extractor of different scales for feature extraction. The feature map of the reconstructed retinal blood vessel image is automatically located and output.

[0102] S33. Input the reconstructed retinal blood vessel image feature map into the classifier, and use a sliding window method to predict each pixel of the feature map one by one to generate two pixel label prediction score maps with the same size as the original image.

[0103] S34, Use The function transforms the predicted scores into a probability distribution;

[0104] S35. Take the index component (0 or 1) where the maximum probability of each pixel is located as the pixel category label to achieve rapid localization of blood vessel structure and obtain a binary image of retinal blood vessel segmentation result (blood vessel region and non-blood vessel region).

[0105] Compared with existing technologies, the attention-based OCTA image retinal vessel segmentation method provided by this invention has the following advantages:

[0106] 1. Each convolutional layer group in the backbone feature extractor and the enhanced feature extractor of this invention consists of a channel-wise convolutional layer and a point-wise convolutional layer, thereby enabling the use of depth-separable convolutional operations based on large convolutional kernels in the retinal vessel feature extraction stage. This allows for better extraction of detailed vascular features, reduces the number of pixels that misclassify the ends of vessels as background, reduces the computational load of convolutional operations, and improves segmentation speed.

[0107] 2. In the downsampling stage, the backbone feature extractor of this invention uses a neighboring block merging operation to reduce the feature map while capturing detailed information around the feature map border, reducing the loss of important information such as blood vessels at the border in feature maps of different scales, and improving segmentation accuracy.

[0108] 3. The structural feature extractor of this invention introduces a convolutional self-attention module, namely the STAM module, to better learn the spatial channel information of feature maps at different scales, capture the complete vascular structure, avoid discontinuities and breaks in retinal vascular segmentation results, and improve the segmentation accuracy.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for retinal vessel segmentation in OCTA images based on an attention mechanism, characterized in that, Includes the following steps: S1. Building a convolutional neural network segmentation model with attention mechanism: S11. The convolutional neural network segmentation model with attention mechanism consists of a backbone feature extractor, a structural feature extractor, an enhancement feature extractor, and a classifier at different scales. The backbone feature extractor at different scales is used to downsample the retinal vascular feature map based on OCTA fundus images four times, obtaining vascular feature maps of four different scales with input feature map size ratios of 1 / 4, 1 / 16, 1 / 64, and 1 / 256, respectively. Vascular detail features are extracted from the four different scale vascular feature maps, and they are sequentially connected in descending order of feature map size. The structural feature extractor is used to further refine the 1 / 4 and 1 / 16 vascular feature maps obtained by the backbone feature extractor. Before channel stitching of the corresponding scale feature maps in the strong feature extractor, the structural features of blood vessels are accurately extracted. The enhanced feature extractor is used to upsample the 1 / 256 high-order feature map obtained by the backbone feature extractor, gradually restoring the feature map size ratio to 1 / 64, 1 / 16, 1 / 4 and 1 / 1, and extracting blood vessel detail features for feature maps of different scales at the same time. The classifier is used to classify the pixel labels according to the blood vessel features extracted from feature maps of different scales. The segmentation network has three input channels and two output channels. The input image size and the output image size are both 512×512, which can achieve end-to-end semantic segmentation. S12. The backbone feature extractor at different scales includes eight convolutional layer groups, four neighboring block merging layers, and four neck convolutional layers. A neighboring block merging layer is placed after every two convolutional layer groups. The four neck convolutional layers are located after the last neighboring block merging layer. Each convolutional layer group consists of a channel-wise convolutional layer with a kernel size of 7×7 and a stride of 1, and a pointwise convolutional layer with a kernel size of 1×1 and a stride of 1. The structural feature extractor includes two attention layers, each including a channel attention submodule and a spatial attention submodule. The enhancement feature extractor includes eight convolutional layer groups and four upsampling layers. Two convolutional layer groups are placed after each upsampling layer. Each convolutional layer group consists of a channel-wise convolutional layer with a kernel size of 7×7 and a stride of 1, and a pointwise convolutional layer with a kernel size of 1×1 and a stride of 1. The classifier consists of a class prediction layer and a... The composition of the regression layer, The regression layer transforms the predicted category scores into a probability distribution; S2. Model Training and Parameter Optimization: S21. The parameters of the convolutional neural network segmentation model with attention mechanism built in step S1 are initialized using the Xavier method. S22. The retinal vessel dataset with retinal vessel segmentation labels after data preprocessing and online enhancement is divided into training set, validation set and test set in a ratio of 7:2:

1. 10-fold cross-validation is used to pre-train the network segmentation model. S23. Input the OCTA image of the same retinal vessel section into the network, and generate the retinal vessel segmentation result through the network forward calculation. The network forward calculation includes convolution operation, nonlinear excitation, probability value transformation and multi-head attention calculation. S24. The classification cross-entropy loss function is used as the optimization objective of the segmentation network. The objective function is defined as follows: ， in, These are classification network parameters. It is a separator tag. It is the predicted probability. The number of pixels in the image. Number of pixel categories For regularization terms, As a regularization factor, The number of model parameters; S25, Optimization Function: The objective function is optimized using stochastic gradient descent, and the parameters of the retinal vessel segmentation network model are updated using error backpropagation. The specific optimization process is as follows: ， ， ， in, Indicates the number of iterations. This indicates gradient calculation. Corresponding to the objective function in step S24 , For use The loss function when the network parameters are... , and These are gradient, momentum, and momentum coefficient, respectively. It is the learning rate; S3. Rapid localization and precise segmentation of retinal vascular structures based on OCTA: S31. Perform sequential data preprocessing and online enhancement processing on the OCTA-based fundus images to obtain the processed images; S32. The image after online enhancement is fed as a three-channel input into a feature extractor consisting of a trunk feature extractor, a structural feature extractor, and an enhancement feature extractor for feature extraction. The feature map of the reconstructed retinal blood vessel image is automatically located and output. S33. Input the reconstructed retinal blood vessel image feature map into the classifier, and use a sliding window method to predict each pixel of the feature map one by one to generate two pixel label prediction score maps with the same size as the original image. S34, Use The function transforms the predicted scores into a probability distribution; S35. Take the index component with the highest probability of each pixel as the pixel category label to achieve rapid localization of blood vessel structure and obtain the binary image of retinal blood vessel segmentation result.

2. The OCTA image retinal vessel segmentation method based on attention mechanism according to claim 1, characterized in that, In the backbone feature extractor of different scales in step S12, each neighboring block merging layer consists of a convolutional layer with a kernel size of 2×2 and a stride of 2, and a convolutional layer with a kernel size of 1×1 and a stride of 1.

3. The OCTA image retinal vessel segmentation method based on attention mechanism according to claim 1, characterized in that, In the structural feature extractor of step S12, the channel attention submodule includes a globally max pooling layer, a globally average pooling layer, a depthwise separable convolutional layer with a kernel size of 1×2, and two convolutional layers with a kernel size of 1×1, all set in a sequential configuration; the spatial attention submodule includes a convolutional layer with a kernel size of 1×1, a multi-head self-attention computation layer, a fully connected layer, and a convolutional layer with a kernel size of 1×1, all set in a sequential configuration.

4. The OCTA image retinal vessel segmentation method based on attention mechanism according to claim 1, characterized in that, In step S12, each upsampling layer in the enhanced feature extractor is a deconvolutional layer with a kernel size of 3×3 and a stride of 2.

5. The OCTA image retinal vessel segmentation method based on attention mechanism according to claim 1, characterized in that, In step S12, the classifier's category prediction layer has a 1×1 convolutional kernel size, 2 convolutional kernels, and a stride of 1.

6. The OCTA image retinal vessel segmentation method based on attention mechanism according to claim 1, characterized in that, In step S22, data preprocessing involves uniformly adjusting the image pixel size to 512×512 before sending it into the network. Online enhancement involves using online data augmentation techniques such as horizontal flipping, vertical flipping, cropping, and rotation at 45°, 90°, 135°, 180°, 225°, 270°, and 315° to increase the number of training data samples to 10 times the initial size.

7. The OCTA image retinal vessel segmentation method based on attention mechanism according to claim 1, characterized in that, The convolution operation in step S23 is as follows: the output feature map corresponding to any convolution kernel in the network. The following formula is used for calculation: ， in, Represents a nonlinear excitation function. Indicates the input channel index number. Indicates the number of input channels. Indicates the first The first convolutional kernel Channel weight matrix, It is a convolution loss operation. Indicates the first One input channel image.

8. The OCTA image retinal vessel segmentation method based on attention mechanism according to claim 1, characterized in that, The nonlinear excitation in step S23 is: using a rectified linear unit. As a nonlinear activation function, it is used to convert the output feature map generated by the convolution kernel. Each value in the rectifier linear unit undergoes a non-linear transformation. The definition is as follows: ， in, Represents the rectified linear unit function. This indicates finding the maximum value. It is an input value.

9. The OCTA image retinal vessel segmentation method based on attention mechanism according to claim 1, characterized in that, In step S23, the probability value is converted to: using The function transforms the predicted scores output by the network into a probability distribution. The function is defined as follows: ， in, Is the pixel belonging to the first The probability of a class Is a certain pixel in the 1st... Predicted scores for each class Represents the number of categories.

10. The OCTA image retinal vessel segmentation method based on attention mechanism according to claim 1, characterized in that, In step S23, during the multi-head attention calculation, the multi-head self-attention mechanism performs the calculation only once, and finally merges the results. The multi-head self-attention mechanism is implemented based on the scaling dot product attention operation, which is defined as follows: ， in, It is an attention function. The input matrix represents the scaled dot product attention. express Dimensions of a matrix and Matrix transformation through multiplication and through function after and The matrices are multiplied to obtain the final output of the self-attention function.

Images