OCTA retinal vessel segmentation method based on Gabor modulation coding improved U-Net

Abstract

The present invention discloses an OCTA retinal blood vessel segmentation method based on Gabor modulation coding improved U-Net, comprising: using Gabor filters to model primary visual cortex simple cells, forming a Gabor convolution path for integrating edge features, and assisting the convolution path to improve feature extraction. At the same Gabor scale, firstly, Gabor convolution kernels in four directions are used to extract contour information from a feature map, so as to obtain texture features at different levels to compensate for the edge features of a blood vessel connection and a fovea region; then the above features are integrated, i.e., stacked according to dimensions; then the integrated features are repeatedly extracted, i.e., 1×1 convolution is used to condense the edge features; finally, the same common convolution coding path is realized to be interconnected, i.e., the interconnection between feature maps of different scales is enhanced through a maximum pooling operation, the edge features are supplemented, and the difference between blood vessels and background is amplified.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to an OCTA retinal blood vessel segmentation method based on Gabor modulation coding improved U-Net. Background Art

[0002] Retinal blood vessels are the main anatomical structures that can be detected in fundus images, and their morphological changes are signals of many fundus and cardiovascular diseases. The microvascular morphological changes shown by optical coherence tomography angiography (OCTA) retinal vascular imaging can assist doctors in studying neurodegenerative diseases such as Alzheimer's disease and mild cognitive impairment. The OCTA-500 dataset contains multi-layer features of the inner limiting membrane, outer plexiform layer, and Bruch's membrane of the retinal layer. For the OCTA-500 dataset, existing methods use the proposed method, which consists of an encoder based on a dynamic Transformer and a decoder based on a lightweight convolution structure. A dynamic token aggregation transformer is proposed to reduce the computational cost of the Transformer, and auxiliary convolution branches are used to assist in the efficient extraction of vascular features. In addition, some scholars have proposed a dual-channel network model containing Transformer branches and convolution branches, which uses Transformer branches to extract non-local features of vascular structures, capture and supplement the global context information missing from convolutional neural networks, and aggregate segmented blood vessels on the same vascular tree.

[0003] The above algorithms all use Transformer encoders to supplement the global contextual information that is difficult to capture by convolutional networks, but this also makes the network parameters large and the training cost high. In addition to the large number of network parameters, the existing OCTA retinal vessel segmentation algorithms still have the following problems: the contour information of retinal vessels can improve the segmentation continuity of vessel edges, but most algorithms cannot make good use of edge information using convolution, resulting in unclear boundaries between vessels and vessels, and between vessels and background in the final segmentation results; and the use of convolution with a relatively single type of receptive field for feature extraction cannot obtain deeper contextual information, resulting in the segmentation results of most existing algorithms having problems with broken or missing vessels. Summary of the invention

[0004] The purpose of the present invention is to provide an OCTA retinal vessel segmentation method based on Gabor modulation coding improved U-Net. According to the multi-directional and multi-scale characteristics of Gabor convolution and the regulating effect of the distance between the central pixel and the surrounding pixels, an OCTA retinal vessel segmentation algorithm based on Gabor modulation coding and center-peripheral convolution improved U-Net is proposed. Aiming at the problem of unclear edge of vascular segmentation, the algorithm constructs a Gabor coding pathway to assist in modulating a pure convolution pathway, uses four directions and Gabor convolutions with increasing scales layer by layer to specifically extract vascular contour information, and fixes the Gabor convolution kernel weights to prevent adding extra burden to network training. Aiming at the problem of a single receptive field and a large number of vascular breakpoints, the algorithm uses center aggregation and center diffusion convolutions to increase and decrease the compactness of peripheral elements and central elements in the feature map respectively, and uses the upper and lower edges of the center and periphery to extract the vascular contour information. The following information provides a priori extraction tendency for convolution and enriches the receptive field form of convolution. The balanced accuracy (Bacc), Dice coefficient, sensitivity (sen), specificity (sp), and Jaccard index of the algorithm on the two public data sets of OCTA-3MM and OCTA-6MM reached 0.9644, 0.9019, 0.9390, 0.9898, 0.8219 and 0.9451, 0.8758, 0.9073, 0.9829, 0.7803 respectively. Compared with the current algorithm, the algorithm effectively extracts the edge contour information of the blood vessels, reduces the blood vessel breakpoints, takes into account the accuracy and completeness of the blood vessel segmentation, and solves the problems in the above-mentioned background technology. The specific technical solution is as follows: An OCTA retinal blood vessel segmentation method based on Gabor modulation coding to improve U-Net includes the following steps:

[0005] Step 1: Obtain the OCTA retinal vessel segmentation datasets OCTA-3MM and OCTA-6MM, and divide the datasets into training set, validation set, and test set;

[0006] Preferably, the datasets OCTA-3MM and OCTA-6MM respectively represent datasets consisting of OCTA projection images with a field of view of 6 mm×6 mm and 3 mm×3 mm, the OCTA-6MM dataset contains 300 OCTA retinal image data with a resolution of 400×400, and the OCTA-3MM dataset contains 200 OCTA retinal image data with a resolution of 304×304;

[0007] The specific division method of the data set into a training set, a validation set and a test set is: the original data set is randomly arranged in shuffle order and divided into a training set, a validation set and a test set in a ratio of 3:1:1.

[0008] Step 2: A model including an encoder and a decoder is constructed based on the U-Net encoder-decoder structure, wherein the encoder and the decoder contain multiple skip connections. The encoder part uses a dual-path structure of Gabor auxiliary coding and U-Net ordinary convolution coding, and the decoder part uses center-peripheral convolution for feature extraction.

[0009] Preferably, the encoder part includes a Gabor auxiliary coding path and a U-Net ordinary double convolution path, which includes a total of four downsampling paths, each of which includes a Gabor modulation coded convolution and a U-Net double convolution structure, and the U-Net convolution coding path is composed of a double convolution structure. The main body of the decoder is 3 upsampling structures, that is, the decoding path includes three upsamplings;

[0010] The encoder first pre-extracts the input through a Gabor convolution with a scale of 3 and an angle of 90°, combines the extracted result with the original input, and then inputs it into the overall network structure, that is, a Gabor filter with a scale of 3 and an angle of 90° is added to the algorithm input part to pre-process the image;

[0011] The Gabor modulation coding pathway is composed of three scales of 3×3, 5×5, and 7×7, and each scale includes four directions of Gabor convolution of 0°, 45°, 90°, and 135°. The pathway uses convolution kernels of three scales from small to large to extract edge features of blood vessels from low to high levels. Feature maps of different scales are fused across scales and levels in the form of dense jump connections, and the auxiliary UNet convolution coding structure supplements edge features.

[0012] The upsampling structure first performs jump connection between the transposed convolution image and the feature map of the corresponding scale, and then uses the center-periphery convolution module and the double convolution module to respectively perform feature refinement extraction. The output of the decoder is reduced in dimension through 1×1 convolution, and the Sigmoid activation function is used to output the activation value of each feature unit.

[0013] Step 3: In the dual-path structure used by the encoder, Gabor convolutions of different scales and orientations are combined and modeled to form a Gabor convolution path that integrates edge features, and the auxiliary convolution path improves feature extraction;

[0014] Preferably, the design of the pathway scale is determined according to the convolution kernel extraction size of the convolution coding pathway and the basic scale of the vascular features, while minimizing the overall computational complexity of the encoder. The present invention uses three relatively small scales of 3, 5, and 7 and Gabor convolution kernels in four directions of 0°, 45°, 90°, and 135° to extract contour information from the feature map, and obtain texture features at different levels to compensate for the edge features of the vascular connection and the fovea area; then the above features are integrated, that is, stacked according to the dimensions; the integrated features are repeatedly extracted, that is, 1×1 convolution is used to concentrate the edge features; finally, the same as the ordinary convolution coding pathway is achieved, that is, the mutual connection between feature maps of different scales is enhanced through the maximum pooling operation, and the edge features are supplemented. Since the Gabor convolution kernel weight is fixed, the Gabor modulation coding pathway can better amplify the differences between blood vessels and background elements, and a larger scale is suitable for higher-level texture features, supplementing more edge contour information; it can bring multi-scale fusion of edge features with lower computational consumption without adding additional parameters.

[0015] Step 4: Construct a center-periphery convolution model in the decoder, use center convolution and peripheral convolution to extract the center and peripheral features of the feature map respectively, and change the weights to form center convolution and peripheral convolution of different strengths to highlight the center and peripheral features;

[0016] Preferably, in view of the fact that general vascular segmentation models are prone to many vascular breakpoints and abnormal points, a central convolution is constructed to strengthen the connection between the center and the periphery to achieve the effect of repairing vascular breakpoints, and a peripheral convolution is constructed to expand the difference between the center and the periphery to achieve the effect of removing abnormal points. The central convolution is reflected in the peripheral elements moving closer to the central elements, while the peripheral convolution is the peripheral elements deviating from the central elements in the opposite direction, thereby achieving the effect of narrowing or enlarging the gap between the peripheral elements and the central elements; by constructing central convolution and peripheral convolution, the central and peripheral features of the feature map are extracted respectively, and the central and peripheral features are highlighted by changing the weights to form central convolution and peripheral convolution of different intensities.

[0017] Step 5: Optimize the overall network model using the loss function that combines the mean square error loss MSE and the focal loss focal;

[0018] Preferably, the focal loss gives more weight to samples that are difficult to classify, alleviating the imbalance between blood vessel segmentation and background elements, which can be expressed as:

[0019]

[0020] in is the classification probability, delay is the balancing variable, is an adjustable focusing parameter. Set to 1, Set to 2, the mean square error loss is the sum of the squares of the distances between the label and the predicted value, and the adopted loss function is further expressed as:

[0021]

[0022] The training set is used for training, and the validation set is used for validation after each training cycle. Finally, the weights obtained from the training set are applied to the data to be tested.

[0023] Step 6: Perform model training on the training set obtained in step 1. After each round of training, verify the weights on the validation set. Apply the obtained training weights to the test data, build evaluation indicators simultaneously, and output the observed segmentation results on the OCTA-3MM and OCTA-6MM datasets.

[0024] Preferably, in order to qualitatively analyze the segmentation effect of the algorithm proposed in the present invention, several algorithms used on the OCTA-3MM and OCTA-6MM data sets are listed for qualitative and quantitative comparison. In order to quantitatively verify the segmentation effect of the algorithm proposed in the present invention, the present invention uses 5 indicators to quantitatively compare different algorithms.

[0025] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0026] (1) Aiming at the black box characteristics of deep learning neural networks, the present invention 6*8 constructs a Gabor modulation coding pathway based on the receptive field characteristics of simple cells in the primary visual cortex, and enhances the interpretability of deep learning based on the biological vision mechanism.

[0027] (2) In view of the problem that most algorithms only use convolution and cannot make good use of edge information, resulting in unclear boundaries between blood vessels and blood vessels, and blood vessels and background in the final segmentation results, the present invention proposes a Gabor modulation coding structure, which can obtain the contour information and edge information of retinal blood vessels to improve the segmentation continuity of blood vessel edges. At the same time, since the Gabor convolution kernel weight is fixed, it can better amplify the differences between blood vessels and background elements, and can bring multi-scale fusion of edge features with low computational consumption without adding additional parameters.

[0028] (3) The existing algorithms use convolution with a relatively single type of receptive field for feature extraction and cannot obtain deeper contextual information, resulting in the problem of broken or missing blood vessels in the segmentation results. The present invention constructs a center-periphery convolution structure. After the structure is applied to the jump connection of the network, the feature maps containing semantic information at different levels are extracted and fused through the center-periphery convolution, taking into account the integrity and locality of feature extraction, thereby achieving the effect of assisting the network to repair blood vessel breakpoints and eliminate abnormal points. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0030] Figure 1 A conceptual diagram of Gabor-assisted coding and center-periphery convolutional network provided in an embodiment of the present invention;

[0031] Figure 2 A structural diagram of the Gabor-assisted coding and center-periphery convolutional network provided in an embodiment of the present invention;

[0032] Figure 3 A single-scale Gabor-assisted coding convolution structure diagram provided in an embodiment of the present invention;

[0033] Figure 4 A structural diagram of a multi-scale Gabor auxiliary coding module provided in an embodiment of the present invention;

[0034] Figure 5 A schematic diagram of a single center and peripheral convolution provided by an embodiment of the present invention;

[0035] Figure 6 A structural diagram of a center-periphery convolution module provided in an embodiment of the present invention;

[0036] Figure 7 A diagram showing the retinal vessel segmentation results of different algorithms provided in an embodiment of the present invention on an OCTA-3MM dataset;

[0037] Figure 8 This is a diagram of retinal vessel segmentation results of different algorithms provided in embodiments of the present invention on the OCTA-6MM dataset. DETAILED DESCRIPTION

[0038] The following will be combined with the attached embodiment of the present invention Figures 1 to 8 , the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] The overall idea of ​​the network structure proposed in the technical solution of the present invention is as follows Figure 1As shown in the figure, considering that the general blood vessel segmentation algorithm is not sufficient for extracting blood vessel edge features, Gabor is added to the input part to preprocess the image, increase the proportion of blood vessel edge features in the original input, and prompt the network to distinguish blood vessel boundaries. In addition, multi-scale and multi-directional Gabor convolution is repeatedly used in the encoder part to provide the network with blood vessel contour edge information, thereby enhancing the network's sensitivity to the trend of blood vessel veins. In view of the problem that general convolutional neural networks are limited by the receptive field and cannot better highlight the segmentation focus, the present invention constructs a center-periphery convolution according to the distance relationship between the central pixel and the surrounding pixels, highlights the focal point extracted by the convolution kernel from both the central and peripheral perspectives, and realizes the organic combination of auxiliary network to make up for blood vessel breakpoints and remove abnormal points, and finally realizes accurate blood vessel segmentation.

[0040] That is, the present invention specifically provides the following technical solution: an OCTA retinal blood vessel segmentation method based on Gabor modulation coding to improve U-Net, specifically comprising the following steps:

[0041] Step 1: Obtain the OCTA retinal vessel segmentation datasets OCTA-3MM and OCTA-6MM, and divide the datasets into training set, validation set, and test set;

[0042] In this embodiment, the OCTA-3MM and OCTA-5MM datasets were downloaded from the official website of OCTA-500. OCTA-500 uses OCTExplorer3.8 software and a hierarchical segmentation method to segment the inner limiting membrane (ILM), outer plexiform layer (OPL) and Bruch's membrane (BM) of the retinal layer from 500 OCT volumes, and manually selects better segmentation results to form a dataset. The pixel-level annotation of retinal vessels includes the vessels generated between ILM and OPL after maximum projection. The retinal vessels have been annotated by five specially trained researchers and reviewed by three senior ophthalmologists to obtain a basic segmentation gold standard. The OCTA-6M and OCTA-3M datasets represent datasets composed of OCTA projection images with a field of view of 6 mm×6 mm and 3 mm×3 mm, respectively. The OCTA-6MM dataset contains 300 OCTA retinal image data with a resolution of 400×400; the OCTA-3MM dataset contains 200 OCTA retinal image data with a resolution of 304×304.

[0043] Exemplarily, the specific method of dividing the data set into a training set, a validation set, and a test set is: randomly shuffle the original data set, and divide it into a training set, a validation set, and a test set in a ratio of 3:1:1, thereby eliminating the randomness of the experimental results to a certain extent.

[0044] Step 2: The model of this method is constructed based on the encoder-decoder structure of U-Net. A dual-channel structure of Gabor-assisted coding and ordinary convolution coding is used in the encoder part. Skip connections are used between the encoder and decoder for information transmission. Center-peripheral convolution is used for refined extraction in the decoder part. Finally, feature dimensionality reduction is performed through 1×1 convolution and the activation value of each feature unit is output using the Sigmoid activation function.

[0045] In this embodiment, combined Figure 2 As shown in (a) of FIG. 1 , the structure proposed in the present invention consists of an encoder and a decoder. The main body of the decoder is a three-upsample structure. The encoder and the decoder contain multiple skip connections. Figure 2 As shown in (b), considering that the general blood vessel segmentation algorithm is not sufficient to extract the blood vessel edge features, the encoder first pre-extracts the input through a Gabor convolution with a scale of 3 and a direction of 90°, and then combines the extraction result with the original input before inputting it into the overall network structure. In order to make full use of the edge texture information of the blood vessels, a Gabor filter with a scale of 3 and an angle of 90° is added to the algorithm input part to pre-process the image, increase the proportion of blood vessel edge features in the original input, and prompt the network to distinguish blood vessel boundaries.

[0046] Exemplarily, the encoder includes a U-Net convolutional coding pathway and a Gabor modulation coding pathway. The U-Net convolutional coding pathway consists of a double convolutional structure, which includes two repeated 3×3 convolutional layers, a Dropblock layer, and a BN layer, and is expanded through a 2×2 maximum pooling downsampling structure. The number of feature channels in each part is 64, 128, and 256, respectively. The Gabor modulation coding pathway consists of three scales (3×3, 5×5, and 7×7), each scale containing four directions (0°, 45°, 90°, and 135°). The pathway uses convolution kernels of three scales from small to large to extract the edge features of blood vessels from low to high levels, so as to achieve the effect of complementing shallow and deep features. Feature maps of different scales are used in the form of dense jump connections to perform cross-scale and cross-level feature fusion, assisting the UNet convolutional coding structure to supplement edge features.

[0047] Exemplary, combined Figure 2As shown in (c), the upsampling structure first jumps the image after the transposed convolution and the feature map of the corresponding scale, and then uses the center-periphery convolution module and the double convolution module to respectively perform feature refinement extraction, which can make up for the vascular breakpoints and remove abnormal points to a certain extent. The output of the decoder is reduced in dimension by 1×1 convolution and the Sigmoid activation function is used to output the activation value of each feature unit. The various stages are shown in detail as follows: The overall network structure consists of an encoder and a decoder, and the encoder and decoder use jump connections for information exchange;

[0048] Specifically, taking the OCTA-3MM dataset as an example, the training image with a size of 304×304×1 obtained in step 1 is The image is input into the network, and features are extracted through Gabor convolution with a scale of 3 and an angle of 90°. Dimensionally stacked with the original input to generate an encoder input of size 304×304×2 , the specific formula is as follows:

[0049]

[0050]

[0051] Preferably, the encoder part includes a Gabor auxiliary coding path and a U-Net ordinary double convolution path, which includes a total of four downsampling paths, each of which includes a Gabor modulation coded convolution and a U-Net double convolution structure;

[0052] Specifically, the first downsampling channel: input The edge features are extracted using Gabor convolution with a scale of 3, angles of 0°, 45°, 90°, and 135°, input channels of 2, output channels of 64, 128, and 256, and maximum pooling scales of 2, 4, and 8, respectively. The resulting image is 152×152×64 in size. , 76×76×128 , 38×38×256 , the specific formula is as follows:

[0053]

[0054]

[0055]

[0056] Second downsampling channel: Input First, use the U-Net double convolution module with a scale of 3, an input channel of 2, and an output channel of 64 for feature extraction. The U-Net double convolution structure is 3×3 convolution with an input of 2, an output of 64, and a padding of 1, a DropBlock layer, a batch normalization layer, a ReLU layer, a 3×3 convolution with an input of 64, an output of 64, and a padding of 1, a DropBlock layer, a batch normalization layer, and a ReLU layer to obtain an intermediate feature of size 304×304×64. , and then use a 2×2 max pooling layer to get an output size of 152×152×64 , and then use Gabor convolution with a scale of 5, angles of 0°, 45°, 90°, and 135°, input channels of 64, output channels of 128 and 256, and maximum pooling scales of 2 and 4 to extract edge features, and get a size of 76×76×128 , 38×38×256 , the specific formula is as follows:

[0057]

[0058]

[0059]

[0060]

[0061]

[0062] The third downsampling channel: extract the Gabor auxiliary coding path And the first double convolution structure extracted Dimension stacking is performed to generate a size of 152×152×128 , feature extraction is performed using a U-Net double convolution module with a scale of 3, an input channel of 128, and an output channel of 128; the U-Net double convolution structure is a 3×3 convolution with an input of 128, an output of 128, and a padding of 1, a DropBlock layer, a batch normalization layer, a ReLU layer, a 3×3 convolution with an input of 128, an output of 128, and a padding of 1, a DropBlock layer, a batch normalization layer, and a ReLU layer, obtaining an intermediate feature of size 152×152×128 , and then use a 2×2 max pooling layer to get an output size of 76×76×128 , and then use Gabor convolution with a scale of 7, angles of 0°, 45°, 90°, 135°, input channels of 128, output channels of 256, and maximum pooling scale of 2 to extract edge features, and get a size of 38×38×256 ; The specific formula is as follows:

[0063]

[0064]

[0065]

[0066]

[0067] The fourth downsampling channel: extract the Gabor auxiliary coding path with a scale of 3 , Gabor-assisted coding pathway extraction with a scale of 5 And the second double convolution structure extracted Dimension stacking is performed to generate a size of 76×76×384 , feature extraction is performed using a U-Net double convolution module with a scale of 3, an input channel of 384, and an output channel of 256. The U-Net double convolution structure is a 3×3 convolution with an input of 384, an output of 256, and a padding of 1, a DropBlock layer, a batch normalization layer, a ReLU layer, a 3×3 convolution with an input of 256, an output of 256, and a padding of 1, a DropBlock layer, a batch normalization layer, and a ReLU layer, obtaining an intermediate feature of size 76×76×256 ; Then 2×2 max pooling layer, get the output size of 38×38×256 ;

[0068] Then the Gabor auxiliary coding path with a scale of 3 is extracted , Gabor-assisted coding pathway extraction with a scale of 5 , Gabor-assisted coding pathway extraction with a scale of 7 And the third double convolution structure extracted Dimension stacking is performed to generate intermediate features of size 38×38×1024 , and finally use the U-Net double convolution module with a scale of 3, an input channel of 1024, and an output channel of 512 for feature extraction. The U-Net double convolution structure is a 3×3 convolution with an input of 1024, an output of 512, and a padding of 1, a DropBlock layer, a batch normalization layer, a ReLU layer, a 3×3 convolution with an input of 512, an output of 512, and a padding of 1, a DropBlock layer, a batch normalization layer, and a ReLU layer, and the output size is 38×38×512. , the specific formula is as follows:

[0069]

[0070]

[0071]

[0072]

[0073]

[0074] Preferably, in this embodiment, the decoding path includes three upsampling steps:

[0075] First upsampling: First, use the transposed convolution with an input channel of 512, an output channel of 256, and a step size of 2 to upsample the encoder's final output Upsample to generate a temporary output of size 76×76×256 ; Secondly, and the intermediate characteristics of the fourth sampling channel Perform skip connection, the output size is 76×76×512 ; Then use the center-peripheral convolution with 512 input channels and 512 output channels Perform fine extraction and generate a 76×76×512 size ; Finally, the U-Net double convolution structure is used for repeated feature extraction. The U-Net double convolution structure is a 3×3 convolution with an input of 512, an output of 256, and a padding of 1, a DropBlock layer, a batch normalization layer, a ReLU layer, a 3×3 convolution with an input of 256, an output of 256, and a padding of 1, a DropBlock layer, a batch normalization layer, and a ReLU layer to obtain an intermediate feature of size 76×76×256 ; The specific formula is as follows:

[0076]

[0077]

[0078]

[0079]

[0080] Second upsampling: First, use the transposed convolution with an input channel of 256, an output channel of 128, and a step size of 2 to upsample the encoder's final output Upsample to generate a temporary output of size 152×152×128 ; Secondly, and the intermediate characteristics of the fourth sampling channel Perform skip connection, and the output size is 152×152×256 Then, we use the center-peripheral convolution with 256 input channels and 256 output channels. Perform fine extraction and generate a 152×152×256 size Finally, the U-Net double convolution structure is used for repeated feature extraction. The U-Net double convolution structure is a 3×3 convolution with an input of 256, an output of 128, and a padding of 1, a DropBlock layer, a batch normalization layer, a ReLU layer, a 3×3 convolution with an input of 128, an output of 128, and a padding of 1, a DropBlock layer, a batch normalization layer, and a ReLU layer, and an intermediate feature with a size of 152×152×128 is obtained. The specific formula is as follows:

[0081]

[0082]

[0083]

[0084]

[0085] The third upsampling: First, the encoder output is processed using a transposed convolution with an input channel of 128, an output channel of 64, and a step size of 2. Upsample to generate a temporary output of size 304×304×64 ; Secondly, and the intermediate characteristics of the fourth sampling channel Perform skip connection, and the output size is 304×304×128 ; Then use the center-peripheral convolution with 128 input channels and 128 output channels Perform fine extraction; generate a size of 304×304×128 ; Finally, the U-Net double convolution structure is used for repeated feature extraction. The U-Net double convolution structure is a 3×3 convolution with an input of 128, an output of 64, and a padding of 1, a DropBlock layer, a batch normalization layer, a ReLU layer, a 3×3 convolution with an input of 64, an output of 64, and a padding of 1, a DropBlock layer, a batch normalization layer, and a ReLU layer, and an intermediate feature with a size of 304×304×64 is obtained. , the specific formula is as follows:

[0086]

[0087]

[0088]

[0089]

[0090] Finally, a 1×1 convolution with 64 input channels and 1 output channel is used for the feature map. Perform feature compression and dimensionality reduction to promote cross-dimensional fusion of features and obtain a feature map with the same size as the input (304×304×1). , output through Sigmoid activation function , get the final retinal blood vessel segmentation prediction map:

[0091]

[0092]

[0093] Step 3: Use Gabor filters to model simple cells in the primary visual cortex. First, use the size of the Gabor convolution kernel to simulate the different scales of the receptive field, and then use the direction of the Gabor convolution kernel to simulate the different orientations of the receptive field.

[0094] In this embodiment, the receptive field of the simple cells of the primary visual cortex has a stronger response to visual stimuli in a specific orientation than in other orientations, and it is generally believed that the receptive field of the lateral geniculate cells is arranged and combined in a specific orientation, so that the receptive field of the primary visual cortex neurons has a direction selection characteristic. Studies have shown that the Gabor filter has good bandpass characteristics and direction sensitivity characteristics, and can detect edge contour information of light and dark contrast. It has been widely used in the modeling of direction selection characteristics. The two-dimensional Gabor function expression is as follows:

[0095]

[0096]

[0097]

[0098] in is the wavelength parameter of the cosine function in the Gabor kernel function, is the standard deviation of the Gaussian factor of the Gabor function, is the direction of the parallel stripes in the Gabor convolution kernel, is the phase shift of the cosine function in the Gabor kernel function, is a constant representing the ratio of the major and minor axes of the receptive field ellipse.

[0099] For example, in this embodiment, Figure 3 The present invention models a single Gabor convolution as follows: Figure 3 As shown in Figure 2, taking the 3×3 Gabor scale as an example, first calculate the corresponding convolution kernel weight parameters according to different scales and angles. , and then use the Gabor convolution kernels in four directions of 0°, 45°, 90°, and 135° to extract the contour information of the feature map to obtain the edge feature map , texture features at different levels are obtained to compensate for the edge features of the vascular junction and the fovea area;

[0100] Then the above features are integrated and stacked according to the dimensions to generate a feature fusion graph ; Then the integrated features are repeatedly extracted, that is, 1×1 convolution is used to condense the edge features to obtain the intermediate feature map Finally, the same convolutional coding pathway is realized, that is, the connection between feature maps of different scales is enhanced through the maximum pooling operation, the edge features are supplemented, the differences between blood vessels and background are amplified, and the output is ; The specific formula is as follows:

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108] Specifically, considering that the convolutional neural network neglects the edge information of blood vessels, the technical solution of the present invention proposes a method of combining Gabor convolutions of different scales and directions to form a Gabor convolution path that integrates edge features, and assists the convolution path to improve feature extraction. In order to adapt to the problem of small blood vessels in OCTA images, the present invention uses a smaller-scale Gabor convolution kernel. In order to focus the extraction on the boundary of the foveal avascular area, four directions with a smaller number but a larger span are used to collect edge features of blood vessels, and the network computing burden can be reduced;

[0109] Exemplarily, the present invention constructs a multi-scale Gabor auxiliary coding module as follows: Figure 4 As shown in the figure, the design of the pathway scale is determined according to the convolution kernel extraction size of the convolution coding pathway and the basic scale of the vascular features, while reducing the overall computational complexity of the encoder as much as possible. The present invention uses three relatively small scales of 3, 5, and 7 and Gabor convolution kernels in four directions of 0°, 45°, 90°, and 135° to extract contour information from the feature map, and obtain texture features at different levels to make up for the edge features of the vascular connection and the fovea area; then the above features are integrated, that is, stacked according to the dimensions; the integrated features are repeatedly extracted, that is, 1×1 convolution is used to concentrate the edge features; finally, the same as the ordinary convolution coding pathway is achieved, that is, the mutual connection between feature maps of different scales is enhanced through the maximum pooling operation, and the edge features are supplemented. Since the Gabor convolution kernel weight is fixed, the Gabor modulation coding pathway can better amplify the difference between blood vessels and background elements, and a larger scale is suitable for higher-level texture features, supplementing more edge contour information; it can bring multi-scale fusion of edge features with lower computational consumption without adding additional parameters.

[0110] Step 4: Propose a center-periphery convolution model, which is based on the distance between the central element and the peripheral element of the convolution kernel, and represents the centripetal and centrifugal states of the peripheral elements by reducing or expanding the distance;

[0111] In this embodiment, since a general blood vessel segmentation model is prone to many blood vessel breakpoints and abnormal points, a central convolution is constructed to strengthen the connection between the center and the periphery to achieve the effect of repairing the blood vessel breakpoints, and a peripheral convolution is constructed to expand the difference between the center and the periphery to achieve the effect of removing abnormal points. The central convolution is manifested as the peripheral elements moving closer to the central elements, while the peripheral convolution is the peripheral elements deviating from the central elements in the opposite direction, thereby achieving the effect of narrowing or enlarging the gap between the peripheral elements and the central elements; specifically, the present invention constructs central convolution and peripheral convolution to respectively extract the central and peripheral features of the feature map, and forms central convolution and peripheral convolution of different intensities by changing the weights to highlight the central and peripheral features.

[0112] For example, taking the convolution kernel size of 3×3 and the scaling factor of 1 / 2 as an example, combined with Figure 5 As shown, the central convolution Reduce the difference between the peripheral elements and the central elements, that is, reduce the distance between the peripheral elements and the central elements by a certain ratio; and the peripheral convolution Increasing the difference between peripheral elements and central elements is manifested by increasing the distance between peripheral elements and central elements at a certain rate. The construction of central and peripheral convolutions is to extract the central and peripheral features of the feature map respectively on the one hand, and to make up for the breakpoints of blood vessel segmentation by reducing the distance between peripheral elements and central elements on the other hand, and to help the network eliminate abnormal points by enlarging the distance between peripheral elements and central elements. Taking a 3×3 convolution kernel with a rate of 1 / 2 as an example, the central convolution and peripheral convolution The calculation formula is as follows:

[0113]

[0114]

[0115]

[0116]

[0117]

[0118]

[0119]

[0120] in Represents the weight value, Represents the element value corresponding to the feature map.

[0121] Exemplary, combined Figure 6 The overall structure of the center-periphery convolution module is as follows Figure 6 As shown in Figure 1, in order to combine the two types of convolution to achieve complementary advantages, the module first includes a dual-channel structure consisting of dual central convolution and dual peripheral convolution to obtain the central output feature map And peripheral output feature map ; Then, the feature maps extracted from the above two paths are fused with the centripetal features and the centrifugal features, and the intermediate feature maps are stacked and output ; Finally, 3×3 convolution is used for feature extraction, and the output is a refined feature map that combines the central and peripheral features. After the structure is applied to the skip connection of the network, the feature maps containing semantic information of different levels are extracted and fused through the center-periphery convolution, taking into account the integrity and locality of feature extraction, and to a certain extent, the effect of assisting the network to repair vascular breakpoints and eliminate abnormal points is achieved; the specific formula is as follows:

[0122]

[0123]

[0124]

[0125]

[0126]

[0127]

[0128] Step 5: Construct a loss function that combines mean square error loss (MSE) and focal loss to optimize the overall network model.

[0129] In this embodiment, since the blood vessels in the OCTA image are widely distributed, but the blood vessels themselves are too thin, there is a class imbalance problem between the blood vessels and the background. Therefore, the present invention uses a combination of mean square error loss (MSE) and focal loss to optimize the model. The focal loss can give more weight to samples that are difficult to classify, which can alleviate the problem of imbalance between blood vessel segmentation and background elements to a certain extent. It can be expressed as:

[0130]

[0131] in is the classification probability, delay is the balancing variable, As an adjustable focusing parameter, based on multiple experiments, Set to 1, Set to 2, the mean square error loss is the sum of the squares of the distances between the label and the predicted value, so the loss function used in this invention can be expressed as:

[0132]

[0133] That is, the present invention performs training through a training set, and performs verification through a verification set after each training cycle, and finally applies the weights obtained through training on the training set to the data to be tested.

[0134] Step 6: Perform model training on the training set obtained in step 1. After each round of training, verify the weights on the validation set. Finally, apply the obtained training weights to the test data, and simultaneously construct evaluation indicators to observe the network output results.

[0135] In this embodiment, the methods proposed in the present invention are all implemented in the Pytorch framework using a single NVIDIA GPU (GeForceRTX3080Ti). In the experimental process of the OCTA-6MM and OCTA-3MM data sets, the batch size set in this paper is 2, and the combination of the MSE loss function and the Focal loss function is used as the loss function of the experiment. The optimizer used is the Adam optimizer, the fixed learning rate is set to 0.0001, the number of training iterations is 100, and the earlystop is set to 45. Considering the overfitting problem of UNet, the retention rate of the Dropblock is set to 0.68, and the discarded block size is 3×3. Finally, the weights obtained on the training set are saved to obtain the trained OCTA retinal vascular segmentation model. Then we load the test set in step 1 onto the trained OCTA retinal vascular segmentation model for segmentation. In order to objectively and scientifically evaluate the segmentation performance of the proposed algorithm, the present invention uses indicators such as Dice coefficient, sensitivity (Sen), specificity (Spe), Jaccard coefficient, and balanced accuracy (Bacc) for quantitative comparison:

[0136]

[0137]

[0138]

[0139]

[0140]

[0141] Among them, TP (true positive) represents the correctly segmented blood vessel points, TN (true negative) represents the correctly segmented background points, FP (false positive) represents the incorrectly segmented background points, and FN (false negative) represents the incorrectly segmented blood vessel points. The Dice coefficient is used to measure the similarity between the predicted image and the label; sensitivity indicates the degree of segmentation of blood vessels and background, reflecting the algorithm's segmentation of small blood vessels. Higher sensitivity means that more small blood vessels are segmented; specificity indicates the ability to identify background elements; Jaccard coefficient is also used to compare the similarities and differences between the predicted image and the label; balanced accuracy is used to measure the accuracy of prediction in unbalanced binary classification tasks. Compared with accuracy, it is more suitable for the situation where blood vessel elements and background elements are unbalanced in this experiment.

[0142] For example, in order to qualitatively analyze the segmentation effect of the algorithm proposed in the present invention, several algorithms used on the OCTA-3MM and OCTA-6MM datasets are listed for qualitative and quantitative comparison. Figure 7 , Figure 8 It can be seen that the method proposed in the present invention effectively utilizes the edge contour information of blood vessels to improve the prediction sensitivity of the network model for small blood vessels, and uses the center-periphery convolution to extract physiological features for the fusion features after jump connection to solve the problem of vascular breakpoints. The organic combination of the above modules improves the effectiveness of the overall segmentation of the network.

[0143] In this embodiment, in order to quantitatively verify the segmentation effect of the algorithm proposed in the present invention, the present invention uses 5 indicators to quantitatively compare different algorithms. As shown in Table 1 and Table 2, the balanced accuracy (Bacc), Dice coefficient, sensitivity (Sen), specificity (Sp), and Jaccard index of the proposed algorithm on the two public data sets of OCTA-3MM and OCTA-6MM reached 0.9644, 0.9019, 0.9390, 0.9898, 0.8219 and 0.9451, 0.8758, 0.9073, 0.9829, 0.7803, respectively, and all indicators have good performance, especially in the two indicators of balanced accuracy and sensitivity. The above results prove the superiority of the algorithm proposed in this article.

[0144] Table 1: Performance comparison of different algorithms on the OCTA-3MM dataset:

[0145]

[0146] Table 2: Performance comparison of different algorithms on the OCTA-6MM dataset:

[0147]

[0148] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0149] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An OCTA retinal vessel segmentation method based on Gabor modulation coding and improved U-Net, characterized by: The following steps are involved: Step 1: Obtain the OCTA retinal vessel segmentation datasets OCTA-3MM and OCTA-6MM, and divide the datasets into training set, validation set, and test set; Step 2: A model including an encoder and a decoder is constructed based on the U-Net encoder-decoder structure, wherein the encoder and the decoder contain multiple skip connections. The encoder part uses a dual-path structure of Gabor auxiliary coding and U-Net ordinary convolution coding, and the decoder part uses center-peripheral convolution for feature extraction. Step 3: In the dual-path structure used by the encoder, Gabor convolutions of different scales and orientations are combined and modeled to form a Gabor convolution path that integrates edge features, and the auxiliary convolution path improves feature extraction; Step 4: Construct a center-periphery convolution model in the decoder, use center convolution and peripheral convolution to extract the center and peripheral features of the feature map respectively, and change the weights to form center convolution and peripheral convolution of different strengths to highlight the center and peripheral features; Step 5: Optimize the overall network model using the loss function that combines the mean square error loss MSE and the focal loss focal; Step 6: Perform model training on the training set obtained in step 1. After each round of training, verify the weights on the validation set and apply the obtained training weights to the test data.

2. The OCTA retinal vessel segmentation method based on Gabor modulation coding improved U-Net according to claim 1, characterized in that: The datasets OCTA-3MM and OCTA-6MM respectively represent datasets consisting of OCTA projection images with a field of view of 6 mm×6 mm and 3 mm×3 mm, the OCTA-6MM dataset contains 300 OCTA retinal image data with a resolution of 400×400, and the OCTA-3MM dataset contains 200 OCTA retinal image data with a resolution of 304×304; The specific division method of the data set into a training set, a validation set and a test set is: the original data set is randomly arranged in shuffle order and divided into a training set, a validation set and a test set in a ratio of 3:1:

1.

3. The OCTA retinal blood vessel segmentation method based on Gabor modulation coding improved U-Net according to claim 2, characterized in that: The step 2 also includes: the encoder part includes a Gabor auxiliary coding path and a U-Net ordinary double convolution path, which includes a total of four downsampling paths, each of which includes a Gabor modulation coded convolution and a U-Net double convolution structure, and the U-Net convolution coding path is composed of a double convolution structure. The main body of the decoder is 3 upsampling structures, that is, the decoding path includes three upsamplings; The encoder first pre-extracts the input through a Gabor convolution with a scale of 3 and an angle of 90°, combines the extracted result with the original input, and then inputs it into the overall network structure, that is, a Gabor filter with a scale of 3 and an angle of 90° is added to the algorithm input part to pre-process the image; The Gabor modulation coding pathway is composed of three scales of 3×3, 5×5, and 7×7, and each scale includes four directions of Gabor convolution of 0°, 45°, 90°, and 135°. The pathway uses convolution kernels of three scales from small to large to extract edge features of blood vessels from low to high levels. Feature maps of different scales are fused across scales and levels in the form of dense jump connections, and the auxiliary UNet convolution coding structure supplements edge features. The upsampling structure first performs jump connection between the transposed convolution image and the feature map of the corresponding scale, and then uses the center-periphery convolution module and the double convolution module to respectively perform feature refinement extraction. The output of the decoder is reduced in dimension through 1×1 convolution, and the Sigmoid activation function is used to output the activation value of each feature unit.

4. The OCTA retinal blood vessel segmentation method based on Gabor modulation coding improved U-Net according to claim 3, characterized in that: The auxiliary convolution pathway in step 3 improves feature extraction by using Gabor filters to model simple cells in the primary visual cortex, firstly using the size of the Gabor convolution kernel to simulate different scales of the receptive field, and secondly using the direction of the Gabor convolution kernel to simulate different orientations of the receptive field.

5. The OCTA retinal vessel segmentation method based on Gabor modulation coding improved U-Net according to claim 4, characterized in that: The step 3 specifically includes: using three smaller scales, 3, 5, and 7, and four directions, 0°, 45°, 90°, and 135°, to extract contour information from the feature map, and obtain texture features at different levels to compensate for the edge features of the vascular connection and the fovea; then integrating the texture features, i.e., stacking them according to the dimensions; then repeatedly extracting the integrated texture features, i.e., using 1×1 convolution to condense the edge features; and finally, achieving intercommunication with the ordinary convolution coding pathway, i.e., enhancing the interconnection between feature maps of different scales through the maximum pooling operation, and supplementing the edge features.

6. The OCTA retinal vessel segmentation method based on Gabor modulation coding improved U-Net according to claim 5, characterized in that: The center-periphery convolution model constructed in step 4 specifically includes: a dual-path structure consisting of dual center convolution and dual peripheral convolution to obtain a center output feature map And peripheral output feature map ; Then, the feature maps extracted from the above two paths are fused with the centripetal features and the centrifugal features, and the intermediate feature maps are stacked and output ; Finally, 3×3 convolution is used for feature extraction, and the output is a refined feature map that combines the central and peripheral features. , and after being applied to the skip connection of the network, the feature maps containing semantic information at different levels are extracted and fused through the center-periphery convolution. The specific formula is as follows: ； ； ； ； ； ； in, represents the central convolution, represents peripheral convolution, represents the activation function layer, represents the batch normalization layer, represents a regularization layer.

7. The OCTA retinal blood vessel segmentation method based on Gabor modulation coding improved U-Net according to claim 6, characterized in that: The calculation formulas for the central convolution and the peripheral convolution in step 4 are as follows: ； ； ； ； ； ； ； in, represents the weight value, Represents the element value corresponding to the feature map, Represents a 3×3 convolution kernel with a magnification of 1 / 2.

8. The OCTA retinal blood vessel segmentation method based on Gabor modulation coding improved U-Net according to claim 7, characterized in that: The step 5 specifically includes: focus loss gives more weight to samples that are difficult to classify, alleviating the imbalance between blood vessel segmentation and background elements, which can be expressed as: ； in is the classification probability, delay is the balancing variable, is an adjustable focusing parameter. Set to 1, Set to 2, the mean square error loss is the sum of the squares of the distances between the label and the predicted value, and the adopted loss function is further expressed as: ； The training set is used for training, and the validation set is used for validation after each training cycle. Finally, the weights obtained from the training set are applied to the data to be tested.

Images