A choroid neovascularization image segmentation method based on a hybrid convolutional network

Abstract

The application discloses a choroid neovascularization image segmentation method based on a hybrid convolutional network, first, collecting fundus OCT scan images and labeling, and forming a data set by using the labeled images; a deep space-time separation hybrid convolutional neural network fusing an attention mechanism is constructed, two-dimensional feature extraction is carried out by using two-dimensional convolution, then the two-dimensional feature extraction is extended to three-dimensional, and then three-dimensional attention deep space-time separation convolution is carried out, and the two-dimensional feature and the three-dimensional feature are aligned and fused; then the hybrid neural network model constructed is trained by using the data set; finally, the trained hybrid neural network model is used to segment choroid neovascularization from the fundus OCT scan images, and a segmentation result is obtained. Through the space-time attention mechanism, local features of the choroid neovascularization image are better extracted, and by using the deep space-time separation convolution, a dimension reduction operation is carried out on the input feature map, so that the calculation parameters can be effectively reduced, thereby reducing the network calculation amount, and the channel attention can be more effectively calculated.

Description

TECHNICAL FIELD

[0001] The present application relates to retinal image segmentation, in particular to a choroidal neovascularization image segmentation method based on a hybrid convolutional network. BACKGROUND

[0002] The tissue structure in the center of the retina is called the macular area, which can determine the quality of vision due to its strong light sensitivity. Choroidal neovascularization refers to proliferative blood vessels from choroidal capillaries, which are mostly found in the macular area.

[0003] There are many kinds of medical image processing methods, and the methods are complex. The imaging principle of medical images is not refined enough, the specificity of individual retinas is large, the gap between individuals is large, and the structures are not the same. The traditional method estimates the choroidal neovascularization area by manual quantification, which is very time-consuming and highly dependent on personal experience, and there is a judgment error.

[0004] With the development of deep learning, image segmentation technology based on deep learning has become an important part of image segmentation. For choroidal neovascularization segmentation, deep learning methods have recently achieved success in this area. However, compared with human organ image segmentation, choroidal neovascularization segmentation has greater difficulties and challenges: 1) there are artifacts and eye structure noise in fundus OCT images 2) using two-dimensional convolutional networks alone will miss features, and three-dimensional convolution is needed to capture image slice information, but using three-dimensional convolution alone has high computational cost. SUMMARY

[0005] The purpose of the present application is to provide a choroidal neovascularization image segmentation method based on a hybrid convolutional network with fine segmentation and low computational complexity.

[0006] Technical scheme: In order to solve the above problems, the present application adopts a choroidal neovascularization image segmentation method based on a hybrid convolutional network, which comprises the following steps:

[0007] (1) Construct a data set; collect fundus OCT scan images, label the choroidal neovascularization in the images, and form a data set from the labeled images;

[0008] (2) Construct a deep spatio-temporal separation hybrid convolutional neural network with fusion attention mechanism, use two-dimensional convolution for two-dimensional feature extraction, then expand the two-dimensional convolution to three-dimensional through two-dimensional deep convolution, and then perform three-dimensional attention deep spatio-temporal separation convolution, align and fuse the features generated by two-dimensional convolution and three-dimensional convolution;

[0009] (3) Use the data set to train the deep spatio-temporal separation hybrid convolutional neural network with fusion attention mechanism constructed to obtain a hybrid neural network model;

[0010] (4) using the trained hybrid neural network model to segment the choroidal neovascularization image from the fundus OCT scan image to obtain a segmentation result.

[0011] Further, the step (2) converts four continuous two-dimensional feature images into a three-dimensional feature vector.

[0012] Further, in the step (2), the obtained three-dimensional feature vector is subjected to a time-space attention mechanism to form a new attention map, and then subjected to a deep space-time separation convolution.

[0013] Further, when the deep space-time separation convolution is performed, a three-dimensional convolution is divided into two separate convolutions, i.e., a 1×Y×Z spatial convolution and an X×1×1 time convolution, and then the three-dimensional deep space-time separation convolution DSTS has:

[0014]

[0015] wherein K P represents a convolution kernel of the point-wise convolution, represents a convolution kernel of the spatial convolution ; and represents a convolution kernel of the time convolution ; represents splicing of two feature maps of the spatial convolution and the time convolution; F” represents a final attention feature map after the space-time attention mechanism; and r represents a dilated convolution operation.

[0016] Further, the calculation process of the final attention feature map F” after the space-time attention mechanism is as follows:

[0017]

[0018]

[0019] wherein F represents an input feature map in the input time attention module, M C (F) represents an output feature map generated after the time attention module, F' represents an input feature map in the input space attention module, M S (F') represents an output feature map generated after the space attention module.

[0020] Further, after the three-dimensional feature vector is input into the time attention module, average pooling and maximum pooling are performed to obtain a maximum-pooled feature and an average-pooled feature Then, a shared network layer composed of a multilayer perceptron containing one hidden layer receives the features subjected to the two kinds of pooling operations, and finally generates a channel attention map M X ∈R X×1×1After the operation of the shared network layer, the sum of the elements is used to combine the output feature vector, and the calculation formula in the time attention module is:

[0021]

[0022] Wherein, σ represents sigmiod function, W0 and W1 are the weights of the multilayer perceptron MLP part, AvgPool(F) represents the average pooling of the input feature F, and MaxPool(F) represents the maximum pooling of the input feature F.

[0023] Further, the three-dimensional feature vector is input into the spatial attention module, and then average pooling and maximum pooling are performed to obtain the maximum pooling feature And the average pooling feature After the standard convolution operation in the convolution layer, the features are coupled to generate the final spatial attention map, and the calculation formula in the spatial attention module is:

[0024]

[0025] Wherein, σ represents sigmiod function, and f represents convolution operation.

[0026] Further, the step (2) of aligning and fusing the features generated by two-dimensional convolution and the features generated by three-dimensional convolution includes the following steps:

[0027] The feature map and the related pixel probability score output from the two-dimensional convolution network are calculated:

[0028] X 2d =f 2d (I 2d ;θ 2d ),X 2d ∈R 4n×256×256×64

[0029] y 2d =f 2dcls (X 2d ;θ 2dcls ),y 2d ∈R 4n×256×256×3

[0030] Wherein, I 2d represents the sample input into the two-dimensional convolution network; n represents the batch size of the input training sample;

[0031] Align the feature map and the probability score in the two-dimensional convolution network with the three-dimensional feature map score map, and the calculation formula is:

[0032] X' 2d =T(X 2d ),X' 2d∈R n×256×256×64

[0033] y' 2d =T(y 2d ),y' 2d ∈R n×256×256×3

[0034] Where T represents the transformation of adjacent slices into three-dimensional data;

[0035] Obtain contextual features y' from a 2D convolutional network via skip connections 2d , the 3D convolutional network performs training on the context pixels of the probability map generated by the 2D convolutional network. At the same time, the probability map generated by the 2D convolutional network provides feedback to the training of the 3D convolutional network. The calculation formula is:

[0036] X 3d =f dsts (I,y' 2d θ 3d )

[0037] Z=X 3d +X' 2d

[0038] Among them, X 3d Represents the output feature map of the 3D convolutional network, and Z represents the 2D and 3D mixed feature map, which refers to the sum of the intra-chip features and inter-chip features in the 2D and 3D networks.

[0039] Furthermore, the two-dimensional and three-dimensional mixed features Z are jointly learned and optimized, and the calculation formula is:

[0040] H=f hff (Z;θ hff )

[0041] y h =f hffcls (H; θ hffcls )

[0042] Among them, H represents the optimized mixed feature, y h Represents the pixel-level prediction probability of the mixed feature fusion layer.

[0043] Furthermore, in the deep spatiotemporal separation hybrid convolutional neural network with integrated attention mechanism, multi-class dice loss and cross entropy loss that are insensitive to class imbalance are used to optimize the imbalance of the segmentation level of 3D choroidal neovascularization. The loss function is:

[0044]

[0045] Where C represents the number of categories, V represents the number of voxels, denotes the predicted probability that the voxel i belongs to the class c, and ε denotes a smoothing factor, denotes the true label that the voxel i belongs to the class c.

[0046] Beneficial effects: The present application has the following advantages over the prior art: the spatial-temporal attention mechanism is used to better extract local features of the choroidal neovascularization image, the deep spatial-temporal separation convolution is used to reduce the calculation parameters by performing dimension reduction operation on the input feature map, thereby reducing the network calculation amount, and the channel attention can be calculated more effectively. The average pooling can effectively fuse the spatial information, and the maximum pooling is more suitable for the attention mechanism, which can find the pixel region closest to the target feature in the feature map. The joint operation of the maximum pooling and the average pooling can refine the feature map more effectively. The spatial attention is more inclined to focus on the position information of the target in the image, which can effectively supplement the temporal attention. After the average pooling and the maximum pooling operations are applied on the channel axis, they are connected to generate a temporary feature map to effectively calculate the spatial attention feature. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 a mixed convolutional network framework of the present application;

[0048] Figure 2 a time attention module operation process diagram in the mixed convolutional network of the present application;

[0049] Figure 3 a spatial attention module operation process diagram in the mixed convolutional network of the present application. DETAILED DESCRIPTION

[0050] The choroidal neovascularization image segmentation method based on the mixed convolutional network in the embodiment includes the following steps:

[0051] (1) Construct a data set; collect fundus OCT scan images, label the choroidal neovascularization in the images, manually label the choroidal neovascularization region in the scan images as a standard, and form a data set by using the labeled images;

[0052] (2) Construct a deep spatial-temporal separation mixed convolutional neural network with a fusion attention mechanism, use two-dimensional convolution to extract two-dimensional features, then expand the two-dimensional convolution to three dimensions through two-dimensional deep convolution, and then perform three-dimensional attention deep spatial-temporal separation convolution to align and fuse the features generated by the two-dimensional convolution and the features generated by the three-dimensional convolution;

[0053] (3) Use the data set to train the deep spatial-temporal separation mixed convolutional neural network with a fusion attention mechanism to obtain a mixed neural network model;

[0054] (4) using the trained hybrid neural network model to segment the choroidal neovascularization from the fundus OCT scan image to obtain a segmentation result.

[0055] In step (2), the network structure is as shown in the figure, including two-dimensional convolution, two-dimensional deep convolution, three-dimensional pointwise convolution, three-dimensional attention deep space-time separation convolution, and the three-dimensional attention deep space-time separation convolution includes a three-dimensional space pyramid module and a space-time attention mechanism module. The hybrid convolution network adopts a coding-decoding structure form, the bottom is composed of two-dimensional deep convolution, and the rest is composed of three-dimensional deep and space-time separation convolution; a space pyramid pooling operation is adopted at the end of the encoder, and the operation is captured by parallel three-dimensional hollow space-time separation convolution of different sizes to capture multi-scale information. Figure 1

[0056] In step (3), the input image labeled is subjected to a down-sampling operation by a common 3x3 two-dimensional convolution to obtain a two-dimensional feature image, and then enters a two-dimensional deep convolution module to expand the deep two-dimensional convolution to three dimensions, and convert four continuous two-dimensional feature images into a three-dimensional feature vector, and reduce the calculation and parameters by independently performing convolution on each input channel.

[0057] In the three-dimensional convolution layer, the input dimension is T F ×W F ×H F ×M feature map F'', and the output dimension is T G ×W G ×H G ×N feature map G, T, W, H represent the time dimension, spatial width and spatial height of the three-dimensional feature map respectively, and M and N represent the number of input and output channels respectively. A standard three-dimensional convolution layer is parameterized by a convolution kernel K S (X×Y×Z×M×N), X, Y, Z are the time dimension and spatial dimension of the convolution kernel respectively, and the output three-dimensional convolution layer can be calculated by the following formula:

[0058]

[0059] In the above formula, r represents the hollow convolution operation.

[0060] For a three-dimensional deep convolution with a convolution kernel K D of X×Y×Z×M, it can be calculated by the following formula:

[0061]

[0062] Then a three-dimensional pointwise convolution with a convolution kernel K P of 1×1×1×M×N is used to combine the output deep convolution and then project it into a new pipe space, which can be represented by the following formula: ​

[0063]

[0064] Deep convolution can effectively reduce the convolution parameters and computational complexity, and is a powerful operation. For example, a tensor with a dimension of 3x3x3, an input channel number of c, and an output channel number of c is convolved, a standard convolution contains 27c 2 parameters, but deep convolution has only 27c parameters, which is c times less than the standard parameters.

[0065] The spatiotemporal attention mechanism better extracts local features of choroidal neovascularization images. In the temporal attention module, as Figure 2 , any channel of the feature map can be regarded as a feature detector, and the role of attention is to find the area features that need to be learned more in the input image. Dimensionality reduction operation on the input feature map can effectively reduce the calculation parameters, thereby reducing the network calculation amount, so that the channel attention can be calculated more effectively. Average pooling can effectively fuse spatial information, while max pooling is more compatible with attention mechanism, which can find the pixel area closest to the target feature in the feature map. The joint operation of max pooling and average pooling can more effectively refine the feature map. After the input feature, average pooling and max pooling are added, the combination of the two operations can effectively fuse the spatial information of the feature map and obtain the attention area feature. The feature maps obtained by their operations can be represented by the following two symbols: and are the max pooling feature and the average pooling feature, respectively. The structure uses a shared network layer composed of a multilayer perceptron containing one hidden layer to receive features after two kinds of pooling operations, and finally generates a channel attention map M X ∈R X×1×1 After the operation of the shared network layer, the sum of the elements is used to combine the output feature vector, and the temporal attention module can be represented as follows:

[0066]

[0067] σ represents the sigmiod function, and W0 and W1 are the weights of the MLP part.

[0068] Unlike the temporal attention module, the spatial attention module, as Figure 3 , is more inclined to focus on the position information of the target in the image, which can effectively supplement the temporal attention. After applying average pooling and max pooling operations on the channel axis, they are connected to generate a temporary feature map to effectively calculate the spatial attention feature. When connecting the feature maps, a convolution layer is used to generate a spatial feature map M S ∈R H×W . The use of max pooling and average pooling operations enables the input feature information to be effectively aggregated, and then two mappings are generated, which can be represented as and denote the max-pooling and average-pooling features respectively. Then the features are coupled by standard convolution operation in the convolution layer to generate the final spatial attention map. The spatial attention module can be represented as:

[0069]

[0070] σ denotes the sigmoid function and f represents the convolution operation.

[0071] By separating the learning of channel attention and spatial attention, the computational complexity can be greatly reduced, and the spatio-temporal features can be effectively captured and learned. The overall process for calculating the final spatio-temporal attention map F" can be represented as:

[0072]

[0073]

[0074] For three-dimensional convolution, more computational complexity is required than for two-dimensional convolution. In order to make the model operation more efficient, a three-dimensional convolution is divided into two separate convolutions, one is a 1×Y×Z spatial convolution and one is an X×1×1 temporal convolution, achieving the purpose of time-space separation (STS). The focus of the spatial convolution is the learning of spatial features, and the focus of the temporal convolution is the learning of temporal features. The parallel time-space separation calculation method is defined as follows:

[0075]

[0076] In the above formula, denotes the convolution kernel of the 1×Y×Z spatial convolution, denotes the convolution kernel of the X×1×1 temporal convolution, and ∪ denotes the concatenation of two feature maps. In the parallel STS module, the two convolutions are executed in parallel in two branches, and then their outputs are connected. This is more effective for anisotropy of the retinal image. In order to further reduce the computational complexity and model parameters, a three-dimensional deep time-space separation convolution is used.

[0077] After the three-dimensional time-space separation convolution operation, the output channels are divided into spatial branches and temporal branches, which focus on the learning of temporal and spatial features respectively. In each branch, spatial / temporal convolution is performed in each channel. After independent feature learning, the outputs of the spatial and temporal branches are connected and fed to a pointwise convolution for feature integration. The three-dimensional deep time-space separation convolution (DSTS) can be represented as:

[0078]

[0079] In the above formula, K Pis the convolution kernel of point-wise convolution, and are the convolution kernel of spatial and temporal convolution respectively, F" is the final attention map after the spatio-temporal attention mechanism. We replace all three-dimensional convolution with depth-wise spatio-temporal convolution to save the computational cost.

[0080] Two-dimensional convolution network with depth-wise convolution can effectively learn high-level plane features, but it ignores the spatial information along the Z dimension. Three-dimensional convolution network can make up for this deficiency, but it needs more computational cost. Therefore, we use two-dimensional and three-dimensional convolution network to jointly fuse and optimize learning, which can better learn the intra-slice and inter-slice features of CNV to better segment the choroidal neovascularization image.

[0081] The feature map and the related pixel probability score output from the two-dimensional convolution network can be expressed as follows:

[0082] X 2d =f 2d (I 2d ; θ 2d ), X 2d ∈R 4n×256×256×64

[0083] y 2d =f 2dcls (X 2d ; θ 2dcls ), y 2d ∈R 4n×256×256×3

[0084] I 2d represents the input sample of the two-dimensional convolution network, and n represents the batch size of the input training sample.

[0085] In order to fuse the mixed features from two-dimensional and three-dimensional convolution networks, the feature size needs to be aligned. The feature map and the probability score in the two-dimensional convolution network are aligned with the three-dimensional feature map score map according to the following formula:

[0086] X′ 2d =T(X 2d ), X′ 2d ∈R n×256×256×64

[0087] y′ 2d =T(y 2d ), y′ 2d ∈R n×256×256×3

[0088] T represents the transformation of adjacent slices to form three-dimensional data.

[0089] The three-dimensional convolutional network part extracts multi-scale features through spatial pyramid pooling, and obtains context features y' from the two-dimensional convolutional network through skip connection 2d The two-dimensional network part generates a feature probability map, and the three-dimensional convolutional network performs training on the context pixels thereof. The probability map generated by the two-dimensional convolutional network can feed back the training of the three-dimensional convolutional network part, solve the problem of heavy calculation burden of self-learning update of the best weight using the three-dimensional network alone, greatly improve the self-learning speed and calculation speed of the three-dimensional network part. The learning process of the three-dimensional network part can be described as:

[0090] X 3d =f dsts (I,y′ 2d ;θ 3d )

[0091] Z=X 3d +X′ 2d

[0092] X 3d indicates the output feature map of the three-dimensional convolutional network part, and Z indicates the two-dimensional and three-dimensional mixed features, which are respectively the sum of the intra-chip features and the inter-chip features in the two-dimensional and three-dimensional networks. Then, the mixed features are jointly learned and optimized in the HFF layer:

[0093] H=f hff (Z;θ hff )

[0094] y h =f hffcls (H;θ hffcls )

[0095] H indicates the optimized mixed features, and y h is the pixel-level prediction probability of the mixed feature fusion layer.

[0096] The three-dimensional choroidal neovascularization segmentation has a serious grade imbalance. In order to solve this problem, the multi-class dice loss and the cross-entropy loss which are not sensitive to class imbalance are used in the mixed convolutional network, as shown in the following formula:

[0097]

[0098] In the above formula, C represents the number of class ratios, V represents the number of voxels, represents the prediction probability of voxel i belonging to class c, and ε represents a smoothing factor, represents the true value label of voxel i belonging to class c.

[0099] Gradient constraint is also introduced to better preserve the CNV boundary gradient constraint G as shown in the following formula:

[0100]

[0101] where N denotes the set of pixel boundaries, g(n) denotes the computation of the gradient, and denote the gradient of pixel n in the x and y direction, respectively.

[0102] The final loss function can be expressed as:

[0103]

Claims

1.A method for choroidal neovascularization image segmentation based on a hybrid convolutional network, characterized in that, Comprise the following steps: (1) Constructing a data set; collecting fundus OCT scan images, labeling the choroidal neovascularization in the images, and forming a data set from the labeled images; (2) Constructing a deep spatio-temporal separation hybrid convolutional neural network with fusion attention mechanism, specifically, two-dimensional convolution is used for two-dimensional feature extraction to obtain a two-dimensional feature image, and then two-dimensional deep convolution is used to expand two-dimensional convolution to three dimensions: four consecutive two-dimensional feature images are converted into a three-dimensional feature vector; then three-dimensional attention deep spatio-temporal separation convolution is performed: the obtained three-dimensional feature vector is subjected to time-space attention mechanism to form a new attention map, and then deep spatio-temporal separation convolution is performed, wherein the deep spatio-temporal separation convolution divides a three-dimensional convolution into two separate convolutions, one being a 1xYxZ spatial convolution and the other being an Xx1x1 temporal convolution, and the three-dimensional deep spatio-temporal separation convolution DSTS has: ； wherein, represents a convolution kernel of a point-wise convolution, represents a convolution kernel of a spatial convolution ; represents a convolution kernel of a temporal convolution ; represents concatenating two feature maps of spatial convolution and temporal convolution; represents a final attention feature map after a spatio-temporal attention mechanism; represents a dilated convolution operation; Align and fuse the features generated by two-dimensional convolution with the features generated by three-dimensional convolution, and learn the pixel-level prediction probability from the fused hybrid features; specifically comprising the following steps: Calculate the feature map and related pixel probability score output from the two-dimensional convolution network: ； ； wherein, represents a sample input to a two-dimensional convolutional network; n represents a batch size of input training samples; represents a pixel probability score in a two-dimensional convolution. Align the feature map and probability score in the two-dimensional convolution network with the three-dimensional feature map score map, first perform three-dimensional transformation on the two-dimensional convolution feature map, the calculation formula is: ； ； Wherein, T represents the transformation of adjacent slices to form three-dimensional data; Obtaining context features from a two-dimensional convolutional network by skip connections A three-dimensional convolutional network performs training on context pixels of a probability map generated by a two-dimensional convolutional network, while the probability map generated by the two-dimensional convolutional network feeds back to the training of the three-dimensional convolutional network, and a calculation formula is: ； ； wherein, represents a three-dimensional convolutional network output feature map, represents a two-dimensional and three-dimensional hybrid feature map, which refers to the sum of intra-slice features and inter-slice features in two-dimensional and three-dimensional networks; represents a pixel probability score in three-dimensional convolution. (3) Use the data set to train the deep spatio-temporal separation hybrid convolutional neural network with fusion attention mechanism constructed to obtain a hybrid neural network model; (4) Use the trained hybrid neural network model to segment the choroidal neovascularization image from the fundus OCT scan image to obtain the segmentation result. 2.The method of claim 1, wherein, The final attention feature map of the spatio-temporal attention mechanism The calculation process is as follows: ； ； wherein, represents an input feature map in the input temporal attention module, represents an output feature map generated by the temporal attention module, represents an input feature map in the input spatial attention module, represents an output feature map generated by the spatial attention module; represents multiplication of two matrix elements. 3.The method of claim 2, wherein, The three-dimensional feature vector is input into the time attention module, and then average pooling and maximum pooling are performed to obtain the maximum pooling feature and the average pooling feature ; then the shared network layer composed of a multilayer perceptron including one hidden layer receives the features after the two pooling operations, and finally generates a channel attention map , , wherein X represents the number of channels, 1 represents the length, and 1 represents the width of the feature map set, after the operation of the shared network layer, the sum of elements is used to combine the output feature vector, and the calculation formula in the time attention module is: ； in, represents the sigmiod function, and are the weights of the MLP part of the multilayer perceptron, Represents the input features Perform average pooling, Represents the input features Perform maximum pooling. 4.The method of claim 3, wherein, The three-dimensional feature vector input space attention module is subjected to average pooling and maximum pooling to obtain maximum pooling features and average pooling features and average pooling features , , represents a feature map set with a channel number of 1, a length of H, and a width of W, the features are coupled through standard convolution operation in a convolution layer to generate a final spatial attention map, and the calculation formula in the spatial attention module is: ； wherein denotes a sigmoid function, represents a convolution operation. 5.The hybrid convolutional network-based choroidal neovascularization image segmentation method of claim 1, wherein, Two-dimensional three-dimensional hybrid features Joint learning and optimization is performed, with the following formula: ； ； where H denotes the optimized mixed features, denotes the pixel probability score in the HFF layer, denotes the pixel-level prediction probability of the mixed feature fusion layer. 6.The hybrid convolutional network-based choroidal neovascularization image segmentation method of claim 5, wherein, In the training of the deep spatio-temporal separation hybrid convolutional neural network with fusion attention mechanism, the multi-class dice loss and cross-entropy loss which are not sensitive to class imbalance are used to optimize the segmentation level imbalance of three-dimensional choroidal neovascularization, and the loss function is: ； wherein, denotes the number of classes, denotes the number of voxels, denotes the predicted probability that a voxel i belongs to a class c, denotes a smoothing factor, denotes the true value label that a voxel i belongs to a class c; denotes a gradient constraint, denotes an activation function in deep learning with respect to a variable .

Images