Retina split image classification method based on fundus optical coherence tomography
By introducing the adaptive feature fusion network ASFF-Neck in the YoLov5 model, the problem of high myopia macular cleavage detection complexity is solved, and a more accurate retinal cleavage classification and detection effect is achieved.
Patent Information
- Application Number
- CN202311224119.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-21
- Publication Date
- 2024-12-24
AI Technical Summary
The detection and classification of macular cleavage of high myopia is complex. The existing optical coherence tomography technology and B-ultrasound images are difficult to clearly display retinal cleavage under refractive interstitial turbidity or interference conditions, resulting in difficulty in classification.
A retinal split image classification method based on fundus optical coherence tomography was designed, and a new neck adaptive feature fusion network ASFF-Neck and YoLov5 model was used to learn spatial filtering and feature fusion through the ASFF module to reduce the number of different levels of Inconsistency improves the effectiveness of feature fusion.
It effectively reduces the characteristic inconsistency in retinal cleavage detection, improves the detection accuracy and classification effect of retinal cleavage, especially when there are cleavages with a larger range and cleavages with a smaller range.
Smart Images
Figure CN119181127A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of ophthalmic imaging and artificial intelligence, and in particular relates to a retinal splitting image classification method based on fundus optical coherence tomography. Background Art
[0002] Myopic macular retinoschisis (MRS) is a common complication of high myopia and one of the important causes of visual impairment and blindness in high myopia. However, MRS classification is complex and quantitative analysis is difficult.
[0003] At present, the commonly used MRS detection methods are optical coherence tomography (OCT) and ocular B-ultrasound. MRS in optical coherence tomography shows thickening of the retinal neuroepithelium in the posterior pole, the presence of a slit-like optical space between the neuroepithelial layers, which is connected by oblique vertical bridge-shaped or columnar light bands, and a moderate degree of reflection in the front of the retinal pigment epithelium (RPE) layer. Under the conditions of opacity or interference of the refractive medium, OCT detection images cannot clearly show retinoschisis, and it is difficult to classify the type of schisis. MRS B-ultrasound images show that the inner surface of the eyeball wall behind the posterior scleral staphyloma is less smooth or thin membranous echo bands attached to the wall. Unlike retinal detachment, when retinoschisis presents a membrane-like bulge, the band echo is strong and thin, and there is no posterior movement, making it difficult to classify the type of schisis. Summary of the invention
[0004] In view of the above problems, the present invention designs a retinoschisis image classification method based on fundus optical coherence tomography, and proposes a retinoschisis detection algorithm based on ASFF-YoLov5 through a new neck Neck adaptive feature fusion network ASFF (Adaptively Spatial Feature Fusion,)-Neck. In the ASFF-Neck part, the ASFF module is used to directly learn how to perform spatial filtering on features at other levels, so as to retain only useful information for combination. When there are large-scale splitting and small-scale splitting in the retinal medical image, the inconsistency between different levels can be effectively reduced, the effectiveness of feature fusion can be improved, and better detection of retinoschisis can be obtained.
[0005] The technical solution adopted by the present invention is:
[0006] A retinal splitting image classification method based on fundus optical coherence tomography includes the following steps:
[0007] S1 collects retinal splitting image data and divides retinal splitting into three types in annotation: internal splitting, external splitting and retinal detachment.
[0008] S2 established the retinal splitting detection model ASFF-YoLov5;
[0009] The retinal splitting detection model includes a sequentially connected input end, a backbone network backbone and a feature fusion network ASFF-Neck.
[0010] S2.1.1 Input terminal
[0011] The image input at the input end is a retinal splitting image containing splitting information collected by S1, and the size of the retinal splitting image is normalized to obtain an image feature matrix X.
[0012] S2.1.2 Backbone network
[0013] The backbone network first increases the network depth through a three-layer cross-stage local network CSP (Cross Stage Partial Network) residual block and a five-layer convolutional batch normalization rectified linear unit CBL (Convolutional BatchNorm ReLU) convolution module, and finally uses the spatial pyramid pooling SPPF (Spatial Pyramid Pooling with Fusion) module to perform multiple maximum pooling and splicing on the convolutional features to increase the receptive field.
[0014] First, input the image feature matrix X, obtain feature C2 through two layers of CBL convolution modules, perform residual processing on C2 through the CSP_1 residual block to obtain feature C3, feature C3 passes through a layer of CBL convolution module to obtain feature C4, and feature C4 passes through CSP_2 to obtain feature C5.
[0015] The CSP residual block, denoted as CSP_k, divides the input features into two parts. One part is processed by a CBL convolution module for dimensionality reduction to obtain a low-dimensional feature map; the other part is processed by a CBL convolution module and k residual units connected in sequence to obtain a high-dimensional feature map. The two dimensional feature maps are spliced and then passed through a CBL convolution module to obtain the output of CSP_k; richer feature information is extracted. The CBL convolution module consists of a standard convolution layer, a batch normalization (BN) layer, and a ReLU activation function, which can perform basic feature extraction and processing.
[0016] Secondly, feature C5 passes through a layer of CBL and CSP_3 to obtain feature C7.
[0017] Finally, feature C7 passes through a CBL module and a SPPF module to obtain feature C9 containing rich semantic information.
[0018] The SPPF module is composed of two layers of CBL convolution modules and three layers of maximum pooling modules cascaded between the two layers of CBL convolution modules. For the input feature A1, the dimension is adjusted through a convolution layer to obtain feature A2. Three small-sized maximum poolings are used for A2 to obtain three features A3. The three A3s and A2 are concatenated to obtain feature A4, which is then passed through a layer of CBL convolution module to obtain the final feature A5. The expressive power of the feature map is enhanced by increasing the network depth and receptive field.
[0019] S2.1.3 Feature fusion network ASFF-Neck takes the three different levels of features extracted by the backbone network as input, and performs feature fusion through several CBL convolution modules and ASFF_n adaptive feature fusion layer.
[0020] Specifically, the two feature maps C5 and C7 at different levels extracted by the backbone network are respectively input into the two adaptive feature fusion layers ASFF_2 after passing through the CBL convolution module, and the new features C 10 and C 11 ; C 10 and C 11 Input CSP_1 to get feature C 12 and C 13 ; Feature C 12 , C 13 And feature C9 cross-input to three adaptive feature fusion layers ASFF_3 to obtain feature C 14 , C 15 and C 16 ; Feature C 14 , C 15 and C 16 The CBL convolution module is used to reduce the channel dimension and scale the feature map. Finally, the sigmoid activation function is used to generate the category and position information of the retinal splitting image prediction box to achieve accurate classification of the retinal splitting target.
[0021] The key idea of ASFF is to adaptively learn the feature fusion spatial weights of feature maps at each scale. The expression of ASFF_n is shown in formula (1):
[0022]
[0023] In the formula, n is the number of features to be fused, α i (i∈(1,n)) is the fusion space weight of the corresponding scale feature map in the ASFF_n module, x i is the feature map of the corresponding scale, Xout It is the feature map output by the adaptive feature fusion layer.
[0024] S3 inputs the collected retinal splitting image data into the retinal splitting detection model for training, and tests the output classification results.
[0025] Beneficial effects of the present invention:
[0026] The present invention designs a feature fusion network ASSF-Neck and proposes a retinal splitting detection algorithm based on ASSF-YoLov5. When there are larger splitting and smaller splitting in retinal medical images, it can effectively reduce the feature inconsistency between different levels, improve the effectiveness of feature fusion, and obtain better retinal splitting detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 This is the network diagram of the ASFF-YoLov5 model;
[0028] Figure 2 This is the block diagram of the ASFF module;
[0029] Figure 3 Illustrations of three types of splitting. DETAILED DESCRIPTION
[0030] The detailed implementation process of the retinal splitting image classification method based on fundus optical coherence tomography is as follows:
[0031] S1 collects retinal splitting image data
[0032] The retinal split OCT images of the present invention are collected from Hangzhou Red Cross Hospital. The data set retina-427 of the present invention includes 350 training sets and 77 test sets. The split conditions are divided into three types in the annotation: internal split, external split and retinal detachment.
[0033] S2 ASFF-YoLov5 model establishment
[0034] The network diagram of the ASFF-YoLov5 model is as follows Figure 1 As shown in the figure, the model is divided into three parts, namely the input end, the backbone network backbone, and the feature fusion network ASFF-Neck.
[0035] S2.1.1 Input terminal
[0036] The image input by the present invention is a medical image containing splitting information. The image is preprocessed to normalize the image size to obtain an image feature matrix X∈(640,640,3).
[0037] S2.1.2backbone
[0038] The backbone of the present invention increases the network depth through three-layer CSP residual blocks (Cross Stage Partial Network) and five-layer CBL (Convolutional BatchNorm ReLU) convolution modules, and uses the SPPF (SpatialPyramid Pooling with Fusion) module to perform multiple maximum pooling and splicing on the convolved features to increase the receptive field.
[0039] Input image feature matrix X∈(640,640,3). First, input image feature matrix X, obtain feature C2 through two layers of CBL convolution modules, perform residual processing on C2 through CSP_1 residual block to obtain feature C3, feature C3 passes through one layer of CBL convolution module to obtain feature C4, and feature C4 passes through CSP_2 to obtain feature C5; as shown in formula (1).
[0040] C5=CSP_2(CBL(CSP_1(CBL(CBL(X))))) (1)
[0041] Among them, CBL represents the standard convolution module, and CSP_i represents the residual module with i residual units. The CSP residual block, denoted as CSP_k, divides the input features into two parts. One part is processed by a CBL convolution module for dimensionality reduction to obtain a low-dimensional feature map; the other part is processed by a CBL convolution module and k residual units connected in sequence to obtain a high-dimensional feature map. The two dimensional feature maps are spliced and then passed through a CBL convolution module to obtain the output of CSP_k; richer feature information is extracted. The CBL convolution module consists of a standard convolution layer, a batch normalization (BN) layer, and a ReLU activation function, which can perform basic feature extraction and processing.
[0042] Secondly, feature C5 passes through a layer of CBL and CSP_3 to obtain feature C7, as shown in formula (2).
[0043] C7=CSP_3(CBL(C5)) (2)
[0044] Finally, feature C7 passes through a CBL module and a SPPF module to obtain feature C9 containing rich semantic information, as shown in formula (3).
[0045] C9=SPPF(CBL(C7)) (3)
[0046] In the formula, the SPPF module is composed of two layers of CBL convolution modules and three layers of maximum pooling modules cascaded between the two layers of CBL convolution modules. For the input feature A1, the dimension is adjusted through a convolution layer to obtain feature A2. Three small-sized maximum poolings are used for A2 to obtain three features A3. The three A3s and A2 are concatenated to obtain feature A4, and then a layer of CBL convolution module is passed to obtain the final feature A5. The expressive ability of the feature map is enhanced by increasing the network depth and receptive field.
[0047] S2.1.3ASFF-Neck
[0048] Specifically, the original YoLov5 uses a feature fusion network that cascades feature pyramids and feature fusion layers. Due to the presence of larger and smaller splits in retinal medical images, there is a conflict when fusion of feature maps at different levels, which reduces the effectiveness of feature fusion. To solve this problem, the present invention designs a new ASFF-Neck feature fusion network. The feature maps C5 and C7 of two different levels extracted by the backbone network are respectively input into two adaptive feature fusion layers ASFF_2 after passing through the CBL convolution module. The new features C5 and C7 are respectively calculated according to the weights. 10 and C 11 ; C 10 and C 11 Input CSP_1 to get feature C 12 and C 13 ; Feature C 12 , C 13 And feature C9 cross-input to three adaptive feature fusion layers ASFF_3 to obtain feature C 14 , C 15 and C 16 ; Feature C 14 , C 15 and C 16 The CBL convolution module is used to reduce the channel dimension and scale the feature map. Finally, the sigmoid activation function is used to generate the category and position information of the retinal splitting image prediction box to achieve accurate classification of the retinal splitting target.
[0049] The three different levels of feature maps extracted by the backbone are used to adaptively learn the fusion space weights of each scale, which gradually makes the feature semantic information of different levels closer, reduces the inconsistency between features at different levels, and improves the effectiveness of feature fusion.
[0050] ASFF_n module structure diagram is as follows Figure 2 As shown in Figure 4, the key idea of ASFF is to adaptively learn the feature fusion spatial weights of feature maps at each scale. The expression of ASFF_n is shown in formula (4):
[0051]
[0052] In the formula, n is the number of features to be fused, α i (i∈(1,n)) is the fusion space weight of the corresponding scale feature map in the ASFF_n module.
[0053] Three features C5, C7, and C9 at different levels are input. Since the semantic gap between features at non-adjacent levels is greater than the semantic gap between features at adjacent levels, directly fusing the three features will have a poor effect. Therefore, this patent adopts a progressive approach in the feature fusion process, which makes the semantic information between features at non-adjacent levels closer in the process of progressive fusion, thereby improving the effect of feature fusion.
[0054] S3 inputs the collected retinal splitting image data into the retinal splitting detection model for training, and tests the output classification results.
[0055] The structure of ASFF-Neck is as follows Figure 1 As shown, three features C5, C7, and C9 at different levels are input. Since the semantic gap between features at non-adjacent levels is greater than the semantic gap between features at adjacent levels, directly fusing the three features will have a poor effect. Therefore, the present invention adopts a progressive approach in the feature fusion process, which makes the semantic information between features at non-adjacent levels closer in the process of progressive fusion, thereby improving the effect of feature fusion.
[0056] S3.1 Dataset
[0057] The retinal split OCT images of the present invention were collected from the Ophthalmology Department of Hangzhou Red Cross Hospital. The dataset retina-427 of the present invention includes 350 training sets and 77 test sets. In the annotation, the split conditions are divided into three types: internal split, external split and retinal detachment. Figure 3 shown.
[0058] S3.2 Experimental environment and parameter settings
[0059] In model training, the present invention uses pytorch as the deep learning framework, and the GPU used in training is NVIDIA TITAN RTX 3090. In each training, the training batch size is 32, and the optimizer uses the adaptive moment estimation optimizer to optimize the loss function. The initial learning rate of the model is set to 0.001, and the weight decay coefficient is 1e-6.
[0060] S3.3 Model performance evaluation indicators
[0061] The present invention uses precision, recall, and mean average precision as indicators to measure model performance. Precision (P) indicates the proportion of samples predicted as positive that are also true values, recall (R) indicates the proportion of samples predicted by the model that are true values when labeled, and mean average precision (mAP) indicates the average value of the precision of all categories.
[0062] S3.3 Model detection performance
[0063] Table 1 Comparison of detection performance of different algorithms on the retina-427 test set Unit: %
[0064]
[0065] As can be seen from Table 1, compared with the SSD and YoLov5s models, the ASSF-YoLov5 model improves the precision by 17.0% and 2.7% respectively, the recall by 12.4% and 2.2% respectively, and the average mean precision by 16.9% and 3.2% respectively.
Claims
1. A method for classifying retinal splitting images based on fundus optical coherence tomography, characterized in that: The following steps are involved: S1 collects retinal splitting image data and divides retinal splitting into three types in annotation: internal splitting, external splitting and retinal detachment; S2 establishes a retinal splitting detection model; The retinoschisis detection model includes a sequentially connected input terminal, a backbone network, and a feature fusion network; S3 inputs the collected retinal splitting image data into the retinal splitting detection model for training, and tests the output classification results.
2. The method for classifying retinoschisis images based on fundus optical coherence tomography according to claim 1, characterized in that: The specific process of S2 is as follows: The image input at the input end of S2.1.1 is the retinal splitting image containing splitting information collected by S1, and the retinal splitting image is size normalized to obtain the image feature matrix X; The backbone network described in S2.1.2 first passes through a three-layer cross-stage local network CSP residual block and a five-layer convolution batch normalization rectified linear unit CBL convolution module, and then uses a spatial pyramid pooling SPPF module to perform multiple maximum pooling and splicing on the features after the CBL convolution module; S2.1.3 Feature fusion network takes the three different levels of features extracted by the backbone network as input, and performs feature fusion through several CBL convolution modules, CSP residual blocks and ASFF_n adaptive feature fusion layers.
3. The method for classifying retinoschisis images based on fundus optical coherence tomography according to claim 2, characterized in that: The CSP residual block described in S2.1.2, denoted as CSP_k, divides the input features into two parts. One part is processed by a CBL convolution module for dimensionality reduction to obtain a low-dimensional feature map; the other part is processed by a CBL convolution module and k residual units connected in sequence to obtain a high-dimensional feature map. The two dimensional feature maps are concatenated and then passed through a CBL convolution module to obtain the output of CSP_k. The CBL convolution module consists of a standard convolution layer, a batch normalization layer and a ReLU activation function; The SPPF module is composed of two layers of CBL convolution modules and three layers of maximum pooling modules cascaded between the two layers of CBL convolution modules, and the input of the latter CBL convolution module is composed of the output of each previous layer concatenated.
4. The method for classifying retinoschisis images based on fundus optical coherence tomography according to claim 3, characterized in that: The backbone network described in S2.1.2 is as follows: First, input the image feature matrix X, and obtain feature C2 through two layers of CBL convolution modules. C2 is processed by CSP_1 residual block to obtain feature C3. Feature C3 is processed by one layer of CBL convolution module to obtain feature C4. Feature C4 is processed by CSP_2 to obtain feature C5. Secondly, feature C5 passes through a layer of CBL and CSP_3 to obtain feature C7; Finally, feature C7 passes through a CBL module and a SPPF module to obtain feature C9 containing rich semantic information.
5. The method for classifying retinoschisis images based on fundus optical coherence tomography according to claim 4, characterized in that: The expression of the ASFF_n adaptive feature fusion layer is as follows: In the formula, n is the number of features to be fused, α i , i∈(1,n) is the fusion space weight of the corresponding scale feature map in the ASFF_n module, x i is the feature map of the corresponding scale, X out It is the feature map output by the adaptive feature fusion layer.
6. The method for classifying retinoschisis images based on fundus optical coherence tomography according to claim 5, characterized in that: S2.1.3 is as follows: The feature maps C5 and C7 of two different levels extracted by the backbone network are respectively input into the two adaptive feature fusion layers ASFF_2 after passing through the CBL convolution module, and the new features C are calculated according to the weights. 10 and C 11 ; C 10 and C 11 Input CSP_1 to get feature C 12 and C 13 ; Feature C 12 , C 13 And feature C9 cross-input to three adaptive feature fusion layers ASFF_3 to obtain feature C 14 , C 15 and C 16 ; Feature C 14 , C 15 and C 16 The CBL convolution module is used to reduce the number of channels and scale the feature map. Finally, the sigmoid activation function is used to generate the category and position information of the retinal splitting image prediction box to achieve the classification of retinal splitting.