An automatic partitioning and recognition method for retinopathy of prematurity based on attention mechanism and deep supervision strategy

By introducing the spatial channel attention module SACAB and deep supervision strategy in the DenseNet121 network, the accuracy of automatic partition identification of retinopathy in premature infants is solved, and the accurate classification of zones I, II and III is achieved, which improves the improvement of zones I, zones II and zones Efficiency and accuracy of ROP diagnosis.

CN112308830BActive Publication Date: 2024-11-08SUZHOU BIGVISION MEDICAL TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202011161035.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-10-27
Publication Date
2024-11-08
Estimated Expiration
2040-10-27

AI Technical Summary

Technical Problem

The prior art is difficult to accurately identify automatic partitioning of three regions of retinopathy (ROP) in premature infants, especially the recognition accuracy of areas I and II is low, and there is a lack of research on areas III, which affects the assessment of ROP severity.

Method used

Using an attention mechanism and deep supervision strategy method, the spatial channel attention module SACAB is set up in the DenseNet121 convolutional neural network and the deep supervision strategy is introduced, combining image preprocessing and data amplification to realize the color photo of retinal fundus in premature babies. The area I/II/III of the image is automatically classified and recognized.

Benefits of technology

It improves the accuracy and efficiency of automatic classification and recognition of ROP three zones, enhances the feature extraction ability of network structure, avoids model overfitting and gradient disappearance problems, and provides a better basis for subsequent ROP automatic diagnosis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112308830B_ABST
    Figure CN112308830B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses an automatic zoning and identification method for retinopathy of prematurity based on an attention mechanism and a deep supervision strategy, including image preprocessing, downsampling the two-dimensional retinal fundus color photo image to 256×256 using bilinear interpolation and performing mean reduction processing; performing online data amplification operations on the data; building a network structure by setting a spatial channel attention module SACAB in the DenseNet121 convolutional neural network and introducing a deep supervision strategy to build a network structure; training and testing the model, using the DenseNet121 convolutional neural network pre-trained on ImageNet as a pre-training model through transfer learning, and training the network structure with data in the training set. After the network structure training is completed, the performance of the network structure is tested with a test set. This application realizes the automatic classification and identification of zone I / II / III in the retinal fundus color photo images of premature infants, laying the foundation for the subsequent automatic diagnosis of ROP.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of retinal image classification methods, and specifically to an automatic zoning and identification method for retinopathy of prematurity based on an attention mechanism and a deep supervision strategy. Background Art

[0002] Retinopathy of Prematurity (ROP) is a retinal vascular proliferative disease that is one of the most dangerous and serious complications of premature infants' eyes and the leading cause of blindness in children worldwide. In most cases, ROP is curable if diagnosed and treated promptly. The International Classification of ROP (ICROP) defines three clinical examination parameters: zone, stage, and additional disease (Plus). ICROP defines clinical ROP zones as I, II, and III according to the location of the lesions. The closer the lesion is to the posterior pole, the more severe the condition. In other words, the lesions in zone I are the most severe and those in zone III are the mildest. Most ROP lesions occur in zone III, followed by zone II. Studies have shown that zone identification is particularly important for severe ROP. However, the accuracy of ophthalmologists' direct zone identification from retinal fundus color photographs is not high and is subjective. Therefore, automatic zone identification of ROP based on retinal fundus color photographs is very important.

[0003] In recent years, with the rapid development of deep learning, many deep learning methods based on convolutional neural networks (CNN) have been applied to medical image analysis. At present, the research work on ROP based on retinal fundus color photos mainly focuses on the identification of additional diseases (Plus) and the screening of ROP, but there are few studies on the partitioning of the three regions of ROP. Among them, Worrall et al. used GoogLeNet pre-trained on ImageNet to classify whether there are additional diseases in ROP. This is the first attempt to use deep neural networks to identify additional diseases. Mao Jianbo et al. used deep learning networks to segment retinal vessels and optic discs, and performed additional disease diagnosis through automatic quantitative analysis of pathological features such as vascular curvature and width. Hu Junjie et al. used the Inception-v2 network pre-trained on ImageNet to identify the presence and severity of ROP. Zhang Guoming et al. used the VGG16 network pre-trained on ImageNet to achieve ROP screening. Peng Yuanyuan et al. used the ResNet18 network pre-trained on ImageNet and combined it with the attention mechanism to achieve automatic screening of ROP. For the study of ROP zone identification, Zhao Jinfeng et al. used ResNet50 pre-trained on the Microsoft COCO dataset to automatically draw the I zone boundary on the retinal fundus color photograph to identify the ROP I zone. The algorithm first detects the optic disc and macula in the image, then locates the optic disc and macula center, and finally divides the I zone according to the ICROP standard. The severity of the disease is judged based on whether the ROP lesion is in the I zone. Although the deep learning framework for automatic identification of ROP I zone has achieved good results, there are still many limitations. First, the algorithm's identification of I zone is based on the detection of the optic disc and macula in the retinal fundus color photograph. Due to the limited field of view of a single fundus color photograph, the clinical analysis of ROP lesions often requires the collection of multiple retinal fundus color photographs from multiple perspectives, that is, the retinal fundus color photographs without the optic disc and macula at the same time cannot be analyzed using this algorithm. Secondly, the performance of the ROP I zone identification method depends on the detection accuracy of the optic disc and macula. However, the macula of newborns is not fully developed, and the corresponding macula structure is not obvious in the fundus color photos, which may lead to low accuracy in macula recognition, thus affecting the recognition accuracy of ROP zone I. Finally, the algorithm only realizes the automatic recognition of zone I, and does not involve the study of automatic recognition of zones II and III. At present, there is no report on the automatic recognition of the three zones of ROP, which is very important for assessing the severity of ROP. Summary of the invention

[0004] The present application aims to solve the above-mentioned technical problems and provide a method for automatic zoning and identification of retinopathy of prematurity based on attention mechanism and deep supervision strategy, so as to realize automatic classification and identification of zone I / zone II / zone III in color fundus images of premature infants' retina, laying the foundation for subsequent automatic diagnosis of ROP.

[0005] To achieve the above objectives, the present application discloses a method for automatic partitioning and identification of retinopathy of prematurity based on an attention mechanism and a deep supervision strategy, the method comprising:

[0006] Image preprocessing: downsample the two-dimensional retinal fundus color image to 256 × 256 using bilinear interpolation and perform mean subtraction processing; perform online data augmentation on the data;

[0007] The network structure is built by setting the spatial channel attention module SACAB in the DenseNet121 convolutional neural network and introducing a deep supervision strategy;

[0008] The training and testing of the model uses the DenseNet121 convolutional neural network pre-trained on ImageNet as a pre-training model through transfer learning, and trains the network structure with the data in the training set. After the training of the network structure is completed, the performance of the network structure is tested with the test set.

[0009] Preferably, the online data augmentation operation performed on the data in the image preprocessing specifically includes: random rotation of 30 degrees, random horizontal flipping, and random vertical flipping.

[0010] Preferably, the network structure includes a 7×7 convolutional layer, a 3×3 maximum pooling layer, a first densely connected layer, a first conversion layer, a spatial channel attention module SACAB, a second densely connected layer, a second conversion layer, a spatial channel attention module SACAB, a third densely connected layer, a third conversion layer, a spatial channel attention module SACAB, a fourth densely connected layer, a global average pooling layer, a third fully connected layer, and a third softmax output layer, wherein the spatial channel attention module SACAB and the second densely connected layer are connected with a global average pooling layer, a first fully connected layer, and a first softmax output layer, and the spatial channel attention module SACAB and the third densely connected layer are connected with a global average pooling layer, a second fully connected layer, and a second softmax output layer.

[0011] Preferably, the spatial channel attention module SACAB includes a channel attention mechanism and a spatial attention mechanism; the channel attention mechanism includes an SE compression excitation module, and the SE compression excitation module includes a compression operation and an excitation operation; the compression operation is completed through the global average pooling layer, and outputs the numerical distribution of C feature maps with a dimension of C×1×1; the excitation operation includes sequentially subjecting the numerical distribution obtained by the compression operation to a C / r×1×1 convolution to obtain a first result with a dimension of C / r×1×1, subjecting the first result to a ReLU activation function layer to output a second result with the same dimension as the first result, subjecting the second result to a C×1×1 convolution to obtain a third result with a dimension of C×1×1, subjecting the third result to a sigmoid function to obtain weight coefficients of C feature maps, performing a dot product between the input feature and the weight coefficient to obtain a dot product feature map, and adding the dot product feature map to the original input feature map to obtain a channel attention feature map; wherein C represents the number of channels and r is the compression rate.

[0012] Preferably, the spatial attention mechanism includes sequentially performing two 3×3 convolutions on the original input features to obtain two convolution feature maps, performing dimension transformation and transposition operations on the first convolution feature map to obtain a first operation result, performing dimension conversion operations on the second convolution feature map to obtain a second operation result, performing matrix multiplication on the first operation result and the second operation result, passing the result of the matrix multiplication through a softmax function to obtain a preliminary spatial attention feature map of size C×W×H, and adding the preliminary spatial attention feature map to the original input features to obtain a final spatial attention feature map, wherein C, W and H are the number of channels, width and height of the convolution feature map, respectively.

[0013] Preferably, the result of feature fusion of the channel attention feature map and the final spatial attention feature map through an averaging operation is used as the output of the spatial channel attention module SACAB.

[0014] Preferably, the deep supervision strategy is introduced after the spatial channel attention module SACAB located after the second conversion layer in the network structure, and the deep supervision strategy is introduced after the spatial channel attention module SACAB located after the third conversion layer in the network structure.

[0015] Preferably, the kernel size of the 7×7 convolutional layer is 7×7, and the step size is 2; the kernel size of the 3×3 maximum pooling layer is 3×3, and the step size is 2; the kernel size of the average pooling layer is 2×2, and the step size is 2.

[0016] Beneficial effects of this application:

[0017] 1. The automatic zoning and identification method of retinopathy of prematurity based on the attention mechanism and deep supervision strategy of the present application can obtain rich and important global and local information through the spatial channel attention module SACAB, so that the network structure can learn the correct lesion characteristics and their positional relationships. At the same time, the present invention adopts a deep supervision strategy to make full use of the shallow information of the network structure, thereby automatically classifying and identifying zone I / II / III in the color fundus images of the premature infants' retina, improving the classification performance of the classification network structure, ensuring the accuracy and efficiency of the zoning identification of retinopathy of prematurity, and laying a good foundation for the subsequent automatic diagnosis of ROP.

[0018] 2. The automatic partitioning and identification method of premature retinopathy based on the attention mechanism and deep supervision strategy of this application uses bilinear interpolation to downsample the two-dimensional retinal fundus color image to 256 × 256 and perform mean reduction processing, which can effectively prevent GPU memory overflow. At the same time, the data augmentation operation can effectively prevent the model from overfitting and enhance the generalization ability of the model.

[0019] 3. The automatic partitioning and identification method of retinopathy of prematurity based on the attention mechanism and deep supervision strategy of this application adds a spatial channel attention module SACAB after the conversion layer of the original DenseNet121 network, outputs a semantically rich feature map, and then outputs the prediction result through the last dense connection layer, global average pooling layer, fully connected layer and softmax layer. At the same time, in the network training process, a deep supervision learning strategy is introduced by adding auxiliary classifiers after the second and third spatial channel attention modules SACAB to make full use of shallow information. The spatial channel attention module SACAB is embedded in the shallower layer of the network structure. On the one hand, it avoids losing more image detail features in the shallow layer, and can obtain feature maps with important features. On the other hand, it can increase the number of convolution kernels, which can increase the depth of the network structure without consuming too much computing resources, making the network structure expression more diversified. At the same time, the introduction of deep supervision strategy can make full use of shallow information, and supervise the backbone network structure by adding auxiliary classifiers in the middle layer, thereby solving the problems of gradient disappearance and slow convergence speed during deep neural network training.

[0020] 4. In the automatic partitioning and identification method of retinopathy of prematurity based on attention mechanism and deep supervision strategy of the present application, the spatial channel attention module SACAB adopts SE compression excitation module as its channel attention mechanism, and automatically obtains the importance of each channel through learning, thereby enhancing important channel information that is beneficial to the target processing object and suppressing some irrelevant detail information. The dot product feature map obtained is then passed through the spatial attention mechanism to further select the lesion focus position and produce a more discriminative feature representation, thereby improving the feature extraction capability of the network structure.

[0021] 5. In the automatic partitioning and identification method of retinopathy of prematurity based on attention mechanism and deep supervision strategy of the present application, the size of the first two-dimensional convolution kernel in the network structure is 7×7 and the step size is 2, the sizes of other two-dimensional convolution kernels are 3×3 and 1×1 respectively, and the step size of the convolution kernel is 1 or 2. The size of the kernel of the 3×3 maximum pooling layer is 3×3 and the step size is 2, and the size of the kernel of the average pooling layer is 2×2 and the step size is 2. Through the setting of this network structure, the depth information can be merged prematurely, and the number of parameters of the network structure can be reduced and the robustness of the network structure can be enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 is a schematic diagram of the network structure in an embodiment of the present application;

[0024] Figure 2 It is a structural diagram of the spatial channel attention module SACAB in an embodiment of the present application;.

[0025] Figure 3 It is a schematic diagram of a color photograph of the retinal fundus of a premature infant identified by partition in an embodiment of the present application;.

[0026] Figure 4 It is a schematic diagram of the structure of the original DenseNet121 convolutional neural network in the embodiment of the present application;.

[0027] Figure 5 It is a structural diagram of the conversion layer in the original DenseNet121 convolutional neural network in the embodiment of the present application;.

[0028] Figure 6 It is a schematic diagram of the structure of the dense module in the original DenseNet121 convolutional neural network in the embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0030] Embodiment: A method for automatic partitioning and identification of retinopathy of prematurity based on an attention mechanism and a deep supervision strategy, the method comprising:

[0031] Image preprocessing: downsample the two-dimensional retinal fundus color image to 256 × 256 using bilinear interpolation and perform mean subtraction processing; perform online data augmentation on the data;

[0032] The network structure is built by setting the spatial channel attention module SACAB in the DenseNet121 convolutional neural network and introducing a deep supervision strategy;

[0033] For model training and testing, the DenseNet121 convolutional neural network pre-trained on ImageNet is used as a pre-training model through transfer learning, and the network structure is trained with the data in the training set. After the network structure training is completed, the performance of the network structure is tested with the test set.

[0034] With the help of the above structure, the automatic zoning and identification method of retinopathy of prematurity based on attention mechanism and deep supervision strategy of the present application can obtain rich and important global and local information through the spatial channel attention module SACAB, so that the network structure can learn the correct lesion characteristics and their positional relationships. At the same time, the present invention adopts a deep supervision strategy to make full use of the shallow information of the network structure, thereby automatically classifying and identifying zone I / II / III in the color fundus images of the premature infants' retina, improving the classification performance of the classification network structure, ensuring the accuracy and efficiency of the zoning identification of retinopathy of prematurity, and laying a good foundation for the subsequent automatic diagnosis of ROP.

[0035] As a preferred implementation of this embodiment, the online data augmentation operation performed on the data in the image preprocessing specifically includes: random rotation of 30 degrees, random horizontal flipping, and random vertical flipping.

[0036] As a preferred implementation of this embodiment, refer to Figure 1As shown, the network structure includes a 7×7 convolutional layer, a 3×3 maximum pooling layer, a first densely connected layer, a first conversion layer, a spatial channel attention module SACAB, a second densely connected layer, a second conversion layer, a spatial channel attention module SACAB, a third densely connected layer, a third conversion layer, a spatial channel attention module SACAB, a fourth densely connected layer, a global average pooling layer, a third fully connected layer, and a third softmax output layer, the spatial channel attention module SACAB and the second densely connected layer are connected with a global average pooling layer, a first fully connected layer, and a first softmax output layer, and the spatial channel attention module SACAB and the third densely connected layer are connected with a global average pooling layer, a second fully connected layer, and a second softmax output layer.

[0037] refer to Figure 2 As shown, the spatial channel attention module SACAB includes a channel attention mechanism and a spatial attention mechanism; the channel attention mechanism includes an SE compression excitation module, and the SE compression excitation module includes a compression operation and an excitation operation; the compression operation is completed through a global average pooling layer, and the numerical distribution of C feature maps with a dimension of C×1×1 is output; the excitation operation includes sequentially passing a C / r×1×1 convolution on the numerical distribution obtained by the compression operation to obtain a first result with a dimension of C / r×1×1, passing the first result through a ReLU activation function layer to output a second result with the same dimension as the first result, passing the second result through a C×1×1 convolution to obtain a third result with a dimension of C×1×1, passing the third result through a sigmoid function to obtain the weight coefficients of C feature maps, performing a dot product of the input feature and the weight coefficient to obtain a dot product feature map, and adding the dot product feature map to the original input feature map to obtain a channel attention feature map, which has important feature information. Wherein, C represents the number of channels, and r is the compression rate. In this embodiment, the C weight coefficients obtained represent the importance of each channel, and these C weight coefficients are obtained through the above-mentioned convolution operation and nonlinear layer learning. The function of the two 1×1 convolutions is to fuse the feature map features of each channel.

[0038] The spatial attention mechanism includes sequentially passing the original input features through two 3×3 convolutions to obtain two convolution feature maps, performing dimension transformation and transposition operations on the first convolution feature map to obtain the first operation result, performing dimension transformation operations on the second convolution feature map to obtain the second operation result, performing matrix multiplication on the first operation result and the second operation result, passing the result of the matrix multiplication through a softmax function to obtain a preliminary spatial attention feature map of size C×W×H, and adding the preliminary spatial attention feature map to the original input features to obtain the final spatial attention feature map, where C, W, and H are the number of channels, width, and height of the convolution feature map, respectively.

[0039] The channel attention feature map and the final spatial attention feature map are fused through an averaging operation as the output of the spatial channel attention module SACAB.

[0040] Preferably, a deep supervision strategy is introduced after the spatial channel attention module SACAB located after the second conversion layer in the network structure, and a deep supervision strategy is introduced after the spatial channel attention module SACAB located after the third conversion layer in the network structure. The advantage of this setting is that on the one hand, shallow layer information can be fully utilized, and on the other hand, the auxiliary classifier added in the middle layer can supervise the backbone network, thereby solving the problems of gradient disappearance and slow convergence speed during deep neural network training.

[0041] Preferably, the kernel size of the 7×7 convolution layer is 7×7, and the step size is 2; the kernel size of the 3×3 maximum pooling layer is 3×3, and the step size is 2; the kernel size of the average pooling layer is 2×2, and the step size is 2. The advantage of this setting is that the depth information can be merged prematurely, and the number of parameters of the network structure can be reduced and the robustness of the network structure can be enhanced.

[0042] refer to Figure 3 As shown, abnormal retinal fundus images of premature infants are divided into zone I, zone II and zone III according to the lesion characteristics and their positional relationship, as shown in (a), (b) and (c) in the figure respectively. In order to reduce the computational cost of the learning task, all two-dimensional fundus color images are downsampled to 256×256 using the bilinear interpolation method, and then the mean is subtracted to improve the difference.

[0043] Combination Figure 2 , Figure 4 , Figure 5 , Figure 6 As shown, the network structure of the present application is improved on the two-dimensional convolutional neural network DenseNet121 model. The original DenseNet121 convolutional neural network is composed of dense connection layers and conversion layers, with 121 two-dimensional convolutional layers, 1 two-dimensional maximum pooling layer, 121 two-dimensional batch normalization layers, 3 average pooling layers, 1 global average pooling layer, 1 fully connected layer, and finally connected to a softmax output layer. Among them, the dense connection layer is composed of dense modules, and the 4 dense connection layers are composed of 6, 12, 24 and 16 dense blocks respectively.

[0044] The present invention adds a spatial channel attention module SACAB after the three conversion layers of the original DenseNet121 network, outputs a semantically rich feature map, and then outputs the prediction result through the last dense connection layer, global average pooling layer, fully connected layer and softmax layer. At the same time, in the process of network structure training, a deep supervision learning strategy is introduced by adding auxiliary classifiers after the second and third spatial channel attention modules SACAB to make full use of shallow information. The spatial channel attention module SACAB is embedded in the shallower layer of the network structure. On the one hand, it avoids losing more image detail features in the shallow layer, and can obtain feature maps with important features. On the other hand, it can increase the number of convolution kernels, which can increase the depth of the network structure without consuming too much computing resources, making the network structure expression more diversified. At the same time, the introduction of a deep supervision strategy can make full use of shallow information, and the backbone network is supervised by adding auxiliary classifiers in the middle layer, thereby solving the problems of gradient disappearance and slow convergence speed in the deep neural network training process.

[0045] Based on the above-mentioned automatic partitioning and identification method of retinopathy of prematurity based on attention mechanism and deep supervision strategy, this application conducted experiments, the specific contents are as follows:

[0046] a) Database

[0047] The present invention uses 662 (82 I area, 299 II area and 281 III area) two-dimensional retinal fundus color images from 148 subjects as a data set, and uses 5-fold cross validation on the entire data set to evaluate the performance of the present invention. In order to reduce the computational cost of the learning task, all two-dimensional fundus color images are downsampled to 256×256 using a bilinear interpolation method, and then the mean is subtracted to improve the difference.

[0048] b) Data augmentation

[0049] In order to prevent overfitting of the model and enhance the generalization ability of the model, the data is augmented online during the training process to increase the diversity of the data, including random rotation of 30 degrees, random horizontal flipping, and random vertical flipping.

[0050] c) Implementation

[0051] The model was trained and tested using the Pytorch integrated environment and three NVIDIA Tesla K40 GPUs with 12GB of storage space. The model was trained by minimizing the cross entropy loss using the backpropagation algorithm, and the Adam optimizer was used to minimize the cost function. The basic learning rate and weight decay were both set to 0.0001. The batch size was set to 16 and the number of epochs was set to 40.

[0052] d) Evaluation indicators.

[0053] In order to quantitatively evaluate the performance of the present invention, three commonly used classification evaluation indicators are used, including accuracy, weighted F1 score, and consistency test coefficient (Kappa). The definitions of accuracy, weighted F1 score, and consistency test coefficient (Kappa) are as follows:

[0054] ;

[0055] ;

[0056] ;

[0057] Where TP, FP, TN and FN represent true positive, false positive, true negative and false negative respectively, W_P and W_R represent weighted precision and weighted recall respectively, Po is the observed consistency rate, and Pe is the opportunity consistency rate.

[0058] (4) Verification and results:

[0059] The present invention uses 5-fold cross validation on the entire dataset to evaluate and compare the original DenseNet121 with the deep attention network proposed in the present invention. In order to demonstrate the effectiveness of SACAB and deep supervised learning strategies, a series of ablation experiments were conducted. The experimental results are shown in Table 1.

[0060] Table 1.

[0061]

[0062] In Table 1, "baseline network" represents the original DenseNet121 convolutional neural network, "baseline network + DANet" represents the addition of a dual attention DANet module to the original DenseNet121 convolutional neural network, "baseline network + SACAB" represents the addition of SACAB to the original DenseNet121 convolutional neural network, "baseline network + deep supervision" represents the introduction of a deep supervision learning strategy in the original DenseNet121 convolutional neural network, and "proposed method" represents the method proposed in this application, namely "baseline network + SACAB + deep supervision". It can be seen that the classification accuracy of the original DenseNet121 convolutional neural network is 85.66%, and the classification accuracy of the present invention can reach 88.52%. The weighted F1 score and consistency test coefficient Kappa of the present invention are 88.50% and 86.99%, respectively, which are 2.86% and 3.16% higher than the original DenseNet121 convolutional neural network, respectively. It can be seen from Table 1 that the spatial channel attention module SACAB module designed in the present invention and the referenced deep supervision strategy can effectively improve the classification accuracy of the original DenseNet121 convolutional neural network.

[0063] In summary, the automatic partitioning and identification method of retinopathy of prematurity based on attention mechanism and deep supervision strategy of the present application, that is, the automatic partitioning and identification method of fundus color photograph images ROP of retinopathy of prematurity, has been realized and verified. Its performance in the experiment is better than the original DenseNet121 convolutional neural network. Therefore, the present application can make a better judgment on the two-dimensional retinal fundus color photograph images. On the other hand, the spatial channel attention module SACAB designed in the present application is not complicated and can be embedded in any other convolutional neural network, so that the feature extraction ability of the network structure is stronger, thereby improving the overall performance of the network structure, which is helpful for the classification and detection of two-dimensional retinal fundus color photograph images, and greatly improves the screening efficiency of two-dimensional retinal fundus color photograph images. The present invention combines image preprocessing, the construction and training of deep neural network models, and testing, which greatly helps the subsequent research on retinopathy of prematurity, such as lesion area segmentation, registration, and automatic diagnosis research of ROP.

[0064] The above description is intended to be illustrative and not limiting. Upon reading the above description, many embodiments and many applications beyond the examples provided will be apparent to those skilled in the art. Therefore, the scope of the present teachings should not be determined with reference to the above description, but should be determined with reference to the appended claims and the full scope of equivalents to which such claims are entitled. For the purpose of comprehensiveness, all articles and references, including disclosures of patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the preceding claims is not intended to be a waiver of such subject matter, nor should it be considered that the applicant has not considered such subject matter to be part of the disclosed application subject matter.

Claims

1. An automatic partitioning and identification method for retinopathy of prematurity based on attention mechanism and deep supervision strategy, characterized in that: The method includes: Image preprocessing: downsample the two-dimensional retinal fundus color image to 256×256 using bilinear interpolation and perform mean subtraction processing; perform online data augmentation operations on the data; The network structure is built by setting the spatial channel attention module SACAB in the DenseNet121 convolutional neural network and introducing a deep supervision strategy to build the network structure; Training and testing of the model: using the DenseNet121 convolutional neural network pre-trained on ImageNet as a pre-training model through transfer learning, and training the network structure with data in the training set. After the training of the network structure is completed, the performance of the network structure is tested with a test set; Among them, the spatial channel attention module SACAB includes a channel attention mechanism and a spatial attention mechanism; the channel attention mechanism includes an SE compression excitation module, and the SE compression excitation module includes a compression operation and an excitation operation; the compression operation is completed through a global average pooling layer, and outputs the numerical distribution of C feature maps with a dimension of C×1×1; the excitation operation includes sequentially subjecting the numerical distribution obtained by the compression operation to a C / r×1×1 convolution to obtain a first result with a dimension of C / r×1×1, subjecting the first result to a ReLU activation function layer to output a second result with the same dimension as the first result, subjecting the second result to a C×1×1 convolution to obtain a third result with a dimension of C×1×1, subjecting the third result to a sigmoid function to obtain weight coefficients of C feature maps, performing a dot product between the input feature and the weight coefficient to obtain a dot product feature map, and adding the dot product feature map to the original input feature map to obtain a channel attention feature map; wherein C represents the number of channels and r is the compression rate.

2. The automatic partitioning and identification method for retinopathy of prematurity based on attention mechanism and deep supervision strategy according to claim 1 is characterized in that: The online data augmentation operation performed on the data in the image preprocessing specifically includes: random rotation of 30 degrees, random horizontal flipping, and random vertical flipping.

3. The automatic partitioning and identification method for retinopathy of prematurity based on attention mechanism and deep supervision strategy according to claim 1 is characterized in that: The network structure includes a 7×7 convolutional layer, a 3×3 maximum pooling layer, a first densely connected layer, a first conversion layer, a spatial channel attention module SACAB, a second densely connected layer, a second conversion layer, a spatial channel attention module SACAB, a third densely connected layer, a third conversion layer, a spatial channel attention module SACAB, a fourth densely connected layer, a global average pooling layer, a third fully connected layer, and a third softmax output layer, which are arranged in sequence. The spatial channel attention module SACAB and the second densely connected layer are connected with a global average pooling layer, a first fully connected layer, and a first softmax output layer. The spatial channel attention module SACAB and the third densely connected layer are connected with a global average pooling layer, a second fully connected layer, and a second softmax output layer. The first conversion layer, the second conversion layer, and the third conversion layer include an average pooling layer.

4. The automatic partitioning and identification method for retinopathy of prematurity based on attention mechanism and deep supervision strategy according to claim 3 is characterized in that: The spatial attention mechanism includes sequentially performing two 3×3 convolutions on the original input features to obtain two convolution feature maps, performing dimension transformation and transposition operations on the first convolution feature map to obtain a first operation result, performing dimension transformation operations on the second convolution feature map to obtain a second operation result, performing matrix multiplication on the first operation result and the second operation result, passing the result of the matrix multiplication through a softmax function to obtain a preliminary spatial attention feature map of size C×W×H, and adding the preliminary spatial attention feature map to the original input features to obtain a final spatial attention feature map, wherein C, W and H are the number of channels, width and height of the convolution feature map, respectively.

5. The method for automatic partitioning and identification of retinopathy of prematurity based on attention mechanism and deep supervision strategy according to claim 4 is characterized in that: The result of feature fusion of the channel attention feature map and the final spatial attention feature map through an averaging operation is used as the output of the spatial channel attention module SACAB.

6. The method for automatic partitioning and identification of retinopathy of prematurity based on attention mechanism and deep supervision strategy according to claim 5 is characterized in that: The deep supervision strategy is introduced after the spatial channel attention module SACAB located after the second conversion layer in the network structure, and the deep supervision strategy is introduced after the spatial channel attention module SACAB located after the third conversion layer in the network structure.

7. The method for automatic partitioning and identification of retinopathy of prematurity based on attention mechanism and deep supervision strategy according to claim 3 is characterized in that: The kernel size of the 7×7 convolutional layer is 7×7, and the step size is 2; the kernel size of the 3×3 maximum pooling layer is 3×3, and the step size is 2; the kernel size of the average pooling layer is 2×2, and the step size is 2.

Citation Information

Patent Citations

  • Attention mechanism-based in-depth learning diabetic retinopathy classification method

    CN108021916A

  • Semi-supervised semantic segmentation method for self-attention adversarial learning

    CN111027575A

  • Yellow River ice semantic segmentation method based on multi-attention mechanism double-flow fusion network

    CN111160311A

  • Premature infant retinal image classification method and device based on attention mechanism

    CN111259982A