Glaucoma image classification method and system based on convolutional neural network

Abstract

The present invention discloses a glaucoma image classification method and system based on a convolutional neural network, the method comprising the following steps: selecting a window with the highest average pixel value from each B-scan image as a retained image through a rolling window; combining all retained images into a three-dimensional stereo image; performing interval sampling on all retained images in the three-dimensional stereo image in the Y-axis direction to obtain a number of images after view transformation; cropping each image after view transformation to obtain a number of cropped images; using all cropped images corresponding to the same group of OCT images to be classified as inputs of a convolutional neural network, and using the output of the convolutional neural network as the classification result of the group of OCT images to be classified. The present invention solves the ambiguity problem and the problem of large data redundancy existing in the OCT image itself, and improves the classification performance of the neural network for glaucoma images of different disease courses.

Description

Technical Field

[0001] The present invention relates to the field of image classification and recognition, and in particular to a glaucoma image classification method and system based on convolutional neural network. Background Art

[0002] Glaucoma is a series of diseases characterized by optic disc atrophy and depression, visual field loss and decreased vision. The optic nerve damage caused by it is irreversible and the early symptoms are hidden. The World Health Organization report shows that glaucoma is the world's leading irreversible blinding eye disease. With the aging of my country's population, the prevalence of glaucoma has increased year by year. Glaucoma is not only common, but also seriously damages the patient's visual function and has a serious impact on the patient's quality of life.

[0003] In the early stages, glaucoma often has no obvious warning symptoms, so even if intraocular pressure increases and visual function is damaged, it is often ignored by patients. However, if the disease is allowed to develop to the middle and late stages and irreversible damage to the optic nerve occurs, it will cause permanent loss of vision in patients. Studies have shown that early treatment with drugs, lasers or surgery in the early stages of glaucoma can effectively prevent or delay the progression of the disease. Therefore, disease screening is an important means of early detection and early diagnosis of glaucoma.

[0004] The diagnosis of glaucoma is based on the unique changes in the eye structure of the optic disc, accompanied by visual field defects. OCT examination is an important way to diagnose glaucoma. With the development of artificial intelligence research, computer-assisted disease diagnosis technology based on deep learning has made great strides. If artificial intelligence technology can be better used to classify and identify OCT images, the diagnosis time of glaucoma can be greatly shortened, the screening cost can be saved to a great extent, and the work efficiency of ophthalmologists can be improved. Therefore, how to use deep learning technology to quickly and accurately classify and identify OCT images to save diagnostic time and material and human resources as much as possible is an issue that needs to be studied urgently.

[0005] In recent years, computer-assisted diagnosis of glaucoma using OCT images has been a hot topic. For example, Muhammad et al. used a deep learning method (HDLM) based on AlexNet combined with wide-field OCT to evaluate its accuracy in clinically distinguishing normal people from glaucoma patients through RNFL thickness. The research data were 102 patients with or suspected of having open-angle glaucoma, with a total of 102 eyes, which were divided into glaucoma (57 eyes) or healthy / suspicious (45 eyes) by two ophthalmologists. The HDLM algorithm can only obtain information from a single wide-field (9×12mm) scanning source OCT scan for each patient. For each subject, 6 lossless png format images were used as input to AlexNet, and the image types were: ①RGC+thickness map; ②RNFL thickness map; ③RGC+probability map; ④RNFL probability map; ⑤frontal projection; ⑥combination of images ②, ③, and ④. A "combined" image was constructed by replacing the red channel of the image with the RNFL probability value, the green channel with the RGC+probability value, and the blue channel with the normalized RNFL thickness value. The convolutional neural network AlexNet is used to extract rich features from these scanned images, and the random forest classifier is used to train a model based on these features, using the image classification results to assist doctors in understanding the extent of glaucomatous damage. Lee et al. studied the detection of glaucomatous changes by deep learning classifiers based on SD-OCT images. This study used the NASNet network architecture, with OCT optic disc and macular images as input, and the image types were: ① GCIPL thickness map; ② GCIPL deviation map; ③ RNFL thickness map; ④ RNFL deviation map. 4032 features were extracted from the above images as predictors for the deep learning classifier, and 4 independent models were trained. In each model, the features of the input image were processed by the Softmax classifier, and the output image was classified as glaucoma. The integrated features of the four images were then used to establish an "overall training model" to assist in the detection of structural changes in glaucoma with high sensitivity and specificity, which was better than the AUC of the single modality training model, but this method requires multiple models to be processed in coordination, which is time-consuming and labor-intensive.

[0006] However, the existing technical methods still have shortcomings. As glaucoma progresses, the patient's retinal nerve fiber layer (RNFL) will gradually become thinner, but due to congenital differences in RNFL between different individuals, it is very common for the RNFL thickness of healthy people to be thinner than that of glaucoma patients. The existing technology only classifies images according to RNFL thickness, without considering the ambiguity of the data. In addition, an OCT data sample contains up to 256 B-scan images, but only a few image areas contain valuable physiological information, and there are only slight differences between multiple adjacent B-scan images. The existing technology uses all OCT data, resulting in high data preprocessing costs, long neural network training time, and does not consider the problem of large data redundancy. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the convolutional neural network-based glaucoma image classification method and system provided by the present invention solve the problems that the existing glaucoma image classification method is inefficient and does not consider the ambiguity of RNFL data, resulting in low accuracy.

[0008] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0009] A glaucoma image classification method based on a convolutional neural network is provided, which comprises the following steps:

[0010] S1, obtaining OCT data of the optic cup and optic disc region in the OCT image group to be classified to obtain a plurality of B-scan images;

[0011] S2, selecting the window with the highest average pixel value from each B-scan image as the retained image by rolling the window;

[0012] S3, combining all the retained images corresponding to the current OCT image to be classified into a three-dimensional stereo image with their width as the X-axis, height as the Y-axis, and thickness as the Z-axis;

[0013] S4, performing interval sampling on all retained images in the three-dimensional stereo image in the Y-axis direction to obtain images after a plurality of view transformations;

[0014] S5, cropping each image after view transformation so that its width and height are equal, and obtaining a number of cropped images accordingly;

[0015] S6. Using all cropped images corresponding to the same group of OCT images to be classified as input of the convolutional neural network, and using the output of the convolutional neural network as the classification result of the OCT image group to be classified; wherein the classification result includes normal eye, early glaucoma, mid-stage glaucoma and late-stage glaucoma.

[0016] Furthermore, in step S1, each B-scan image has a width of 512 pixels and a height of 992 pixels; a group of OCT images to be classified includes 256 B-scan images.

[0017] Furthermore, the specific method for obtaining the retained image in step S2 is:

[0018] A rolling window with a step size of 1, a height of 320 pixels, and a width of 512 pixels is used to slide from top to bottom in each B-scan image, and the window with the highest average pixel value in each B-scan image is selected as the retained image corresponding to the B-scan image.

[0019] Furthermore, the width, height, and thickness of the three-dimensional image are 512 pixels, 320 pixels, and 256 pixels, respectively.

[0020] Furthermore, in step S4, the step length of the interval sampling is 20 pixels, and 16 images after view transformation are obtained, and the width and height of each image after view transformation are 512 pixels and 256 pixels respectively.

[0021] Furthermore, the width and height of the cropped image are both 256 pixels.

[0022] Furthermore, the convolutional neural network includes a backbone network and a classifier connected in sequence; wherein the backbone network is ResNet18 or MedicalNet; and the classifier is a Softmax classifier.

[0023] Provided is a glaucoma image classification system based on a convolutional neural network, which includes a data acquisition module, a redundancy removal module, an image combination module, an interval sampling module, an image cropping module and an image classification module;

[0024] A data acquisition module, used for acquiring OCT data of the optic cup and optic disc region in the OCT image group to be classified to obtain a plurality of B-scan images;

[0025] A redundancy removal module is used to select the window with the highest average pixel value from each B-scan image as a retained image through a rolling window;

[0026] An image combination module is used to combine all the reserved images corresponding to the current OCT image to be classified into a three-dimensional stereo image with the width of all the reserved images as the X axis, the height as the Y axis, and the thickness as the Z axis;

[0027] An interval sampling module is used for performing interval sampling on all retained images in the three-dimensional stereo image in the Y-axis direction to obtain images after a plurality of view transformations;

[0028] An image cropping module is used to crop each image after view transformation so that its width and height are equal, and a number of cropped images are obtained accordingly;

[0029] The image classification module is used to use all cropped images corresponding to the same group of OCT images to be classified as inputs of the convolutional neural network, and use the output of the convolutional neural network as the classification result of the OCT image group to be classified; wherein the classification result includes normal eyes, early glaucoma, mid-stage glaucoma and late-stage glaucoma.

[0030] Furthermore, the step size of the scroll window is 1, the height is 320 pixels, and the width is 512 pixels; the step size of the interval sampling is 20 pixels.

[0031] Furthermore, the convolutional neural network includes a backbone network and a classifier connected in sequence; wherein the backbone network is ResNet18 or MedicalNet; and the classifier is a Softmax classifier.

[0032] The beneficial effects of the present invention are as follows: the present invention uses view transformation, rolling window and interval sampling methods to transform OCT images, thereby solving the ambiguity problem and large data redundancy problem existing in the OCT images themselves, and adopts convolutional neural network to classify the transformed OCT images, thereby improving the classification performance of the neural network for glaucoma images of different disease courses. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a schematic diagram of the process of this method;

[0034] Figure 2 The following are sample images of OCT image data with ambiguity. (a) is a sample OCT image of a healthy person, and (b) is a sample OCT image of a glaucoma patient.

[0035] Figure 3 OCT images at different viewing angles; (a) and (b) are B-scan images at horizontal viewing angles; (c) and (d) are C-scan images at vertical viewing angles;

[0036] Figure 4 Schematic diagram of the retained image selected by the rolling window in the embodiment; wherein (a) is a three-dimensional stereo image corresponding to a combination of several B-scan images; and (b) is a three-dimensional stereo image corresponding to a combination of retained images. DETAILED DESCRIPTION

[0037] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0038] like Figure 1 As shown, the glaucoma image classification method based on convolutional neural network includes the following steps:

[0039] S1, obtaining OCT data of the optic cup and optic disc region in the OCT image group to be classified to obtain a plurality of B-scan images;

[0040] S2, selecting the window with the highest average pixel value from each B-scan image as the retained image by rolling the window;

[0041] S3, combining all the retained images corresponding to the current OCT image to be classified into a three-dimensional stereo image with their width as the X-axis, height as the Y-axis, and thickness as the Z-axis;

[0042] S4, performing interval sampling on all retained images in the three-dimensional stereo image in the Y-axis direction to obtain images after a plurality of view transformations;

[0043] S5, cropping each image after view transformation so that its width and height are equal, and obtaining a number of cropped images accordingly;

[0044] S6. Using all cropped images corresponding to the same group of OCT images to be classified as input of the convolutional neural network, and using the output of the convolutional neural network as the classification result of the OCT image group to be classified; wherein the classification result includes normal eye, early glaucoma, mid-stage glaucoma and late-stage glaucoma.

[0045] The glaucoma image classification system based on convolutional neural network includes a data acquisition module, a redundancy removal module, an image combination module, an interval sampling module, an image cropping module and an image classification module;

[0046] A data acquisition module, used for acquiring OCT data of the optic cup and optic disc region in the OCT image group to be classified to obtain a plurality of B-scan images;

[0047] A redundancy removal module is used to select the window with the highest average pixel value from each B-scan image as a retained image through a rolling window;

[0048] An image combination module is used to combine all the reserved images corresponding to the current OCT image to be classified into a three-dimensional stereo image with the width of all the reserved images as the X axis, the height as the Y axis, and the thickness as the Z axis;

[0049] An interval sampling module is used for performing interval sampling on all retained images in the three-dimensional stereo image in the Y-axis direction to obtain images after a plurality of view transformations;

[0050] An image cropping module is used to crop each image after view transformation so that its width and height are equal, and a number of cropped images are obtained accordingly;

[0051] The image classification module is used to use all cropped images corresponding to the same group of OCT images to be classified as inputs of the convolutional neural network, and use the output of the convolutional neural network as the classification result of the OCT image group to be classified; wherein the classification result includes normal eyes, early glaucoma, mid-stage glaucoma and late-stage glaucoma.

[0052] In this embodiment, each B-scan image has a width of 512 pixels and a height of 992 pixels; a group of OCT images to be classified includes 256 B-scan images.

[0053] The specific method for obtaining the retained image is: use a rolling window with a step size of 1, a height of 320 pixels, and a width of 512 pixels to slide from top to bottom in each B-scan image, and select the window with the highest average pixel value in each B-scan image as the retained image corresponding to the B-scan image.

[0054] The width, height and thickness of the 3D image are 512 pixels, 320 pixels and 256 pixels respectively. The step size of the interval sampling is 20 pixels, and 16 images after view transformation are obtained, and the width and height of each image after view transformation are 512 pixels and 256 pixels respectively. The width and height of the cropped image are both 256 pixels.

[0055] The convolutional neural network includes a backbone network and a classifier connected in sequence; the backbone network is ResNet18 or MedicalNet; and the classifier is a Softmax classifier.

[0056] In the specific implementation process, the ambiguity of OCT image judgment of glaucoma is as follows Figure 2 As shown, the retinal layer of healthy people is thinner than that of glaucoma patients. According to prior knowledge, Figure 2 (a) There is a greater probability of being diagnosed as a glaucoma patient, which is contrary to the actual situation. If only based on the change pattern of RNFL during the course of glaucoma, in deep learning technology, healthy people may have a greater probability of being classified as glaucoma patients, which will seriously affect the accuracy of glaucoma image classification. Based on this, Figure 3 As shown, the present invention performs view transformation on OCT data. Since A-scan and B-scan have similar imaging angles and both observe OCT data from a horizontal perspective, it is not very meaningful to convert B-scan to A-scan. Furthermore, the present invention considers converting the B-scan view to a C-scan view, transforming the perspective from the horizontal direction to the vertical direction, and obtaining a coarse-grained C-scan image from the B-scan three-dimensional data. The C-scan image has three-dimensional information of the retinal structure. After the above view transformation operation, the problem caused by the ambiguity of the data itself is effectively avoided, and it is conducive to extracting the features of glaucoma image classification under the new view. Specifically:

[0057] Since the effective physiological area in the B-scan image occupies less than half of the overall image area, the data redundancy is large. Therefore, in order to remove the redundant area in the image, the present invention uses a rolling window with a step size of 1, a height of 320 pixels, and a width of 512 pixels to slide from top to bottom in each B-scan image, and selects the window with the highest average pixel value in each B-scan image as the retained image corresponding to the B-scan image. The results are shown in Figure 2. Figure 4 shown.

[0058] The reason why the rolling window can effectively remove redundant areas in the image is that the pixel intensity index measured by the rolling window in this embodiment is highly consistent with the high-value area of ​​the image, and the spatial relationship between the high-value areas of the image can still maintain a corresponding strong correlation in the pixel distribution intensity, for example, the similarity under the spatial Euclidean distance is mapped to the Euclidean distance of the pixel intensity value.

[0059] Since multiple adjacent B-scan images are very similar, with only minor differences between them, the B-scan images in the 3D image are sampled and cropped at intervals, with a sampling interval of 20 pixels set to remove redundant similar images from the data. Each sample data after interval sampling contains only 16 images after view transformation (i.e., C-scan images), which accelerates the training and reasoning of the neural network.

[0060] In one embodiment of the present invention, in order to verify the effectiveness of the above method, relevant experiments on glaucoma image classification using B-scan images and C-scan images were carried out on the public dataset GAMMA published at the OMIA8 conference and a private dataset. The experimental results are shown in Table 1.

[0061] Table 1: Kappa value / accuracy of glaucoma image classification using B-scan images and C-scan images

[0062]

[0063] As can be seen from Table 1, whether on the public dataset GAMMA or the private dataset, and in different model architectures, the C-scan image under the vertical perspective performs better than the B-scan image under the horizontal perspective in terms of consistency check and accuracy, which proves the effectiveness of the view transformation method proposed in the present invention.

[0064] In this embodiment, in order to prove that the present invention has a better effect on glaucoma image classification, a comparative experiment was conducted using the public data set GAMMA and the relevant methods commonly used for classification. The experimental results are shown in Table 2. The training speed of the model is in s / epoch, that is, how many seconds it takes for the entire training set to be trained once through the neural network model.

[0065] Table 2: Comparative experiment between the present invention and the existing algorithm

[0066]

[0067] It can be seen from Table 2 that the accuracy of glaucoma image classification using the present invention is significantly higher than that of the existing algorithm; in addition, after data preprocessing with rolling windows and interval sampling, the model training speed of the present invention is still nearly 6.5 times faster than that of the fastest ResNet method, which greatly reduces the training cost, proving that the present invention has a better effect on glaucoma image classification.

[0068] In summary, the present invention uses view transformation, rolling window and interval sampling methods to transform OCT images, which solves the ambiguity problem and large data redundancy problem of OCT images themselves, and uses convolutional neural network to classify the transformed OCT images, thereby improving the classification performance of neural network for glaucoma images of different disease courses.

Claims

1. A glaucoma image classification method based on convolutional neural network, characterized in that: The following steps are involved: S1, obtaining OCT data of the optic cup and optic disc region in the OCT image group to be classified to obtain a plurality of B-scan images; S2, selecting the window with the highest average pixel value from each B-scan image as the retained image by rolling the window; S3, combining all the retained images corresponding to the current OCT image to be classified into a three-dimensional stereo image with their width as the X-axis, height as the Y-axis, and thickness as the Z-axis; S4, performing interval sampling on all retained images in the three-dimensional stereo image in the Y-axis direction to obtain images after a plurality of view transformations; S5, cropping each image after view transformation so that its width and height are equal, and obtaining a number of cropped images accordingly; S6, using all cropped images corresponding to the same group of OCT images to be classified as inputs of the convolutional neural network, and using the output of the convolutional neural network as the classification result of the OCT image group to be classified; wherein the classification result includes normal eye, early glaucoma, mid-stage glaucoma and late-stage glaucoma; In step S1, each B-scan image has a width of 512 pixels and a height of 992 pixels; a set of OCT images to be classified includes 256 B-scan images; The specific method for obtaining the retained image in step S2 is: A rolling window with a step size of 1, a height of 320 pixels, and a width of 512 pixels is used to slide from top to bottom in each B-scan image, and the window with the highest average pixel value in each B-scan image is selected as the retained image corresponding to the B-scan image; The width, height, and thickness of the 3D stereo image are 512 pixels, 320 pixels, and 256 pixels, respectively; In step S4, the step length of the interval sampling is 20 pixels, and 16 images after view transformation are obtained, and the width and height of each image after view transformation are 512 pixels and 256 pixels respectively; The width and height of the cropped image are both 256 pixels; The convolutional neural network includes a backbone network and a classifier connected in sequence; the backbone network is ResNet18 or MedicalNet; and the classifier is a Softmax classifier.

2. A glaucoma image classification system based on a convolutional neural network used in the glaucoma image classification method based on a convolutional neural network according to claim 1, characterized in that: It includes a data acquisition module, a redundancy removal module, an image combination module, an interval sampling module, an image cropping module and an image classification module; A data acquisition module, used for acquiring OCT data of the optic cup and optic disc region in the OCT image group to be classified to obtain a plurality of B-scan images; A redundancy removal module is used to select a window with the highest average pixel value from each B-scan image as a retained image through a rolling window; An image combination module is used to combine all the reserved images corresponding to the current OCT image to be classified into a three-dimensional stereo image with the width of all the reserved images as the X axis, the height as the Y axis, and the thickness as the Z axis; An interval sampling module is used for performing interval sampling on all retained images in the three-dimensional stereo image in the Y-axis direction to obtain images after a plurality of view transformations; An image cropping module is used to crop each image after view transformation so that its width and height are equal, and a number of cropped images are obtained accordingly; An image classification module, used to use all cropped images corresponding to the same group of OCT images to be classified as inputs of a convolutional neural network, and use the output of the convolutional neural network as a classification result of the group of OCT images to be classified; The classification results include normal eyes, early glaucoma, middle glaucoma and late glaucoma.

Images