Corneal disease image detection and classification method and device based on local and global information

Abstract

The application discloses a corneal disease image detection and classification method and device based on local and global information, and the method comprises the following steps: segmenting an image to obtain a superpixel lesion area set; extracting local features and global features; obtaining superpixel lesion area feature representation combined with local and global information; performing classification training by using global main features combined with local features, and making a prediction on a corneal disease slit lamp image; performing unsupervised clustering and category representation on the superpixel lesion area, and visually displaying abnormal lesion areas in the corneal disease slit lamp image. The corneal disease image detection and classification method and device based on local and global information can effectively extract global features and lesion area local features of the slit lamp image, perform feature fusion on the abnormal areas, realize image classification and abnormal area display.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a corneal disease image detection and classification method and device based on local and global information. BACKGROUND

[0002] Corneal diseases are one of the major causes of blindness in the world, which are manifested in clinical as obvious blurred vision, pain, light irritation and severe visual impairment. Fast clinical examination greatly depends on the professional quality of doctors and has low accuracy, while relatively accurate laboratory examination takes a long time and delays the treatment process. With the development of computer vision, it has achieved remarkable and effective practical application value in the fields of transportation and industry, and also has great application prospect in the field of medical image auxiliary diagnosis. The use of deep learning network to analyze and process corneal disease data images can provide abnormal feature area display, which can be used as an important auxiliary tool in clinical diagnosis, greatly improving the accuracy and efficiency of clinical diagnosis. At the same time, the corneal disease abnormal area feature detection device combined with deep neural network can realize the transformation from "expert sinking" to "technology sinking", effectively improve the diagnosis efficiency, relieve the diagnosis pressure of hospitals and improve the patient experience.

[0003] The lesion areas of different pathogen categories have various differences, and these subtle abnormal feature areas often highlight important discriminant indicators. Although the traditional image classification algorithm based on deep learning directly extracts features from the whole image and maps them to a one-dimensional category vector for classification has a certain feasibility, the single modeling of the global image often ignores the local information and the key feature information of the abnormal lesion area which is small and decisive. SUMMARY

[0004] The application provides a corneal disease image detection and classification method and device based on local and global information, which adopts the following technical scheme:

[0005] A corneal disease image detection and classification method based on local and global information comprises the following steps:

[0006] S1: applying a superpixel segmentation algorithm to obtain a superpixel lesion area set; using a VGG16 model to obtain basic deep features, and designing a superpixel average pooling module to extract local features and global features in combination with the superpixel lesion area set;

[0007] S2: performing sequence splicing processing on the global features and the local features in the channel dimension, using an attention module to perform information fusion, and obtaining superpixel lesion area feature representation combined with local and global information;

[0008] S3: using the global main body features combined with the local features for classification training, and making classification prediction on the corneal disease slit lamp images;

[0009] S4: Unsupervised clustering and class representation are performed on the superpixel lesion area, and abnormal lesion areas in the corneal disease slit lamp image are visualized and displayed.

[0010] Further, step S1 comprises:

[0011] S11: Circular region detection is performed on the corneal disease slit lamp image, the basic region contour of the eyeball is obtained, and the eyeball region positioning and cutting are performed according to the contour; the number of superpixel regions m is set, and a superpixel segmentation algorithm is applied to obtain mask data, which is shown in Mask,

[0012]

[0013] where h and w represent the height and width of the slit lamp image, respectively, and m u,v represents the superpixel region label to which the pixel point belongs, and the processed images are used to construct a data set data = {image i , mask i}, wherein represents the corneal disease slit lamp image, and mask represents the superpixel segmentation mask.

[0014] S12: A deep neural network feature extractor is applied to perform feature extraction on the corneal disease slit lamp image, and for an image image i , the feature extractor extracts a corresponding feature map F where c represents the feature dimension, h' represents the height of the feature map, and w' represents the width of the feature map. The deep neural network feature extractor is based on the deep convolutional network VGG16, retains the benchmark units of conv1, conv2, and conv3 in the deep convolutional network VGG16, removes the Max-pooling layers in conv4 and conv5 to maintain the resolution of the feature map, and removes the fully connected layer.

[0015] S13: The feature map F i of each image image i is upsampled by bilinear interpolation to the same resolution size as image i , to obtain a feature map F′ where h represents the height of the feature map F′ i , and w represents the width of the feature map F′ i . The superpixel average pooling unit is applied to process the feature map F′ i , and each element F′ i in the feature map F′ i(u, v) according to the superpixel label mask i (u, v) are accumulated and added to obtain the feature value of the corresponding superpixel region, and the superpixel average pooling module is represented as:

[0016]

[0017] Each picture image i The superpixel feature set SP is obtained from the above i ={sp0, sp1,..., sp m-1}; In order to keep consistent, the global average pooling is performed on the whole picture to obtain the global feature

[0018] Further, step S2 comprises:

[0019] S21: concatenating the global feature f i and the superpixel feature set SP i ={sp0, sp1,..., sp m-1} in the channel dimension to obtain a new feature set

[0020] S22: applying a local-global attention module to the feature set g i for fusion processing: the feature g i goes through the query, key, and value layer Embedding to obtain Multi-head attention processing is performed thereon, and for each head, the attention score matrix M is calculated by the following formula h_i

[0021]

[0022] and finally obtains the fused feature

[0023] S23: SP' i =g' i [1: ], which is represented as a superpixel region feature set fused with local-global information.

[0024] Further, step S3 comprises:

[0025] S31: constructing a fully connected classifier FC for g' i [0] to obtain the output representation of , wherein pre i indicates the prediction probability value of the classification result of each corneal disease of the image, and K indicates the number of corneal diseases to be predicted, and wherein g'i [0] The global feature is taken as the main body, and the overall representation of the local region feature of the superpixel of the image is fused;

[0026] S32: Cross-entropy loss is used as the loss function of the model architecture, which is defined as follows: pre i represents the corneal disease classification prediction probability vector after local-global feature fusion, label i represents the true diagnosis label of the image, j represents the jth corneal disease category, and the network is trained by minimizing the loss, so that the model can give a prediction result for the corneal disease slit image in the prediction data.

[0027] Further, step S4 comprises:

[0028] S41: All superpixel region features in each superpixel region feature set SP' i are extracted into a new data set Global_Local_SP = {glsp0, glsp1,..., glsp z}, where z represents the total number of superpixel regions in the training set;

[0029] S42: K-Means unsupervised clustering is performed on the data set Global_Local_SP, K = 30 is set to obtain the cluster center, and for each corneal disease slit lamp image image i , combined with the superpixel segmentation mask mask i , the cluster class identifier of each superpixel lesion local region on the image is obtained.

[0030] S43: For the multi-head attention matrix M, the following formula is used to calculate the abnormal lesion region matrix, and the superpixel region in the image is assigned an abnormal weight value and displayed using the following formula:

[0031] show(u, v) = M'[mask(u, v)]

[0032] That is, each superpixel region in the image is assigned and displayed according to the attention matrix mask, where show(u, v) represents the abnormal feature region display image.

[0033] A corneal disease image detection and classification device based on local and global information comprises:

[0034] A superpixel feature extraction module applies a superpixel segmentation algorithm to obtain a superpixel lesion region set; a VGG16 model is used to obtain basic deep features, and a superpixel average pooling module is designed to combine the superpixel lesion region set to extract local features and global features.​

[0035] an information fusion module, configured to perform sequence splicing processing on the global features and the local features in a channel dimension, perform information fusion by using an attention module, and obtain super-pixel lesion region feature representation combining local and global information;

[0036] a classification module, configured to perform classification training by using the global main body features combined with the local features, and make a prediction on the corneal disease slit lamp image;

[0037] an abnormal feature detection module, configured to perform unsupervised clustering and class representation on the super-pixel lesion region, and perform visual display on the abnormal lesion region in the corneal disease slit lamp image.

[0038] Further, the super-pixel division module comprises:

[0039] a corneal region positioning sub-module, configured to perform circular region detection on the corneal disease slit lamp image, obtain a basic region contour of the eyeball, and perform eyeball region positioning and cutting by slightly expanding the contour; set a super-pixel region number parameter m, and apply a super-pixel segmentation algorithm to obtain mask data, which is shown in a Mask,

[0040]

[0041] wherein h and w respectively represent the height and width of the slit lamp image, m u,v represents a super-pixel region label to which the pixel point belongs, and a data set data={image i , mask i} is constructed after all images are processed, wherein represents the corneal disease slit lamp image, represents the super-pixel segmentation mask.

[0042] a feature extraction sub-module, configured to apply a constructed deep neural network feature extractor to perform feature extraction on the corneal disease slit lamp image, and for a Zhang image image i , the feature extractor extracts a corresponding feature map wherein c represents a feature dimension, h' represents the height of the feature map, and w' represents the width of the feature map, the deep neural network feature extractor is based on a deep convolutional network VGG16, retains benchmark units of conv1, conv2 and conv3 in the deep convolutional network VGG16, removes Max-pooling layers in conv4 and conv5 to maintain the resolution of the feature map, and removes a fully connected layer;

[0043] a super-pixel lesion feature set acquisition sub-module, configured to acquire a super-pixel lesion feature set for each Zhang image image ifeature map F i is up-sampled to the same resolution size of image i by bilinear interpolation, obtaining a feature map where h represents the height of the feature map F′ i and w represents the width of the feature map F′ i A superpixel average pooling unit is applied to process the feature map F′ i Each element F′ i (u, v) in the feature map F′ i is accumulated and added according to its superpixel label mask i (u, v) to obtain the feature value of the corresponding superpixel region, and the superpixel average pooling module is represented as:

[0044]

[0045] Each picture image i The superpixel feature set SP i = {sp0, sp1,..., sp m-1} is obtained by the above; and the global feature f

[0046] Further, the information fusion module comprises:

[0047] A feature sequence data acquisition submodule is configured to splice the global feature f i and the superpixel feature set SP i = {sp0, sp1,..., sp m-1} in the channel dimension to obtain a new feature set g

[0048] An attention feature fusion submodule is configured to apply a local-global attention module to perform fusion processing on the feature set g i After the feature g i undergoes query, key, and value layer Embedding, respectively, g is obtained. Multi-head attention processing is performed on g h_i

[0049]

[0050] and finally the fused feature g′

[0051] Further, the classification module comprises:

[0052] A prediction submodule is configured to predict g′ i[0] Construct a fully connected classifier FC to model, apply LogSoftMax function to normalize, get the output expression of , wherein pre i represents the prediction probability value of the classification result of each corneal disease of the image, K represents the number of corneal disease categories to be predicted, and g′ i [0] The global feature is mainly used to fuse the local region feature of the superpixel of the image.

[0053] The network training submodule is used to use cross-entropy loss as the loss function of the model architecture, which is defined as follows: pre i represents the corneal disease classification prediction probability vector fused by the local-global feature, and label i represents the true diagnosis label of the image, j represents the jth corneal disease category, and the network is trained by minimizing the loss, so that the model can give a prediction result for the corneal disease slit image in the prediction data.

[0054] Further, the abnormal feature detection module comprises:

[0055] The local-global feature fusion superpixel feature extraction submodule is used to extract all the superpixel region features in each superpixel region feature set SP′ i to a new data set Global_Local_SP={glsp0, glsp1,..., glsp z}, wherein z represents the number of all superpixel regions in the entire training set;

[0056] The superpixel lesion unsupervised clustering submodule is used to perform K-Means unsupervised clustering on the data set Global_Local_SP, set K=30 to obtain the clustering center, and for each corneal disease slit lamp image image i , combined with the superpixel segmentation mask mask i , the clustering class identifier is performed on each superpixel lesion local region on the image.

[0057] The abnormal region positioning display submodule is used to calculate the abnormal lesion region matrix by using for the multi-head attention matrix M, and the abnormal weight value of the superpixel region in the image is assigned and displayed by using the following formula:

[0058] show(u, v)=M′[mask(u, v)]

[0059] That is, each superpixel region in the image is assigned a mask according to the attention matrix, where show(u, v) represents the abnormal feature region display image.

[0060] The application has the advantages that the provided corneal disease image detection and classification method and device based on local and global information can effectively extract global features and local features of a lesion area of a slit lamp image, perform feature fusion of an abnormal area, and realize image classification and abnormal area display.

[0061] The application also has the advantages that the provided corneal disease image detection and classification method and device based on local and global information can relieve the defect of neglecting small and important feature information of a local area and an abnormal lesion area by applying a deep neural network to model fusion combined with local-global feature information, thereby relieving the defect of single modeling of a global image. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 is a model diagram of the corneal disease image detection and classification method based on local and global information of the application;

[0063] Figure 2 is a flowchart of the corneal disease image detection and classification method based on local and global information of the application;

[0064] Figure 3 is a schematic diagram of the corneal disease image detection and classification device based on local and global information of the application;

[0065] Figure 4 is a schematic diagram of an electronic device for implementing the corneal disease image detection and classification method based on local and global information of the application;

[0066] Figure 5 is a display result schematic diagram of the application. DETAILED DESCRIPTION

[0067] The application will be specifically described below in combination with the drawings and specific embodiments.

[0068] As shown in the drawings, Figures 1-2 a corneal disease image detection and classification method based on local and global information of the application includes the following steps:

[0069] S1: a superpixel segmentation algorithm is applied to obtain a superpixel lesion area set; a VGG16 model is used to obtain basic deep features, and a superpixel average pooling module is designed to extract local features and global features in combination with the superpixel lesion area set;

[0070] S2: Global and local features are sequentially concatenated along the channel dimension, and information is fused using an attention module to obtain a superpixel lesion region feature representation that combines local and global information;

[0071] S3: Use global subject features that combine local features for classification training to make classification predictions for slit lamp images of corneal diseases;

[0072] S4: Perform unsupervised clustering and category representation on superpixel lesion regions, and visualize abnormal lesion regions in slit-lamp images of corneal diseases.

[0073] In a preferred embodiment, step S1 includes:

[0074] S11: Perform circular region detection on the slit-lamp image of corneal disease to obtain the basic contour of the eyeball, and perform small-scale outward cropping of the eyeball region based on this contour; set the superpixel region number parameter m, and apply a superpixel segmentation algorithm to obtain mask data, the representation of which is shown in Mask.

[0075]

[0076] Where h and w represent the height and width of the slit lamp image, respectively, and m u,v ∈{0, 1, ..., m-1} represents the superpixel region label to which the pixel belongs. After processing all images, a dataset data = {image i mask i},in This represents a slit-lamp image of corneal disease. This represents a superpixel segmentation mask.

[0077] S12: A constructed deep neural network feature extractor is used to extract features from slit-lamp images of corneal diseases. For each image... i The feature extractor extracts the corresponding feature map. Where c represents the feature dimension, h′ represents the height of the feature map, and w′ represents the width of the feature map, the deep neural network feature extractor is based on the deep convolutional network VGG16, retains the baseline units of conv1, conv2, and conv3 in the deep convolutional network VGG16, removes the Max-pooling layers in conv4 and conv5 to maintain the resolution of the feature map, and removes the fully connected layers.

[0078] S13: For each image i Feature map F i Adjust it to match the image by bilinear interpolation upsampling. iThe same resolution size, get feature map Where h represents the height of the feature map F' i , w represents the width of the feature map F' i , and the feature map F' i is processed by applying a superpixel average pooling unit, so that each element F' i (u, v) in the feature map F' i is accumulated and added according to its superpixel label mask i (u, v) to obtain the feature value of the corresponding superpixel region, and the superpixel average pooling module is represented as:

[0079]

[0080] Each picture image i The superpixel feature set SP i = {sp0, sp1,..., sp m-1} is obtained according to the above; and the global feature is obtained by performing global average pooling on the entire image.

[0081] As a preferred embodiment, step S2 comprises:

[0082] S21: concatenating the global feature f i and the superpixel feature set SP i = {sp0, sp1,..., sp m-1} in the channel dimension to obtain a new feature set g

[0083] S22: applying a local-global attention module to fuse the feature set g i : after the feature g i goes through the query, key, and value layer Embedding, respectively, it obtains , and the multi-head attention processing is performed thereon, and for each head, the attention score matrix M h_i is calculated by the following formula:

[0084]

[0085] and finally obtains the fused feature g'

[0086] S23: SP' i = g' i [1: ], which is represented as a superpixel region feature set fused with local-global information.

[0087] As a preferred embodiment, step S3 comprises:

[0088] S31: g′ i [0] is modeled by constructing a fully connected classifier FC, and a LogSoftMax function is applied to normalize it, obtaining the output representation , wherein pre i represents the predicted probability value of the image for the classification result of each corneal disease, K represents the number of corneal disease categories to be predicted, and g′ i [0] is mainly based on global features, and integrates the overall representation of the local region features of the image superpixels;

[0089] S32: Cross-entropy loss is used as the loss function of the model architecture, which is defined as follows: pre i represents the corneal disease classification prediction probability vector after local-global feature fusion, label i represents the true diagnosis label of the image, and j represents the jth corneal disease category. The network is trained by minimizing the loss, so that the model can give a prediction result for the corneal disease slit image in the prediction data.

[0090] As a preferred embodiment, step S4 comprises:

[0091] S41: All superpixel region features in each superpixel region feature set SP′ i are extracted into a new data set Global_Local_SP = {glsp0, glsp1,..., glsp z}, wherein z represents the total number of superpixel regions in the training set;

[0092] S42: K-Means unsupervised clustering is performed on the data set Global_Local_SP, K = 30 is set to obtain the cluster center, and for each corneal disease slit lamp image image i , combined with the superpixel segmentation mask mask i , the cluster class identifier is obtained for each superpixel lesion local region on the image.

[0093] S43: For the multi-head attention matrix M, the following formula is used to obtain the abnormal lesion region matrix, and the abnormal weight value of the superpixel region in the image is assigned and displayed using the following formula:

[0094] show(u, v) = M′[mask(u, v)]

[0095] That is, each superpixel region in the image is assigned and displayed according to the attention matrix mask, wherein show(u, v) represents the abnormal feature region display image.​

[0096] As shown in Figure 3 It is a kind of corneal disease image detection classification device based on local and global information, comprising: superpixel feature extraction module, information fusion module, classification module and abnormal feature detection module.

[0097] Specifically, the superpixel feature extraction module is used to obtain a superpixel lesion area set by applying a superpixel segmentation algorithm;The VGG16 model is used to obtain the basic deep features, and the superpixel average pooling module is designed to combine the superpixel lesion area set to extract local features and global features.Information fusion module is used to perform sequence splicing processing on the channel dimension of global features and local features, and information fusion is performed by using attention module to obtain superpixel lesion area feature representation combined with local and global information.Classification module is used to perform classification training by using global main feature combined with local feature, and make prediction for corneal disease slit lamp image.Abnormal feature detection module is used for unsupervised clustering and class representation of superpixel lesion area, and abnormal lesion area in corneal disease slit lamp image is visualized.

[0098] As a preferred embodiment, the superpixel feature extraction module comprises: corneal region positioning submodule, feature extraction submodule and superpixel lesion feature set acquisition submodule.

[0099] The corneal region positioning submodule is used for circular region detection of corneal disease slit lamp image, obtaining the basic region contour of eyeball, and positioning and cutting the eyeball region by small range expansion according to the contour;The number of superpixel region parameters m is set, and the superpixel segmentation algorithm is applied to obtain mask data, which is shown as Mask,

[0100]

[0101] Where h, w represent the height and width of slit lamp image respectively, m u，v ∈{0,1,...m-1} represents the superpixel region label to which the pixel point belongs, and the data set data={image i ,mask i} is constructed after processing all images. Corneal disease slit lamp image is represented by image Superpixel segmentation mask is represented by mask.

[0102] The feature extraction submodule is used to apply the constructed deep neural network feature extractor to extract features from the corneal disease slit lamp image.For the image image i , the feature extractor extracts the corresponding feature map wherein c represents the feature dimension, h' represents the height of the feature map, and w' represents the width of the feature map, the deep neural network feature extractor is based on a deep convolutional network VGG16, retains the benchmark units of conv1, conv2 and conv3 in the deep convolutional network VGG16, removes the Max-pooling layers in conv4 and conv5 to maintain the resolution of the feature map, and removes the fully connected layers at the same time;

[0103] The superpixel lesion feature set acquisition submodule is configured to acquire a feature map F i for each image image i The feature map F i is adjusted to the same resolution size as the image image by bilinear interpolation upsampling, to obtain a feature map F' i wherein h represents the height of the feature map F' i and w represents the width of the feature map F' i The superpixel average pooling unit is applied to process the feature map F' i Each element F' i (u, v) in the feature map F' i is accumulated and added according to the superpixel label mask i (u, v) to obtain a feature value of the corresponding superpixel region, and the superpixel average pooling module is represented as:

[0104]

[0105] Each image image i is obtained by the superpixel feature set SP i = {sp0, sp1,..., sp m-1}; and global features f

[0106] As a preferred embodiment, the information fusion module comprises a feature sequence data acquisition submodule, an attention feature fusion submodule and a feature division submodule.

[0107] The feature sequence data acquisition submodule is configured to splice the global features f i and the superpixel feature set SP i = {sp0, sp1,..., sp m-1} in the channel dimension to obtain a new feature set g

[0108] The attention feature fusion submodule is configured to apply a local-global Attention module to perform fusion processing on the feature set g i The feature g iAfter query, key, value layer Embedding, respectively get Multi-head attention processing is performed thereon, and an attention score matrix M is calculated for each head by the following formula h_i

[0109]

[0110] And finally get fusion features

[0111] The feature division sub-module divides SP′ i = g′ i [1: ], which is a superpixel region feature set that fuses local-global information.

[0112] As a preferred embodiment, the classification module comprises a prediction sub-module and a network training sub-module.

[0113] The prediction sub-module is used to model the construction of a fully connected classifier FC for g′ i [0] and normalize it by applying a LogSoftMax function to obtain the output representation of pre , wherein pre i represents the predicted probability value of the classification result of each corneal disease of the image, and K represents the number of corneal disease categories to be predicted, wherein g′ i [0] is mainly based on global features and fuses the overall representation of the superpixel local region features of the image.

[0114] The network training sub-module is used to use cross-entropy loss as the loss function of the model architecture, which is defined as follows: pre i represents the corneal disease classification prediction probability vector after local-global feature fusion, label i represents the true diagnosis label of the image, and j represents the jth corneal disease category. The network is trained by minimizing the loss, so that the model can give a prediction result for the corneal disease slit image in the prediction data.

[0115] As a preferred embodiment, the abnormal feature detection module comprises a local-global feature fusion superpixel feature extraction sub-module, a superpixel lesion unsupervised clustering sub-module, and an abnormal region positioning and display sub-module.

[0116] The local-global feature fusion superpixel feature extraction sub-module is used to extract all superpixel region features in each superpixel region feature set SP′ i to a new data set Global_Local_SP = {glsp0, glsp1,..., glspz ), where z represents the number of all superpixel regions in the entire training set.

[0117] The superpixel lesion unsupervised clustering submodule is used for K-Means unsupervised clustering of the data set Global_Local_SP, setting K = 30 to obtain the clustering center, and performing clustering class identification on each superpixel lesion in the corneal disease slit lamp image image i in combination with the superpixel segmentation mask mask i .

[0118] The abnormal region positioning and display submodule is used for calculating the abnormal lesion region matrix M' by using for the multi-head attention matrix M, and performing abnormal weight assignment and display on the superpixel regions in the image by using the following formula:

[0119] show(u, v) = M'[mask(u, v)]

[0120] That is, each superpixel region in the image is assigned and displayed according to the attention matrix corresponding mask, where show(u, v) represents the abnormal feature region display image.

[0121] As shown in Figure 4 , the application further discloses an electronic device, including a memory and a processor, the memory is used for storing one or more computer instructions, wherein the one or more computer instructions are executed by the processor to realize the foregoing corneal disease image detection and classification method based on local and global information.

[0122] Specifically, the computer program can be divided into one or more modules / units, one or more modules / units are stored in the memory 401 and executed by the processor 402, and the I / O interface transmission of data is completed by the input interface 405 and the output interface 406, so as to complete the application, one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which is used to describe the execution process of the computer program in the computer device.

[0123] The computer device can be a desktop computer, a notebook computer, a palm computer, and a cloud server, etc. The computer device can include, but is not limited to, the memory 401 and the processor 402. Those skilled in the art can understand that the embodiment is only an example of the computer device, and does not constitute a limitation on the computer device, and can include more or fewer components, or combine certain components, or different components, for example, the computer device can also include an inputter 407, a network access device, a bus, etc.

[0124] The processor 402 can be a central processing unit (CPU), and can also be other general-purpose processors 402, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor 402 can be a microprocessor or can also be any conventional processor.

[0125] The memory 401 can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. The memory 401 can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. provided on the computer device. Further, the memory 401 can include both the internal storage unit and the external storage device of the computer device. The memory 401 is used to store computer programs and other programs and data required by the computer device, and can also be used to temporarily store the output of the outputter 408.

[0126] The embodiment is tested on the corneal disease image data set provided by an ophthalmic hospital. The method is related to the practice of the four most common infectious keratitis in the clinic: bacterial keratitis, fungal keratitis, viral keratitis, and acanthamoeba keratitis. The embodiment processes the data in the corneal disease image data set, and finally obtains 3015 corneal disease images, each of which corresponds to only one of the four categories of bacterial keratitis, fungal keratitis, viral keratitis, and acanthamoeba keratitis.

[0127] The present application detects abnormal feature regions of slit lamp images of corneal diseases and judges the pathogen categories. In the pathogen category judgment, the discrimination accuracy is used for evaluation. The experimental results are shown in Table 1, and it can be seen that the classification method of the present application has good classification effect and can preliminarily judge the pathogen category to provide assistance for doctors.

[0128] Table 1 Recognition result F1 value of different corneal disease images

[0129]

[0130] In the abnormal feature area detection, the main purpose is to provide the abnormal and prominent micro-cancer area for the doctors, and the display in various categories of corneal diseases is as shown in Figure 5 It can be seen that in different categories of keratitis diseases, the present application accurately locates the abnormal feature area, providing strong auxiliary support for doctors.

[0131] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the above examples do not limit the present application in any form, and any technical solutions obtained by equivalent replacement or equivalent transformation fall within the protection scope of the present application.

Claims

1. A corneal disease image detection classification method based on local and global information, characterized by, The method comprises the following steps: S1: obtaining a superpixel lesion region set by applying a superpixel segmentation algorithm; obtaining basic deep features by using a VGG16 model, and designing a superpixel average pooling module to combine the superpixel lesion region set to extract local features and global features; S2: performing sequence splicing processing on the global features and the local features in the channel dimension, performing information fusion by using an attention module, and obtaining superpixel lesion region feature representation combined with local and global information; S3: performing classification training by using global main features combined with local features, and making classification prediction on the corneal disease slit lamp image; S4: performing unsupervised clustering and class representation on the superpixel lesion region, and visually displaying the abnormal lesion region in the corneal disease slit lamp image.

2. The corneal disease image detection and classification method based on local and global information according to claim 1, wherein the step S1 comprises: S11: performing circular region detection on the corneal disease slit lamp image, obtaining a basic region contour of the eyeball, and performing eyeball region positioning and cutting by slightly expanding the contour; setting a superpixel region number parameter m, applying a superpixel segmentation algorithm to obtain mask data, and the mask data is shown as Mask.

3. The corneal disease image detection and classification method based on local and global information according to claim 2, wherein the step S2 comprises: where h, w represent the height and width of the slit lamp image, respectively, m u,v ∈ {0, 1, …, m - 1} represents the superpixel region label to which the pixel point belongs, and all images are processed to construct a dataset data = {image i , mask i}, wherein represents a corneal disease slit lamp image, represents a superpixel segmentation mask; S12: applying the constructed deep neural network feature extractor to the corneal disease slit lamp image for feature extraction, for the zoom image image i , the feature extractor extracts the corresponding feature map where c represents the feature dimension, h' represents the height of the feature map, and w' represents the width of the feature map. The deep neural network feature extractor is based on the deep convolutional network VGG16, retains the benchmark units of conv1, conv2, and conv3 in the deep convolutional network VGG16, removes the Max-pooling layers in conv4 and conv5 to maintain the resolution of the feature map, and removes the fully connected layers. S13: For each image image i , a feature map F i is obtained by applying a convolutional neural network to image i , and then up-sampling it to the same resolution size as image by bilinear interpolation, obtaining a feature map F′ i , where h represents the height of the feature map F′ i , and w represents the width of the feature map F′ i , and a superpixel average pooling unit is applied to the feature map F′ i , so that each element F′ i (u, v) in the feature map F′ i is accumulated and added according to its superpixel label mask (u, v) to obtain a feature value corresponding to the superpixel region, and the superpixel average pooling module is represented as: Each image image i From the above, the superpixel feature set SP i = {sp0, sp1, …, sp m-1 For consistency, the global average pooling is performed on the entire image to obtain the global feature 4. The corneal disease image detection and classification method based on local and global information according to claim 3, wherein the step S3 comprises:

5. The corneal disease image detection and classification method based on local and global information according to claim 4, wherein the step S4 comprises: S21: concatenating the global feature f and the superpixel feature set SP in the channel dimension, to obtain a new feature set i and the superpixel feature set SP i = {sp0, sp1, …, sp m-1} S22: apply the local-global attention module to the feature set g i fusion processing: feature g i After query, key, and value layer Embedding, respectively get Multi-head attention processing is performed thereon, and for each head, an attention score matrix M is calculated by the following formula h_i and finally get the fusion features S23: Let SP' = SP - S i = g' i [1: ], which is represented as a set of superpixel region features that fuse local-global information. show(u,v)=M′[mask(u,v)] That is, each superpixel region in the image is assigned and displayed according to the mask corresponding to the attention matrix, wherein show(u,v) represents an abnormal feature region display image. S31: g' = g + g' (1) i [0] Construct a fully connected classifier FC to model, apply LogSoftMax function to normalize it, get The output expression, where pre i represents the prediction probability value of the classification result of each corneal disease of the image, K represents the number of corneal diseases to be predicted, and g' i [0] The global feature is the main body, and the overall representation of the local region feature of the superpixel of the image is fused. S32: Cross-entropy loss is used as the loss function of the model architecture, which is defined as follows: pre i represents the corneal disease classification prediction probability vector after local-global feature fusion, label i represents the true diagnosis label of the image, j represents the jth corneal disease category, and the network is trained by minimizing the loss, so that the model can give a prediction result for the corneal disease slit image in the prediction data. It comprises: a superpixel feature extraction module, which obtains a superpixel lesion region set by applying a superpixel segmentation algorithm; obtains basic deep features by using a VGG16 model, and designs a superpixel average pooling module to combine the superpixel lesion region set to extract local features and global features; S41: Extract all the superpixel region features in each superpixel region feature set SP' to a new data set Global_Local_SP = {glsp0, glsp1,..., glsp i}, where z represents the total number of superpixel regions in the entire training set. z} S42: K-Means unsupervised clustering is performed on the data set Global_Local_SP, K=30 is set to obtain the clustering center, and for each corneal disease slit lamp image image i , combined with the superpixel segmentation mask mask i , the local lesion of each superpixel on the image is clustered and classified. S43: For the multi-head attention matrix M, use The abnormal lesion region matrix is calculated, and the superpixel region in the image is assigned an abnormal weight and displayed using the following formula: an information fusion module, which is used for performing sequence splicing processing on the global features and the local features in the channel dimension, performing information fusion by using an attention module, and obtaining superpixel lesion region feature representation combined with local and global information; a classification module, which is used for performing classification training by using global main features combined with local features, and making prediction on the corneal disease slit lamp image; 6. A corneal disease image detection classification apparatus based on local and global information, characterized by, an abnormal feature detection module, which is used for performing unsupervised clustering and class representation on the superpixel lesion region, and visually displaying the abnormal lesion region in the corneal disease slit lamp image.

7. The corneal disease image detection and classification device based on local and global information according to claim 6, wherein the superpixel feature extraction module comprises: ​ ​ ​ ​ ​ The corneal region positioning sub-module detects a circular region of the corneal disease slit lamp image, acquires a basic region contour of the eyeball, and performs small-range outer expansion of the eyeball region positioning and cutting according to the contour; a superpixel region quantity parameter m is set, a superpixel segmentation algorithm is applied to obtain mask data, and the mask data is shown as Mask, where h, w represent the height and width of the slit lamp image, respectively, m u,v represents the superpixel region label that the pixel belongs to, and its value range is m u,v ∈{0, 1, …, m-1}, and after processing all images, a data set data = {image i , mask i} is constructed, where represents the corneal disease slit lamp image, represents the superpixel segmentation mask; The feature extraction sub-module extracts features of the corneal disease slit lamp image by using the constructed deep neural network feature extractor, and the feature extraction sub-module extracts features of the image i The feature extractor extracts the corresponding feature map Wherein c represents the feature dimension, h' represents the height of the feature map, and w' represents the width of the feature map. The deep neural network feature extractor is based on the deep convolutional network VGG16, retains the benchmark units of conv1, conv2 and conv3 in the deep convolutional network VGG16, removes the Max-pooling layer in conv4 and conv5 to maintain the resolution of the feature map, and removes the fully connected layer. The superpixel lesion feature set acquisition submodule processes each image. i Feature map F i Adjust it to match the image by bilinear interpolation upsampling. i The same resolution size yields the feature map. Where h represents the feature map F′ i The height and w represent the feature map F′ i The width of the feature map F′ i Apply superpixel average pooling units to process the feature map F′ i Each element F′ in i (u,v) is determined by its superpixel label mask. i The features of the corresponding superpixel region are obtained by summing (u,v). The superpixel average pooling module is represented as follows: Each image image i From the above, the superpixel feature set SP i = {sp0, sp1, …, sp m-1}; To maintain consistency, the global average pooling is performed on the entire image to obtain the global feature 8. The corneal disease image detection and classification device based on local and global information according to claim 7, wherein, The information fusion module comprises: The feature sequence data acquisition submodule is configured to concatenate the global feature f i and the superpixel feature set SP i ={sp0, sp1, …, sp m-1} in the channel dimension to obtain a new feature set The attention feature fusion sub-module is configured to apply a local-global attention module to the feature set g i fusion processing is performed on the feature g i After query, key, and value layer Embedding, the following are obtained Multi-head attention processing with 6 heads is performed on the feature g, and the attention score matrix M is calculated for each head by the following formula h_i and finally get the fusion features The feature division module divides the SP′ i = g′ i [1: ], which is represented as a set of superpixel region features that fuse local-global information.

9. The corneal disease image detection and classification device based on local and global information according to claim 8, wherein, The classification module comprises: a prediction sub-module, configured to predict g′ i [0] constructs a full connection classifier FC to model, and applies a LogSoftMax function to normalize the full connection classifier FC to obtain an output representation of the full connection classifier FC, wherein pre i represents a prediction probability value of the image for the classification result of each corneal disease, and K represents a preset number of corneal disease categories to be predicted, wherein g′ i [0] takes the global feature as a main body, and fuses the global representation of the local region feature of the superpixel of the image.​ The network training sub-module is configured to use cross-entropy loss as the loss function of the model architecture, which is defined as follows: pre i represents the corneal disease classification prediction probability vector after local-global feature fusion, label i represents the true diagnosis label of the image, j represents the jth corneal disease category, and the network is trained by minimizing the loss, so that the model can give a prediction result for the corneal disease slit image in the prediction data.

10. The corneal disease image detection and classification device based on local and global information according to claim 9, wherein, The abnormal feature detection module comprises: a local-global feature fusion superpixel feature extraction sub-module, configured to extract all superpixel region features in each superpixel region feature set SP′ i into a new data set Global_Local_SP = {glsp0, glsp1,..., glsp z z}, where z represents the number of all superpixel regions in the entire training set; The superpixel lesion unsupervised clustering submodule is configured to perform K-Means unsupervised clustering on the data set Global_Local_SP, set K=30 to obtain clustering centers, and perform clustering class identification on each superpixel lesion in each corneal lesion slit lamp image image i in combination with a superpixel segmentation mask mask i . The abnormal region positioning display submodule is configured to, for the multi-head attention matrix M, utilize to perform calculation to obtain an abnormal lesion region matrix, and utilize the following formula to perform abnormal weight assignment on the superpixel regions in the image and display: show(u,v)=M′[mask(u,v)] That is, each superpixel region in the image is assigned and displayed according to the mask corresponding to the attention matrix, wherein show(u,v) represents an abnormal feature region display image.

Images