A mobile-based fundus image classification method and system

Abstract

The present invention discloses a mobile-based fundus image classification method and system, which relates to the field of computer technology. The method comprises: constructing a basic fundus image recognition layer based on DenseNet; constructing an image classifier, and constructing a basic fundus image classification model based on the basic fundus image recognition layer and the image classifier; obtaining a mobile-side fundus image classification model through knowledge distillation based on the basic fundus image classification model; collecting and acquiring fundus images of target users to obtain target fundus images; performing data enhancement and expansion on the target fundus images to obtain a target fundus image set; and inputting the target fundus image set into the mobile-side fundus image classification model to obtain image classification results. The present invention solves the technical problems in the prior art of poor timeliness of retinal fundus image recognition and classification, inconvenience in convenient use on mobile terminals, and single function, thereby achieving the technical effect of improving the timeliness and accuracy of fundus image recognition and classification.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a mobile-based fundus image classification method and system. Background Art

[0002] Retinal fundus images are an important means of observing and diagnosing ophthalmic diseases. With the development of computer technology, computers and software have empowered the medical industry, and have also contributed to a certain role in the field of retinal fundus image analysis, serving as an auxiliary means to improve the efficiency and effectiveness of image analysis.

[0003] Currently, deep learning is generally used to build more complex network models to analyze and process retinal fundus images, and the analysis and processing is generally targeted at a single disease type.

[0004] The existing technology for analyzing and processing retinal fundus images requires a complex network model, resulting in low computational efficiency and long response time. It is not possible to perform lightweight and fast image analysis and processing on mobile devices. In addition, the technology generally only analyzes and processes a single disease based on fundus images, resulting in technical problems such as large computing resource usage, poor timeliness and convenience, and relatively single functions. Summary of the Invention

[0005] The purpose of this application is to provide a mobile-based fundus image classification method and system to solve the technical problems in the existing technology of retinal fundus image analysis and processing, such as large computing resource usage, poor timeliness and convenience, and relatively single functions.

[0006] In view of the above problems, the present application provides a mobile-based fundus image classification method and system.

[0007] In the first aspect, the present application provides a mobile-based fundus image classification method, the method comprising: constructing a basic fundus image recognition layer based on DenseNet; constructing an image classifier, and constructing a basic fundus image classification model based on the basic fundus image recognition layer and the image classifier; obtaining a mobile-end fundus image classification model through knowledge distillation based on the basic fundus image classification model; collecting and acquiring fundus images of target users to obtain target fundus images; performing data enhancement and expansion on the target fundus images to obtain a target fundus image set; inputting the target fundus image set into the mobile-end fundus image classification model to obtain image classification results.

[0008] On the other hand, the present application also provides a mobile-based fundus image classification system for executing a mobile-based fundus image classification method as described in the first aspect, wherein the system includes: a basic recognition layer construction module for constructing a basic fundus image recognition layer based on DenseNet; a basic recognition model construction module for constructing an image classifier, and constructing a basic fundus image classification model based on the basic fundus image recognition layer and the image classifier; a mobile-end recognition model construction module for obtaining a mobile-end fundus image classification model through knowledge distillation based on the basic fundus image classification model; a fundus image acquisition module for acquiring fundus images of target users to obtain target fundus images; an image enhancement and expansion module for performing data enhancement and expansion on the target fundus images to obtain a target fundus image set; and an image recognition and classification module for inputting the target fundus image set into the mobile-end fundus image classification model to obtain image classification results.

[0009] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0010] The technical solution provided in this application is based on DenseNet, which constructs a basic fundus image recognition layer with large computing resources locally, and can ensure the maximum information transmission between each layer in the network. By densely connecting the feature maps of all network layers, feature reuse is achieved, the efficiency and accuracy of image analysis and processing are improved, and an image classifier is constructed to achieve classification, recognition and prediction of multiple diseases, and improve the functional dimension of the combination of retinal fundus images and deep learning. Then, through knowledge distillation and the Teacher-Student mode, a lightweight but high-performance mobile fundus image classification model is obtained, which realizes fast, intelligent and accurate fundus image recognition and analysis processing on the mobile terminal. Reduce the computing resource usage of the model on the mobile terminal, improve the response efficiency, then collect and obtain the retinal fundus image of the current user, perform data enhancement and expansion, improve the image data dimension, and thus improve the accuracy of image data analysis and processing, input the mobile terminal fundus image classification model, and obtain the image classification result of the retinal fundus image of the current user. This application realizes more complex fundus image analysis and processing based on deep learning on the mobile terminal, which is more lightweight and convenient, and can perform image recognition and classification of multiple diseases, with richer functions, achieving the technical effect of reducing the computing resource usage of fundus image recognition processing, improving the timeliness and convenience of mobile terminal use, and enriching the fundus image recognition and processing functions.

[0011] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in this application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and a person of ordinary skill in the art can obtain other drawings based on the provided drawings without any creative work.

[0013] Figure 1 A flowchart of a mobile-based fundus image classification method provided in an embodiment of the present application;

[0014] Figure 2 A schematic diagram of the basic network structure of the basic fundus image recognition layer in a mobile-based fundus image classification method provided in an embodiment of the present application;

[0015] Figure 3 A schematic diagram of a process for obtaining a basic fundus image classification model in a mobile-based fundus image classification method provided in an embodiment of the present application;

[0016] Figure 4 A schematic diagram of a process for obtaining a mobile-terminal fundus image classification model in a mobile-terminal fundus image classification method provided in an embodiment of the present application;

[0017] Figure 5 A schematic structural diagram of a mobile-based fundus image classification system provided in an embodiment of the present application.

[0018] Explanation of the accompanying drawings: basic recognition layer construction module 11, basic recognition model construction module 12, mobile terminal recognition model construction module 13, fundus image acquisition module 14, image enhancement and expansion module 15, image recognition and classification module 16. DETAILED DESCRIPTION

[0019] The present application provides a mobile-based fundus image classification method and system to solve the technical problems in the prior art of retinal fundus image analysis and processing, such as large computing resource usage, poor timeliness and convenience, and relatively single functions.

[0020] The acquisition, storage, use, and processing of data in this application's technical solution comply with relevant national laws and regulations.

[0021] Below, the technical solutions in this application will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this application, rather than all the embodiments of this application. It should be understood that this application is not limited to the example embodiments described herein. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. It should also be noted that, for the convenience of description, only the parts related to this application, rather than all of them, are shown in the accompanying drawings.

[0022] Example 1

[0023] like Figure 1 As shown, the present application provides a mobile-based fundus image classification method, the method comprising:

[0024] S100: Based on DenseNet, builds a basic fundus image recognition layer;

[0025] In an embodiment of the present application, based on DenseNet in deep learning, a mapping path is created from the shallow layer to the deep layer of the network. Under the premise of ensuring maximum information transmission between layers in the network, feature reuse can be achieved by densely connecting the feature maps of all network layers, thereby improving image processing efficiency and quality.

[0026] Step S100 of the method provided in the embodiment of the present application includes:

[0027] S110: Acquire multiple fundus images of multiple disease types to obtain multiple sample fundus images;

[0028] S120: performing data enhancement and expansion on the multiple sample fundus images to obtain multiple sample fundus image sets;

[0029] S130: performing feature data identification on the plurality of sample fundus image sets respectively to obtain a plurality of sample feature data sets;

[0030] S140: Constructing a basic network structure of the basic fundus image recognition layer based on DenseNet, wherein the basic network structure includes:

[0031] X i =H i ([X0, X1, ..., X i-1 ])

[0032] Among them, X i is the output of layer i, [X0,X1,…,X i-1 ] is X0 to X i-1 The concatenation of all output feature vectors of the layer, H iis the nonlinear transformation performed on the i-th layer, and the nonlinear transformation H is a combination of BN+ReLU+Conv(3×3);

[0033] S150: Using the multiple sample fundus image sets and the multiple sample feature data sets, supervised training is performed on the basic fundus image recognition layer until convergence or the accuracy reaches a preset requirement.

[0034] Specifically, fundus images of multiple diseases of multiple categories are collected and acquired, where, illustratively, the multiple diseases include: dry age-related macular degeneration, wet age-related macular degeneration, branch retinal vein occlusion, central retinal vein occlusion, central serous chorioretinopathy, non-proliferative diabetic retinopathy, proliferative diabetic retinopathy, macular epiretinal membrane, macular hole, vitreous opacity, vitreous hemorrhage, pathological myopia, optic disc-related disease, retinitis pigmentosa, peripheral retinal degeneration, retinal detachment, other, and poor quality. The multiple diseases mentioned above are all diseases that can be diagnosed and confirmed based on retinal fundus images.

[0035] Based on the medical records of multiple patients who have previously been treated for multiple types of diseases, multiple fundus images of multiple patients can be collected to obtain multiple sample fundus images.

[0036] Preferably, the above-mentioned multiple retinal fundus images are collected based on micro-clear CRO. Micro-clear CRO uses a 532nm wavelength laser and a 785nm wavelength laser to combine the laser images of the two wavelengths into a multi-color photo. It has strong penetrating ability and is not easily absorbed / reflected by the refractive medium. The edge collection obtains retinal fundus images with fundus lesion tissue characteristics at different levels.

[0037] Data enhancement and expansion are performed on the multiple sample fundus images, wherein, illustratively, the multiple sample fundus images are respectively rotated, horizontally flipped, vertically inverted, and scaled to obtain multiple processed sample fundus images, and clustering is performed to obtain multiple sample fundus image sets, which can improve the data dimension of the retinal fundus images used to construct the model, thereby improving the accuracy and generalization of the model.

[0038] Feature data identification is performed on multiple sample fundus image sets, wherein, based on the data identification, data identification is performed on the original retinal fundus images and the processed retinal fundus images in the multiple sample fundus image sets respectively to obtain multiple sample feature data sets as data for constructing the model.

[0039] Based on DenseNet in deep learning, the basic network structure of the basic fundus image recognition layer is constructed. The basic network structure includes:

[0040] Xi =H i ([X0, X1, ..., X i-1 ])

[0041] Among them, X i is the output of layer i, [X0,X1,…,X i-1 ] is X0 to X i-1 The concatenation of all output feature vectors of the layer, H i The nonlinear transformation H is a combination of BN+ReLU+Conv(3×3). The nonlinear transformation H is also called Denseblock.

[0042] Figure 2 A possible basic network structure of the basic fundus image recognition layer in the embodiment of the present application is shown as follows: Figure 2 In the basic network structure of the basic fundus image recognition layer, the number of Denseblocks in the first layer N1 = 6, the number of Denseblocks in the second layer N2 = 12, the number of Denseblocks in the third layer N3 = 64, and the number of Denseblocks in the fourth layer N4 = 48.

[0043] Based on the constructed basic fundus image recognition layer, the basic fundus image recognition layer is supervised trained using the above-mentioned multiple sample fundus image sets and multiple sample feature data sets until the training converges or the accuracy reaches the preset requirements, which can be an accuracy of 80% or 90%, etc.

[0044] Based on the trained basic fundus image recognition layer, the user's retinal fundus image set is input into it, and after multi-layer convolution processing and analysis, it can finally output relatively accurate corresponding multiple feature data.

[0045] S200: constructing an image classifier, and constructing a basic fundus image classification model based on the basic fundus image recognition layer and the image classifier;

[0046] Specifically, in order to realize the identification and classification of multiple diseases based on retinal fundus images, the embodiment of the present application also constructs an image classifier for classifying and identifying multiple diseases, which is used to perform classification and identification based on multiple feature data output by the basic fundus image recognition layer, and finally obtain the disease category corresponding to the user.

[0047] like Figure 3 As shown, step S200 in the method provided in the embodiment of the present application includes:

[0048] S210: Randomly selecting M sample feature data sets from the multiple sample feature data sets with replacement based on a random forest algorithm as a first constructed data set;

[0049] S220: Using the first constructed dataset to construct a first sub-model of the image classifier;

[0050] S230: Randomly selecting M sample feature data sets from the plurality of sample feature data sets again with replacement as a second constructed data set;

[0051] S240: Using the second constructed data set to construct a second sub-model of the image classifier;

[0052] S250: Continue to construct multiple sub-models of the image classifier, integrate the multiple sub-models, and obtain the image classifier;

[0053] S260: Constructing the basic fundus image classification model according to the basic fundus image recognition layer and the image classifier.

[0054] In the embodiment of the present application, the image classifier is constructed based on the random forest algorithm.

[0055] Specifically, M sample feature data sets are randomly selected with replacement from the plurality of sample feature data sets as a first construction data set to construct a first sub-model of the image classifier. M is a positive integer less than the number of the plurality of sample feature data sets, preferably 2 / 3 of the number of the plurality of sample feature data sets.

[0056] Step S220 in the method provided in the embodiment of the present application includes:

[0057] S221: Randomly select a sample feature data in the first constructed data set as a first partitioning threshold;

[0058] S222: Constructing a first-level classification node of the first sub-model based on the first division threshold, wherein the first-level classification node can perform binary classification on the input feature data;

[0059] S223: Randomly select a sample feature data in the first constructed data set again as a second segmentation threshold;

[0060] S224: constructing a secondary classification node of the first sub-model based on the second division threshold, wherein the secondary classification node can perform binary classification on the classification result of the primary classification node;

[0061] S225: Based on the first constructed data set, continue to construct the multi-level classification nodes of the first sub-model;

[0062] S226: Acquire label values ​​of the plurality of sample fundus images to obtain a plurality of label values;

[0063] S227: Obtain multiple classification results obtained by performing multi-level binary classification on the multi-level classification node, and set different disease classification results for the multiple classification results according to the multiple label values ​​to obtain the first sub-model.

[0064] Specifically, based on the random forest algorithm, a sample feature data is randomly selected in the above-mentioned first constructed data set as the first partitioning threshold, and based on the first partitioning threshold, a first-level classification node of the first sub-model is constructed, wherein the first-level classification node can perform binary classification on the feature data input into the first sub-model, and divide the feature data greater than the first partitioning threshold into one category, and divide the feature data not greater than the first partitioning threshold into another category, so as to obtain a binary classification result.

[0065] A sample feature data is randomly selected again in the above-mentioned first constructed data set as the second partitioning threshold. Based on the second partitioning threshold, a secondary classification node of the first sub-model is constructed, wherein the secondary classification node can further perform binary classification on the binary classification result of the first classification node to obtain four classification results.

[0066] In this way, the multi-level classification nodes of the first sub-model are continuously constructed until the number of levels of the multi-level classification nodes reaches a preset value, wherein the preset value can be set according to the number of sample feature data in the first constructed data set and the number of disease classifications.

[0067] Based on the multi-level classification nodes, multiple feature data input can be subjected to multiple binary classifications to obtain multiple classification results. Each classification result includes a feature data interval. The label values ​​of multiple sample fundus images are collected to obtain multiple label values. The label value is the true value of whether the user has a certain disease among the above-mentioned multiple diseases according to the actual diagnosis of the multiple sample fundus images.

[0068] By using the multiple label values ​​and setting different label values ​​for the multiple classification results according to the characteristic data in the multiple classification results, the disease classification results of the multiple classification results are obtained, and the first sub-model is constructed.

[0069] In this way, M sample feature data sets are randomly selected with replacement from the multiple sample feature data sets to serve as the second constructed data set. The second constructed data set is then used to construct a second sub-model of the image classifier. The first sub-model and the second sub-model are constructed using different data but the same construction method. Therefore, the specific performance of the first sub-model and the second sub-model is different.

[0070] Continue to build multiple sub-models of the image classifier, obtain multiple sub-models with different construction data and relatively poor performance, integrate multiple sub-models with relatively poor performance, and obtain an image classifier with relatively good performance and greater stability.

[0071] Based on the constructed image classifier, the feature data obtained by the basic fundus image recognition layer based on the fundus image analysis and processing is input into it, and then the feature data is input into multiple sub-models. After analysis, processing and classification by multiple sub-models, multiple disease classification results are obtained. Combining the disease classification results of multiple sub-models, the probability of the fundus image corresponding to the input feature data suffering from various diseases can be obtained as reference data to complete the recognition processing and multi-disease classification of retinal fundus images, which is more accurate and intelligent.

[0072] Based on the constructed basic fundus image recognition layer and image classifier, the above-mentioned basic fundus image classification model is further constructed.

[0073] Step S260 in the method provided in the embodiment of the present application includes:

[0074] S261: Connecting the basic fundus image recognition layer and the image classifier through a global AvgPooling layer;

[0075] S262: Iteratively train the connected basic fundus image recognition layer and the image classifier according to a basic model loss function to obtain the basic fundus image classification model, wherein the basic model loss function is as follows:

[0076]

[0077]

[0078]

[0079] y_pred m =round(output m )

[0080] diff = y_true - y_pred

[0081] Among them, output m is the probability of the mth disease type output by the image classifier, round is the rounding function, y_pred m is the rounded output m The value of y_true is the actual label value of the mth disease in onehot encoding form, diff is the difference between y_true and y_pred, sum i(x) is the number of values ​​i in the statistic x, and muliti_loss is the loss function of the basic model.

[0082] Specifically, based on the constructed basic fundus image recognition layer and image classifier, the basic fundus image recognition layer and image classifier are passed through the global AvgPooling layer. Among them, the basic fundus image recognition layer has undergone preliminary supervised training and has not been supervised learned together with the image classifier. The accuracy is low. Therefore, it is necessary to further iteratively train the connected basic fundus image recognition layer and image classifier.

[0083] Specifically, the connected basic fundus image recognition layer and image classifier are iteratively trained based on the basic model loss function, and the basic model loss function is as follows:

[0084]

[0085]

[0086]

[0087] y_pred m =round(output m )

[0088] diff = y_true - y_pred

[0089] Among them, output m is the probability of the mth disease type output by the image classifier, round is the rounding function, y_pred m is the rounded output m The value of y_true is the actual label value of the mth disease in onehot encoding form, diff is the difference between y_true and y_pred, sum i (x) is the number of values ​​i in the statistic x, and muliti_loss is the basic model loss function.

[0090] During the iterative training process, the connected basic fundus image recognition layer and image classifier are used to perform error calculation according to the above-mentioned basic model loss function. The above-mentioned multiple sample fundus image sets and the true label values ​​of multiple diseases are used. The training is iteratively trained until fitting, and the training is completed to obtain the constructed basic fundus image classification model.

[0091] The embodiment of the present application constructs a basic fundus image recognition layer based on DenseNet and an image classifier based on the random forest algorithm to obtain a basic fundus image classification model, which can ensure the maximum analysis and acquisition of characteristic information of retinal fundus images, obtain characteristic data, and further perform classification and identification of multiple diseases, set corresponding loss functions, and after iterative training, have a high accuracy rate, can realize the recognition and processing of retinal fundus images, and perform classification and identification of multiple diseases, with richer functions.

[0092] S300: Obtaining a mobile fundus image classification model through knowledge distillation based on the basic fundus image classification model;

[0093] like Figure 4 As shown, step S300 in the method provided in the embodiment of the present application includes:

[0094] S310: constructing a mobile-end network structure of a mobile-end fundus image recognition layer in the mobile-end fundus image classification model based on DenseNet, wherein the number of nonlinear transformations in the mobile-end network structure is less than the number of nonlinear transformations in the basic network structure;

[0095] S320: Connecting the mobile terminal fundus image recognition layer and the image classifier to obtain the mobile terminal fundus image classification model;

[0096] S330: Iteratively train the mobile-end fundus image classification model according to the mobile-end loss function and the basic fundus image classification model, wherein the mobile-end loss function is as follows:

[0097] L=αL t +(1-α)L s

[0098]

[0099]

[0100] Among them, L t is the teacher loss of the basic fundus image classification model, L s is the student loss of the mobile fundus image classification model, α is the weight, N is the number of disease categories for multiple diseases, and p′ i is the probability of the i-th disease type output by the basic fundus image classification model, q′ i is the probability of the i-th disease type output by the mobile fundus image classification model after knowledge distillation learning, c i is the label value of the i-th disease in the mobile fundus image classification model, q iIt is the probability of the i-th disease type output after label value learning.

[0101] Specifically, based on DenseNet, a mobile network structure of the mobile fundus image recognition layer in the mobile fundus image classification model is constructed, wherein the network structure within the mobile network structure is similar to the above-mentioned basic network structure. The difference is that the mobile network structure is more lightweight, and the number of nonlinear transformations H in the mobile network structure is smaller than the number of nonlinear transformations H in the basic network structure, and the computing resources occupied are smaller. When performing image processing and analysis on the mobile terminal, the response speed is faster and more convenient.

[0102] For example, in the mobile terminal network structure, the number of Denseblocks in the first layer is 6, the number of Denseblocks in the second layer is 12, the number of Denseblocks in the third layer is 24, and the number of Denseblocks in the fourth layer is 16.

[0103] Based on the image classifier in the aforementioned content, the mobile fundus image recognition layer and the image classifier are connected to obtain a mobile fundus image classification model.

[0104] Based on knowledge distillation in machine learning, the Teacher-Student model is adopted. The complex basic fundus image classification model mentioned above is used as the Teacher, and the mobile fundus image classification model is used as the Student. The mobile fundus image classification model is trained according to the mobile loss function, so that the lightweight mobile fundus image classification model also has good performance and generalization.

[0105] Among them, the mobile terminal loss function is as follows:

[0106] L=αL t +(1-α)L s

[0107]

[0108]

[0109] Among them, L t is the teacher loss of the basic fundus image classification model, L s is the student loss of the mobile fundus image classification model, α is the weight, preferably 0.9, N is the number of disease categories for multiple diseases, and p′ i is the probability of the i-th disease type output by the basic fundus image classification model, q′ i is the probability of the i-th disease type output by the mobile fundus image classification model after knowledge distillation learning, c iis the true label value of the i-th disease type in the mobile fundus image classification model, q i It is the probability of suffering from the i-th disease type output after label value learning.

[0110] Based on the multiple sample fundus image sets and multiple label values ​​in the aforementioned content as training data, the mobile fundus image classification model is iteratively trained using the Teacher-Student mode according to the mobile loss function. During the iterative supervised training process, the network is trained to fit according to the multi-disease recognition loss error output by itself, and a trained mobile fundus image classification model is obtained.

[0111] The embodiment of the present application constructs a complex and large basic fundus image classification model, and based on the Teacher-Student model of knowledge distillation, constructs a lightweight mobile fundus image classification model on the mobile terminal, which can meet the requirements of lightweight and fast retinal fundus image recognition and classification, and achieve the technical effect of improving the efficiency and response speed of retinal fundus image recognition and classification on the mobile terminal, while ensuring a certain degree of accuracy.

[0112] S400: Acquire a target user's fundus image to obtain a target fundus image;

[0113] Based on the constructed mobile fundus image classification model, the user's retinal fundus images can be identified and classified.

[0114] Specifically, a retinal fundus image of the current target user is collected and acquired to obtain a target fundus image.

[0115] S500: performing data enhancement and expansion on the target fundus image to obtain a target fundus image set;

[0116] Step S500 in the method provided in the embodiment of the present application includes:

[0117] S510: rotating, horizontally flipping, vertically inverting, and scaling the target fundus image to obtain a plurality of expanded images;

[0118] S520: Obtain the target fundus image set according to the target fundus image and the multiple expanded images.

[0119] Specifically, data augmentation and expansion are performed on the target fundus image, including rotation, horizontal flipping, vertical inversion, and scaling, to obtain multiple augmented images. Based on the target fundus image and the multiple augmented images, a target fundus image set for the target user is obtained. This data augmentation and expansion increases the dimensionality of the fundus image data, thereby improving the stability and accuracy of fundus image recognition and classification.

[0120] S600: Inputting the target fundus image set into the mobile fundus image classification model to obtain image classification results.

[0121] The target fundus image set is input into the mobile fundus image classification model, first into the mobile fundus image recognition layer within the mobile fundus image classification model, and after convolution feature extraction and feature reuse, multiple feature data of multiple fundus images in the target fundus image set are finally output. Then, the multiple feature data are input into the image classifier within the mobile fundus image classification model, and the probability information of each fundus image corresponding to various types of diseases is output. Combined with the multiple fundus images in the target fundus image set, the probability information of the current target user's fundus image corresponding to various types of diseases is calculated and obtained as the image classification result and as reference data for fundus image recognition.

[0122] After experiments on the recognition and classification of retinal fundus images using a mobile fundus image classification model, the method provided in the embodiment of the present application takes about 0.2 seconds to recognize and classify each retinal fundus image, which is highly efficient and accurate. The AUC value for the recognition and classification of retinal fundus images of various types of diseases reaches above 0.95, which has high timeliness and accuracy.

[0123] In summary, the mobile-based fundus image classification method provided by this application has the following technical effects:

[0124] The technical solution provided in this application is based on DenseNet to construct a basic fundus image recognition layer that occupies a large amount of computing resources locally, and can ensure the maximum information transmission between each layer in the network. By densely connecting the feature maps of all network layers, feature reuse is achieved, the efficiency and accuracy of image analysis and processing are improved, and an image classifier is constructed to achieve classification, recognition and prediction of multiple diseases, and improve the functional dimension of the combination of retinal fundus images and deep learning. Then, through knowledge distillation and the Teacher-Student mode, a lightweight but high-performance mobile fundus image classification model is obtained, which realizes fast, intelligent and accurate fundus image recognition and analysis processing on the mobile terminal. , reduce the computing resource occupancy of the model on the mobile terminal, improve the response efficiency, then collect and obtain the retinal fundus image of the current user, perform data enhancement and expansion, improve the image data dimension, and then improve the accuracy of image data analysis and processing, input the mobile terminal fundus image classification model, and obtain the image classification result of the retinal fundus image of the current user. This application realizes more complex fundus image analysis and processing based on deep learning on the mobile terminal, which is more lightweight and convenient, and can identify and classify multiple diseases, with richer functions, achieving the technical effect of reducing the computing resource occupancy of fundus image recognition processing, improving the timeliness and convenience of mobile terminal use, and enriching the fundus image recognition and processing functions.

[0125] Example 2

[0126] Based on the same inventive concept as the mobile-based fundus image classification method in the aforementioned embodiment, Figure 5 As shown, the present application also provides a mobile-based fundus image classification system, the system comprising:

[0127] A basic recognition layer construction module 11 is used to construct a basic fundus image recognition layer based on DenseNet;

[0128] A basic recognition model construction module 12 is used to construct an image classifier, and to construct a basic fundus image classification model based on the basic fundus image recognition layer and the image classifier;

[0129] A mobile terminal recognition model construction module 13 is used to obtain a mobile terminal fundus image classification model through knowledge distillation based on the basic fundus image classification model;

[0130] The fundus image acquisition module 14 is used to acquire the fundus image of the target user and obtain the target fundus image;

[0131] An image enhancement and expansion module 15 is configured to perform data enhancement and expansion on the target fundus image to obtain a target fundus image set;

[0132] The image recognition and classification module 16 is configured to input the target fundus image set into the mobile fundus image classification model to obtain an image classification result.

[0133] Furthermore, the basic identification layer construction module 11 is also used to implement the following functions:

[0134] Collect and acquire multiple fundus images of multiple types of diseases respectively to obtain multiple sample fundus images;

[0135] Performing data enhancement and expansion on the multiple sample fundus images to obtain multiple sample fundus image sets;

[0136] Performing feature data identification on the multiple sample fundus image sets respectively to obtain multiple sample feature data sets;

[0137] Based on DenseNet, a basic network structure of the basic fundus image recognition layer is constructed, wherein the basic network structure includes:

[0138] X i =H i ([X0, X1, ..., X i-1 ])

[0139] Among them, X iis the output of layer i, [X0,X1,…,X i-1 ] is X0 to X i-1 The concatenation of all output feature vectors of the layer, H i is the nonlinear transformation performed on the i-th layer, and the nonlinear transformation H is a combination of BN+ReLU+Conv(3×3);

[0140] The basic fundus image recognition layer is supervised and trained using the multiple sample fundus image sets and the multiple sample feature data sets until convergence or the accuracy reaches a preset requirement.

[0141] Furthermore, the basic recognition model building module 12 is also used to implement the following functions:

[0142] Based on the random forest algorithm, randomly selecting M sample feature data sets from the multiple sample feature data sets with replacement as a first constructed data set;

[0143] Using the first constructed dataset, construct a first sub-model of the image classifier;

[0144] Randomly selecting M sample feature data sets from the multiple sample feature data sets with replacement as a second constructed data set;

[0145] Using the second constructed data set to construct a second sub-model of the image classifier;

[0146] Continue to construct multiple sub-models of the image classifier, integrate the multiple sub-models, and obtain the image classifier;

[0147] The basic fundus image classification model is constructed based on the basic fundus image recognition layer and the image classifier.

[0148] The basic fundus image classification model is constructed based on the basic fundus image recognition layer and the image classifier, including:

[0149] Connecting the basic fundus image recognition layer and the image classifier through a global AvgPooling layer;

[0150] According to the basic model loss function, the connected basic fundus image recognition layer and the image classifier are iteratively trained to obtain the basic fundus image classification model, wherein the basic model loss function is as follows:

[0151]

[0152]

[0153]

[0154] y_pred m =round(output m )

[0155] diff = y_true - y_pred

[0156] Among them, output m is the probability of the mth disease type output by the image classifier, round is the rounding function, y_pred m is the rounded output m The value of y_true is the actual label value of the mth disease in onehot encoding form, diff is the difference between y_true and y_pred, sum i (x) is the number of values ​​i in the statistic x, and muliti_loss is the loss function of the basic model.

[0157] The first constructed dataset is used to construct a first sub-model of the image classifier, including:

[0158] Randomly selecting a sample feature data in the first constructed data set as a first segmentation threshold;

[0159] Based on the first partition threshold, constructing a first-level classification node of the first sub-model, wherein the first-level classification node can perform binary classification on the input feature data;

[0160] Randomly selecting a sample feature data in the first constructed data set again as a second segmentation threshold;

[0161] Based on the second division threshold, constructing a secondary classification node of the first sub-model, wherein the secondary classification node can perform binary classification on the classification result of the primary classification node;

[0162] Based on the first constructed data set, continue to construct the multi-level classification nodes of the first sub-model;

[0163] Acquiring label values ​​of the plurality of sample fundus images to obtain a plurality of label values;

[0164] A plurality of classification results obtained by performing multi-level binary classification on the multi-level classification node are obtained, and different disease classification results are set for the plurality of classification results according to the plurality of label values ​​to obtain the first sub-model.

[0165] Furthermore, the mobile terminal recognition model building module 13 is also used to implement the following functions:

[0166] Based on DenseNet, construct a mobile network structure of a mobile fundus image recognition layer in the mobile fundus image classification model, wherein the number of nonlinear transformations in the mobile network structure is less than the number of nonlinear transformations in the basic network structure;

[0167] Connecting the mobile terminal fundus image recognition layer and the image classifier to obtain the mobile terminal fundus image classification model;

[0168] The mobile terminal fundus image classification model is iteratively trained according to the mobile terminal loss function and the basic fundus image classification model, wherein the mobile terminal loss function is as follows:

[0169] L=αL t +(1-α)L s

[0170]

[0171]

[0172] Among them, L t is the teacher loss of the basic fundus image classification model, L s is the student loss of the mobile fundus image classification model, α is the weight, N is the number of disease categories for multiple diseases, and p′ i is the probability of the i-th disease type output by the basic fundus image classification model, q′ i is the probability of the i-th disease type output by the mobile fundus image classification model after knowledge distillation learning, c i is the label value of the i-th disease in the mobile fundus image classification model, q i It is the probability of the i-th disease type output after label value learning.

[0173] Furthermore, the image enhancement and expansion module 15 is also used to implement the following functions:

[0174] Rotating, horizontally flipping, vertically inverting, and scaling the target fundus image to obtain a plurality of expanded images;

[0175] The target fundus image set is obtained according to the target fundus image and the multiple expanded images.

[0176] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. Figure 1The mobile-based fundus image classification method and specific examples in Example 1 are also applicable to the mobile-based fundus image classification system of this embodiment. Through the above detailed description of the mobile-based fundus image classification method, those skilled in the art can clearly understand the mobile-based fundus image classification system of this embodiment, so for the sake of brevity, it will not be described in detail here. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0177] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to be embodied in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A mobile-based fundus image classification method, characterized in that: The method comprises: Based on DenseNet, build a basic fundus image recognition layer; Constructing an image classifier, and constructing a basic fundus image classification model based on the basic fundus image recognition layer and the image classifier; According to the basic fundus image classification model, a mobile fundus image classification model is obtained through knowledge distillation; Acquire a target user's fundus image to obtain a target fundus image; Performing data enhancement and expansion on the target fundus image to obtain a target fundus image set; Inputting the target fundus image set into the mobile fundus image classification model to obtain an image classification result; According to the basic fundus image recognition layer and the image classifier, constructing and obtaining the basic fundus image classification model includes: Connecting the basic fundus image recognition layer and the image classifier through a globalAvgPooling layer; According to the basic model loss function, the connected basic fundus image recognition layer and the image classifier are iteratively trained to obtain the basic fundus image classification model, wherein the basic model loss function is as follows: y_pred m =round(output m ) diff = y_true - y_pred Among them, output m is the probability of the mth disease type output by the image classifier, round is the rounding function, y_pred m is the rounded output m The value of y_true is the actual label value of the mth disease in onehot encoding form, diff is the difference between y_true and y_pred, sum i (x) is the number of values ​​i in the statistic x, and muliti_loss is the loss function of the basic model.

2. The method according to claim 1, characterized in that The basic fundus image recognition layer is constructed based on DenseNet, including: Collect and acquire multiple fundus images of multiple types of diseases respectively to obtain multiple sample fundus images; Performing data enhancement and expansion on the multiple sample fundus images to obtain multiple sample fundus image sets; Performing feature data identification on the multiple sample fundus image sets respectively to obtain multiple sample feature data sets; Based on DenseNet, a basic network structure of the basic fundus image recognition layer is constructed, wherein the basic network structure includes: X i =H i ([X0, X1,..., X i-1 ]) Among them, X i is the output of layer i, [X0, X1, ..., X i-1 ] is X0 to X i-1 The concatenation of all output feature vectors of the layer, H i is the nonlinear transformation performed on the i-th layer, and the nonlinear transformation H is a combination of BN+ReLU+Conv(3×3); The basic fundus image recognition layer is supervised and trained using the multiple sample fundus image sets and the multiple sample feature data sets until convergence or the accuracy reaches a preset requirement.

3. The method according to claim 2, characterized in that The constructing of the image classifier comprises: Based on the random forest algorithm, randomly selecting M sample feature data sets from the multiple sample feature data sets with replacement as a first constructed data set; Using the first constructed dataset, construct a first sub-model of the image classifier; Randomly selecting M sample feature data sets from the multiple sample feature data sets with replacement as a second constructed data set; Using the second constructed data set to construct a second sub-model of the image classifier; Continue to construct multiple sub-models of the image classifier, integrate the multiple sub-models, and obtain the image classifier; The basic fundus image classification model is constructed based on the basic fundus image recognition layer and the image classifier.

4. The method according to claim 3, characterized in that Using the first constructed dataset to construct a first sub-model of the image classifier includes: Randomly selecting a sample feature data in the first constructed data set as a first segmentation threshold; Based on the first partition threshold, constructing a first-level classification node of the first sub-model, wherein the first-level classification node can perform binary classification on the input feature data; Randomly selecting a sample feature data in the first constructed data set again as a second segmentation threshold; Based on the second division threshold, constructing a secondary classification node of the first sub-model, wherein the secondary classification node can perform binary classification on the classification result of the primary classification node; Based on the first constructed data set, continue to construct the multi-level classification nodes of the first sub-model; Acquiring label values ​​of the plurality of sample fundus images to obtain a plurality of label values; A plurality of classification results obtained by performing multi-level binary classification on the multi-level classification node are obtained, and different disease classification results are set for the plurality of classification results according to the plurality of label values ​​to obtain the first sub-model.

5. The method according to claim 3, characterized in that According to the basic fundus image classification model, a mobile fundus image classification model is obtained through knowledge distillation, including: Based on DenseNet, construct a mobile network structure of a mobile fundus image recognition layer in the mobile fundus image classification model, wherein the number of nonlinear transformations in the mobile network structure is less than the number of nonlinear transformations in the basic network structure; Connecting the mobile terminal fundus image recognition layer and the image classifier to obtain the mobile terminal fundus image classification model; The mobile terminal fundus image classification model is iteratively trained according to the mobile terminal loss function and the basic fundus image classification model, wherein the mobile terminal loss function is as follows: L=αL t +(1-α)L s Among them, L t is the teacherloss of the basic fundus image classification model, L s is the student loss of the mobile fundus image classification model, α is the weight, N is the number of disease categories for multiple diseases, and p′ i is the probability of the i-th disease type output by the basic fundus image classification model, q′ i is the probability of the i-th disease type output by the mobile fundus image classification model after knowledge distillation learning, c i is the label value of the i-th disease in the mobile fundus image classification model, q i It is the probability of the i-th disease type output after label value learning.

6. The method according to claim 1, characterized in that Performing data enhancement and expansion on the target fundus image to obtain a target fundus image set includes: Rotating, horizontally flipping, vertically inverting, and scaling the target fundus image to obtain a plurality of expanded images; The target fundus image set is obtained according to the target fundus image and the multiple expanded images.

7. A system using the mobile-based fundus image classification method according to any one of claims 1 to 6, characterized in that: The system comprises: Basic recognition layer construction module, used to build a basic fundus image recognition layer based on DenseNet; A basic recognition model construction module is used to construct an image classifier, and to construct a basic fundus image classification model based on the basic fundus image recognition layer and the image classifier; A mobile terminal recognition model construction module is used to obtain a mobile terminal fundus image classification model through knowledge distillation based on the basic fundus image classification model; The fundus image acquisition module is used to acquire the fundus image of the target user and obtain the target fundus image; An image enhancement and expansion module, configured to perform data enhancement and expansion on the target fundus image to obtain a target fundus image set; The image recognition and classification module is used to input the target fundus image set into the mobile fundus image classification model to obtain image classification results.

Images