Eye disease recognition method, model training method, device, equipment and medium

Abstract

The invention provides a fundus image-based eye disease recognition method and device, a model training method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining one or more fundus images to be recognized; performing feature extraction on each acquired eye fundus image by using a deep convolutional neural network; the deep convolutional neural network comprises a trained segmentation model and a trained classification model, and is used for extracting the feature map of each fundus image, and carrying out segmentation processing and classification processing on the extracted feature map of each fundus image to obtain a segmentation result and a classification result of each fundus image, and the deep convolutional neural network comprises a trained segmentation model and a trained classification model; the classification model is trained based on model parameters of the trained segmentation model; and performing fusion prediction on the segmentation result and the classification result of each fundus image to obtain a recognition result of eye diseases in each fundus image. By using the deep convolutional neural network provided by the invention to identify the eye fundus image, the accuracy of eye disease identification based on the eye fundus image can be improved.

Description

Technical Field

[0001] The present invention relates to the field of medical image classification technology, and in particular to an eye disease recognition method based on fundus images, a segmentation model training method, a classification model training method, a device, an electronic device and a storage medium. Background Art

[0002] With the advancement of technology and the popularization of electronic products, the incidence of myopia is getting higher and higher. Myopia with a degree greater than 600 degrees is called high myopia, which is very likely to cause pathological myopia and may also cause complications such as retinal detachment, which may lead to permanent visual impairment or even blindness.

[0003] In related technologies, the trained model can only screen out pathological myopia, but cannot identify eye diseases such as retinal detachment. This is because fundus images of retinal detachment are difficult to collect and the amount of data that can be obtained is small. Therefore, the characteristics of retinal detachment data cannot be fully learned during model training, so the classification effect of retinal detachment data is poor.

[0004] Therefore, whether pathological myopia and retinal detachment and other eye diseases can be accurately screened based on fundus images is a technical problem that needs to be solved. Summary of the invention

[0005] The present invention provides an eye disease recognition method based on fundus images, a segmentation model training method, a classification model training method, an apparatus, an electronic device, and a computer-readable storage medium, so as to at least solve the technical problem in the related art that the efficiency of eye disease classification and recognition is poor due to the inability to accurately screen out eye diseases such as pathological myopia and retinal detachment. The technical solution of the present invention is as follows:

[0006] According to a first aspect of an embodiment of the present invention, there is provided an eye disease recognition method based on a fundus image, comprising:

[0007] Acquire one or more fundus images to be identified;

[0008] Using a deep convolutional neural network to extract features from each acquired fundus image to obtain a feature map of each fundus image, and performing segmentation processing and classification processing on the feature map of each fundus image to obtain a segmentation result and a classification result of each fundus image, wherein the deep convolutional neural network includes: a trained segmentation model and a classification model, and the classification model is trained based on model parameters of the trained segmentation model;

[0009] The segmentation result and the classification result of each fundus image are fused and predicted to obtain the recognition result of the eye disease in each fundus image.

[0010] Optionally, before extracting features from each acquired fundus image using a deep convolutional neural network, the method further includes:

[0011] Pre-training the segmentation model in the deep convolutional neural network, and continuing to train the classification model in the deep convolutional neural network based on the parameters of the trained segmentation model;

[0012] The trained parameters of the classification model are used as parameters of the trained deep convolutional neural network.

[0013] Optionally, the pre-training of the segmentation model in the deep convolutional neural network includes:

[0014] Acquire a first training set, the first training set comprising: a plurality of sample fundus images, the plurality of sample fundus images comprising a first annotated image of an eye disease and a second annotated image with no obvious lesions, wherein the first annotated image of an eye disease at least comprises: annotated data of pathological myopia and retinal detachment; the second annotated image with no obvious lesions comprises: a normal fundus image and a leopard fundus image;

[0015] Inputting the plurality of sample fundus images into an initial segmentation model, extracting features of lesions related to eye diseases in each input fundus image, as well as features of normal fundus images without obvious lesions and leopard fundus images;

[0016] Comparing the extracted features corresponding to the lesion features and the normal fundus image and the leopard fundus image without obvious lesions with the corresponding first annotated image and the second annotated image respectively;

[0017] Calculate the loss value based on the gap in the comparison results;

[0018] Based on the loss value, the parameters of the segmentation model are updated through the back propagation mechanism. After multiple iterations, the segmentation model converges, and the output image when the segmentation model converges is upsampled to restore it to the size of the original sample fundus image to obtain a trained segmentation model.

[0019] Optionally, continuing to train the pre-classification model in the deep convolutional neural network based on the trained parameters of the segmentation model includes:

[0020] Acquire a second training set, the second training set comprising: a plurality of sample fundus images, the plurality of sample fundus images comprising classification labels of eye diseases and non-eye diseases, wherein the classification labels at least include: no obvious lesions, leopard-like fundus, diffuse choroidal atrophy, patchy choroidal atrophy, macular atrophy, and retinal detachment;

[0021] Inputting the plurality of sample fundus images into an initial classification model, wherein the parameters of the initial classification model are the parameters of the trained segmentation model;

[0022] Extracting classification features related to eye diseases and non-eye diseases in each of the plurality of sample fundus images;

[0023] Performing dimension and classification training on the extracted classification features through a fully connected layer, and comparing the training results with the classification labels;

[0024] Calculate the loss value based on the gap in the comparison results;

[0025] Based on the loss value, the parameters of the classification model are updated through the back propagation mechanism. After multiple iterations, the results of each iteration are processed twice by the fully connected layer for dimension and classification until the classification model converges to obtain a trained classification model.

[0026] Optionally, the segmenting process is performed on the feature map of each fundus image to obtain the segmentation result of each fundus image, including:

[0027] Upsampling the feature map of each fundus image and outputting a segmentation result;

[0028] Calculating the sum of the pixel points of the lesion area of ​​each fundus image in the segmentation result;

[0029] When the sum of the pixel points in the lesion area is greater than a first preset threshold, determining that the segmentation result of the corresponding fundus image is a fundus disease; or

[0030] When the sum of the pixel points in the lesion area is not greater than the first preset threshold, it is determined that the segmentation result of the corresponding fundus image is non-fundus disease.

[0031] Optionally, the classifying process is performed on the feature map of each fundus image to obtain a classification result of each fundus image, including:

[0032] The feature map of each fundus image is processed in sequence by dimension and classification through a fully connected layer to obtain a corresponding classification result;

[0033] When the classification result is retinal detachment, calculating the classification confidence of the retinal detachment, and when the classification confidence of the retinal detachment is greater than a second threshold, determining that the classification result is retinal detachment; or

[0034] When the classification result is pathological myopia, calculating the classification confidence of pathological myopia, and when the classification confidence of pathological myopia is greater than a third threshold, determining that the classification result is pathological myopia; or

[0035] When the classification result is that no obvious lesions are found, the classification confidence of the no obvious lesions is calculated, and when the classification confidence of the no obvious lesions is greater than a fourth threshold, it is determined that the classification result is that no obvious lesions are found.

[0036] Optionally, the performing fusion prediction on the segmentation result and the classification result of each fundus image to obtain the recognition result of the eye disease in each fundus image includes:

[0037] If the segmentation result and classification result of each fundus image are the same, outputting the same result; or

[0038] If the segmentation result and the classification result of each fundus image are different, a logical judgment is performed on the segmentation result and the classification result of each fundus image to obtain the recognition result of the eye disease in each fundus image.

[0039] Optionally, performing logical judgment on the segmentation result and classification result of each fundus image to obtain the recognition result of the eye disease in each fundus image includes:

[0040] If the segmentation result includes retinal detachment, performing an OR logic operation on the segmentation result and the classification result to obtain a prediction result including retinal detachment; or

[0041] If any one of diffuse choroidal atrophy, patchy choroidal atrophy and macular atrophy exists in the classification result, the classification result and the segmentation result are subjected to an OR logic operation to obtain a prediction result of pathological myopia;

[0042] If the segmentation result is that no obvious lesions are found, and the classification result is retinal detachment and the probability of the classification result is greater than the classification threshold, the segmentation result and the classification result are ORed to obtain a prediction result of retinal detachment; or

[0043] If the segmentation result is pathological myopia and the classification result is no obvious lesion, the segmentation result and the classification result are ORed to obtain a prediction result of pathological myopia; or

[0044] If the segmentation result is pathological myopia, the classification result is retinal detachment and the probability of the classification result is greater than the classification threshold, a prediction result of pathological myopia or retinal detachment is obtained.

[0045] According to a second aspect of an embodiment of the present invention, a segmentation model training method is provided, the method comprising:

[0046] Acquire a first training set, the first training set comprising: a plurality of sample fundus images, the plurality of sample fundus images comprising: a first annotated image of an eye disease and a second annotated image without obvious lesions, wherein the first annotated image of an eye disease at least comprises: annotated data of pathological myopia and retinal detachment; the second annotated image without obvious lesions comprises: a normal fundus image and a leopard fundus image;

[0047] Inputting the plurality of sample fundus images into an initial segmentation model;

[0048] Extract the features of lesions related to eye diseases in each input fundus image, as well as the features of normal fundus images and leopard fundus images without obvious lesions;

[0049] Comparing the extracted lesion features and features of a normal fundus image without obvious lesions and a leopard fundus image with the corresponding first annotated image and the second annotated image respectively;

[0050] Calculate the loss value based on the gap in the comparison results;

[0051] Based on the loss value, the parameters of the segmentation model are updated through the back propagation mechanism. After multiple iterations, the segmentation model converges, and the output image when the segmentation model converges is upsampled to restore it to the size of the original sample fundus image to obtain a trained segmentation model.

[0052] According to a third aspect of an embodiment of the present invention, a classification model training method is provided, the method comprising:

[0053] Acquire a second training set, the second training set comprising: a plurality of sample fundus images, the plurality of sample fundus images comprising classification labels of eye diseases and non-eye diseases, wherein the classification labels at least include: no obvious lesions, leopard-like fundus, diffuse choroidal atrophy, patchy choroidal atrophy, macular atrophy, and retinal detachment;

[0054] Inputting the plurality of sample fundus images into an initial classification model, wherein parameters of the initial classification model are parameters of a trained segmentation model;

[0055] Extracting classification features related to eye diseases and non-eye diseases in each of the plurality of sample fundus images;

[0056] The extracted classification features are trained in terms of dimension and classification through a fully connected layer;

[0057] Compare the training result with the classification label, and calculate the loss value according to the difference between the comparison results;

[0058] Based on the loss value, the parameters of the classification model are updated through the back propagation mechanism. After multiple iterations, the results of each iteration are processed twice by the fully connected layer for dimension and classification until the classification model converges to obtain a trained classification model.

[0059] According to a fourth aspect of an embodiment of the present invention, there is provided an eye disease identification device based on a fundus image, comprising:

[0060] An acquisition module, used for acquiring one or more fundus images to be identified;

[0061] A processing module, used to extract features from each acquired fundus image using a deep convolutional neural network to obtain a feature map of each fundus image; and to perform segmentation processing and classification processing on the feature map of each fundus image to obtain a segmentation result and a classification result of each fundus image, wherein the deep convolutional neural network includes: a trained segmentation model and a classification model, and the classification model is trained based on model parameters of the trained segmentation model;

[0062] The fusion prediction module is used to perform fusion prediction on the segmentation result and the classification result of each fundus image to obtain the recognition result of the eye disease in each fundus image.

[0063] Optionally, the device further comprises:

[0064] A first training module is used to pre-train a segmentation model in the deep convolutional neural network before the processing module uses the deep convolutional neural network to extract features from each acquired fundus image; and

[0065] A second training module, used for continuing to train the classification model in the deep convolutional neural network based on the trained parameters of the segmentation model;

[0066] A determination module is used to use the parameters of the classification model trained by the second training module as the parameters of the trained deep convolutional neural network.

[0067] Optionally, the first training module includes:

[0068] A first acquisition module is used to acquire a first training set, wherein the first training set includes: a plurality of sample fundus images, wherein the plurality of sample fundus images include a first annotated image of an eye disease and a second annotated image without obvious lesions, wherein the first annotated image of an eye disease includes at least annotated data of pathological myopia and retinal detachment; and the second annotated image without obvious lesions includes a normal fundus image and a leopard fundus image;

[0069] A first input module, used for inputting the plurality of sample fundus images into an initial segmentation model;

[0070] The first extraction module is used to extract the features of lesions related to eye diseases in each input fundus image, as well as the features of normal fundus images without obvious lesions and leopard fundus images;

[0071] A first comparison module is used to compare the extracted lesion features and features of a normal fundus image without obvious lesions and a leopard fundus image with the corresponding first and second annotation images respectively;

[0072] A first calculation module, used for calculating the loss value according to the difference of the comparison result;

[0073] A first iteration module, used to update the parameters of the segmentation model through a back propagation mechanism based on the loss value, after multiple iterations, until the segmentation model converges;

[0074] The sampling processing module is used to upsample the output image when the segmentation model converges, restore it to the size of the original sample fundus image, and obtain a trained segmentation model.

[0075] Optionally, the second training module includes:

[0076] A second acquisition module is used to acquire a second training set, wherein the second training set includes: a plurality of sample fundus images, wherein the plurality of sample fundus images include classification labels of eye diseases and non-eye diseases, wherein the classification labels include at least: no obvious lesions, leopard-like fundus, diffuse choroidal atrophy, patchy choroidal atrophy, macular atrophy, and retinal detachment;

[0077] A second input module, used for inputting the plurality of sample fundus images into an initial classification model; wherein the parameters of the initial classification model are the parameters of the trained segmentation model;

[0078] A second extraction module is used to extract classification features related to eye diseases and non-eye diseases in each of the plurality of sample fundus images;

[0079] A second comparison module is used to perform dimension and classification training on the extracted classification features through a fully connected layer, and compare the training results with the classification labels;

[0080] The second calculation module is used to calculate the loss value according to the difference of the comparison results;

[0081] A second iteration module is used to update the parameters of the classification model through a back propagation mechanism based on the loss value, after multiple iterations;

[0082] The dimension classification processing module is used to perform dimension and classification processing on the result of each iteration in two fully connected layers in sequence until the classification model converges to obtain a trained classification model.

[0083] Optionally, the processing module includes: a segmentation processing module and a classification processing module, wherein:

[0084] The segmentation processing module comprises:

[0085] An upsampling module, used for upsampling the feature map of each fundus image and then outputting a segmentation result;

[0086] A calculation module, used for calculating the sum of the pixel points of the lesion area of ​​each fundus image in the segmentation result;

[0087] A first disease determination module, configured to determine that the segmentation result of the corresponding fundus image is a fundus disease when the sum of the pixel points in the lesion area is greater than a first preset threshold value; or to determine that the segmentation result of the corresponding fundus image is a non-fundus disease when the sum of the pixel points in the lesion area is not greater than the first preset threshold value;

[0088] The classification processing module comprises:

[0089] A classification module, used for sequentially performing dimension and classification processing on the feature map of each fundus image through a fully connected layer to obtain a corresponding classification result;

[0090] The second disease determination module is used to calculate the classification confidence of retinal detachment when the classification result is retinal detachment, and determine that the classification result is retinal detachment when the classification confidence of the retinal detachment is greater than a second threshold; or to calculate the classification confidence of pathological myopia when the classification result is pathological myopia, and determine that the classification result is pathological myopia when the classification confidence of pathological myopia is greater than a third threshold; or to calculate the classification confidence of no obvious lesions when the classification result is no obvious lesions, and determine that the classification result is no obvious lesions when the classification confidence of no obvious lesions is greater than a fourth threshold.

[0091] Optionally, the fusion prediction module includes: a first fusion prediction module and / or a second fusion prediction module, wherein:

[0092] The first fusion prediction module is used for outputting the same result if the segmentation result and classification result of each fundus image are the same;

[0093] The second fusion prediction module is used to perform logical judgment on the segmentation result and classification result of each fundus image when the segmentation result and classification result of each fundus image are different, so as to obtain the recognition result of the eye disease in each fundus image.

[0094] Optionally, the second fusion prediction module includes:

[0095] a first logic calculation module, configured to, when the segmentation result and the classification result of each fundus image are different, perform an OR logic operation on the segmentation result and the classification result to obtain a prediction result including retinal detachment if the segmentation result includes retinal detachment; or

[0096] A second logic calculation module is used for, when the segmentation result and the classification result of each fundus image are different, if any one of diffuse choroidal atrophy, patchy choroidal atrophy and macular atrophy exists in the classification result, performing a logic OR operation on the classification result and the segmentation result to obtain a prediction result of pathological myopia;

[0097] A third logic calculation module is used for, when the segmentation result and the classification result of each fundus image are different, if the segmentation result is that no obvious lesions are found, and the classification result is retinal detachment and the probability of the classification result is greater than the classification threshold, then performing an OR operation on the segmentation result and the classification result to obtain a prediction result of retinal detachment;

[0098] a fourth logic calculation module, for, when the segmentation result and the classification result of each fundus image are different, if the segmentation result is pathological myopia and the classification result is no obvious lesion, performing an OR operation on the segmentation result and the classification result to obtain a prediction result of pathological myopia;

[0099] The fifth logic calculation module is used to, when the segmentation result and the classification result of each fundus image are different, if the segmentation result is pathological myopia and the classification result is no obvious lesion, perform an OR operation on the segmentation result and the classification result to obtain a prediction result of pathological myopia.

[0100] According to a fifth aspect of an embodiment of the present invention, a segmentation model training device is provided, the device comprising:

[0101] An acquisition module is used to acquire a first training set, wherein the first training set includes: a plurality of sample fundus images, wherein the plurality of sample fundus images include: a first annotated image of an eye disease and a second annotated image without obvious lesions, wherein the first annotated image of an eye disease includes at least annotated data of pathological myopia and retinal detachment; and the second annotated image without obvious lesions includes: a normal fundus image and a leopard fundus image;

[0102] An input module, used for inputting the plurality of sample fundus images into an initial segmentation model;

[0103] An extraction module, used to extract features of lesions related to eye diseases in each input fundus image, as well as features of normal fundus images without obvious lesions and leopard fundus images;

[0104] A comparison module, used for comparing the extracted lesion features and features of the normal fundus image without obvious lesions and the leopard fundus image with the corresponding first annotated image and the second annotated image respectively;

[0105] A calculation module, used for calculating the loss value according to the difference of the comparison results;

[0106] An iteration module, used to update the parameters of the segmentation model through a back propagation mechanism based on the loss value, until the segmentation model converges after multiple iterations;

[0107] The sampling processing module is used to upsample the output image when the segmentation model converges, restore it to the size of the original sample fundus image, and obtain a trained segmentation model.

[0108] According to a sixth aspect of an embodiment of the present invention, a classification model training device is provided, the device comprising:

[0109] An acquisition module is used to acquire a second training set, wherein the second training set includes: a plurality of sample fundus images, wherein the plurality of sample fundus images include classification labels of eye diseases and non-eye diseases, wherein the classification labels include at least: no obvious lesions, leopard-like fundus, diffuse choroidal atrophy, patchy choroidal atrophy, macular atrophy, and retinal detachment;

[0110] An input module, used for inputting the plurality of sample fundus images into an initial classification model, wherein the parameters of the initial classification model are the parameters of the trained segmentation model;

[0111] An extraction module, used for extracting classification features related to eye diseases and non-eye diseases in each of the plurality of sample fundus images;

[0112] A comparison module, used for training the extracted classification features in terms of dimension and classification through a fully connected layer;

[0113] A calculation module, used to compare the training result with the classification label, and calculate the loss value according to the difference of the comparison result;

[0114] An iteration module, for updating the parameters of the classification model through a back propagation mechanism based on the loss value, after multiple iterations;

[0115] The dimension classification processing module is used to perform dimension and classification processing on the result of each iteration in two fully connected layers in sequence until the classification model converges to obtain a trained classification model.

[0116] According to a seventh aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0117] processor;

[0118] a memory for storing instructions executable by the processor;

[0119] Wherein, the processor is configured to execute the instructions to implement the eye disease identification method based on fundus images as described above or the segmentation model training method as described above and the classification model training method as described above.

[0120] According to an eighth aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the eye disease identification method based on fundus images as described above, or the segmentation model training method as described above, and the classification model training method as described above.

[0121] According to a ninth aspect of an embodiment of the present invention, a computer program product is provided, comprising a computer program or instructions, which, when executed by a processor of an electronic device, implements the eye disease identification method based on fundus images as described above, or the segmentation model training method as described above, and the classification model training method as described above.

[0122] The technical solution provided by the embodiments of the present invention brings at least the following beneficial effects:

[0123] In an embodiment of the present invention, one or more fundus images to be identified are obtained; a deep convolutional neural network is used to extract features from each fundus image obtained to obtain a feature map of each fundus image; segmentation processing and classification processing are respectively performed on the feature map of each fundus image to obtain a segmentation result and a classification result of each fundus image; the segmentation result and the classification result of each fundus image are fused and predicted to obtain an identification result of an eye disease in each fundus image. The present invention uses a deep convolutional neural network to obtain a feature map of each fundus image, and fusion prediction is performed on the segmentation result and the classification result of the segmentation processing and classification processing based on the feature map to obtain an identification result of an eye disease in each fundus image, thereby improving the accuracy of identifying eye diseases based on fundus images, maintaining the final accuracy of eye disease prediction at a high level, and ensuring the health and safety of users.

[0124] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0125] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention, and do not constitute an improper limitation of the present invention. In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0126] Figure 1 This is a flow chart of a method for identifying eye diseases based on fundus images provided by an embodiment of the present invention.

[0127] Figure 2 A flow chart of segmentation model training provided in an embodiment of the present invention.

[0128] Figure 3 A flowchart of a classification model training provided by an embodiment of the present invention.

[0129] Figure 4 A schematic diagram of performing fusion prediction based on the results of a classification model and a segmentation model provided in an embodiment of the present invention.

[0130] Figure 5 It is a block diagram of an eye disease identification device based on fundus images provided by an embodiment of the present invention.

[0131] Figure 6 This is another block diagram of an eye disease identification device based on fundus images provided by an embodiment of the present invention.

[0132] Figure 7 It is a block diagram of the first training module provided by an embodiment of the present invention.

[0133] Figure 8 It is a block diagram of an electronic device provided by an embodiment of the present invention.

[0134] Fig. 9 It is a block diagram of a device for identifying eye diseases based on fundus images provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0135] In order to enable ordinary persons in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0136] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0137] In recent years, research on computer vision, deep learning, machine learning, image processing, image recognition and other technologies based on artificial intelligence has made important progress. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial Intelligence is a comprehensive discipline involving many types of technologies such as chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, neural networks, etc. Computer vision, as an important branch of artificial intelligence, specifically allows machines to recognize the world. Computer vision technology usually includes face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, target detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, robot navigation and positioning and other technologies. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, automatic driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, and smart temperature measurement.

[0138] Figure 1 is a flow chart of a method for identifying eye diseases based on fundus images provided by an embodiment of the present invention, such as Figure 1 As shown, the eye disease recognition method based on fundus images includes the following steps:

[0139] Step 101: Acquire one or more fundus images to be identified.

[0140] Step 102: Use a deep convolutional neural network to extract features from each acquired fundus image to obtain a feature map of each fundus image, and perform segmentation processing and classification processing on the feature map of each fundus image to obtain a segmentation result and a classification result of each fundus image, wherein the deep convolutional neural network includes: a trained segmentation model and a classification model, and the classification model is trained based on the model parameters of the trained segmentation model.

[0141] Step 103: performing fusion prediction on the segmentation result and classification result of each fundus image to obtain the recognition result of the eye disease in each fundus image.

[0142] The eye disease identification method based on fundus images described in the present invention can be applied to terminals, servers, etc., without limitation herein, and its terminal implementation device can be a smart phone, a laptop computer, a tablet computer, a personal computer and other electronic devices, without limitation herein.

[0143] Combine the following Figure 1 , the specific implementation steps of an eye disease identification method based on fundus images provided by an embodiment of the present invention are described in detail.

[0144] In step 101, one or more fundus images to be identified are acquired.

[0145] In this step, the fundus image of the user can be obtained by professional fundus photography equipment (such as fundus photography equipment in a hospital, etc.), or by an environmentally friendly smoking room or a public area for fundus collection. Of course, the fundus image of the user can also be obtained by photographing the fundus of the user with a fundus camera, etc., which is not limited in this embodiment.

[0146] Optionally, in this embodiment, the user's fundus may be examined by artificial intelligence (AI) fundus image recognition technology or an artificial fundus screening platform to obtain the user's fundus image.

[0147] In this embodiment, one or more acquired fundus images are used as fundus images to be identified.

[0148] In step 102, a deep convolutional neural network is used to perform feature extraction on each acquired fundus image to obtain a feature map of each fundus image, and segmentation preprocessing and classification preprocessing are performed on the feature map of each fundus image to obtain a segmentation result and a classification result of each fundus image, wherein the deep convolutional neural network includes: a trained segmentation model and a classification model, and the classification model is trained based on the model parameters of the trained segmentation model.

[0149] In this step, feature extraction is performed on each acquired fundus image through a pre-trained deep convolutional neural network to obtain a feature map corresponding to each fundus image. The deep convolutional neural network may include: a segmentation model and a classification model.

[0150] In this step, the trained deep convolutional neural network is first used to extract features from all areas in each fundus image, such as retinal detachment features and pathological myopia features. Feature extraction can also be performed on specific areas in each fundus image, where the specific areas may include: optic disc, macula, optic cup, etc., which are not limited in this embodiment.

[0151] It should be noted that, although the related technology can extract the image features of pathological myopia fundus images, retinal detachment fundus images, and images without obvious lesions and leopard fundus images through the segmentation model. However, since the segmentation model only uses a small number of pathological myopia images and a small number of negative images as training samples during the training process, the segmentation model does not fully utilize the image information of a large number of pathological myopia fundus images, fundus images without obvious lesions, and leopard fundus images collected, so the generalization performance of the segmentation model needs to be improved. Based on this, the embodiment of the present invention classifies and annotates all the collected data and trains the classification network. The basic network of the segmentation network is used as the basic network of the classification network, and a fully connected layer and softmax are added after the last feature layer to train the classification model. The categories of the classification model include: no obvious lesions, leopard fundus, diffuse choroidal atrophy, patchy choroidal atrophy, macular atrophy and retinal detachment, a total of 6 classification categories. In the process of training the classification network, the parameters of the trained segmentation model are used as the basic feature layer parameters, and the basic feature layer parameters are fixed. Only the last added fully connected layer is trained, which saves training costs and can quickly obtain the classification model.

[0152] That is to say, this step can simultaneously perform auxiliary diagnosis of pathological myopia and retinal detachment, two myopia-related eye diseases, through the trained deep convolutional neural network, and weigh the annotation according to the actual amount of data collected to save the annotation cost, and enable the classification model to achieve better recognition accuracy. However, due to the low incidence of retinal detachment, the corresponding retinal detachment data is difficult to collect, and the amount of data that can be obtained is small; while the incidence of pathological myopia is relatively high relative to retinal detachment, data collection is relatively easy, and the amount of data that can be obtained is large. In this case, a segmentation model is used for less retinal detachment data. In order to allow the segmentation model to simultaneously extract the characteristics of pathological myopia, the data type is weighed and labeled when the data is labeled, that is, all retinal detachment data is segmented and labeled before training the segmentation model, and an appropriate amount of pathological myopia data is taken for segmentation and labeling, which can save the time and labor cost of labeling. At the same time, the trained segmentation network can better learn the characteristics of the two diseases of pathological myopia and retinal detachment, and finally have a higher recognition accuracy.

[0153] Wherein, the feature map of each fundus image is subjected to segmentation preprocessing and classification preprocessing (which may also be understood as segmentation preprocessing and classification preprocessing) to obtain a segmentation result and a classification result of each fundus image, including:

[0154] Among them, in this step, the feature map of each fundus image is segmented (i.e., segmentation preprocessing) to obtain the segmentation result of each fundus image, including:

[0155] First, upsample the feature map of each fundus image and then output the segmentation result; calculate the sum of the pixel points of the lesion area of ​​each fundus image in the segmentation result; then, determine whether the sum of the pixel points of the lesion area is greater than the first preset threshold; if the sum of the pixel points of the lesion area is greater than the first preset threshold, determine that the segmentation result of the corresponding fundus image is a fundus disease; if the sum of the pixel points of the lesion area is not greater than the first preset threshold, determine that the segmentation result of the corresponding fundus image is not a fundus disease. Among them, the first preset threshold can be any integer between 15 and 25, such as 20. Of course, this embodiment is not limited to the above process in specific applications, and other segmentation preprocessing methods can also be used, which is not limited by this embodiment.

[0156] Among them, in this step, the feature map of each fundus image is classified (i.e., classification preprocessing) to obtain the classification result of each fundus image, including:

[0157] First, the feature map of each fundus image is processed in dimension and classification in turn through the fully connected layer to obtain the corresponding classification result; then, it is determined which type of fundus lesion the classification result is. Specifically, it includes:

[0158] If the classification result is retinal detachment, the classification confidence of the retinal detachment is calculated, and when the classification confidence of the retinal detachment is greater than a second threshold (such as 0.85, etc., but in actual applications, it is not limited to this, it can also be 0, 7 or 0.8 or 0.9, etc.), the classification result is determined to be retinal detachment. Among them, the classification confidence is the probability of the predicted category given by the classifier, and whether the result of the classifier itself is reliable is judged based on the confidence. If there are multiple classifiers for decision fusion, then the confidence is needed to weight the results given by each classifier (which may be probabilities or category labels). After these weighted single classifier results are fused through different decision criteria (such as DS, LOP, LOGP, etc.), the final classification result is given.

[0159] If the classification result is pathological myopia, the classification confidence of pathological myopia is calculated, and when the classification confidence of pathological myopia is greater than a third threshold (for example, 0.65, but in actual applications, it is not limited to this, it can also be 0.63 or 0.68 or 0.7, etc.), the classification result is determined to be pathological myopia.

[0160] If the classification result is no obvious lesions, the classification confidence of no obvious lesions is calculated, and when the classification confidence of no obvious lesions is greater than a fourth threshold (for example, 0.6, but in actual applications, it is not limited to this, it can also be 0.55 or 0.58 or 0.62, etc.), the classification result is determined to be no obvious lesions.

[0161] Of course, this embodiment is not limited to the above process in specific applications, and other segmentation preprocessing methods may also be used, which is not limited in this embodiment.

[0162] In step 103, the segmentation result and the classification result of each fundus image are fused and predicted to obtain the recognition result of the eye disease in each fundus image.

[0163] Among them, in this step, a logical judgment is performed on the segmentation result and the classification result of each fundus image to obtain the recognition result of the eye disease in each fundus image. That is to say, the results of the segmentation model and the classification model are combined for fusion prediction, and specifically, a judgment logic is performed, and the judgment logic is that since the segmentation model is more effective for retinal detachment, the result of retinal detachment is mainly concluded by the segmentation model; and the classification model is more effective for pathological myopia and images without obvious lesions, and the results of pathological myopia and no obvious lesions are mainly concluded by the classification model. When the result of the segmentation model and the result of the classification model are the same, the same result is output; when the results of the two are different, a logical judgment is performed, specifically including; if the segmentation result includes retinal detachment, the segmentation result and the classification result are operated with an OR logic operation to obtain a prediction result including retinal detachment; wherein, the OR logic operation is that if the segmentation result and the classification result are represented by A+B, when A and B are both 0, its value is 0, otherwise it is 1; or

[0164] If any one of diffuse choroidal atrophy, patchy choroidal atrophy and macular atrophy exists in the classification result, the classification result and the segmentation result are subjected to an OR logic operation to obtain a prediction result of pathological myopia;

[0165] The results in other cases are explained as follows: if the segmentation result is no disease and the classification result is retinal detachment and the probability of the classification result is greater than the classification threshold, the two results are ORed to obtain the result of retinal detachment; if the segmentation result is pathological myopia and the classification result is no disease, the two results are ORed to obtain the result of pathological myopia; if the segmentation result is pathological myopia and the classification result is retinal detachment and the probability of the classification result is greater than the classification threshold, the result of pathological myopia or retinal detachment is obtained, and the patient is advised to go to the hospital for treatment.

[0166] If both the classification result and the segmentation result indicate that no obvious lesions are found, a prediction result indicating that no obvious lesions are found is obtained.

[0167] Of course, in practical applications, it is not limited to the above situation. Specifically, prediction can be performed according to the above logical judgment operation, which will not be explained one by one in this embodiment.

[0168] That is to say, under normal circumstances, the result of the segmentation model and the result of the classification model may be the same or different. In the same case, for example, the output result of the segmentation model is retinal detachment, and the output result of the classification model is also retinal detachment, then the output result is retinal detachment, that is, the output result of the logical operation judgment is retinal detachment; in different cases, for example, the output result of the segmentation model is retinal detachment, and the output result of the classification model is pathological myopia, the above-mentioned logical operation judgment is required to output the result after the logical judgment.

[0169] In an embodiment of the present invention, one or more fundus images to be identified are first obtained; a deep convolutional neural network is used to extract features from each fundus image to obtain a feature map of each fundus image; and segmentation processing and classification processing are respectively performed on the feature map of each fundus image to obtain a segmentation result and a classification result of each fundus image; the segmentation result and the classification result of each fundus image are fused and predicted to obtain the identification result of the eye disease in each fundus image. In other words, the present invention performs segmentation processing and classification processing on the feature map of each fundus image obtained, and fusion prediction is performed on the classification result and the segmentation result to obtain the identification result of the eye disease in each fundus image, thereby improving the accuracy of identifying eye diseases based on fundus images, so that the final accuracy of eye disease prediction is maintained at a high level, and the health and safety of users are guaranteed.

[0170] Optionally, in another embodiment, based on the above embodiment, before using the deep convolutional neural network to extract features from each acquired fundus image, the method further includes: pre-training the segmentation model in the deep convolutional neural network, and continuing to train the classification model in the deep convolutional neural network based on the parameters of the trained segmentation model; and using the parameters of the trained classification model as parameters of the trained deep convolutional neural network.

[0171] That is to say, in the embodiment of the present invention, the incidence of retinal detachment is relatively low, such as 0.0001. Due to its low incidence, fundus images of retinal detachment are difficult to collect, and the acquired data of retinal detachment are relatively small. The incidence of pathological myopia is relatively high, such as about 2%. Relatively speaking, fundus images of pathological myopia are also easier to collect, and the amount of data that can be collected is also large. In the case of less retinal detachment data, if only the classification network is used to train the disease recognition model of pathological myopia and retinal detachment, the disease recognition model cannot fully learn the characteristics of the retinal detachment data, and the classification effect of the retinal detachment data will be poor.

[0172] Therefore, the present invention implements the use of image segmentation network for training. Its purpose is to enable the model to learn the characteristics of retinal detachment and pathological myopia at the same time. Before training the network, the data of retinal detachment and pathological myopia are segmented and labeled at the same time. Considering the high cost of image segmentation and labeling, in order to balance the learning effect of the segmentation model on the characteristics of the two diseases of pathological myopia and retinal detachment, the data of the two diseases are weighed and labeled according to the amount of data, that is, all the data of retinal detachment with a small amount are segmented and labeled, and then some representative data are selected from all the data of pathological myopia, and the number of selected data is about 1.5 times the total amount of retinal detachment data. The selected pathological myopia data is also segmented and labeled, which will greatly reduce the labeling cost. That is, when there is less retinal detachment data and more pathological myopia data, the two categories of data are weighed and labeled, which reduces the labeling cost, and the model can simultaneously identify pathological myopia and retinal detachment.

[0173] Then, the labeled retinal detachment data and case myopia data are taken as positive samples, and the normal fundus images and leopard fundus images without obvious lesions, which are equal to the number of positive samples, are taken as negative samples, and these data are used to train the segmentation model. Figure 2 As shown, Figure 2 A flow chart of a segmentation model training provided by an embodiment of the present invention. Figure 2 As shown in Figure 1, the process of pre-training the segmentation model includes:

[0174] Step 201: Obtain a first training set, wherein the first training set includes: a plurality of sample fundus images, and a first annotated image of eye diseases and a second annotated image of normal fundus images without obvious lesions and leopard fundus images in the plurality of sample fundus images, wherein the first annotated image of eye diseases includes at least annotated data of pathological myopia and retinal detachment;

[0175] Step 202: input the first training set as input to a deep convolutional neural network (such as a ResNet101 network, etc.); that is, input the plurality of sample fundus images into an initial segmentation model (i.e., a ResNet101 network) for training;

[0176] During the training process, the features of lesions related to eye diseases and the features of normal fundus images and leopard fundus images without obvious lesions are extracted from each input fundus image, and the corresponding feature map is output (i.e., step 203, where the size of the feature map output is small, which is one-N times the size of the original image, for example, one-32 times the size of the original image), and the features of lesions related to eye diseases and the features of normal fundus images and leopard fundus images without obvious lesions in the extracted feature map are compared with the corresponding first annotation map and the second annotation map, respectively, and a loss value LOSS is calculated according to the difference between the comparison results (step 204), and based on the loss value, the parameters of the segmentation model are updated through a back propagation mechanism, and after multiple iterations, until the segmentation model converges, the output feature map of the segmentation model when it converges is upsampled (step 205) to restore it to the size of the original sample fundus image, and the segmentation result is output (i.e., step 206), thereby obtaining a trained segmentation model.

[0177] ResNet101 has two basic blocks, namely Conv Block and Identity Block. The input and output dimensions of Conv Block are different, so they cannot be connected in series. Its function is to change the dimension of the network. The input and output dimensions of Identity Block are the same, so they can be connected in series to deepen the network.

[0178] Resnet101 uses 4 modules composed of residual blocks, corresponding to the bottleneck of the above network. Each module uses several residual blocks with the same number of output channels. In the first residual block of each module, the number of channels of the previous module is doubled and the height and width are reduced.

[0179] ResNet, also known as residual neural network, refers to the idea of ​​adding residual learning to traditional convolutional neural networks. It solves the problems of gradient diffusion and precision degradation (training set) in deep networks, allowing the network to become deeper and deeper, ensuring both accuracy and speed.

[0180] In this embodiment, the segmentation model obtained through the above training process can simultaneously learn the lesion features of pathological myopia and retinal detachment, and because the amount of training data for these two diseases is relatively balanced, the model has a strong ability to extract features for both diseases.

[0181] The trained segmentation model can extract the image features of pathological myopia fundus images, retinal detachment fundus images, and images without obvious lesions and leopard fundus images. However, since the segmentation model only uses a small number of pathological myopia images and a small number of negative images as training samples during the training process, the segmentation model does not fully utilize the image information of a large number of pathological myopia fundus images, fundus images without obvious lesions, and leopard fundus images collected, and the generalization performance of the model needs to be improved. Based on this, all the collected data are classified and labeled, and the classification network is trained. Based on this, in this implementation, the basic parameters of the segmentation network are used as the basic parameters of the classification network, and a fully connected layer and softmax are added after the last feature layer to train the classification model. The categories of the classification model include: no obvious lesions, leopard fundus, diffuse choroidal atrophy, patchy choroidal atrophy, macular atrophy and retinal detachment, a total of 6 classification categories. In the process of training the classification network, the feature layer parameters of the above-mentioned trained segmentation model are fixed, and only the last added fully connected layer is trained, which saves the training cost and can quickly obtain the classification model. The training process of the classification network is as follows Figure 3 As shown, Figure 3 A flow chart of a classification model training provided by an embodiment of the present invention. Figure 3 As shown, the process of pre-training the classification model includes: based on the parameters of the trained segmentation model, continuing to train the classification model in the deep convolutional neural network, including:

[0182] Step 301: Obtain a second training set, wherein the second training set includes: a plurality of sample fundus images and classification labels of eye diseases and non-eye diseases in the plurality of sample fundus images, wherein the classification labels at least include: no obvious lesions, leopard-like fundus, diffuse choroidal atrophy, patchy choroidal atrophy, macular atrophy, and retinal detachment;

[0183] Step 302: The second training set is used as input to a deep convolutional neural network (ResNet101 network, which is based on Figure 1 Parameters of the segmentation model trained in the training), that is, inputting the multiple sample fundus images into the initial classification model for training, wherein the parameters of the initial classification model are the parameters of the trained segmentation model;

[0184] During the training process, classification features related to eye diseases and non-eye diseases are extracted from each of the multiple sample fundus images, and the corresponding feature map is output (step 303). The extracted classification features are sequentially subjected to dimension and classification optimization training through two fully connected layers (i.e., fully connected FC layer 304 and FC layer 305), and the training results are compared with the classification labels of the eye diseases and non-eye diseases. The loss value is calculated according to the difference in the comparison results, and based on the loss value LOSS (step 306), the parameters of the classification model are updated through a back propagation mechanism. After multiple iterations, the results of each iteration are sequentially subjected to dimension and classification processing through two fully connected layers until the classification model converges and the classification result is output (step 307), so that it is considered that the class model has been trained.

[0185] Since there are a large number of images of pathological myopia, images without obvious lesions and leopard fundus images, the classification model makes full use of the image information of these images. Therefore, on the premise of being able to identify fundus images without disease, fundus images of pathological myopia and fundus images of retinal detachment, the classification model has a good classification effect on pathological myopia and fundus images without disease.

[0186] Since the classification network and the segmentation network are based on the feature extraction network, after the classification network training is completed, combined with the upsampling layer of the segmentation network, one network can be used to obtain the classification prediction results and the segmentation prediction results at the same time. The final model structure is as follows Figure 4 , Figure 4 A schematic diagram of a method for performing fusion prediction based on the results of a classification model and a segmentation model provided by an embodiment of the present invention. The method comprises:

[0187] Step 401: Acquire one or more fundus images to be identified.

[0188] Step 402: Input one or more fundus images to be identified into a pre-trained deep convolutional neural network (here, a trained classification model), perform feature extraction on each fundus image obtained, and output a feature map corresponding to each fundus image (i.e., step 403), wherein the deep convolutional neural network includes: a trained classification model, and the classification model is trained based on the model parameters of the trained segmentation model.

[0189] In this step, the specific training process of the segmentation model and the classification model is detailed in the corresponding process above and will not be repeated here.

[0190] Step 404: up-sample the feature map corresponding to each output fundus image.

[0191] Step 405: Output the segmentation result of each fundus image.

[0192] Step 406 and step 407: perform two fully connected FC layers on the feature map corresponding to each output fundus image, and perform dimension and classification optimization training in turn.

[0193] Step 408: Output the classification result of each fundus image.

[0194] Step 409: Perform logical judgment on the segmentation result and classification result of each fundus image to obtain the recognition result of the eye disease in each fundus image.

[0195] Step 410: Output the recognition result.

[0196] In the embodiment of the invention, before performing logical judgment on the output results of the above-mentioned classification model and the output results of the segmentation model, it is necessary to perform preliminary processing on the results of the model.

[0197] Because the training of the segmentation network may be affected by the noise in the image, after using the segmentation network to obtain the segmentation result, the segmentation result is first preprocessed, that is, to determine whether the sum of the pixels in the lesion area in the obtained segmentation map is greater than 20. If it is greater than 20, the result of the segmentation model is judged as the corresponding fundus disease, otherwise the segmentation result is judged as no obvious lesion.

[0198] When training the classification network, there are fewer retinal detachment data in the training data, and more images of pathological myopia and images without fundus lesions. Therefore, the model's ability to judge retinal detachment is relatively poor, and its ability to judge images of pathological myopia and images without fundus lesions is stronger. After obtaining the classification results, preliminary preprocessing is also required, namely:

[0199] If the result of the classification model is retinal detachment, and the classification confidence of retinal detachment is greater than 0.85, then the classification result of the model is considered to be retinal detachment, otherwise the result of the classification model is abandoned;

[0200] If the result of the classification model is pathological myopia, and the classification confidence of pathological myopia is greater than 0.65, then the prediction result of the classification model is considered to be pathological myopia, otherwise the result of the classification model is abandoned;

[0201] If the result of the classification model is that no obvious lesions are found and the classification confidence of no obvious lesions is greater than 0.6, then the prediction result of the classification model is considered to be no obvious lesions. Otherwise, the result of the classification model is also abandoned.

[0202] The model combines the results of the segmentation model and the classification model for fusion prediction in the final prediction. The main judgment logic is that the segmentation model is more effective for retinal detachment, and the results of retinal detachment are mainly drawn by the segmentation model; the classification model is more effective for pathological myopia and images without obvious lesions, and the results of pathological myopia and images without obvious lesions are mainly drawn by the classification model. Specifically:

[0203] If the result of the segmentation model includes retinal detachment, the result of the segmentation model and the result of the classification model are ORed to obtain the result of retinal detachment;

[0204] If any one of diffuse choroidal atrophy, patchy choroidal atrophy and macular atrophy exists in the classification model, the result of the classification model and the result of the segmentation model are OR-operated to obtain the prediction result of pathological myopia;

[0205] If it is other cases, the results of the segmentation model and the classification model are XORed. When the results of the two models are different, an OR operation is performed.

[0206] When the results of both models are no obvious lesions, the final prediction result of the model is no obvious lesions.

[0207] In the implementation of the present invention, the segmentation network and the classification network share the previous feature extraction layer, and the same network is used to obtain the classification and segmentation results at the same time, so that the model training cost is reduced and the model prediction time is also shorter. At the same time, the respective characteristics of the classification model and the segmentation model are utilized to improve the accuracy of identifying eye diseases based on fundus images, so that the prediction accuracy of the final model is maintained at a high level and the health and safety of users is guaranteed.

[0208] In the implementation of the present invention, a deep convolutional neural network is used to simultaneously perform auxiliary diagnosis of pathological myopia and retinal detachment, two myopia-related eye diseases, and weigh the labeling according to the actual amount of data collected to save labeling costs, and enable the model to achieve better recognition accuracy. Since the incidence of retinal detachment is very low, the corresponding retinal detachment data is difficult to collect, and the amount of data that can be obtained is small; while the incidence of pathological myopia is relatively high relative to retinal detachment, data collection is relatively easy, and the amount of data that can be obtained is large. In this case, a segmentation model is used for less retinal detachment data. In order to allow the segmentation model to simultaneously extract the characteristics of pathological myopia, the labeling is weighed according to the amount of data of the disease type when the data is labeled, that is, all retinal detachment data is segmented and labeled before training the segmentation model, and an appropriate amount of pathological myopia data is taken for segmentation and labeling, which can save labeling time and labor costs. At the same time, the trained segmentation network can better learn the characteristics of the two diseases, pathological myopia and retinal detachment, and finally have a higher recognition accuracy.

[0209] In an embodiment of the present invention, a training method combining image classification and image segmentation is used simultaneously. After the image segmentation network is trained, the model parameters in front of the segmentation network are fixed, and only the newly added fully connected layer at the end of the network is trained to train the classification network. After the segmentation network is trained, the network has a certain ability to extract features and can simultaneously extract the features of pathological myopia and retinal detachment. When training the classification network, the model parameters of the segmentation network are fixed, and the last fully connected layer of the classification network is trained separately, which can reduce the training cost while ensuring the accuracy of the model. When the final prediction is made, the same network can be used to obtain the segmentation result and the classification result at the same time, which reduces the prediction time when the model is used and improves the user experience.

[0210] In the embodiment of the present invention, after the network training is completed, the classification results and segmentation results obtained by the network are combined for fusion prediction. When making the final decision, the results of the classification and segmentation models are combined to make the final prediction. In the embodiment of the present invention, the classification results and segmentation results of the network are combined for fusion prediction, and the final disease recognition effect is better, and the recognition accuracy of pathological myopia and retinal detachment is higher.

[0211] The embodiment of the present invention also provides a segmentation model training method, the method comprising:

[0212] 1) Obtaining a first training set, the first training set comprising: a plurality of sample fundus images, the plurality of sample fundus images comprising: a first annotated image of an eye disease and a second annotated image without obvious lesions, wherein the first annotated image of an eye disease at least comprises: annotated data of pathological myopia and retinal detachment; the second annotated image without obvious lesions comprises: a normal fundus image and a leopard fundus image;

[0213] 2) inputting the plurality of sample fundus images into an initial segmentation model;

[0214] 3) Extract the features of lesions related to eye diseases in each input fundus image, as well as the features of normal fundus images and leopard fundus images without obvious lesions;

[0215] 4) comparing the extracted lesion features and features of the normal fundus image without obvious lesions and the leopard fundus image with the corresponding first annotated image and the second annotated image respectively;

[0216] 5) Calculate the loss value based on the difference in the comparison results;

[0217] 6) Based on the loss value, the parameters of the segmentation model are updated through a back propagation mechanism, and after multiple iterations, the segmentation model converges, and the output image when the segmentation model converges is upsampled to restore it to the size of the original sample fundus image, thereby obtaining a trained segmentation model.

[0218] In this embodiment, the implementation process of each step is detailed in the implementation process of the corresponding step in the above embodiment, which will not be repeated here.

[0219] In the implementation of the present invention, based on the situation that there is less retinal detachment data and more pathological myopia data, the two categories of data are weighed and labeled, which not only reduces the labeling cost, but also enables the segmentation model to simultaneously identify pathological myopia and retinal detachment. That is to say, this embodiment uses the labeled retinal detachment data and case myopia data as positive samples, and takes normal fundus images and leopard fundus images without obvious lesions that are equivalent to the number of positive samples as negative samples, and uses these data to train the segmentation model. The segmentation model obtained through the above training process can simultaneously learn the lesion features of pathological myopia and retinal detachment, and because the amount of training data for these two diseases is relatively balanced, the model has a strong ability to extract features for these two diseases.

[0220] In addition, an embodiment of the present invention further provides a classification model training method, the method comprising:

[0221] 1) obtaining a second training set, wherein the second training set includes: a plurality of sample fundus images, wherein the plurality of sample fundus images include classification labels of eye diseases and non-eye diseases, wherein the classification labels at least include: no obvious lesions, leopard-like fundus, diffuse choroidal atrophy, patchy choroidal atrophy, macular atrophy, and retinal detachment;

[0222] 2) inputting the plurality of sample fundus images into an initial classification model, wherein the parameters of the initial classification model are the parameters of the trained segmentation model;

[0223] 3) extracting classification features related to eye diseases and non-eye diseases in each of the plurality of sample fundus images;

[0224] 4) Training the extracted classification features in terms of dimension and classification through a fully connected layer;

[0225] 5) Compare the training results with the classification labels, and calculate the loss value according to the difference in the comparison results;

[0226] 6) Based on the loss value, the parameters of the classification model are updated through the back propagation mechanism. After multiple iterations, the results of each iteration are processed twice by the fully connected layer for dimension and classification until the classification model converges to obtain a trained classification model.

[0227] In this embodiment, the implementation process of each step is detailed in the implementation process of the corresponding step in the above embodiment, which will not be repeated here.

[0228] In the embodiment of the present invention, the parameters of the trained segmentation model are used as the basic feature layer parameters, and the basic feature layer parameters are fixed, and only the last added fully connected layer is trained, which saves training costs and can quickly obtain a classification model.

[0229] It should be noted that, for the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that this implementation disclosure is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present invention.

[0230] See also Figure 5 , is a block diagram of an eye disease identification device based on fundus images provided by an embodiment of the present invention. Figure 5 The device comprises: an acquisition module 501, a processing module 502 and a fusion prediction module 503, wherein:

[0231] The acquisition module 501 is used to acquire one or more fundus images to be identified;

[0232] The processing module 502 is used to extract features from each acquired fundus image using a deep convolutional neural network to obtain a feature map of each fundus image; and to perform segmentation processing and classification processing on the feature map of each fundus image to obtain a segmentation result and a classification result of each fundus image;

[0233] The fusion prediction module 503 is used to perform fusion prediction on the segmentation results and classification results of each fundus image to obtain the recognition results of eye diseases in each fundus image, wherein the deep convolutional neural network includes: a trained segmentation model and a classification model, and the classification model is trained based on the model parameters of the trained segmentation model.

[0234] Optionally, in another embodiment, based on the above embodiment, the device further includes: a first training module 601, a second training module 602 and a determination module 603, and its structural diagram is shown in FIG. Figure 6 As shown,

[0235] The first training module 601 is used to pre-train the segmentation model in the deep convolutional neural network before the processing module uses the deep convolutional neural network to extract features from each acquired fundus image; and

[0236] The second training module 602 is used to continue training the classification model in the deep convolutional neural network based on the parameters of the trained segmentation model;

[0237] The determination module 603 is used to use the parameters of the classification model trained by the second training module 602 as the parameters of the trained deep convolutional neural network.

[0238] Optionally, in another embodiment, based on the above embodiment, the first training module 601 includes: a first acquisition module 701, a first input module 702, a first extraction module 703, a first comparison module 704, a first calculation module 705, a first iteration module 706 and a sampling processing module 707; the second training module 602 includes: a second acquisition module 708, a second input module 709, a second extraction module 710, a second comparison module 711, a second calculation module 712, a second iteration module 713 and a dimensional classification processing module 714, wherein.

[0239] The first acquisition module 701 is used to acquire a first training set, wherein the first training set includes: a plurality of sample fundus images, wherein the plurality of sample fundus images include a first annotated image of an eye disease and a second annotated image without obvious lesions, wherein the first annotated image of an eye disease includes at least annotated data of pathological myopia and retinal detachment; and the second annotated image without obvious lesions includes a normal fundus image and a leopard fundus image;

[0240] The first input module 702 is used to input the plurality of sample fundus images into an initial segmentation model;

[0241] The first extraction module 703 is used to extract the features of lesions related to eye diseases in each input fundus image, as well as the features of normal fundus images without obvious lesions and leopard fundus images;

[0242] The first comparison module 704 is used to compare the extracted lesion features and the features of the normal fundus image without obvious lesions and the leopard fundus image with the corresponding first annotated image and the second annotated image respectively;

[0243] The first calculation module 705 is used to calculate the loss value according to the difference of the comparison result;

[0244] The first iteration module 706 is used to update the parameters of the segmentation model through a back propagation mechanism based on the loss value, and perform multiple iterations until the segmentation model converges;

[0245] The sampling processing module 707 is used to perform up-sampling processing on the output image when the segmentation model converges, and restore it to the size of the original sample fundus image to obtain a trained segmentation model;

[0246] The second acquisition module 708 is used to acquire a second training set, wherein the second training set includes: a plurality of sample fundus images, wherein the plurality of sample fundus images include classification labels of eye diseases and non-eye diseases, wherein the classification labels at least include: no obvious lesions, leopard-like fundus, diffuse choroidal atrophy, patchy choroidal atrophy, macular atrophy, and retinal detachment;

[0247] The second input module 709 is used to input the plurality of sample fundus images into an initial classification model, wherein the parameters of the initial classification model are the parameters of the trained segmentation model.

[0248] The second extraction module 710 is used to extract classification features related to eye diseases and non-eye diseases in each of the plurality of sample fundus images.

[0249] The second comparison module 711 is used to perform dimension and classification training on the extracted classification features through a fully connected layer, and compare the training results with the classification labels.

[0250] The second calculation module 712 is used to calculate the loss value according to the difference of the comparison result.

[0251] The second iteration module 713 updates the parameters of the classification model based on the loss value through a back propagation mechanism. After multiple iterations,

[0252] The dimension classification processing module 714 is used to perform dimension and classification processing on each iteration result in two fully connected layers in sequence until the classification model converges to obtain a trained classification model.

[0253] Optionally, in another embodiment, based on the above embodiment, the processing module 503 includes: a segmentation processing module and a classification processing module (not shown in the figure), wherein:

[0254] The segmentation processing module includes: an upsampling module, a calculation module and a first disease determination module, wherein:

[0255] The upsampling module is used to upsample the feature map of each fundus image and then output a segmentation result;

[0256] The calculation module is used to calculate the sum of the pixel points of the lesion area of ​​each fundus image in the segmentation result;

[0257] The first disease determination module is used to determine that the segmentation result of the corresponding fundus image is a fundus disease when the sum of the pixel points in the lesion area is greater than a first preset threshold; or to determine that the segmentation result of the corresponding fundus image is not a fundus disease when the sum of the pixel points in the lesion area is not greater than the first preset threshold;

[0258] The classification processing module includes: a classification module and a second disease determination module, wherein:

[0259] The classification module is used to sequentially perform dimension and classification processing on the feature map of each fundus image through a fully connected layer to obtain a corresponding classification result;

[0260] The second disease determination module is used to calculate the classification confidence of retinal detachment when the classification result is retinal detachment, and determine that the classification result is retinal detachment when the classification confidence of the retinal detachment is greater than a second threshold; or calculate the classification confidence of pathological myopia when the classification result is pathological myopia, and determine that the classification result is pathological myopia when the classification confidence of pathological myopia is greater than a third threshold; or calculate the classification confidence of no obvious lesions when the classification result is no obvious lesions, and determine that the classification result is no obvious lesions when the classification confidence of no obvious lesions is greater than a fourth threshold.

[0261] Optionally, in another embodiment, based on the above embodiment, the fusion prediction module includes: a first fusion prediction module and / or a second fusion prediction module, wherein:

[0262] The first fusion prediction module is used for outputting the same result if the segmentation result and classification result of each fundus image are the same;

[0263] The second fusion prediction module is used to perform logical judgment on the segmentation result and classification result of each fundus image when the segmentation result and classification result of each fundus image are different, so as to obtain the recognition result of the eye disease in each fundus image.

[0264] Optionally, in another embodiment, based on the above embodiment, the second fusion prediction module includes: a first logic calculation module, a second logic calculation module, a third logic calculation module, a fourth logic calculation module and a fifth logic calculation module, wherein:

[0265] The first logic calculation module is used for, when the segmentation result and the classification result of each fundus image are different, if the segmentation result includes retinal detachment, performing an OR logic operation on the segmentation result and the classification result to obtain a prediction result including retinal detachment; or

[0266] The second logic calculation module is used for, when the segmentation result and the classification result of each fundus image are different, if any one of diffuse choroidal atrophy, patchy choroidal atrophy and macular atrophy exists in the classification result, performing a logic OR operation on the classification result and the segmentation result to obtain a prediction result of pathological myopia;

[0267] The third logic calculation module is used for, when the segmentation result and the classification result of each fundus image are different, if the segmentation result is that no obvious lesions are found, and the classification result is retinal detachment and the probability of the classification result is greater than the classification threshold, then performing an OR operation on the segmentation result and the classification result to obtain a prediction result of retinal detachment;

[0268] The fourth logic calculation module is used for, when the segmentation result and the classification result of each fundus image are different, if the segmentation result is pathological myopia and the classification result is no obvious lesion, performing an OR operation on the segmentation result and the classification result to obtain a prediction result of pathological myopia;

[0269] The fifth logic calculation module is used to, when the segmentation result and the classification result of each fundus image are different, if the segmentation result is pathological myopia and the classification result is no obvious lesion, perform an OR operation on the segmentation result and the classification result to obtain a prediction result of pathological myopia.

[0270] The embodiment of the present invention also provides a segmentation model training device, which includes: an acquisition module, an input module, an extraction module, a comparison module, a calculation module, an iteration module and a sampling processing module; wherein,

[0271] The acquisition module is used to acquire a first training set, wherein the first training set includes: a plurality of sample fundus images, wherein the plurality of sample fundus images include: a first annotated image of an eye disease and a second annotated image without obvious lesions, wherein the first annotated image of an eye disease includes at least annotated data of pathological myopia and retinal detachment; and the second annotated image without obvious lesions includes: a normal fundus image and a leopard fundus image;

[0272] The input module is used to input the plurality of sample fundus images into an initial segmentation model;

[0273] The extraction module is used to extract the features of lesions related to eye diseases in each input fundus image, as well as the features of normal fundus images without obvious lesions and leopard fundus images;

[0274] The comparison module is used to compare the extracted lesion features and features of the normal fundus image without obvious lesions and the leopard fundus image with the corresponding first annotated image and the second annotated image respectively;

[0275] The calculation module is used to calculate the loss value according to the difference of the comparison results;

[0276] The iteration module is used to update the parameters of the segmentation model through a back propagation mechanism based on the loss value, and perform multiple iterations until the segmentation model converges;

[0277] The sampling processing module is used to perform up-sampling processing on the output image when the segmentation model converges, restore it to the size of the original sample fundus image, and obtain a trained segmentation model.

[0278] Optionally, an embodiment of the present invention further provides a classification model training device, the device comprising: an acquisition module, an input module, an extraction module, a comparison module, a calculation module, an iteration module and a dimension classification processing module, wherein:

[0279] The acquisition module is used to acquire a second training set, wherein the second training set includes: a plurality of sample fundus images, wherein the plurality of sample fundus images include classification labels of eye diseases and non-eye diseases, wherein the classification labels at least include: no obvious lesions, leopard-like fundus, diffuse choroidal atrophy, patchy choroidal atrophy, macular atrophy, and retinal detachment;

[0280] The input module is used to input the plurality of sample fundus images into an initial classification model, wherein the parameters of the initial classification model are the parameters of the trained segmentation model;

[0281] The extraction module is used to extract classification features related to eye diseases and non-eye diseases in each of the plurality of sample fundus images;

[0282] The comparison module is used to perform dimension and classification training on the extracted classification features through a fully connected layer;

[0283] The calculation module is used to compare the training result with the classification label and calculate the loss value according to the difference of the comparison result;

[0284] The iterative module is used to update the parameters of the classification model through a back propagation mechanism based on the loss value, after multiple iterations; dimension classification

[0285] The dimension classification processing module is used to perform dimension and classification processing on each iteration result in two fully connected layers in sequence until the classification model converges to obtain a trained classification model.

[0286] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0287] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0288] Optionally, an embodiment of the present invention further provides an electronic device, including:

[0289] processor;

[0290] a memory for storing instructions executable by the processor;

[0291] Wherein, the processor is configured to execute the instructions to implement the eye disease identification method based on fundus images as described above.

[0292] Optionally, an embodiment of the present invention further provides a computer-readable storage medium, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the eye disease identification method based on fundus images as described above.

[0293] Optionally, an embodiment of the present invention further provides a computer program product, including a computer program or instructions, which, when executed by a processor of an electronic device, implements the above-mentioned method for identifying eye diseases based on fundus images.

[0294] See also Figure 8 , is a block diagram of an electronic device 800 provided in an embodiment of the present invention. For example, the electronic device 800 may be a mobile terminal or a server, and the electronic device is a mobile terminal as an example for explanation in the embodiment of the present invention. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0295] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0296] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0297] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0298] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0299] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0300] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0301] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0302] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0303] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0304] In an embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned fundus image-based eye disease identification method.

[0305] In an embodiment, a computer-readable storage medium is also provided, and when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device 800 can perform the above-mentioned method for identifying eye diseases based on fundus images. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0306] In an embodiment, a computer program product is also provided, including a computer program or instructions, which, when executed by the processor 820 of the electronic device 800, enables the electronic device 800 to perform the above-mentioned method for identifying eye diseases based on fundus images.

[0307] Fig. 9 is a block diagram of an apparatus 900 for identifying eye diseases based on fundus images provided by an embodiment of the present invention. For example, the apparatus 900 may be provided as a server. Fig. 9 , the apparatus 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932 for storing instructions, such as an application, that can be executed by the processing component 922. The application stored in the memory 932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute the instructions to perform the above method.

[0308] The device 900 may also include a power supply component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0309] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0310] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for identifying eye diseases based on fundus images, characterized in that: include: Acquire one or more fundus images to be identified; Using a deep convolutional neural network to extract features from each acquired fundus image to obtain a feature map of each fundus image, performing segmentation processing and classification processing on the feature map of each fundus image to obtain a segmentation result and a classification result of each fundus image, wherein the deep convolutional neural network includes: a trained segmentation model and a classification model, and the classification model is trained based on model parameters of the trained segmentation model; The segmentation result and the classification result of each fundus image are fused and predicted to obtain the recognition result of the eye disease in each fundus image.

2. The method for identifying eye diseases based on fundus images according to claim 1, characterized in that: Before extracting features from each acquired fundus image using a deep convolutional neural network, the method further includes: Pre-training the segmentation model in the deep convolutional neural network, and continuing to train the classification model in the deep convolutional neural network based on the parameters of the trained segmentation model; The parameters of the trained classification model are used as parameters of the trained deep convolutional neural network.

3. The method for identifying eye diseases based on fundus images according to claim 2, characterized in that: The pre-training of the segmentation model in the deep convolutional neural network includes: Acquire a first training set, the first training set comprising: a plurality of sample fundus images, the plurality of sample fundus images comprising: a first annotated image of an eye disease and a second annotated image without obvious lesions, wherein the first annotated image of an eye disease at least comprises: annotated data of pathological myopia and retinal detachment; the second annotated image without obvious lesions comprises: a normal fundus image and a leopard fundus image; Inputting the plurality of sample fundus images into an initial segmentation model, extracting features of lesions related to eye diseases in each input fundus image, as well as features of normal fundus images without obvious lesions and leopard fundus images; Comparing the extracted lesion features and features of a normal fundus image without obvious lesions and a leopard fundus image with the corresponding first annotated image and the second annotated image respectively; Calculate the loss value based on the gap in the comparison results; Based on the loss value, the parameters of the segmentation model are updated through the back propagation mechanism. After multiple iterations, the segmentation model converges, and the output image when the segmentation model converges is upsampled to restore it to the size of the original sample fundus image to obtain a trained segmentation model.

4. The method for identifying eye diseases based on fundus images according to claim 3, characterized in that: The step of continuing to train the pre-classification model in the deep convolutional neural network based on the trained parameters of the segmentation model includes: Acquire a second training set, the second training set comprising: a plurality of sample fundus images, the plurality of sample fundus images comprising classification labels of eye diseases and non-eye diseases, wherein the classification labels at least include: no obvious lesions, leopard-like fundus, diffuse choroidal atrophy, patchy choroidal atrophy, macular atrophy, and retinal detachment; Inputting the plurality of sample fundus images into an initial classification model, wherein the parameters of the initial classification model are the parameters of the trained segmentation model; Extracting classification features related to eye diseases and non-eye diseases from each of the multiple sample fundus images, and performing dimension and classification training on the extracted classification features through a fully connected layer; Compare the training result with the classification label, and calculate the loss value according to the difference between the comparison results; Based on the loss value, the parameters of the classification model are updated through the back propagation mechanism. After multiple iterations, the results of each iteration are processed twice by the fully connected layer for dimension and classification until the classification model converges to obtain a trained classification model.

5. The method for identifying eye diseases based on fundus images according to any one of claims 1 to 4, characterized in that: The segmentation process is performed on the feature map of each fundus image to obtain the segmentation result of each fundus image, including: Upsampling the feature map of each fundus image and outputting a segmentation result; Calculating the sum of the pixel points of the lesion area of ​​each fundus image in the segmentation result; When the sum of the pixel points in the lesion area is greater than a first preset threshold, the segmentation result of the corresponding fundus image is determined to be a fundus disease; or when the sum of the pixel points in the lesion area is not greater than the first preset threshold, the segmentation result of the corresponding fundus image is determined to be a non-fundus disease.

6. The method for identifying eye diseases based on fundus images according to any one of claims 1 to 4, characterized in that: The classifying process is performed on the feature map of each fundus image to obtain the classification result of each fundus image, including: The feature map of each fundus image is processed in sequence by dimension and classification through a fully connected layer to obtain a corresponding classification result; When the classification result is retinal detachment, calculating the classification confidence of the retinal detachment, and when the classification confidence of the retinal detachment is greater than a second threshold, determining that the classification result is retinal detachment; or When the classification result is pathological myopia, calculating the classification confidence of pathological myopia, and when the classification confidence of pathological myopia is greater than a third threshold, determining that the classification result is pathological myopia; or When the classification result is that no obvious lesions are found, the classification confidence of the no obvious lesions is calculated, and when the classification confidence of the no obvious lesions is greater than a fourth threshold, it is determined that the classification result is that no obvious lesions are found.

7. The method for identifying eye diseases based on fundus images according to any one of claims 1 to 4, characterized in that: The segmentation result and classification result of each fundus image are fused and predicted to obtain the recognition result of the eye disease in each fundus image, including: If the segmentation result and classification result of each fundus image are the same, outputting the same result; or If the segmentation result and the classification result of each fundus image are different, a logical judgment is performed on the segmentation result and the classification result of each fundus image to obtain the recognition result of the eye disease in each fundus image.

8. The method for identifying eye diseases based on fundus images according to claim 7, characterized in that: The logical judgment is performed on the segmentation result and the classification result of each fundus image to obtain the recognition result of the eye disease in each fundus image, including: If the segmentation result includes retinal detachment, performing an OR logic operation on the segmentation result and the classification result to obtain a prediction result including retinal detachment; or If any one of diffuse choroidal atrophy, patchy choroidal atrophy and macular atrophy exists in the classification result, the classification result and the segmentation result are subjected to an OR logic operation to obtain a prediction result of pathological myopia; or If the segmentation result is that no obvious lesions are found, and the classification result is retinal detachment and the probability of the classification result is greater than the classification threshold, the segmentation result and the classification result are ORed to obtain a prediction result of retinal detachment; or If the segmentation result is pathological myopia and the classification result is no obvious lesion, the segmentation result and the classification result are ORed to obtain a prediction result of pathological myopia; or If the segmentation result is pathological myopia, the classification result is retinal detachment and the probability of the classification result is greater than the classification threshold, a prediction result of pathological myopia or retinal detachment is obtained.

9. A segmentation model training method, characterized in that: The method comprises: Acquire a first training set, the first training set comprising: a plurality of sample fundus images, the plurality of sample fundus images comprising: a first annotated image of an eye disease and a second annotated image without obvious lesions, wherein the first annotated image of an eye disease at least comprises: annotated data of pathological myopia and retinal detachment; the second annotated image without obvious lesions comprises: a normal fundus image and a leopard fundus image; Inputting the plurality of sample fundus images into an initial segmentation model; Extract the features of lesions related to eye diseases in each input fundus image, as well as the features of normal fundus images and leopard fundus images without obvious lesions; Comparing the extracted lesion features and features of a normal fundus image without obvious lesions and a leopard fundus image with the corresponding first annotated image and the second annotated image respectively; Calculate the loss value based on the gap in the comparison results; Based on the loss value, the parameters of the segmentation model are updated through the back propagation mechanism. After multiple iterations, the segmentation model converges, and the output image when the segmentation model converges is upsampled to restore it to the size of the original sample fundus image to obtain a trained segmentation model.

10. A classification model training method, characterized in that: The method comprises: Acquire a second training set, the second training set comprising: a plurality of sample fundus images, the plurality of sample fundus images comprising classification labels of eye diseases and non-eye diseases, wherein the classification labels at least include: no obvious lesions, leopard-like fundus, diffuse choroidal atrophy, patchy choroidal atrophy, macular atrophy, and retinal detachment; Inputting the plurality of sample fundus images into an initial classification model, wherein the parameters of the initial classification model are the parameters of the trained segmentation model; Extracting classification features related to eye diseases and non-eye diseases in each of the plurality of sample fundus images; The extracted classification features are trained in terms of dimension and classification through a fully connected layer; Compare the training result with the classification label, and calculate the loss value according to the difference between the comparison results; Based on the loss value, the parameters of the classification model are updated through the back propagation mechanism. After multiple iterations, the results of each iteration are processed twice by the fully connected layer for dimension and classification until the classification model converges to obtain a trained classification model.

11. An eye disease identification device based on fundus images, characterized in that: include: An acquisition module, used for acquiring one or more fundus images to be identified; A processing module, used for extracting features from each acquired fundus image using a deep convolutional neural network to obtain a feature map of each fundus image, performing segmentation preprocessing and classification preprocessing on the feature map of each fundus image to obtain a segmentation result and a classification result of each fundus image, wherein the deep convolutional neural network includes: a trained segmentation model and a classification model, and the classification model is trained based on model parameters of the trained segmentation model; The fusion prediction module is used to perform fusion prediction on the segmentation result and the classification result of each fundus image to obtain the recognition result of the eye disease in each fundus image.

12. A segmentation model training device, characterized in that: The device comprises: An acquisition module is used to acquire a first training set, wherein the first training set includes: a plurality of sample fundus images, wherein the plurality of sample fundus images include: a first annotated image of an eye disease and a second annotated image without obvious lesions, wherein the first annotated image of an eye disease includes at least annotated data of pathological myopia and retinal detachment; and the second annotated image without obvious lesions includes: a normal fundus image and a leopard fundus image; An input module, used for inputting the plurality of sample fundus images into an initial segmentation model; An extraction module, used to extract features of lesions related to eye diseases in each input fundus image, as well as features of normal fundus images without obvious lesions and leopard fundus images; A comparison module, used for comparing the extracted lesion features and features of the normal fundus image without obvious lesions and the leopard fundus image with the corresponding first annotated image and the second annotated image respectively; A calculation module, used for calculating the loss value according to the difference of the comparison results; An iteration module, used to update the parameters of the segmentation model through a back propagation mechanism based on the loss value, until the segmentation model converges after multiple iterations; The sampling processing module is used to upsample the output image when the segmentation model converges, restore it to the size of the original sample fundus image, and obtain a trained segmentation model.

13. A classification model training device, characterized in that: The device comprises: An acquisition module is used to acquire a second training set, wherein the second training set includes: a plurality of sample fundus images, wherein the plurality of sample fundus images include classification labels of eye diseases and non-eye diseases, wherein the classification labels include at least: no obvious lesions, leopard-like fundus, diffuse choroidal atrophy, patchy choroidal atrophy, macular atrophy, and retinal detachment; An input module, used for inputting the plurality of sample fundus images into an initial classification model, wherein the parameters of the initial classification model are the parameters of the trained segmentation model; An extraction module, used for extracting classification features related to eye diseases and non-eye diseases in each of the plurality of sample fundus images; A comparison module, used for training the extracted classification features in terms of dimension and classification through a fully connected layer; A calculation module, used to compare the training result with the classification label, and calculate the loss value according to the difference of the comparison result; An iteration module, for updating the parameters of the classification model through a back propagation mechanism based on the loss value, after multiple iterations; The dimension classification processing module is used to perform dimension and classification processing on the result of each iteration in two fully connected layers in sequence until the classification model converges to obtain a trained classification model.

14. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the eye disease identification method based on fundus images as described in any one of claims 1 to 8 or the segmentation model training method as described in claim 9 or the classification model training method as described in claim 10.

15. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the eye disease identification method based on fundus images as described in any one of claims 1 to 8, the segmentation model training method as described in claim 9, or the classification model training method as described in claim 10.

Images