Eye fundus image anomaly recognition method based on semi-supervised learning and related equipment

Abstract

The invention provides a semi-supervised learning-based eye fundus image anomaly recognition method and related equipment, and the method comprises the steps: carrying out the class label supervision training of a small number of labeled images with class labels, obtaining an undetermined classification model, continuing to obtain different enhanced images through different image enhancement modes for a large number of unlabeled images, and carrying out the classification of the undetermined classification model. And respectively carrying out semi-supervised mutual learning on the to-be-determined classification model to obtain a high-performance classification model for realizing analysis of the to-be-identified eye fundus image, rapidly and accurately obtaining type prediction information of the eye fundus image, and outputting an anomaly identification result of the eye fundus image according to the type prediction information.

Description

Technical Field

[0001] The present application mainly relates to the field of image processing, and in particular to a method for identifying fundus image abnormalities based on semi-supervised learning and related equipment. Background Art

[0002] Diabetic retinopathy (DR) is a common retinal vascular disease that causes severe vision loss and is one of the three major complications of diabetes. Professional doctors are usually required to identify abnormal pathological features in fundus images to determine the severity of DR lesions. However, this manual method of identifying fundus abnormalities cannot meet the needs of massive fundus DR abnormality identification in a timely manner.

[0003] In order to quickly and accurately realize DR abnormality screening, it is proposed to perform supervised learning on a large number of annotated images to obtain a classification model for identifying whether the fundus image belongs to the DR abnormality category, so as to realize rapid DR abnormality screening of the patient's fundus images, and assist doctors to quickly and accurately determine the patient's condition and its treatment method.

[0004] However, medical images require professional doctors to spend a lot of time and energy on manual labeling. It is usually difficult to obtain a sufficient number of labeled images. In addition, since different doctors have different labeling standards, there are noisy labels in the obtained labeled images, which reduces the reliability of the labeled images and the accuracy of the classification model, thereby affecting the classification accuracy of whether the fundus image belongs to the DR abnormality category. Summary of the invention

[0005] In order to solve the above problems, this application proposes the following technical solutions:

[0006] On the one hand, the present application proposes a fundus image recognition method based on semi-supervised learning, the method comprising:

[0007] obtaining a fundus image to be identified;

[0008] Inputting the fundus image into a trained classification model to obtain category prediction information of the fundus image; wherein the classification model is obtained by using different enhanced images obtained by performing different image enhancement processes on unlabeled images, and performing semi-supervised mutual learning on the pending classification model, and the pending classification model is obtained by supervised training of category label correction on the labeled image;

[0009] Based on the category prediction information, an abnormality recognition result of the fundus image is output.

[0010] Optionally, the obtaining of the pending classification model by supervised training of class label correction on the annotated image includes:

[0011] Acquire a set of annotated images; each annotated image contained in the set of annotated images has a category label;

[0012] According to the category label correction network, the predicted category labels of each labeled image are corrected to obtain a pending classification model;

[0013] Wherein, the category label correction network includes a convolutional network and a curriculum learning network.

[0014] Optionally, the method of correcting the predicted category labels of the annotated images according to the category label correction network to obtain a pending classification model includes:

[0015] Performing feature extraction on each of the annotated images through the convolutional network to obtain a first feature vector corresponding to the annotated image;

[0016] Performing principal component analysis data dimensionality reduction processing on a plurality of first eigenvectors corresponding to each of the annotated images to obtain a second eigenvector corresponding to the annotated image;

[0017] Performing clustering processing on the plurality of second feature vectors to obtain feature vector clusters of respective plurality of prediction levels; the higher the prediction level, the more complex the data of the second feature vector in the feature vector cluster;

[0018] Based on the prediction level, the curriculum learning network performs category prediction on the second feature vector after the feature vector clusters belonging to the corresponding prediction level are sequentially added to obtain a category prediction value corresponding to the annotated image;

[0019] Obtaining a correction matrix according to the category prediction values;

[0020] According to the correction matrix, the corrected category label corresponding to the annotated image is obtained to realize the prediction and correction of the category label of the annotated image corresponding to the part after adding the next sequential prediction level, until the prediction and correction of the category label of the annotated image corresponding to the last prediction level is enhanced to obtain the pending classification model.

[0021] Optionally, obtaining a correction matrix according to the category prediction value, and obtaining a corrected category label corresponding to the annotated image according to the correction matrix includes:

[0022] Determining the maximum category prediction value of each category label from the obtained category prediction values ​​of the labeled image;

[0023] Determining the annotated image having the maximum category prediction value as a target annotated image corresponding to the category label;

[0024] From the obtained category prediction values ​​of the annotated images, determining the target annotated images corresponding to the category labels are predicted as mispredicted values ​​of other predicted category labels;

[0025] Using the misprediction value of the target labeled image, a correction matrix corresponding to the category label is obtained;

[0026] By forward correction, the predicted category label of the corresponding annotated image is corrected based on the correction matrix to obtain the corrected category label of the annotated image.

[0027] Optionally, during the supervised training of the pending classification model:

[0028] Obtaining a corresponding pending prediction loss according to the category prediction value of each labeled image;

[0029] The undetermined prediction loss is multiplied by the correction matrix to obtain a prediction loss corresponding to the labeled image.

[0030] Optionally, using different enhanced images obtained by performing different image enhancement processes on the unlabeled images to respectively perform semi-supervised mutual learning on the trained pending classification models to obtain the classification model includes:

[0031] Obtain a set of unlabeled images;

[0032] The unlabeled images in the unlabeled image set are processed respectively by a strong enhancement method and a weak enhancement method to obtain a strongly enhanced image and a weakly enhanced image of the same unlabeled image;

[0033] Based on the undetermined classification model obtained through training, respectively, class prediction is performed on the strongly enhanced image and the weakly enhanced image to obtain a first class prediction value and a second class prediction value of the corresponding unlabeled image;

[0034] Based on the second category prediction value, obtaining a pseudo category label corresponding to the unlabeled image;

[0035] Obtaining the prediction loss of the corresponding unlabeled image by using the pseudo category label and the second category prediction value in a consistency regularization manner;

[0036] Obtaining a total prediction loss based on the prediction loss of the labeled image and the prediction loss of the unlabeled image;

[0037] The classification model is trained by minimizing the total prediction loss.

[0038] Optionally, outputting the abnormality recognition result of the fundus image according to the category prediction information includes:

[0039] If the category prediction information includes an abnormality prediction value indicating that the fundus image belongs to an abnormal category, determining that the abnormality prediction value is greater than the prediction threshold, and obtaining that an abnormal region of the abnormal category exists in the fundus image;

[0040] According to the preset output requirements, an abnormality identification result including at least the abnormal area and the corresponding abnormal category is obtained;

[0041] The abnormality recognition result is output to mark the abnormal area on the fundus image.

[0042] On the other hand, the present application also proposes a fundus image abnormality recognition device based on semi-supervised learning, the device comprising:

[0043] A fundus image acquisition module, used for acquiring a fundus image to be identified;

[0044] A category prediction information obtaining module is used to input the fundus image into a trained classification model to obtain category prediction information of the fundus image; wherein the classification model is obtained by using different enhanced images obtained by performing different image enhancement processes on unlabeled images, and performing semi-supervised mutual learning on the pending classification model, and the pending classification model is obtained by supervised training of category label correction on the labeled image;

[0045] The abnormality recognition result output module is used to output the abnormality recognition result of the fundus image according to the category prediction information.

[0046] On the other hand, the present application also proposes an electronic device, the electronic device comprising: at least one data communication port, at least one memory and at least one processor, wherein:

[0047] The data communication port is used to obtain the fundus image to be identified;

[0048] The processor is used to load and execute the computer instruction code stored in the memory to implement the following steps:

[0049] obtaining a fundus image to be identified;

[0050] Inputting the fundus image into a trained classification model to obtain category prediction information of the fundus image; wherein the classification model is obtained by using different enhanced images obtained by performing different image enhancement processes on unlabeled images, and performing semi-supervised mutual learning on the pending classification model, and the pending classification model is obtained by supervised training of category label correction on the labeled image;

[0051] Based on the category prediction information, an abnormality recognition result of the fundus image is output.

[0052] Optionally, the electronic device further includes:

[0053] An image acquisition device, used for acquiring fundus images to be identified;

[0054] An output device, used for outputting an abnormality recognition result of the fundus image;

[0055] The data communication port is also used to realize the communication connection between the electronic device and the server, and obtain the classification model that has been trained by the server. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0057] Figure 1 A flowchart of an optional embodiment of the method for identifying fundus image abnormalities based on semi-supervised learning proposed in this application;

[0058] Figure 2 A flowchart of another optional embodiment of the method for identifying fundus image abnormalities based on semi-supervised learning proposed in this application;

[0059] Figure 3 A schematic diagram of an optional structure of a category label correction network applicable to the fundus image abnormality recognition method based on semi-supervised learning proposed in the present application;

[0060] Figure 4 A schematic diagram of an optional structure of a semi-supervised learning network applicable to the method for identifying fundus image abnormalities based on semi-supervised learning proposed in the present application;

[0061] Figure 5 A flowchart diagram of another optional embodiment of the method for identifying fundus image abnormalities based on semi-supervised learning proposed in this application;

[0062] Figure 6 A schematic diagram of the structure of an optional embodiment of the fundus image abnormality recognition device based on semi-supervised learning proposed in this application;

[0063] Figure 7 A schematic diagram of the structure of another optional embodiment of the fundus image abnormality recognition device based on semi-supervised learning proposed in the present application;

[0064] Figure 8The present invention is a schematic diagram of the hardware structure of an optional example of an electronic device suitable for the fundus image abnormality recognition method based on semi-supervised learning proposed in the present application. DETAILED DESCRIPTION

[0065] In response to the technical problems described in the background technology section, and the technical problems existing in the application of abnormal screening of fundus images such as DR by the classification model based on supervised learning, this application proposes a classification model based on semi-supervised learning to achieve abnormal screening of fundus images. Since semi-supervised learning is committed to improving the performance of classification algorithms based on a small number of labeled images, it can use a large number of related unlabeled images to reduce the dependence of artificial intelligence methods on labeled data.

[0066] Based on the above analysis, the present application proposes to train the initial classification network by performing category label correction on a small number of annotated images, that is, by correcting the noise labels existing in the annotated images, a small number of annotated images with corrected labels are obtained to train the classification network. For the classification network obtained from each training, a large number of unlabeled images can be used, combined with pseudo-labels and consistency regularization ideas, to perform semi-supervised mutual learning on the classification network, thereby improving the category prediction accuracy of the classification network, and obtaining a classification model that can accurately identify whether the fundus image belongs to a preset abnormal category such as DR (Diabetic Retinopathy), thereby realizing abnormal screening of the patient's fundus images, assisting doctors to quickly and accurately determine the patient's condition and its treatment plan.

[0067] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. For the convenience of description, only the parts related to the relevant inventions are shown in the drawings. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. That is to say, based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0068] Wherein, flow chart is used in the present application to illustrate the operation performed by the system according to the embodiment of the present application.It should be understood that the preceding or following operation is not necessarily performed accurately in order.On the contrary, each step can be processed in reverse order or simultaneously.At the same time, other operations can also be added to these processes, or a certain step or several steps of operation can be removed from these processes.

[0069] Reference Figure 1, which is a flow chart of an optional embodiment of the fundus image abnormality recognition method based on semi-supervised learning proposed in the present application. The method can be applicable to electronic devices, which can be servers such as independent physical servers, server clusters integrating multiple physical servers, or cloud servers with cloud computing capabilities, and / or terminal devices with certain data processing capabilities, such as smart medical devices, robots, desktop computers, etc. Of course, in some scenarios, the terminal device can also interact with the server to implement the fundus image abnormality recognition method based on semi-supervised learning proposed in the present application. The present application does not limit the application environment architecture of the method.

[0070] Based on the above analysis, if Figure 1 As shown, the method for identifying fundus image abnormalities based on semi-supervised learning proposed in this embodiment may include:

[0071] Step S11, obtaining a fundus image to be identified;

[0072] In the actual application of this application, medical equipment such as a fundus camera can be used to collect fundus images of the patient's eyes to obtain a retinal fundus image of the eye, which is referred to as a fundus image in this application. By analyzing abnormal pathological features of the fundus, it is determined whether an abnormal area, such as a diabetic retinopathy (DR) area, appears, thereby determining whether the fundus image belongs to an image of the corresponding abnormal category. If necessary, the abnormal level (such as DR level) can be further determined based on the identified abnormal pathological features, thereby assisting ophthalmologists to accurately understand the condition and formulate a correct treatment plan. It should be noted that this application does not limit the method for implementing the acquisition of fundus images.

[0073] Optionally, in the case where the medical device for collecting fundus images has data processing capabilities, the medical device can be used as an electronic device to execute the fundus image abnormality recognition method based on semi-supervised learning proposed in this application; if the medical device does not have data processing capabilities, the fundus image collected from a certain eye can be sent to an electronic device (such as a terminal or server, etc.) through wired communication or wireless communication, so that the electronic device executes the fundus image abnormality recognition method based on semi-supervised learning proposed in this application for each fundus image received, and feeds back the abnormal recognition result to the terminal output. Of course, the implementation methods of step S11 include but are not limited to these two implementation methods, which can be determined according to the application environment, and this application does not give detailed examples one by one.

[0074] Step S12, inputting the fundus image into the trained classification model to obtain category prediction information of the fundus image; wherein the classification model is obtained by using different enhanced images obtained by processing the unlabeled image in different image enhancement methods, and performing semi-supervised mutual learning on the pending classification model, and the pending classification model is obtained by supervised training of class label correction on the labeled image;

[0075] In combination with the above description of the technical solution of the present application, among the open source abnormal fundus image sets (such as DR fundus image sets, etc.) used in the present application, a small number are annotated images with category labels (such as whether they belong to abnormal categories such as DR), and a large number are unlabeled images without category labels. Since annotated images are usually labeled by ophthalmologists based on their professional skills, there may be noisy labels. In this regard, the present application proposes to correct the category labels of the small number of annotated images directly obtained, and then conduct supervised training using the annotated images with the corrected category labels to improve the performance of the resulting pending classification model.

[0076] Afterwards, a large number of unlabeled images are enhanced by different image enhancement methods to obtain different enhanced unlabeled image sets, that is, enhanced image sets composed of unlabeled enhanced images, and semi-supervised mutual learning is performed on the classification models to be determined, so as to obtain a classification model that can accurately identify whether the fundus image belongs to the corresponding abnormal category. It can be seen that the training of the classification model of the present application reduces the dependence on a large number of labeled images, and improves the reliability of the model training data by correcting the noise labels existing in the manual annotations, and combines a sufficient number of unlabeled images to ensure the accuracy of the classification model learned by semi-supervision.

[0077] Based on this, the collected fundus image to be identified is input into the classification model obtained according to the above-mentioned training method, so that the category prediction information of the fundus image can be obtained quickly and accurately, such as the predicted probability or predicted score of the fundus image belonging to an abnormal category (such as an abnormal category with DR pathological characteristics), or whether the fundus image obtained thereby belongs to or does not belong to an abnormal category, etc. The present application does not impose any restrictions on the content of the category prediction information, which can be determined based on the network structure of the classification model obtained through training.

[0078] Step S13: outputting the abnormality recognition result of the fundus image according to the category prediction information.

[0079] Combined with the above description of the category prediction information, it can be directly used as the abnormal recognition result of the fundus image to be identified; or the category prediction information can be analyzed, such as comparing the prediction value of the fundus image contained therein belonging to the abnormal category with the prediction threshold to determine whether the fundus image belongs to the abnormal category, etc. This application does not limit the content of the abnormal recognition results and their output method, which can be determined according to the circumstances.

[0080] Reference Figure 2 , is a flow chart of another optional embodiment of the method for identifying fundus image abnormalities based on semi-supervised learning proposed in this application. This embodiment can describe an optional training implementation method of the classification model proposed above, such as Figure 2 As shown, the method may include:

[0081] Step S21, obtaining a labeled image set and an unlabeled image set; each labeled image contained in the labeled image set has a category label;

[0082] In the practical application of this application, a set of annotated images and a set of unannotated images can be obtained from an open-source fundus image set of abnormal categories such as diabetic retinopathy DR features. For the annotated images, an ophthalmologist can annotate a small number of unannotated images in the abnormal fundus image set with image-level category labels. Since the required number is small, the accuracy of the annotated category labels is not required, that is, there is no strict requirement for the professional skills of ophthalmologists. Ordinary ophthalmologists can spend very little time annotating category labels and quickly obtain a small number of annotated images, that is, ophthalmologists determine whether the fundus image is a sample fundus image of the corresponding abnormal category. Therefore, there may be noisy labels in the annotated image set obtained by this application, that is, category labels with incorrect abnormal category labels for fundus images.

[0083] From the above analysis, it can be understood that the unlabeled image set contains a large number of unlabeled images, that is, sample fundus images whose belonging to abnormal categories have not been determined. Compared with a large number of labeled images used for supervised learning classification models, the data acquisition method described in step S21 of the present application greatly shortens the efficiency of acquiring training sample data and improves data reliability.

[0084] Step S22, correcting the predicted category labels of each annotated image according to the category label correction network to obtain a pending classification model;

[0085] In an embodiment of the present application, a noise-resistant deep learning framework for medical image classification based on label correction is used to process the uncertainty of category labels in annotated images, correct noisy labels, and improve the accuracy of category labels in annotated images.

[0086] Optional, see Figure 3 The schematic diagram of the structure of a category label correction network is shown. The category label correction network proposed in the present application may include a convolutional network and a curriculum learning network. In order to improve the efficiency and accuracy of label correction, the present application proposes to update the labels step by step in order from high to low according to the prediction accuracy of the category label of each annotated image (that is, the difficulty of the category prediction of the corresponding annotated image, or the complexity of the data it contains or the distribution density of the data in the feature space, etc.).

[0087] Therefore, for a small number of annotated images, after the feature processing of the annotated images by the above-mentioned convolutional network and the feature vector of each annotated image is obtained, the distribution density of the data in the feature space can be used to measure the complexity of the data to cluster the feature vectors, that is, the data complexity of each feature vector is sorted in an unsupervised manner, so that each annotated image in the entire annotated image set is divided into multiple prediction levels, each prediction level corresponds to a feature vector clustering cluster, and the data complexity of each feature vector belonging to the same feature vector clustering cluster is basically the same, and the complexity of feature vectors of different prediction levels is different. Under normal circumstances, the higher the prediction level, the greater the difficulty of category prediction of the corresponding feature vector (or the corresponding annotated image), that is, the greater the error between the predicted category label obtained by category prediction and the true category label. This application does not limit the number of prediction levels, which can be determined according to the circumstances.

[0088] Afterwards, since the category labels of annotated images with higher prediction levels are the most difficult to correct, it is easy to make mistakes if they are corrected too early, which will have a negative impact on the accuracy of the classification model. Figure 3 As shown, in the course learning network, the present application first corrects the category labels of the labeled images with the lowest prediction level, such as correcting simple errors, and based on the labeled images with the corrected category labels obtained, the labeled images (i.e., samples) belonging to the low prediction level are updated, so as to train the classification network (e.g., Figure 3 The softmax network in the network is used to optimize network parameters and improve the accuracy of category prediction. After that, the scope of the correction data can be expanded to annotated images with higher prediction levels (i.e., more complex data). Combined with the updated samples obtained previously, the category labels of the annotated images of this prediction level are corrected. This is repeated until the category labels of all annotated images are corrected and a pending classification model is generated.

[0089] It can be seen that the proposed network based on the category label correction for each labeled image (based on Figure 3In the correction process, the softmax output of the classification network determines the category prediction value of each labeled image belonging to different category labels, and determines the category label corresponding to the maximum category prediction value as the predicted category label of the labeled image. In the correction process, a category label correction strategy designed to be sorted in ascending order of difficulty is used, from the category label correction task of simple samples to the category label correction task of more difficult samples, and the correction work is performed after gradually increasing the samples. For the difficulty or complexity of the label image task, the distribution density of the image features in the feature space, that is, the clustering category density, can be used to determine the complexity, and the difficulty sorting of the labeled images can be achieved in an unsupervised manner. Then, according to the above correction method, the correction of the predicted category labels of the labeled images is achieved, thereby improving the prediction accuracy of the classification network. The implementation details can be combined with Figure 3 The network structure shown is fixed and will not be described in detail in this application.

[0090] Step S23, in the process of correcting the predicted category label of the annotated image, obtaining the prediction loss of the annotated image;

[0091] Following the above analysis, the present application performs probabilistic correction on the manually annotated category labels according to the above method, and trains the classification network at the same time to predict and update the category label distribution of the annotated image to improve the reliability and accuracy of the category label prediction. However, due to the limited number of annotated images used for supervised learning of the classification network, the performance of the classification network is affected. Therefore, after the pending classification model is learned through the above supervised learning, the present application can continue to train it with a larger number of samples to improve the accuracy of image classification.

[0092] Step S24, processing each unlabeled image included in the unlabeled image set by a strong enhancement method and a weak enhancement method, respectively, to obtain a strongly enhanced image and a weakly enhanced image of the same unlabeled image;

[0093] This application proposes to use a semi-supervised learning (SSL) method to further train the above-mentioned undetermined classification model obtained by training the labeled images with corrected category labels to obtain a classification model with higher performance. Figure 4 As shown in the schematic diagram of the semi-supervised learning network structure, based on the idea of ​​consistency regularization, the semi-supervised learning network is usually composed of two branch networks, which respectively process samples with different disturbances (unlabeled images in this embodiment), and learn in combination with a preset loss function, so that the two branch networks have consistent prediction results for the same sample, that is, the prediction category for the same unlabeled image is the same.

[0094] Based on this, for the implementation method of the input perturbation of the above two branch networks, the present application proposes to use different image enhancement methods to respectively implement weak perturbations and strong perturbations of the same input. That is to say, for each unlabeled image in the unlabeled image set, it can be processed using a weak enhancement (Weakly-augmented) method and a strong enhancement (Strongly-augmented) method, respectively, to obtain strongly enhanced data and weakly enhanced images of the same unlabeled image, which are used to respectively input the pending classification models obtained by the above training for category prediction. The implementation process of the image enhancement processing is not described in detail in this application.

[0095] Optionally, the above-mentioned weak enhancement method can adopt a standard flipping and movement enhancement strategy to perform simple enhancement processing on the unlabeled image. In this way, after the electronic device determines the flipping parameters (such as 50% flipping probability, etc.) and movement parameters (the first translation probability in the horizontal direction of the image and the second translation probability in the vertical direction, the first translation probability and the second translation probability can both be less than or equal to 12.5%) used for weak image enhancement, it can randomly flip and randomly translate each unlabeled image contained in the unlabeled image set according to the flipping parameters and movement parameters to obtain a weakly enhanced image of the corresponding unlabeled image. Exemplarily, for each unlabeled image, the present application randomly flips the image with a probability of 50%, and randomly translates the unlabeled image in the vertical and horizontal directions by up to 12.5% ​​to obtain a weakly enhanced image.

[0096] For the above-mentioned strong enhancement method, each unlabeled image contained in the unlabeled image set can be enhanced based on a variety of automatic data enhancement methods (AutoAugment) to obtain a strongly enhanced image of the corresponding unlabeled image. For example, RandAugment and CTAugment are used to process the unlabeled image, and then Cutout is used to output the strongly enhanced image. For example, according to a preset size window (such as a square window, etc.), after randomly deleting the corresponding size area in the unlabeled image, the deleted area is filled with grayscale or black. It can be seen that compared with the corresponding weakly enhanced image, the strongly enhanced image obtained in the present application is a severely distorted image of the unlabeled image. The classifier with the same structure is used to predict its category, and the error of the category prediction result of the strongly enhanced image is larger.

[0097] Step S25, performing category prediction on the strongly enhanced image and the weakly enhanced image respectively according to the undetermined classification model obtained through training, and obtaining the first category prediction value and the second category prediction value of the corresponding unlabeled image;

[0098] Combined with the above description of semi-supervised learning, after applying different intensity perturbations to each unlabeled image, the two branch networks will predict its category respectively. Therefore, Figure 4As shown, the strongly enhanced image and the weakly enhanced image of the same unlabeled image are respectively input into the undetermined classification model of the same structure for classification prediction, and the category prediction values ​​(such as prediction probability) of the corresponding input images belonging to different categories are obtained, that is, the category prediction is performed after applying different disturbances to the same unlabeled image, and the category prediction values ​​such as the prediction probability that the labeled image belongs to the abnormal category are obtained.

[0099] Step S26, obtaining a pseudo category label of the corresponding unlabeled image according to the second category prediction value;

[0100] In combination with the above analysis, the classification network with the same structure is used to perform category prediction on the strongly enhanced image and the weakly enhanced image obtained by the enhancement processing of the same unlabeled image. Compared with the category prediction result of the strongly enhanced image, the category prediction result of the weakly enhanced image of the same unlabeled image is more accurate. The present application can use the category prediction value of this branch network (that is, the second category prediction value mentioned above) to generate a pseudo-label (Pseudo-label), that is, only retain the unlabeled image whose highest category prediction value obtained by the classification network is higher than the threshold, and designate it as the category to which the maximum category prediction value belongs, that is, the category label, which ensures the reliability of the pseudo-label to a certain extent, and can also use the strong perturbation of another branch network to bring better learning effect to the classification network.

[0101] Therefore, according to the pseudo-labeling method (Pseudo-labeling) in semi-supervised learning, that is, threshold screening and strengthening the category prediction results of the pending classification model to generate pseudo category labels for unlabeled images, the second category prediction value of each unlabeled image belonging to each category, that is, the second category prediction value of the weakly enhanced image, can be compared with a preset probability threshold (that is, threshold), and the predicted category whose category prediction probability is greater than the preset probability threshold is determined as the pseudo category label of the unlabeled image, such as Figure 4 Among the second category prediction values ​​of the same unlabeled image belonging to different categories shown in , the category corresponding to the second category prediction value above the dotted line is determined as the pseudo category label of the unlabeled image.

[0102] Step S27, using a consistency regularization method, using the pseudo category label and the first category prediction value of the same unlabeled image to obtain the prediction loss of the corresponding unlabeled image;

[0103] Step S28, obtaining a total prediction loss based on the prediction loss of the labeled image and the prediction loss of the unlabeled image obtained by the current round of training of the classification network;

[0104] Step S29, training a classification model by minimizing the total prediction loss.

[0105] For consistency regularization in semi-supervised learning, the pending classification model performs category prediction on different perturbation versions of the same unlabeled input image (i.e., enhanced images after strong and weak enhancement) to output similar category prediction results, that is, the category prediction values ​​are basically the same, etc. This application does not elaborate on the idea of ​​consistency regularization. Optionally, after completing a round of training of the pending classification model based on the corrected labeled image, the above-mentioned semi-supervised learning steps can be performed. After the unlabeled weakly enhanced image is predicted by the classification network and the pseudo-category label of the corresponding unlabeled image is generated, the pending classification model can be forced to obtain consistent category prediction values ​​for the corresponding unlabeled strongly enhanced image based on the idea of ​​consistency regularization, and the pending classification model can continue to be trained to optimize the performance of the classification network to obtain a classification model with high category prediction accuracy.

[0106] In the embodiment of the present application, in the classification problem of fundus images, X={x b ,p b}: b∈(1,…,B) represents a set of labeled images with category labels, B can represent the number of labeled images contained in the labeled image set, x b Can represent the labeled image, p b It can represent the category label of the b-th annotated image and can be represented by one-hot encoding (i.e., one-bit effective encoding), that is, a one-dimensional binary vector.

[0107] Correspondingly, for the unlabeled image set that can be applied to the above strong enhancement and weak enhancement processing, U = u b :b∈(1,……,μB) means, u b Can represent unlabeled images, μ can represent a hyperparameter, the ratio of unlabeled images, which determines the relative size of X and U, that is, the number of images contained in the unlabeled image set obtained in this application. This application does not limit its numerical value, which can be determined according to the situation.

[0108] In practical applications, it can be obtained by m (y|x) represents the category prediction result of the undetermined classification model for the input x, which can be the distribution of the category prediction values ​​of the predicted input x belonging to different categories. The cross entropy of the distributions p and q of the two category prediction results can be expressed as H(p,q), which measures the difference between the two category prediction results. It can be seen that the loss function of this application can be composed of two cross entropy loss terms, a supervision loss L for annotated images s , and the other is the unsupervised loss L for unlabeled images u .

[0109] Therefore, in the prediction loss calculation process of the above labeled image, the standard cross entropy loss function shown in the following formula (2) can be used:

[0110]

[0111] Combined with the above analysis, after using the undetermined classification model of the same network structure to classify and predict the enhanced images of different perturbations, the category prediction probability of the enhanced image belonging to different categories (i.e., the above-mentioned one category prediction value) is obtained. After that, the pseudo category label of the corresponding unlabeled image can be determined based on the second category prediction value of the weakly enhanced image, so that it can be applied to the cross entropy loss function shown in formula (1) to calculate its prediction loss. Among them, for the strong enhancement processing and weak enhancement processing of the unlabeled image, the corresponding strong enhancement image and weak enhancement image are respectively obtained by and α(·).

[0112] Combined with the above analysis, q b =p m (y|α(u b )) can represent the second category prediction probability of the weakly enhanced image, and the prediction category corresponding to the maximum category prediction probability is taken as the input unlabeled image u b The pseudo-category label of Afterwards, the application forces the pending classification model to predict the category of the strongly enhanced image in accordance with the pseudo category label, and obtains the pseudo category label of the weakly enhanced image according to the method described above. After that, for the strong enhancement processing u b , we can enforce the cross entropy loss on the output of the given classification model, i.e., the prediction loss L for unlabeled images u :

[0113]

[0114] In the above formula (2), represents a strongly enhanced image; It can represent the first category prediction value of the strongly enhanced image, such as the prediction probability, etc.; τ can be a scalar hyperparameter, which represents the preset confidence threshold for the pseudo category label. This application does not limit its value. In the continuous training process of the classification network, the cross entropy loss of the unlabeled image can be minimized to make the category prediction results of the two branch networks in the semi-supervised learning consistent, thereby improving the classification accuracy.

[0115] For the entire training process of the classification model, the present application can combine the prediction loss of labeled images and the prediction loss of unlabeled images to adjust the network parameters of each network, optimize the pending classification model, reduce the total prediction loss, and obtain the classification model. The implementation process of this embodiment will not be described in detail here.

[0116] In some embodiments, in combination with the above description of the category label correction network, its network backbone can select any convolutional neural network, such as Figure 3 The network structure shown in the figure can be a pure convolutional network such as ConNeXt to improve the reasoning speed and accuracy. Each annotated image is input into the convolutional network for feature extraction to obtain the corresponding feature vector to form the FC layer. After the principal component analysis PCA (Principal Component Analysis) data dimension reduction processing, the data complexity is sorted in an unsupervised manner, and the feature vectors of these annotated images are divided into multiple prediction levels belonging to different data complexities. Through the course learning network, the category labels of the annotated images corresponding to the corresponding prediction levels are corrected in order from simple to difficult, that is, the prediction level is from low to high, to realize the training of the classification network therein. After obtaining a more accurate classification network, continue to add the annotated images of the next prediction level, and repeat this process to realize the category label correction of all annotated images.

[0117] Based on the above analysis, refer to Figure 5 The flowchart of another optional embodiment of the method for identifying fundus image abnormalities based on semi-supervised learning proposed in the present application is shown, and the method may include:

[0118] Step S51, obtaining a labeled image set and an unlabeled image set; each labeled image contained in the labeled image set has a category label;

[0119] Step S52, extracting features from each annotated image through a convolutional network in the category label correction network to obtain a first feature vector of the corresponding annotated image;

[0120] Step S53, performing principal component analysis data dimensionality reduction processing on the plurality of first eigenvectors to obtain second eigenvectors corresponding to the annotated image;

[0121] As analyzed above, the convolutional network ConvNeXt extracts features from the input labeled image, performs PCA dimensionality reduction operation on each first eigenvector in the output FC layer, and outputs the corresponding second eigenvector. The processing implementation process is not described in detail in this embodiment.

[0122] Step S54, clustering the second feature vectors of each annotated image to obtain feature vector clusters of multiple prediction levels; the higher the prediction level, the more complex the data of the second feature vector in the corresponding feature vector cluster;

[0123] In this embodiment, the distribution density of the data in the annotated image in the feature space can be used to measure the complexity of the data, thereby indicating the difficulty of the corresponding annotated image being predicted by the category, thereby achieving the sorting (i.e., clustering) of the corresponding second feature vectors, and according to the sorting result, the obtained second feature vectors are divided into multiple feature vector clustering clusters, each feature vector clustering cluster belongs to a prediction level. Figure 3 The network structure shown can be divided into five prediction levels in this way, such as level 1, level 2, level 3, level 4 and level 5. The higher the prediction level, the more difficult it is to determine the true category of the corresponding labeled image. However, it is not limited to these five prediction levels and can be determined according to actual conditions.

[0124] Step S55, performing category prediction on each second feature vector after sequentially adding feature vector clusters belonging to the corresponding prediction level based on the prediction level through the curriculum learning network in the category label correction network, and obtaining a category prediction value of the corresponding labeled image;

[0125] In the process of correcting the category labels of annotated images with different prediction levels, in order to improve the correction reliability and the accuracy of the corrected category labels, as analyzed above, after the curriculum learning network obtains the feature vector clusters of different prediction levels, it can try to correct the wrong category labels of the corresponding annotated images from the second feature vector of the low prediction level, and obtain the corresponding part of the category standard. The corrected annotated image trains the classification network to optimize the performance of the classification network.

[0126] Step S56, determining the maximum category prediction value of each category label from the obtained category prediction values ​​of the annotated image;

[0127] Step S57, determining the annotated image with the maximum category prediction value as the target annotated image corresponding to the category label;

[0128] In the supervised learning process of annotated images in the embodiment of the present application, in order to correct the predicted classification labels obtained by classification prediction, the most reliable annotated image of the classification prediction result of each category label can be obtained, as well as the misprediction value of the classification network predicting it as other predicted category labels, that is, the category prediction value of each annotated image belonging to a different category (that is, a preset category label) predicted by the classification network is used to construct a correction matrix to correct the predicted category labels of other annotated images. Based on this, the present application can use the method described by the following formula (3) to determine the perfect sample corresponding to each category label i, that is, the target annotated image of the image category represented by the category label:

[0129]

[0130] In the above formula (3), for each annotated image in the above annotated image set, the category prediction of the second feature vector input can be performed through the softmax classification network according to the method described above to determine the category prediction value of the corresponding annotated image, that is, the category prediction value of the annotated image belonging to various category labels i, thereby determining the maximum category prediction value as the predicted category label of the corresponding annotated image. It is the probability distribution learned by the neural network. It is the predicted category label after the annotated image is affected by noise, that is, the predicted category label to be corrected.

[0131] For each category label i, as shown in the above formula (3), the labeled image with the largest category prediction value can be determined as the perfect sample from the category predictions of the corresponding different labeled images belonging to the category label i output by softmax, and is recorded as the target labeled image x i , which means that the reliability and accuracy of the target labeled image belonging to the category label i are the highest, and can be used to achieve effective supervised learning of the classification network.

[0132] Step S58, determining the misprediction value of the target labeled image corresponding to each category label being predicted as other category labels based on the category prediction value of the currently obtained partial labeled image;

[0133] Step S59, using the misprediction value of the target labeled image to obtain a correction matrix corresponding to the category label;

[0134] Step S510, by forward correction, based on the correction matrix, the category label of the corresponding annotated image is corrected to obtain the corrected category label of the annotated image, so as to correct the category label of the annotated image corresponding to the next sequential prediction level, until the category label of the annotated image corresponding to the last prediction level is corrected to obtain the pending classification model;

[0135] The present application can combine the cross entropy calculation method described in formula (1) above to obtain the undetermined prediction loss of the annotated image. At the same time, in order to classify the predicted category label of the corrected annotated image by the classification network, the present application can also obtain the category prediction result of the target annotated image corresponding to each category label. After that, it is determined that the target labeled image is predicted to be of other categories (such as the category prediction value belonging to different category labels). j The category prediction probability (i.e., the category that was wrongly predicted) is the wrong prediction value, and the conversion matrix is ​​constructed to realize the correction of the predicted category label, which is recorded as the correction matrix T corresponding to the category label ij :

[0136]

[0137] Step S511, obtaining the prediction loss of the labeled image by using the cross entropy loss function and the correction matrix of the category label and the category prediction value of each labeled image;

[0138] Among them, based on the cross entropy loss function, the category prediction value of each labeled image output by the given classification model is processed to obtain the cross entropy loss L of the labeled image s After that, the cross entropy loss L s The correction matrix T with the above category labels ij Multiply them together to get the prediction loss L of the corresponding labeled image s ← , that is, the category prediction loss of the annotated image after the category label is corrected:

[0139]

[0140] Step S512, using different enhanced images obtained by performing different image enhancement processes on the unlabeled images, respectively, semi-supervised mutual learning is performed on the trained pending classification models to obtain a classification model.

[0141] In the semi-supervised learning process described in the above embodiment, the prediction loss L of the unlabeled image obtained in each round of training is obtained. u After that, you can follow (L s ← +λ u L u ) This calculation method obtains the total prediction loss, λ u It can represent the loss weight of the unlabeled image, and this application does not limit its value. After that, the network parameters can be trained further by minimizing the total prediction loss to obtain the classification model.

[0142] Optionally, during the entire training process, including the above-mentioned supervised learning process and semi-supervised learning process, during the supervised learning process, each correction of the predicted category label performed by the curriculum learning network on the labeled image can obtain the prediction loss of the labeled image according to the method described above, and based on the pending classification model obtained from this training, semi-supervised learning training is performed on the unlabeled image to obtain the prediction loss of the unlabeled image, and the network parameters are adjusted in combination with the total prediction loss, and the classification network with the adjusted network parameters is then used to continue to perform the next correction of the predicted category label on the labeled image. The implementation process is not described in detail in this application.

[0143] In practical applications, the training and implementation process of the above-mentioned classification model can be implemented by a server or a terminal device with certain data processing capabilities. In this way, after collecting images of a patient's eye to be identified and obtaining its fundus image, it can be input into the classification model obtained by the above-mentioned training, and the category prediction value of the fundus image belonging to the abnormal category, such as DR prediction probability or score, is output, and it is compared with the prediction threshold (such as the prediction probability critical value for determining that the fundus image belongs to the DR category), and it is determined that the obtained abnormal category prediction value is greater than the prediction threshold, and the classification result of the fundus image belonging to the abnormal category is obtained; otherwise, the classification result of the fundus image not belonging to the abnormal category can be obtained.

[0144] Optionally, after obtaining the abnormal classification result of the fundus image according to but not limited to the above method, the abnormal identification result containing at least the classification result can be obtained according to the preset output requirements, and the abnormal identification result can be output to display the classification result on the fundus image to assist the doctor to quickly and accurately determine the patient's condition. Among them, the preset output requirements may include requirements on at least one aspect of the content of the abnormal identification result and its representation form, output form, etc. The present application does not limit its content and it can be determined according to the circumstances. Exemplarily, the abnormal identification result of the fundus image can be required to be output in the form of a report to display the fundus image and its classification result in the report. If necessary, the abnormal category prediction value of the fundus image can also be displayed for the doctor's reference to determine the treatment plan for the patient.

[0145] Reference Figure 6 , is a schematic structural diagram of an optional embodiment of a fundus image recognition device based on semi-supervised learning proposed in this application, and the device may include:

[0146] A fundus image acquisition module 61, used to obtain a fundus image to be identified;

[0147] The category prediction information obtaining module 62 is used to input the fundus image into a trained classification model to obtain category prediction information of the fundus image; wherein the classification model is obtained by using different enhanced images obtained by performing different image enhancement processes on unlabeled images, and respectively performing semi-supervised mutual learning on the pending classification model, and the pending classification model is obtained by supervised training of category label correction on the labeled image;

[0148] The abnormality recognition result output module 63 is used to output the abnormality recognition result of the fundus image according to the category prediction information.

[0149] Optional, such as Figure 7 As shown, the undetermined classification model training module 64 for obtaining the undetermined classification model by performing supervised training for class label correction on the annotated image may include:

[0150] The annotated image set acquisition unit 651 is used to acquire an annotated image set; each annotated image contained in the annotated image set has a category label;

[0151] The undetermined classification model obtaining unit 652 is used to correct the predicted category labels of each labeled image according to the category label correction network to obtain the undetermined classification model; wherein the category label correction network includes a convolutional network and a curriculum learning network;

[0152] Optionally, the undetermined classification model obtaining unit 652 may include:

[0153] A first feature vector obtaining unit, configured to extract features from each of the annotated images through the convolutional network to obtain a first feature vector corresponding to the annotated image;

[0154] A second eigenvector obtaining unit is used to perform principal component analysis data dimensionality reduction processing on the plurality of first eigenvectors corresponding to the annotated images to obtain a second eigenvector corresponding to the annotated image;

[0155] A clustering processing unit, used for performing clustering processing on the plurality of second feature vectors to obtain feature vector clusters of respective plurality of prediction levels; the higher the prediction level, the more complex the data of the second feature vectors in the feature vector cluster;

[0156] a category prediction value obtaining unit, configured to perform category prediction on the second feature vector after sequentially adding the feature vector clusters belonging to the corresponding prediction level through the curriculum learning network based on the prediction level, to obtain a category prediction value corresponding to the annotated image;

[0157] A correction matrix obtaining unit, used for obtaining a correction matrix according to the category prediction value;

[0158] The correction training unit is used to obtain the corrected category label corresponding to the annotated image according to the correction matrix, so as to realize the prediction and correction of the category label of the annotated image corresponding to the part after adding the next sequential prediction level, until the prediction and correction of the category label of the annotated image corresponding to the last prediction level is enhanced, and the pending classification model is obtained.

[0159] Optionally, the correction matrix obtaining unit may include:

[0160] A first determining unit, configured to determine the maximum category prediction value of each category label from the obtained category prediction values ​​of the annotated image;

[0161] A second determining unit, configured to determine the annotated image having the maximum category prediction value as a target annotated image corresponding to the category label;

[0162] A third determining unit is used to determine, from the obtained category prediction values ​​of the annotated images, the misprediction value of the target annotated image corresponding to each category label being predicted as other predicted category labels;

[0163] A first obtaining unit, configured to obtain a correction matrix corresponding to a category label using the misprediction value of the target labeled image;

[0164] Based on this, the correction training unit may include:

[0165] The category label obtaining unit is used to correct the predicted category label of the corresponding annotated image based on the correction matrix in a forward correction manner to obtain the corrected category label of the annotated image.

[0166] The above-mentioned pending classification model training module 64 may also include:

[0167] A pending prediction loss obtaining unit, used to obtain the corresponding pending prediction loss according to the category prediction value of each labeled image;

[0168] The prediction loss obtaining unit is used to multiply the pending prediction loss by the correction matrix to obtain the prediction loss corresponding to the labeled image.

[0169] Based on the above analysis, if Figure 7 As shown, the classification model training module 65 for performing semi-supervised mutual learning on the undetermined classification models obtained by training using different enhanced images obtained by performing different image enhancement processes on the unlabeled images to obtain the classification models may include:

[0170] An unlabeled image set acquisition unit 651, used to acquire an unlabeled image set;

[0171] The image enhancement processing unit 652 is used to process each unlabeled image included in the unlabeled image set by a strong enhancement method and a weak enhancement method, respectively, to obtain a strongly enhanced image and a weakly enhanced image of the same unlabeled image;

[0172] A category prediction module 653 is used to perform category prediction on the strongly enhanced image and the weakly enhanced image respectively according to the undetermined classification model obtained through training, so as to obtain a first category prediction value and a second category prediction value corresponding to the unlabeled image;

[0173] A pseudo category label obtaining unit 654 is used to obtain a pseudo category label corresponding to the unlabeled image according to the second category prediction value;

[0174] A consistency regularization loss obtaining unit 655 is used to obtain the prediction loss of the corresponding unlabeled image by using the pseudo category label and the second category prediction value through a consistency regularization method;

[0175] A total prediction loss obtaining unit 656, configured to obtain a total prediction loss based on the prediction loss of the labeled image and the prediction loss of the unlabeled image;

[0176] The training unit 657 is used to train the classification model by minimizing the total prediction loss.

[0177] In some embodiments, the abnormality identification result output module 63 may include:

[0178] a classification result obtaining unit, configured to, if the category prediction information includes an abnormality prediction value indicating that the fundus image belongs to an abnormal category, determine that the abnormality prediction value is greater than the prediction threshold, and obtain a classification result indicating that the fundus image belongs to the abnormal category;

[0179] An abnormality identification result obtaining unit, used for obtaining an abnormality identification result including at least the abnormal area and the corresponding abnormal category according to a preset output requirement;

[0180] The abnormality recognition result output unit is used to output the abnormality recognition result so as to mark the abnormal area on the fundus image.

[0181] It should be noted that the various modules, units, etc. in the above-mentioned device embodiments can be stored in the memory as program modules, and the processor executes the above-mentioned program modules stored in the memory to implement the corresponding functions. For the functions implemented by each program module and its combination, as well as the technical effects achieved, please refer to the description of the corresponding parts of the above-mentioned method embodiments, which will not be repeated in this embodiment.

[0182] The present application also provides a computer-readable storage medium on which a computer program can be stored. The computer program can be called and loaded by a processor to implement the various steps of the fundus image abnormality recognition method based on semi-supervised learning described in the above embodiment. The specific implementation process can refer to the description of the corresponding part of the above embodiment, which will not be repeated in this embodiment.

[0183] Reference Figure 8 , is a schematic diagram of the hardware structure of an optional example of an electronic device applicable to the method for identifying fundus image abnormalities based on semi-supervised learning proposed in the present application, such as Figure 8 As shown, the electronic device may include: a communication module 81, a memory 82 and a processor 83, wherein:

[0184] The number of each of the communication module 81, the memory 82 and the processor 83 can be at least one, and the communication module 81, the memory 82 and the processor 83 can all be connected to a communication bus to achieve data interaction with each other through the communication bus. The specific implementation process can be determined according to the requirements of the specific application scenario. This application does not limit the communication connection method between the components in the electronic device.

[0185] The communication module 81 may include a wireless communication module that can use a wireless communication network to realize data interaction, such as a WIFI module, a 5G / 6G (fifth generation mobile communication network / sixth generation mobile communication network) module, a GPRS module, etc., to realize wireless communication between the electronic device and other devices, such as obtaining fundus images to be identified, unlabeled images, trained classification models, etc., outputting DR recognition results of fundus images, etc. The communication content and its communication method can be determined according to the fundus image abnormality recognition method based on semi-supervised learning proposed in this application, and this application will not be described in detail. Of course, the above-mentioned communication module 81 may also include a data interface for realizing wired communication, so that the electronic device can be connected to other devices through a corresponding type of data cable to realize data transmission between different devices.

[0186] It should be understood that the above-mentioned communication module 81 can generally include a communication interface for realizing data interaction between internal components of the electronic device, such as a USB interface, a serial / parallel port, a multimedia transmission interface, etc. The present application does not limit the type and quantity of the communication module 81.

[0187] In an embodiment of the present application, the memory 82 can be used to store a program for implementing the method for identifying fundus image abnormalities based on semi-supervised learning described in any of the above method embodiments; the processor 83 can load and execute the program stored in the memory 82 to implement the various steps of the method for identifying fundus image abnormalities based on semi-supervised learning proposed in any of the above method embodiments of the present application. The specific implementation process can refer to the description of the corresponding part of the corresponding embodiment above, and will not be repeated in this embodiment.

[0188] In practical applications, the memory 82 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device. The processor 83 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, etc. The types of the memory 82 and the processor 83 may be determined according to actual needs.

[0189] It should be understood that Figure 8 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiment of the present application. In practical applications, the electronic device may include Figure 8 More components shown, or a combination of certain components, can be determined according to the product type of the electronic device. When the electronic device is a terminal device, the electronic device may also include: an image acquisition device for acquiring fundus images to be identified, such as a camera; an output device for outputting abnormal identification results of fundus images, such as a display and / or a speaker and / or a connected printing device, etc. In this case, through at least one communication module 81 (such as the above-mentioned wireless communication module), a communication connection between the electronic device and the server can be achieved, and the classification model trained by the server can be obtained. Regarding the implementation process of the server training the classification model, reference can be made to the description of the corresponding part of the above embodiment.

[0190] Optionally, in order to facilitate input operations on the terminal device, the electronic device may also include at least one input component such as a microphone, input function buttons, a keyboard, a mouse, and a sensor module including multiple sensors, a power supply component, etc. The composition structure of the electronic device can be determined according to the application scenario, and this application does not list them one by one here.

[0191] Finally, it should be noted that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, parts or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0192] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0193] Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two. The following terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0194] In addition, the various embodiments in this specification are described in a progressive or parallel manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the devices and electronic devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0195] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying abnormal fundus images based on semi-supervised learning, characterized in that: The method comprises: obtaining a fundus image to be identified; Inputting the fundus image into a trained classification model to obtain category prediction information of the fundus image; wherein the classification model is obtained by using different enhanced images obtained by performing different image enhancement processes on unlabeled images, and performing semi-supervised mutual learning on the pending classification model, and the pending classification model is obtained by supervised training of category label correction on the labeled image; Based on the category prediction information, an abnormality recognition result of the fundus image is output.

2. The method according to claim 1, characterized in that: The method of obtaining a pending classification model by performing supervised training of class label correction on the annotated image includes: Acquire a set of annotated images; each annotated image contained in the set of annotated images has a category label; According to the category label correction network, the predicted category labels of each labeled image are corrected to obtain a pending classification model; Wherein, the category label correction network includes a convolutional network and a curriculum learning network.

3. The method according to claim 2, characterized in that The method of correcting the predicted category labels of each annotated image based on the category label correction network to obtain a pending classification model includes: Performing feature extraction on each of the annotated images through the convolutional network to obtain a first feature vector corresponding to the annotated image; Performing principal component analysis data dimensionality reduction processing on a plurality of first eigenvectors corresponding to each of the annotated images to obtain a second eigenvector corresponding to the annotated image; Performing clustering processing on the plurality of second feature vectors to obtain feature vector clusters of respective plurality of prediction levels; the higher the prediction level, the more complex the data of the second feature vector in the feature vector cluster; Based on the prediction level, the curriculum learning network performs category prediction on the second feature vector after the feature vector clusters belonging to the corresponding prediction level are sequentially added to obtain a category prediction value corresponding to the annotated image; Obtaining a correction matrix according to the category prediction values; According to the correction matrix, the corrected category label corresponding to the annotated image is obtained to realize the prediction and correction of the category label of the annotated image corresponding to the part after adding the next sequential prediction level, until the prediction and correction of the category label of the annotated image corresponding to the last prediction level is enhanced to obtain the pending classification model.

4. The method according to claim 3, characterized in that The step of obtaining a correction matrix based on the category prediction value, and obtaining a corrected category label corresponding to the annotated image based on the correction matrix includes: Determining the maximum category prediction value of each category label from the obtained category prediction values ​​of the labeled image; Determining the annotated image having the maximum category prediction value as a target annotated image corresponding to the category label; From the obtained category prediction values ​​of the annotated images, determining the target annotated images corresponding to the category labels are predicted as mispredicted values ​​of other predicted category labels; Using the misprediction value of the target labeled image, a correction matrix corresponding to the category label is obtained; By forward correction, the predicted category label of the corresponding annotated image is corrected based on the correction matrix to obtain the corrected category label of the annotated image.

5. The method according to claim 4, characterized in that During the supervised training of the pending classification model: Obtaining a corresponding pending prediction loss according to the category prediction value of each labeled image; The undetermined prediction loss is multiplied by the correction matrix to obtain a prediction loss corresponding to the labeled image.

6. The method according to any one of claims 2 to 5, characterized in that: The method uses different enhanced images obtained by performing different image enhancement processes on the unlabeled images to respectively perform semi-supervised mutual learning on the trained pending classification models to obtain the classification models, including: Obtain a set of unlabeled images; The unlabeled images in the unlabeled image set are processed respectively by a strong enhancement method and a weak enhancement method to obtain a strongly enhanced image and a weakly enhanced image of the same unlabeled image; Based on the undetermined classification model obtained through training, respectively, class prediction is performed on the strongly enhanced image and the weakly enhanced image to obtain a first class prediction value and a second class prediction value of the corresponding unlabeled image; Based on the second category prediction value, obtaining a pseudo category label corresponding to the unlabeled image; Obtaining the prediction loss of the corresponding unlabeled image by using the pseudo category label and the second category prediction value in a consistency regularization manner; Obtaining a total prediction loss based on the prediction loss of the labeled image and the prediction loss of the unlabeled image; The classification model is trained by minimizing the total prediction loss.

7. The method according to claim 6, characterized in that Outputting the abnormality recognition result of the fundus image according to the category prediction information includes: If the category prediction information includes an abnormality prediction value indicating that the fundus image belongs to an abnormal category, determining that the abnormality prediction value is greater than the prediction threshold, and obtaining that an abnormal region of the abnormal category exists in the fundus image; According to the preset output requirements, an abnormality identification result including at least the abnormal area and the corresponding abnormal category is obtained; The abnormality recognition result is output to mark the abnormal area on the fundus image.

8. A fundus image abnormality recognition device based on semi-supervised learning, characterized in that: The device comprises: A fundus image acquisition module, used for acquiring a fundus image to be identified; A category prediction information obtaining module is used to input the fundus image into a trained classification model to obtain category prediction information of the fundus image; wherein the classification model is obtained by using different enhanced images obtained by performing different image enhancement processes on unlabeled images, and performing semi-supervised mutual learning on the pending classification model, and the pending classification model is obtained by supervised training of category label correction on the labeled image; The abnormality recognition result output module is used to output the abnormality recognition result of the fundus image according to the category prediction information.

9. An electronic device, characterized in that: The electronic device comprises: at least one data communication port, at least one memory and at least one processor, wherein: The data communication port is used to obtain the fundus image to be identified; The processor is used to load and execute the computer instruction code stored in the memory to implement the following steps: obtaining a fundus image to be identified; Inputting the fundus image into a trained classification model to obtain category prediction information of the fundus image; wherein the classification model is obtained by using different enhanced images obtained by performing different image enhancement processes on unlabeled images, and performing semi-supervised mutual learning on the pending classification model, and the pending classification model is obtained by supervised training of category label correction on the labeled image; Based on the category prediction information, an abnormality recognition result of the fundus image is output.

10. The electronic device according to claim 9, characterized in that: The electronic device further comprises: An image acquisition device, used for acquiring fundus images to be identified; An output device, used for outputting an abnormality recognition result of the fundus image; The data communication port is also used to realize the communication connection between the electronic device and the server, and obtain the classification model that has been trained by the server.

Images