Eye image recognition system, method, apparatus, and storage medium

Abstract

The application provides an eye image recognition system, method and device and a storage medium, and relates to the technical field of image recognition.The system comprises an image acquisition device and an eye image recognition device.The image acquisition device is used to acquire an eye image to be recognized.The eye image recognition device is used to receive the eye image to be recognized, encode the eye image to be recognized based on an image encoder, and obtain an image feature vector;the image feature vector is sequentially matched with a normal eye text feature vector and an abnormal eye text feature vector, each eye disease text feature vector and each eye disease subtype text feature vector;the disease category recognition result of the eye image to be recognized is obtained according to the first matching result, the second matching result and the third matching result;and the corresponding severity grading result is obtained in combination with the image feature vector.The application can reduce the dependence on the device, improve the recognition accuracy and improve the model expansion performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to an eye image recognition system, method, device and storage medium. BACKGROUND

[0002] At present, the detection and diagnosis of anterior segment diseases mainly rely on professional medical equipment and manual judgment of doctors, and the basic process includes image acquisition, image preprocessing and feature extraction and analysis.

[0003] Among them, image acquisition is to obtain high-resolution anterior segment images by using professional equipment such as slit lamp, biological microscope, etc. Image preprocessing is to remove noise, normalize, enhance contrast (such as adaptive histogram equalization CLAHE) and smooth (such as Gaussian filter) of the collected image, so as to facilitate subsequent analysis. Feature extraction and analysis is to extract high-dimensional image features by using traditional image processing methods (such as edge detection, region segmentation) or deep learning technology (such as convolutional neural network CNN), and to diagnose and classify the lesion area according to the experience of doctors. Among them, after extracting high-dimensional image features in feature extraction and analysis, the disease categories are usually predicted by a classifier design through a fully connected layer and a softmax layer, and the diagnostic results are output. The classification results are optimized and corrected to form the final diagnostic report or disease assessment.

[0004] However, in the above process, the professional equipment required for image acquisition limits the wide screening and early detection of anterior segment diseases. When designing the classifier, the automatic classification method based on images mainly relies on visual information. On the one hand, it is difficult to achieve fine-grained classification, and the quantitative analysis of disease severity is insufficient, which is difficult to provide objective indicators for clinical practice. On the other hand, when new categories are added, the entire model usually needs to be retrained, which is costly and inefficient. In addition, it is also difficult to give accurate recognition results based on images collected by non-professional equipment (such as ordinary cameras, mobile phone cameras, etc.). SUMMARY

[0005] The embodiments of the present application provide an eye image recognition system, method, device and storage medium to solve the problems of strong device dependence, poor expansibility and insufficient disease assessment in existing eye image recognition.

[0006] In a first aspect, the embodiments of the present application provide an eye image recognition system, comprising: an image acquisition device and an eye image recognition device, the image acquisition device is used to acquire an eye image to be recognized, the eye image recognition device comprises a memory and a processor, the memory stores a computer program, and the processor realizes the following operations when executing the computer program:

[0007] receive an eye image to be identified, and encode the eye image to be identified based on an image encoder to obtain an image feature vector;

[0008] match the image feature vector with a normal eye text feature vector and an abnormal eye text feature vector respectively to obtain a first matching result and a first comprehensive feature vector corresponding to the first matching result;

[0009] match the first comprehensive feature vector with each eye disease text feature vector to obtain a second matching result and a second comprehensive feature vector corresponding to the second matching result;

[0010] match the second comprehensive feature vector with a disease subtype text feature vector of each eye disease to obtain a third matching result;

[0011] obtain a disease category identification result of the eye image to be identified according to the first matching result, the second matching result and the third matching result;

[0012] obtain a severity grading result corresponding to the disease category identification result according to the disease category identification result and the image feature vector.

[0013] In a possible implementation, matching the image feature vector with a normal eye text feature vector and an abnormal eye text feature vector respectively to obtain a first matching result and a first comprehensive feature vector corresponding to the first matching result comprises:

[0014] calculate a first cosine similarity between the image feature vector and the normal eye text feature vector, and calculate a second cosine similarity between the image feature vector and the abnormal eye text feature vector;

[0015] determine a first matching result according to the first cosine similarity and the second cosine similarity;

[0016] embed a target text feature vector corresponding to the first matching result into the image feature vector to obtain a first comprehensive feature vector corresponding to the first matching result, the target text feature vector being the normal eye text feature vector or the abnormal eye text feature vector.

[0017] In a possible implementation, before encoding the eye image to be identified based on an image encoder to obtain an image feature vector, the method further comprises:

[0018] preprocess the eye image to be identified, the preprocessing comprising image normalization processing, image equalization processing and Gaussian filter denoising processing.

[0019] In a possible implementation, the normal eye text feature vector is obtained by encoding, by a text encoder, symptom descriptions and detection data of a normal eye;

[0020] The non-normal eye text feature vector is obtained by encoding, by a text encoder, symptom descriptions and detection data of a non-normal eye.

[0021] The text feature vector of each eye disease is obtained by encoding, by a text encoder, symptom descriptions and detection data of each eye disease.

[0022] The text feature vector of a disease subtype of each eye disease is obtained by encoding, by a text encoder, symptom descriptions and detection data of the disease subtype of each eye disease.

[0023] In a possible implementation, the severity grading result corresponding to the disease category recognition result is obtained according to the disease category recognition result and the image feature vector, and the severity grading result corresponding to the disease category recognition result is obtained according to the disease category recognition result.

[0024] A binary classification model of a corresponding eye disease is selected according to the disease category recognition result, and is denoted as a target binary classification model.

[0025] A lesion feature vector is obtained by performing feature extraction on the image feature vector based on a feature extraction module of the target binary classification model.

[0026] A normal eye feature center is obtained based on a normal eye sample image adopted by the target binary classification model.

[0027] A target distance between the lesion feature vector and the normal eye feature center is calculated.

[0028] The target distance is subjected to severity grading mapping to obtain a severity grading result corresponding to the disease category recognition result.

[0029] In a possible implementation, the distance is subjected to severity grading mapping to obtain a severity grading result corresponding to the disease category recognition result, and the severity grading result corresponding to the disease category recognition result is obtained according to a normalization result of the target distance.

[0030] The target distance is normalized based on a maximum distance and a minimum distance, and a severity grading result corresponding to the disease category recognition result is obtained according to a normalization result, where the maximum distance is a distance between a feature vector of a target non-normal eye sample image farthest from the normal eye feature center in a non-normal eye sample image adopted by the target binary classification model and the normal eye feature center, and the minimum distance is a distance between a feature vector of a target normal eye sample image farthest from the normal eye feature center in a normal eye sample image adopted by the target binary classification model and the normal eye feature center.

[0031] In a possible implementation, the image acquisition device comprises a mobile terminal and a light supplement device, or an image sensor and a light supplement device, and the light supplement device comprises light sources of different frequency bands.

[0032] In a second aspect, an embodiment of the present application provides an eye image recognition method, applied to the eye image recognition system as described in the first aspect or any possible implementation of the first aspect, and the method comprises:

[0033] receiving an eye image to be recognized, and encoding the eye image to be recognized based on an image encoder to obtain an image feature vector;

[0034] matching the image feature vector with a normal eye text feature vector and an abnormal eye text feature vector respectively to obtain a first matching result and a first comprehensive feature vector corresponding to the first matching result;

[0035] matching the first comprehensive feature vector with each eye disease text feature vector respectively to obtain a second matching result and a second comprehensive feature vector corresponding to the second matching result;

[0036] matching the second comprehensive feature vector with each disease subtype text feature vector of each eye disease respectively to obtain a third matching result;

[0037] obtaining a disease category recognition result of the eye image to be recognized according to the first matching result, the second matching result and the third matching result;

[0038] obtaining a severity grading result corresponding to the disease category recognition result according to the disease category recognition result and the image feature vector.

[0039] In a third aspect, an embodiment of the present application provides an eye image recognition device, comprising:

[0040] an image acquisition and processing module, configured to receive an eye image to be recognized, and encode the eye image to be recognized based on an image encoder to obtain an image feature vector;

[0041] a first matching module, configured to match the image feature vector with a normal eye text feature vector and an abnormal eye text feature vector respectively to obtain a first matching result and a first comprehensive feature vector corresponding to the first matching result;

[0042] a second matching module, configured to match the first comprehensive feature vector with each eye disease text feature vector respectively to obtain a second matching result and a second comprehensive feature vector corresponding to the second matching result;

[0043] The third matching module is configured to match the second comprehensive feature vector with a disease subtype text feature vector of each eye disease respectively to obtain a third matching result;

[0044] The recognition module is configured to obtain a disease category recognition result of the eye image to be recognized according to the first matching result, the second matching result and the third matching result.

[0045] The severity grading module is configured to obtain a severity grading result corresponding to the disease category recognition result according to the disease category recognition result and the image feature vector.

[0046] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium storing a computer program, which, when executed by a processor, implements the method in the implementation manner of the second aspect.

[0047] In the embodiment of the present application, the image recognition device receives the eye image to be recognized and encodes the eye image to be recognized based on the image encoder to obtain the image feature vector. Then, the image feature vector is matched with the normal eye text feature vector and the non-normal eye text feature vector respectively to obtain the first matching result and the first comprehensive feature vector corresponding to the first matching result. Then, the first comprehensive feature vector is matched with the text feature vector of each eye disease respectively to obtain the second matching result and the second comprehensive feature vector corresponding to the second matching result. Then, the second comprehensive feature vector is matched with the disease subtype text feature vector of each eye disease respectively to obtain the third matching result. Finally, the disease category recognition result of the eye image to be recognized is obtained according to the first matching result, the second matching result and the third matching result. The severity grading result corresponding to the disease category recognition result is obtained according to the disease category recognition result and the image feature vector. The eye disease recognition result corresponding to the eye image to be recognized can be accurately given through the step-by-step transmission of the first comprehensive feature vector and the second comprehensive feature vector without limiting the image acquisition device for obtaining the eye image to be recognized. Moreover, the eye disease recognition result is subdivided according to the normal eye, the non-normal eye, each eye disease, the disease subtype of each eye disease and the severity level corresponding to the disease subtype of each eye disease, which can realize fine-grained classification and help to accurately quantify the severity of the eye disease and provide objective indicators for the clinic. In addition, since the image feature vector is matched with the corresponding text feature vector step by step in the embodiment of the present application, when a new category is added, only the corresponding sample image and text feature vector need to be trained, and the whole model does not need to be retrained, which helps to reduce the model training cost and improve the efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a schematic diagram of an eye image recognition system provided by an embodiment of the present application;

[0049] Figure 2 is a structural schematic diagram of an eye image recognition model provided by an embodiment of the present application;

[0050] Figure 3 is an implementation flowchart of an eye image recognition method provided by an embodiment of the present application;

[0051] Figure 4 is a structural schematic diagram of an eye image recognition device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0052] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0053] Referring to Figure 1 , a schematic diagram of an eye image recognition system provided by an embodiment of the present application is shown, as shown in Figure 1 , the eye image recognition system of this embodiment includes an image acquisition device 2 and an eye image recognition device 1, the image acquisition device 2 is used to acquire an eye image to be recognized, and the eye image recognition device 1 includes a processor 10 and a memory 11. The memory 11 stores a computer program 12. The processor 10 implements the specific steps of eye image recognition when executing the computer program 12. Alternatively, the processor 10 executes the computer program 12 to realize the functions of each module / unit in each device embodiment described below.

[0054] For example, the computer program 12 can be divided into one or more modules / units, which are stored in the memory 11 and executed by the processor 10 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 12 in the eye image recognition device 1.

[0055] The eye image recognition device 1 can include, but is not limited to, the processor 10, the memory 11. Those skilled in the art can understand that Figure 1 is only an example of the eye image recognition device 1 and does not constitute a limitation on the eye image recognition device 1, and can include more or fewer components than the illustration, or combine certain components, or different components, for example, the eye image recognition device 1 can also include an input / output device, a network access device, a bus, etc.

[0056] The processor 10 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0057] The memory 11 can be an internal storage unit of the eye image recognition device 1, such as a hard disk or a memory of the eye image recognition device 1. The memory 11 can also be an external storage device of the eye image recognition device 1, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the eye image recognition device 1. Further, the memory 11 can also include both the internal storage unit and the external storage device of the eye image recognition device 1. The memory 11 is used to store the computer program 12 and other programs and data required by the eye image recognition device 1. The memory 11 can also be used to temporarily store data that has been output or will be output.

[0058] For example, the image acquisition device 2 can be a non-professional eye image acquisition device such as a mobile terminal or an image sensor, such as a mobile phone, a tablet, a single-lens reflex camera, etc.

[0059] In order to improve the eye image recognition effect without professional equipment, the image acquisition device 2 can also include a mobile terminal and a light supplement device, or an image sensor and a light supplement device, wherein the light supplement device can include light sources of different frequency bands, such as visible light, blue light (such as 465-490 nm), near-infrared light (such as 800-1000 nm), ultraviolet light (such as 200-400 nm), short-wave infrared light (such as 1400-2500 nm), etc. For example, the light sources of different frequency bands can be LED lamps of different frequency bands, which are low in cost, easy to obtain, and do not increase the difficulty of obtaining the eye image to be recognized.

[0060] On this basis, in combination with Figure 2 and Figure 3 , the steps of eye image recognition are described as follows:

[0061] The eye image to be recognized is received, and the eye image to be recognized is encoded based on an image encoder to obtain an image feature vector.

[0062] The image feature vector is matched with the normal eye text feature vector and the non-normal eye text feature vector respectively, to obtain a first matching result and a first comprehensive feature vector corresponding to the first matching result.

[0063] The first comprehensive feature vector is matched with each eye disease text feature vector respectively, to obtain a second matching result and a second comprehensive feature vector corresponding to the second matching result.

[0064] The second comprehensive feature vector is matched with each disease subtype text feature vector of the eye disease respectively, to obtain a third matching result.

[0065] According to the first matching result, the second matching result and the third matching result, a disease category recognition result of the eye image to be recognized is obtained.

[0066] According to the disease category recognition result and the image feature vector, a severity classification result corresponding to the disease category recognition result is obtained.

[0067] In an embodiment, before the image feature vector is obtained by encoding the eye image to be recognized based on an image encoder, the method can further include:

[0068] The eye image to be recognized is preprocessed, and the preprocessing includes image normalization processing, image equalization processing and Gaussian filter denoising processing.

[0069] For example, the eye image to be recognized can be an anterior segment image collected by a mobile phone camera, a single-lens reflex camera or other image collection devices. After the anterior segment image is obtained, the collected anterior segment image can be normalized, adaptively histogram equalized (for example, contrast limited adaptive histogram equalization, CLAHE) and Gaussian filtered and denoised. Then, a normalized image feature vector can be extracted by using a deep neural network such as a Vision Transformer (ViT) and a convolutional neural network as an image encoder. .

[0070] The image normalization processing is to scale the pixel value to the interval [0, 1], and the formula is:

[0071] ;

[0072] In the formula, is the maximum pixel value in the image, is the minimum pixel value in the image. This can improve the model convergence speed, reduce the computational complexity and reduce the influence of data scale difference.

[0073] In the image equalization processing, an adaptive histogram equalization (e.g., CLAHE) method can be used to enhance the local region of the image to improve the low contrast problem.

[0074] In the Gaussian filter denoising processing, a Gaussian filter can be used to smooth the image to reduce noise, and the formula is as follows:

[0075] ;

[0076] Then, the preprocessed image will be input to the subsequent modules to ensure the consistency of features and image quality in various acquisition environments.

[0077] On this basis, a pre-trained deep neural network (such as ViT and convolutional neural network) can be used as an image encoder to extract features from the preprocessed image to obtain a normalized feature vector .

[0078] Meanwhile, during the feature extraction of the preprocessed image, multi-scale fusion (e.g., using a feature pyramid network FPN and a skip connection) can be used to ensure the effective combination of low-level details and high-level semantics.

[0079] In an embodiment, the normal eye text feature vector is obtained by encoding the symptom description and detection data of the normal eye based on a text encoder.

[0080] The non-normal eye text feature vector is obtained by encoding the symptom description and detection data of the non-normal eye based on a text encoder.

[0081] The text feature vector of each eye disease is obtained by encoding the symptom description and detection data of each eye disease based on a text encoder.

[0082] The text feature vector of the disease subtype of each eye disease is obtained by encoding the symptom description and detection data of the disease subtype of each eye disease based on a text encoder.

[0083] In combination Figure 2 with FIG. 1, in the embodiment, a Transformer model can be used as a text encoder to encode the preset limited terms at each level to generate the corresponding text feature vectors at each level .

[0084] For example, the limited terms can include: first-level limited terms (representing whether the eye is normal, such as “normal eye”, “non-normal eye”, etc.), second-level limited terms (representing the major category of eye diseases, such as “cataract”, “keratitis”, “pterygium”, “glaucoma”, etc.), and third-level limited terms (representing the fine classification of eye diseases or the subtype of eye diseases, such as “xx cataract”, “yy keratitis”, etc.).

[0085] For example, in order to improve the recognition accuracy, in addition to the words representing whether the eye is normal, the first level qualifier can also include the symptom description and some easily obtained detection data of the normal eye, as well as the symptom description of the abnormal eye and the corresponding easily obtained detection data. In addition to the words representing the major category of eye diseases, the second level qualifier can also include the symptom description and the corresponding easily obtained detection data of each eye disease. Similarly, in addition to the words representing the fine classification of eye diseases, the third level qualifier can also include the symptom description and the corresponding easily obtained detection data of the disease subtype of each eye disease.

[0086] In this embodiment, the image and the text are respectively converted into vector representations by the encoding module (i.e., the image encoder and the text encoder), which lays a foundation for subsequent contrastive learning and multi-level classification.

[0087] In an embodiment, the image feature vector is matched with the normal eye text feature vector and the abnormal eye text feature vector respectively, and a first matching result and a first comprehensive feature vector corresponding to the first matching result are obtained, including:

[0088] The first cosine similarity between the image feature vector and the normal eye text feature vector is calculated, and the second cosine similarity between the image feature vector and the abnormal eye text feature vector is calculated.

[0089] The first matching result is determined according to the first cosine similarity and the second cosine similarity.

[0090] The target text feature vector corresponding to the first matching result is embedded into the image feature vector to obtain the first comprehensive feature vector corresponding to the first matching result, and the target text feature vector is the normal eye text feature vector or the abnormal eye text feature vector.

[0091] On this basis, the process of matching the first comprehensive feature vector with each eye disease text feature vector respectively to obtain a second matching result and a second comprehensive feature vector corresponding to the second matching result, and the process of matching the second comprehensive feature vector with each disease subtype text feature vector of the eye disease respectively to obtain a third matching result are similar to the process of matching the image feature vector with the normal eye text feature vector and the abnormal eye text feature vector respectively to obtain the first matching result and the first comprehensive feature vector corresponding to the first matching result, which will not be described here.

[0092] Before matching, the feature sharing can be trained based on the contrastive learning mechanism, specifically including:

[0093] With InfoNCE Loss, a loss function applied to image-text pairs, the cosine similarity between image feature vectors and corresponding text feature vectors of the text descriptors of the images is optimized. The "image-text pairs" are fed into two encoders: image encoder and text encoder. The goal is to have higher similarity for the correct "image-text pairs" and lower similarity for the incorrect "image-text pairs".

[0094] The mathematical expression of InfoNCE Loss is:

[0095] The overall loss: .

[0096] The image loss: .

[0097] The text loss: .

[0098] Where:

[0099] is the normalized encoding vector of the th image, i.e., the image feature vector corresponding to the th image.

[0100] is the normalized encoding vector of the th text, i.e., the text feature vector corresponding to the th text.

[0101] is the cosine similarity.

[0102] is the temperature parameter, which can be used to control the sharpness of the distribution (usually a trainable parameter).

[0103] is the batch size, there are image-text pairs in each batch.

[0104] This embodiment has the following characteristics:

[0105] Adopting bidirectional contrast learning (bidirectional InfoNCE):

[0106] Having the highest similarity (maximum cosine similarity) for the correct image-text pairs.

[0107] Having lower similarity for the incorrect image-text pairs.

[0108] All samples are negative samples for each other:

[0109] In one batch, each sample is treated as a negative example of other samples, which can greatly increase the number of negative samples, so that the model learns stronger discrimination.

[0110] Normalization processing:

[0111] Before calculating the cosine similarity, the image feature vector and the text feature vector are normalized to ensure that their norms are 1, so that the contrast learning is more stable.

[0112] In addition, by introducing a skip connection and multi-scale feature fusion, the low-level and fine-grained features of the image encoder can be shared and continuously transmitted in the multi-level classification process, thereby improving the overall classification accuracy and robustness.

[0113] The step-by-step matching process includes:

[0114] First-level classification: compare the image feature vector of the eye image to be identified with the text feature vectors corresponding to "normal eyes" and "abnormal eyes" respectively to determine whether the eye image to be identified is normal. If the highest similarity is "normal eyes", output normal; if the highest similarity is "abnormal eyes", output abnormal, and embed the text feature vector corresponding to "abnormal eyes" into the image feature vector to form a first comprehensive feature vector for entering the second-level classification.

[0115] Second-level classification: for abnormal eyes, calculate the similarity between the first comprehensive feature vector transmitted from the previous level and the text feature vectors corresponding to various eye disease categories (such as "cataract", "keratitis", "pterygium", "glaucoma", etc.), output the probability through the softmax layer, determine the specific disease category, and embed the text feature vector corresponding to the determined eye disease into the first comprehensive feature vector to form a second comprehensive feature vector for entering the third-level classification.

[0116] Third-level classification: after determining the disease category, calculate the similarity between the second comprehensive feature vector and the text feature vectors corresponding to the sub-definitions (such as "xx cataract", "yy cataract") under this category to complete the fine division of the disease.

[0117] In the entire step-by-step classification process, the shared image bottom features are used to ensure the continuity and discrimination consistency of the information through the feature transmission module, thereby improving the overall classification accuracy and robustness.

[0118] In one embodiment, the severity classification result corresponding to the disease category recognition result is obtained according to the disease category recognition result and the image feature vector, including:

[0119] ​According to the disease category recognition result, a binary classification model corresponding to the eye disease is selected, and is denoted as a target binary classification model.

[0120] A feature extraction module based on the target binary classification model performs feature extraction on the image feature vector to obtain a lesion feature vector.

[0121] Based on the normal eye sample image adopted by the target binary classification model, a normal eye feature center is obtained.

[0122] The distance between the lesion feature vector and the normal eye feature center is calculated, and is denoted as a target distance.

[0123] The target distance is mapped to a severity classification result corresponding to the disease category recognition result.

[0124] In an embodiment, the target distance is mapped to a severity classification result corresponding to the disease category recognition result, including:

[0125] The target distance is normalized based on the maximum distance and the minimum distance, and the severity classification result corresponding to the disease category recognition result is obtained according to the normalized result, wherein the maximum distance is the distance between the feature vector of the farthest target non-normal eye sample image in the non-normal eye sample image adopted by the target binary classification model and the normal eye feature center, and the minimum distance is the distance between the feature vector of the farthest target normal eye sample image in the normal eye sample image adopted by the target binary classification model and the normal eye feature center.

[0126] For example, the binary classification model can be pre-trained first: for each specific disease (such as cataract, keratitis, etc.), a binary classification model is constructed and pre-trained respectively to distinguish the disease from the normal eye. Cross-entropy loss and contrast or triplet loss can be used during training to ensure that the lesion features and normal state are clearly distinguished in the feature space.

[0127] On this basis, a gating mechanism is adopted, that is, according to the secondary and tertiary classification results, a soft decision is made on the classification results through a lightweight fully connected network (for example, FCN), and a corresponding binary classification model (i.e., the target binary classification model) is dynamically selected.

[0128] Then, feature extraction and distance calculation are performed, and the specific process includes:

[0129] The selected binary classification model is used to extract the lesion feature vector based on the image feature vector extract the lesion feature vector .

[0130] Calculate the feature center of the normal eye of the target binary classification model :

[0131] ;

[0132] wherein, is the number of normal eye sample images used in training the target binary classification model, is the image feature vector of the i-th normal eye sample image used in training the target binary classification model.

[0133] Calculate the Euclidean distance between the to-be-identified eye image and the feature center:

[0134] ;

[0135] Disease severity normalization: for example, map the distance to a disease severity score of 1-100 level (this value can be selected according to actual conditions, for example, the distance can also be mapped to a disease severity score of 1-5 level, etc.):

[0136] ;

[0137] wherein, is the maximum distance, i.e., the distance between the feature vector of the non-normal eye sample image farthest from the normal eye feature center in the target binary classification model and the normal eye feature center, is the minimum distance, i.e., the distance between the feature vector of the normal eye sample image farthest from the normal eye feature center in the target binary classification model and the normal eye feature center, and can be statistically based on training data and dynamically adjusted through online updating strategies.

[0138] This module can ensure that dedicated models are used for feature extraction and disease quantification for different diseases, providing objective and quantitative auxiliary decision-making basis for clinics.

[0139] In the above modules, the following optimization strategies can also be integrated to improve the overall system performance:

[0140] Data augmentation: in the training process, perform various data augmentation operations such as rotation, scaling, flipping, color jittering, etc. on the images, simulate different lighting and shooting conditions, and improve the robustness of the model.

[0141] Multi-scale fusion: use FPN and skip connection to ensure that low-level fine-grained information and high-level semantic information are fully fused, and improve the detection ability of subtle lesion areas.

[0142] Attention mechanism: add CBAM or SE module in the image encoder to strengthen the feature expression of key areas (such as the anterior segment lesion area) and suppress background interference. ​

[0143] Online updating module: design online incremental learning strategy, adopt exponential moving average (EMA) to update feature center and score mapping parameters, ensure the system continuously optimize, adapt to the changing environment and population distribution in practical application, this kind of dynamic updating mechanism can effectively deal with data drift and environmental change in long-term deployment, improve the long-term stability and accuracy of the model.

[0144] Compared with the prior art, the embodiment of the application realizes similarity calculation between images and text qualifiers by introducing contrast learning technology, and realizes the following improvements by utilizing the advantages of shared and transmitted bottom features in multi-level classification:

[0145] High-precision multi-level classification: first-level judgment of normal and abnormal eyes, second-level identification of disease categories, and third-level completion of disease subdivision; Ensure continuous transmission of fine-grained information, significantly improve the diagnosis accuracy.

[0146] Flexible scalability: when adding new disease categories, only the corresponding qualifiers and text data need to be added, without the need to retrain the entire model.

[0147] Objective quantitative disease assessment: combined with the gated binary classification model and feature distance calculation, the severity of the disease is quantified as, for example, 1-100 levels, providing a reliable basis for clinical decision-making.

[0148] Device structure optimization: based on the traditional image acquisition and processing module, a text encoder, a contrast learning module and a gated binary classification module are added to build an image and text dual-channel system, realizing information fusion and efficient collaboration between modules.

[0149] In summary, the present application proposes a method for anterior segment disease classification and disease severity grading based on contrast learning to solve the problems of device dependence, strong subjectivity, poor scalability and insufficient disease assessment in traditional methods. The anterior segment images collected by ordinary mobile phones realize low-cost and non-invasive early screening. Through the multi-level classification strategy (first level: normal / abnormal; second level: disease category; third level: disease subdivision), fine and high-precision automatic diagnosis is realized. With the help of contrast learning technology, the cosine similarity between images and preset qualifier text is used to capture fine-grained discriminative features and improve the classification robustness. A gated binary classification model is used to extract special lesion features for different diseases, and through feature center calculation and distance mapping, the objective quantification of disease severity (e.g. 1-100 levels) is realized. Through multi-scale feature fusion, data enhancement, attention mechanism and online updating, the robustness and generalization ability of the model under different collection conditions are enhanced, and the model has the advantages of flexible expansion and convenient maintenance. The precision, scalability and clinical practicability of automated detection are significantly improved.

[0150] Reference Figure 3The following is a method embodiment of the present application, which is applied to the eye image recognition system of any of the above embodiments. For details not described in detail, please refer to the corresponding system embodiments described above.

[0151] Step 301, receiving an eye image to be identified, and encoding the eye image to be identified based on an image encoder to obtain an image feature vector.

[0152] Step 302, matching the image feature vector with the normal eye text feature vector and the non-normal eye text feature vector respectively, obtaining a first matching result and a first comprehensive feature vector corresponding to the first matching result.

[0153] Step 303, matching the first comprehensive feature vector with each eye disease text feature vector respectively, obtaining a second matching result and a second comprehensive feature vector corresponding to the second matching result.

[0154] Step 304, matching the second comprehensive feature vector with the disease subtype text feature vector of each eye disease respectively, obtaining a third matching result.

[0155] Step 305, obtaining a disease category recognition result of the eye image to be identified according to the first matching result, the second matching result and the third matching result.

[0156] Step 306, obtaining a severity grading result corresponding to the disease category recognition result according to the disease category recognition result and the image feature vector.

[0157] The present embodiment has the same beneficial effects as the eye image recognition system provided in the above embodiments, and will not be described here.

[0158] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0159] The following is a device embodiment of the present application. For details not described in detail, please refer to the corresponding method embodiments described above.

[0160] Figure 4 The structure of the eye image recognition device provided by the embodiment of the present application is shown. For the convenience of description, only the part related to the embodiment of the present application is shown, and the details are as follows:

[0161] As shown in Figure 4 , the eye image recognition device comprises:

[0162] The image acquisition and processing module 41 is configured to receive an eye image to be identified, and encode the eye image to be identified based on an image encoder to obtain an image feature vector.

[0163] The first matching module 42 is configured to match the image feature vector with the normal eye text feature vector and the non-normal eye text feature vector respectively, to obtain a first matching result and a first comprehensive feature vector corresponding to the first matching result.

[0164] The second matching module 43 is configured to match the first comprehensive feature vector with each eye disease text feature vector respectively, to obtain a second matching result and a second comprehensive feature vector corresponding to the second matching result.

[0165] The third matching module 44 is configured to match the second comprehensive feature vector with each disease subtype text feature vector of the eye disease respectively, to obtain a third matching result.

[0166] The recognition module 45 is configured to obtain a disease category recognition result of the eye image to be recognized according to the first matching result, the second matching result and the third matching result.

[0167] The severity grading module 46 is configured to obtain a severity grading result corresponding to the disease category recognition result according to the disease category recognition result and the image feature vector.

[0168] For the convenience and brevity of description, only the division of the above functional modules / units is exemplified, and in actual application, the above functions can be completed by different functional modules / units according to needs. The above modules / units can be realized in the form of hardware, software or a combination of hardware and software.

[0169] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method in each method embodiment described above is realized.

[0170] The embodiment of the present application further provides a computer program product, which comprises a computer program. When the computer program is executed by a processor, the method in each method embodiment described above is realized.

[0171] The computer program comprises computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electric carrier wave signal, telecommunication signal and software distribution medium, etc.

[0172] In the above embodiments, the description of each embodiment is focused on, and the part not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments. If there is no special description and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be mutually referred to, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0173] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An ocular image recognition system, characterized by, The image acquisition device is used to acquire an eye image to be identified, and the eye image recognition device includes a memory and a processor. Receiving the eye image to be identified, and encoding the eye image to be identified based on an image encoder to obtain an image feature vector; Matching the image feature vector with a normal eye text feature vector and an abnormal eye text feature vector respectively, embedding the abnormal eye text feature vector into the image feature vector when the image feature vector matches the abnormal eye text feature vector, obtaining a first matching result and a first comprehensive feature vector corresponding to the first matching result; Matching the first comprehensive feature vector with each eye disease text feature vector respectively, obtaining a second matching result and a second comprehensive feature vector corresponding to the second matching result; Matching the second comprehensive feature vector with each disease subtype text feature vector of each eye disease respectively, obtaining a third matching result; Obtaining a disease category recognition result of the eye image to be identified according to the first matching result, the second matching result and the third matching result; Obtaining a severity classification result corresponding to the disease category recognition result according to the disease category recognition result and the image feature vector. Matching the image feature vector with a normal eye text feature vector and an abnormal eye text feature vector respectively, embedding the abnormal eye text feature vector into the image feature vector when the image feature vector matches the abnormal eye text feature vector, obtaining a first matching result and a first comprehensive feature vector corresponding to the first matching result, including:

2. The ocular image recognition system of claim 1, wherein, Calculating a first cosine similarity between the image feature vector and the normal eye text feature vector, and calculating a second cosine similarity between the image feature vector and the abnormal eye text feature vector; If the second cosine similarity is greater than the first cosine similarity, it is determined that the image feature vector matches the abnormal eye text feature vector, and the abnormal eye is taken as the first matching result; Embedding the abnormal eye text feature vector into the image feature vector to obtain a first comprehensive feature vector corresponding to the first matching result. Before the image acquisition device is used to acquire an eye image to be identified, and the eye image recognition device includes a memory and a processor, the method further includes:

3. The ocular image recognition system of claim 1, wherein, Pretreating the eye image to be identified, the pretreatment including image normalization processing, image equalization processing and Gaussian filter noise reduction processing.

4. The eye image recognition system according to claim 1, wherein The normal eye text feature vector is obtained by encoding the symptom description and detection data of the normal eye based on a text encoder; The abnormal eye text feature vector is obtained by encoding the symptom description and detection data of the abnormal eye based on a text encoder; The eye disease text feature vector is obtained by encoding the symptom description and detection data of each eye disease based on a text encoder; ​ The text feature vector for each subtype of eye disease is obtained by encoding the symptom description and detection data of each subtype of eye disease using a text encoder.

5. The ocular image recognition system of claim 1, wherein, Based on the disease category identification result and the image feature vector, the severity rating result corresponding to the disease category identification result is obtained, including: Based on the disease category identification results, a binary classification model corresponding to the eye disease is selected and denoted as the target binary classification model; The feature extraction module based on the target binary classification model extracts features from the image feature vector to obtain the lesion feature vector; Based on the normal eye sample images used by the target binary classification model, the feature center of the normal eye is obtained; Calculate the distance between the lesion feature vector and the center of the normal eye feature vector, and denote it as the target distance; The target distance is mapped to a severity level to obtain the severity level result corresponding to the disease category identification result.

6. The ocular image recognition system of claim 5, wherein, The target distance is mapped to a severity level to obtain the severity level result corresponding to the disease category identification result, including: The target distance is normalized based on the maximum distance and the minimum distance. The severity classification result corresponding to the disease category identification result is obtained according to the normalization result. The maximum distance is the distance between the feature vector of the target abnormal eye sample image that is farthest from the feature center of the normal eye in the abnormal eye sample image used by the target binary classification model and the feature center of the normal eye. The minimum distance is the distance between the feature vector of the target normal eye sample image that is farthest from the feature center of the normal eye in the normal eye sample image used by the target binary classification model and the feature center of the normal eye.

7. The ocular image recognition system of claim 1, wherein, The image acquisition device includes a mobile terminal and a supplementary lighting device, or an image sensor and a supplementary lighting device, wherein the supplementary lighting device includes light sources of different frequency bands.

8. An eye image recognition method characterized by comprising: Applied to the eye image recognition system as described in any one of claims 1-7, the method comprises: Receive an image of an eye to be identified, and encode the image of the eye to be identified based on an image encoder to obtain an image feature vector; The image feature vector is matched with the normal eye text feature vector and the abnormal eye text feature vector respectively. When the image feature vector and the abnormal eye text feature vector match, the abnormal eye text feature vector is embedded into the image feature vector to obtain a first matching result and a first comprehensive feature vector corresponding to the first matching result. The first comprehensive feature vector is matched with the text feature vector of each eye disease to obtain the second matching result and the second comprehensive feature vector corresponding to the second matching result; The second comprehensive feature vector is matched with the text feature vector of each disease subtype of eye disease to obtain the third matching result; The disease category identification result of the eye image to be identified is obtained based on the first matching result, the second matching result and the third matching result; The severity rating result corresponding to the disease category identification result is obtained based on the disease category identification result and the image feature vector.

9. An ocular image recognition apparatus characterized by comprising: include: An image acquisition and processing module is configured to receive an eye image to be identified and encode the eye image to be identified based on an image encoder to obtain an image feature vector; A first matching module is configured to match the image feature vector with a normal eye text feature vector and a non-normal eye text feature vector respectively, embed the non-normal eye text feature vector into the image feature vector when the image feature vector matches the non-normal eye text feature vector, and obtain a first matching result and a first comprehensive feature vector corresponding to the first matching result; A second matching module is configured to match the first comprehensive feature vector with each eye disease text feature vector respectively, obtain a second matching result and a second comprehensive feature vector corresponding to the second matching result; A third matching module is configured to match the second comprehensive feature vector with each eye disease subtype text feature vector respectively, and obtain a third matching result; An identification module is configured to obtain a disease category identification result of the eye image to be identified according to the first matching result, the second matching result and the third matching result; A severity grading module is configured to obtain a severity grading result corresponding to the disease category identification result according to the disease category identification result and the image feature vector.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in claim 8.

Images