A method for associating eye image with symptom information

Through deep residual neural network training of multi-view multi-stage image-level classifiers, the characteristics of diabetes-related eye images were explored, and the fuzziness and instability of existing eye diagnosis methods were solved, and non-invasive and accurate diabetes diagnosis was achieved.

CN113378794BActive Publication Date: 2024-11-19CAPITALBIO CORP +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110780736.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-09
Publication Date
2024-11-19
Estimated Expiration
2041-07-09

AI Technical Summary

Technical Problem

Due to the lack of information and visual presentation, the existing visual diagnosis methods lead to ambiguity, subjectivity and instability, which limits the accuracy and safety of diabetes diagnosis, and multiple blood tests bring about physical trauma and infection risks. .

Method used

Deep residual neural network is used to train multi-view multi-stage image-level classifiers, and through image high-dimensional feature extraction and preprocessing technology, a diabetic sample-level classifier is established to explore disease-related eye images characteristics and areas, and provide important visual reference areas. Used for diabetes diagnosis.

Benefits of technology

A non-invasive diagnosis is achieved, avoiding the risk of physical trauma and infection in the subject, and improving the accuracy and safety of diabetes diagnosis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113378794B_ABST
    Figure CN113378794B_ABST
Patent Text Reader

Abstract

The present application discloses a method and device for associating information between eye images and symptom information, specifically, collecting image samples; preprocessing the image samples to obtain an eye image diagram; training a deep residual neural network using a training sample set to obtain a multi-view multi-stage image-level classifier; establishing a diabetes sample-level classifier based on the image-level classifier by extracting high-dimensional features of the image; statistically analyzing the image samples to obtain important eye image areas associated with symptom information in each image sample; and calculating important reference areas corresponding to the symptom information based on the image-level classifier. Specifically, the technical solution of the present application can provide doctors with an information processing solution, through which invasive diagnosis of the subject is not required, thereby not only avoiding physical trauma to the subject, but also avoiding infection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of medical systems, and more specifically, to a method and device for associating eye images with symptom information. Background Art

[0002] For a long time, traditional medicine represented by Chinese medicine has focused on the macroscopic level of "exterior" in diagnosis, such as observation, smell, questioning, and palpation; while modern medicine focuses on the microscopic level of "interior" such as biochemistry, immunity, molecules, and genes. In order to achieve accurate diagnosis of "exterior and interior are the same", it should be an accurate diagnosis that combines tradition and modernity, and it should be an accurate diagnosis that combines macro and micro. Only by achieving the true "exterior and interior are the same" can we achieve true precision health management.

[0003] Traditional medicine believes that the slight color changes of the human body due to disease can be seen by observing the eyes. Modern medicine also believes that changes in the conjunctival microcirculation can reflect the condition of the whole body. No matter what kind of disease, there will be corresponding changes in the conjunctival microcirculation of the eye. Therefore, eye diagnosis is not only an important technology for "preventive treatment", but also an excellent way to combine Chinese and Western medicine.

[0004] However, due to the lack of carriers to present eye images and lack of information-based and visual presentation forms, previous eye diagnosis had to rely on the doctor's naked eye observation and personal experience to complete. Naturally, various different schools of eye diagnosis were formed, which resulted in great ambiguity, subjectivity and instability. This brought many inconveniences to the clinical, teaching and scientific research of eye diagnosis, and severely limited the role of eye diagnosis.

[0005] At present, the diagnosis of diabetes requires multiple blood draws to test blood sugar, which not only increases the time and economic costs of the examinee and medical staff, but most importantly, this diagnostic method is invasive, which not only causes physical trauma to the examinee, but also easily causes infection to the examinee. Summary of the invention

[0006] In view of this, the present application provides a method and device for associating eye images with symptom information, which is used to provide doctors with a diagnostic tool to avoid causing physical trauma to the subject and to prevent the subject from getting infected.

[0007] In order to achieve the above objectives, the proposed solution is as follows:

[0008] A method for associating eye images with symptom information, the method comprising the steps of:

[0009] Collecting image samples, wherein the image samples include binocular eye images of diabetic people and binocular eye images of normal people;

[0010] Preprocessing the image sample to obtain an eye image, wherein the eye image includes an iris portion and / or a sclera portion based on an iris center;

[0011] The deep residual neural network is trained using the training sample set to obtain a multi-view multi-stage image-level classifier; a diabetes sample-level classifier is established based on the multi-view multi-stage image-level classifier by extracting high-dimensional features of the image;

[0012] Performing statistical analysis on the image samples to obtain important eye image areas associated with symptom information in each of the binocular eye image images;

[0013] The image-level classifier is used to explore the features and regions of disease-related eye images, and obtain important reference regions corresponding to the symptom information.

[0014] Optionally, the preprocessing of the image sample to obtain an eye image comprises the steps of:

[0015] Marking the iris center of each of the binocular eye images;

[0016] performing foreground extraction processing on the binocular eye image according to the iris center, so that only the sclera image and the iris image are retained in the binocular eye image;

[0017] The binocular eye image after foreground extraction processing is subjected to extraction processing to obtain the iris part and the sclera part

[0018] The iris part and the sclera part are enhanced to obtain the eye image.

[0019] Optionally, the method of training a deep residual neural network using a training sample set to obtain a multi-view multi-stage image-level classifier comprises the following steps:

[0020] Performing data preparation based on the training sample set to obtain a training set and a test set;

[0021] Inputting the training sample set into the deep residual neural network for model training to obtain an initial model;

[0022] The initial model is tested and adjusted using the test set to obtain the multi-view multi-stage image-level classifier.

[0023] Optionally, the exploration of disease-related eye image features and regions using the image-level classifier to obtain important reference regions corresponding to the symptom information comprises the steps of:

[0024] Generate vector sets according to the eye image diagrams respectively, perform statistical analysis on the vector sets, and generate a P value matrix;

[0025] Inputting the specific eye image into the diabetes image-level classifier and calculating the importance of each position of the image to the classifier to obtain an importance matrix in the specific eye image;

[0026] A fusion processing unit is used to fuse the P value matrix and the importance matrix with the iris center as the origin to obtain the visualized important reference area.

[0027] An information association device for eye image and symptom information, the information association device comprising:

[0028] A data acquisition module, used for acquiring image samples, wherein the image samples include binocular eye images of diabetic patients and binocular eye images of normal patients;

[0029] A data preprocessing module, used for preprocessing the image sample to obtain an eye image, wherein the eye image includes an iris part and / or a sclera part based on the iris center;

[0030] A model training module is used to train a deep residual neural network using a training sample set to obtain a multi-view multi-stage image-level classifier;

[0031] A model processing module, used for establishing a diabetes sample-level classifier based on the multi-view multi-stage image-level classifier by extracting high-dimensional features of the image;

[0032] The correlation analysis module is used to use the image-level classifier to explore the disease-related eye image features and areas to obtain important reference areas corresponding to the symptom information.

[0033] Optionally, the data preprocessing module includes:

[0034] An iris marking unit, used for marking the iris center of each binocular eye image;

[0035] a foreground extraction unit, configured to perform foreground extraction processing on the binocular eye image according to the iris center, so that only the sclera image and the iris image are retained in the binocular eye image;

[0036] An image extraction unit is used to extract the binocular eye image after foreground extraction processing to obtain the iris part and the sclera part.

[0037] The enhancement processing unit is used to perform enhancement processing on the iris part and the sclera part to obtain the eye image.

[0038] Optionally, the model training module includes:

[0039] A data preparation unit, used to prepare data based on the training sample set to obtain a training set and a test set;

[0040] A training execution unit, used for inputting the training sample set into the deep residual neural network for model training to obtain an initial model;

[0041] A model adjustment unit is used to use the test set to test and adjust the initial model to obtain the multi-view multi-stage image-level classifier.

[0042] Optionally, the classification execution module includes:

[0043] A statistical analysis unit, used for generating vector sets according to the eye image diagrams, performing statistical analysis on the vector sets, and generating a P value matrix;

[0044] A fusion processing unit, used for performing fusion processing on the P value matrix to obtain the important area associated with the symptom information;

[0045] A first classification unit, used for inputting the specific eye image into the diabetes image-level classifier to obtain an importance matrix in the specific eye image;

[0046] The second classification unit is used to perform fusion processing on the importance matrix with the iris center as the origin to obtain the visualized important reference area.

[0047] It can be seen from the above technical solutions that the present application discloses a method and device for associating information between eye images and symptom information, specifically collecting image samples; preprocessing the image samples to obtain an eye image diagram; training the deep residual neural network using a training sample set to obtain a multi-view multi-stage image-level classifier; establishing a diabetes sample-level classifier based on the image-level classifier by extracting high-dimensional features of the image; inputting a specific eye image into the diabetes image-level classifier and calculating the importance of each position of the image to the classifier to obtain an important reference area corresponding to the symptom information. Specifically, the technical solution of the present application can provide doctors with an information processing solution, through which the subject does not need to be invasively diagnosed, thereby not only avoiding physical trauma to the subject, but also avoiding infection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flowchart of a method for associating eye images with symptom information according to an embodiment of the present application;

[0050] Figure 2 This is a sample example diagram of an embodiment of the present application;

[0051] Figure 3 This is an example diagram of a foreground segmentation result of an embodiment of the present application;

[0052] Figure 4 This is an example diagram of iris extraction results in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of the transfer learning process at each stage of an embodiment of the present application;

[0054] Figure 6 This is a heat map of the P value difference between the eye images of diabetics and healthy people in the embodiment of the present application;

[0055] Figure 7 A schematic diagram of an important reference area of ​​an eye diagram according to an embodiment of the present application;

[0056] Figure 8 This is a block diagram of another device for associating eye image and symptom information according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] Embodiment 1

[0059] Figure 1 This is a flow chart of a method for associating eye images with symptom information according to an embodiment of the present application.

[0060] like Figure 1 As shown, the information association method provided in this embodiment includes the following steps:

[0061] S1. Collect image samples.

[0062] The image samples include binocular eye images of diabetic patients and binocular eye images of normal patients, and the demographic information includes demographic information of diabetic patients and demographic information of normal patients. The binocular eye images include the upward view, downward view, left view and right view of the binocular eyes, such as Figure 2It is worth pointing out that since the image samples are of diabetic people and normal people, the image samples here need to be sufficient in number.

[0063] S2. Preprocess the image samples to obtain an eye image.

[0064] In the specific processing process, the center of the eyeball, that is, the iris center, is first automatically marked. The iris, sclera and iris center are automatically segmented using image processing algorithms, where the image processing algorithms include but are not limited to image enhancement algorithms, corrosion algorithms, dilation algorithms, edge detection algorithms, and division of connected areas.

[0065] Then, foreground extraction is performed to remove the skin, eyelashes, and edge of the eye image acquisition device in the binocular eye image, leaving only the sclera image and iris image. And under the premise of maintaining the image ratio, the image size is normalized to 512x512x3 pixels, such as Figure 3 The methods used include but are not limited to active contour models (SNAKE, level set method, etc.), graph segmentation methods (GraphCut, GrabCut and Random Walk, etc.), threshold segmentation methods (Otsu method), etc.

[0066] Then, the image obtained by the above operation is accurately extracted. The approximate iris part is first extracted through the image processing algorithm, and then the iris radius of a single image is calculated through polar coordinate transformation. The iris part is masked and the sclera part is calculated at the same time. The result is as follows: Figure 4 Specifically, the image processing algorithm includes but is not limited to image enhancement, corrosion algorithm, expansion algorithm, edge detection algorithm, and division of connected areas.

[0067] Finally, the obtained image is enhanced to obtain an eye image. The eye image here may include only the image of the iris part or the image of the sclera part, or may include the image of the iris part and the image of the sclera part. The enhancement processing method includes but is not limited to image correction (histogram equalization, normalization, white balance, grayscale correction and transformation, image smoothing, etc.), illumination distortion (random change of image brightness, contrast, saturation, etc.) and geometric distortion (random image scaling, cropping, flipping, rotation), local random occlusion of the image (block occlusion, grid occlusion), etc. The eye image here is finally scaled to 224x224 pixels.

[0068] S3. Use the training sample set to train the deep residual neural network.

[0069] Specifically, multi-view, multi-stage transfer learning is used to train the deep residual neural network, that is, the deep residual neural network is trained using the training sample set obtained above to obtain a multi-view, multi-stage image-level classifier for diabetes. The specific steps are as follows:

[0070] First, data preparation is performed based on the above training sample set. That is, the training set and the test set are randomly divided according to the ID of the sample. Taking the training set as an example, for each sample, the multivariate eye image corresponding to the sample is divided into G and L = {L p}, p∈[1, 2, ..., m] is a subset. That is, G is a set of all visual eye images, which is used for the model to find the full-area eye image features, and each subset of L contains a certain specific eye image of the relative position of the inner eye corner and the eyeball (or the eye image after horizontal mirroring), which is used to find the local eye image features. The data partitioning strategy includes but is not limited to K-fold cross validation, random sampling without replacement, stratified random sampling, stratified equal-ratio random sampling, etc. The multivariate eye image includes the sclera image and iris image obtained by the above operation, and their enhanced images. The eye image images in the training sample set here include images from multiple angles, such as multi-angle eye images such as left, right, upper left, lower left, upper right, lower right, upper, and lower, so that the information contained in the sample is richer and the final multi-view multi-stage image-level classifier is more accurate.

[0071] Then, the training set is input into the above residual deep neural network for training to obtain the initial model. The model training process here is divided into T stages. In the first stage, the ImageNet pre-trained model is used to fine-tune the model with images in G and L respectively to obtain model G 1 and in, Only images with viewpoint p are trained and predicted.

[0072] As in the first stage, in each subsequent stage, the model obtained in the previous stage is used as the initial model of the current stage, and the model is fine-tuned using the images in G and L respectively. In particular, in order to improve the practicality of the model, in the last few stages, only the data of the categories that the model in the previous stage classified more incorrectly can be used to fine-tune the model. For example, if the model in the T0 stage has a low classification accuracy for a certain type of diabetes and a certain type of healthy people, then in the subsequent T-T0 stages, only the data of these two subclasses in the training set can be used to fine-tune the model.

[0073] Specifically, if the model obtained in the previous stage consists of m sub-models, that is, Then in the i+1 stage, the m models are fine-tuned using the data in G, and the image L with a viewing angle of p is used pFine-tuning the network For T stages, this step will yield 2 T Each group of models consists of 1 or m sub-models. Taking m = 8 and T = 4 as an example, the transfer learning process at each stage is as follows: Figure 5 shown.

[0074] Specifically, T is an integer greater than 0; batch size is an integer with factors of 2 and 5; the learning rate range is (0.0000001, 1); optimization methods include but are not limited to Ranger, Adam, SGD, RMSProp, AdaGrad, etc.; the deep residual network includes the following modules in sequence: convolution layer, normalization, activation layer, pooling layer, 8 submodules, random drop, softmax layer. Each submodule includes in sequence: convolution layer, normalization layer, activation layer, convolution layer, normalization layer, residual connection layer, activation layer. The convolution kernel size of the convolution layer includes but is not limited to integers such as 3, 5, 7, and the probability of random drop includes but is not limited to 0.3, 0.5, 0.7.

[0075] Finally, the initial model is tested using the validation set, and the samples with incorrect predictions are sorted out. From this part of the samples, a relatively balanced data sample set is randomly selected to fine-tune the previous model, thus finally obtaining the above-mentioned multi-view multi-stage image-level classifier.

[0076] S4. Obtain a diabetes sample-level classifier by extracting high-dimensional features of the image.

[0077] According to the high-dimensional features of the model of 2×m unenhanced eye images of each subject, a diabetes sample-level classifier is established based on the multi-view multi-stage image-level classifier to calculate the probability that the subject is predicted to be a diabetic patient.

[0078] Specifically, high-dimensional features include but are not limited to feature maps and feature vectors of the middle layer of the model, image prediction results of the last layer, etc., and classifiers include but are not limited to voting, logistic regression, random forest, support vector machine, decision tree, adaptive enhancement, extreme gradient boosting, etc. In addition, this step can also extract high-dimensional features from the iris part of the image, and model it together with other high-dimensional features to generate a diabetes sample-level classifier.

[0079] S5. Input the specific eye image into the diabetes image-level classifier to obtain the important reference area.

[0080] Firstly, vector sets are generated according to the eye images of the left eye and the right eye respectively.

[0081] Taking the iris center as the origin, for the same coordinate position (x, y)∈[-224, 224] 2, respectively calculate the pixel value of the monocular eye image of diabetes and healthy people at this position in the R, G, and B channels to form a vector set in, Represents the pixel set vector of channel C at the coordinate (x, y) position of the left (right) eye image of all diabetic patients, Represents the pixel set vector of channel C at the coordinate (x, y) position of the left (right) eye image of all healthy people.

[0082] Then, statistical analysis is performed on the vector sets generated by the eye images of the left eye and the right eye respectively to generate a P value matrix, and the P value matrices of the two eyes are fused.

[0083] Specifically, a difference test method including but not limited to the Mann-whitney test and T test is used to test each set of vectors. Perform a two-sided test on the null hypothesis that the means of the two groups of vectors are equal and calculate its P value Where C∈[R,G,B],(x,y)∈[-224,224] 2 . Select the part with P value less than the specified significance level (α) as the P value matrix of the left eye / right eye. That is:

[0084]

[0085] When fusing the P value matrix, the right eye P value matrix of the R, G, and B channels is horizontally mirrored and dot-multiplied with the right eye P value matrix to generate the P value matrix P0∈[0, α 2 ] 224*224*3 . Make a transformation The P values ​​were normalized to [0, 255] to visualize the P value heatmap, i.e., the significant regions associated with symptom information.

[0086] like Figure 6 As shown, the lighter the color, the more significant the difference between the diabetic and healthy people's eye images at the corresponding position, and the position where the color is not black indicates that there are significant differences in the left and right eye images of the diabetic and healthy people's eye images at the corresponding position.

[0087] Then, for a specific eye image X to be identified, the specific eye image is input into the above-mentioned image-level classifier, and the importance of each pixel in X is defined by calculating the derivative of the classifier at each pixel in X. The importance of all pixels in each eye image forms an importance matrix M. Specifically, when the given eye image is a right eye image, its importance matrix is ​​horizontally mirrored.

[0088] Finally, for all the obtained eye image importance matrices {M i}, and the fusion is performed with the iris center as the origin. Where i∈{1, 2, …, n} represents the i-th image. Specifically, for the pixel point (x, y)∈[-224, 224] 2 , average the elements of all importance matrices at this position to get the fusion matrix. Similarly, the fusion matrix is ​​normalized to [0, 255] and visualized to obtain the above important reference areas, such as Figure 7 shown.

[0089] The fusion method of the above-mentioned importance matrices includes but is not limited to first horizontally mirroring the importance matrix of each left eye image and then averaging all the importance matrices, first averaging the importance matrices of the left and right eye image images respectively, and then horizontally mirroring the left eye average fusion matrix and fusing it with the right eye average fusion matrix.

[0090] It can be seen from the above technical scheme that this embodiment provides a method for associating information between eye images and symptom information, specifically collecting image samples; preprocessing the image samples to obtain an eye image diagram; using the training sample set to train the deep residual neural network to obtain a multi-view multi-stage image-level classifier; by extracting high-dimensional features of the image, a diabetes sample-level classifier is established based on the image-level classifier; a specific eye image is input into the diabetes image-level classifier to obtain an important reference area corresponding to the symptom information. Specifically, the technical scheme of the present application can provide doctors with an information processing scheme, through which the subject does not need to be invasively diagnosed, thereby not only avoiding physical trauma to the subject, but also avoiding infection.

[0091] Embodiment 2

[0092] Figure 8 This is a block diagram of a device for associating eye image and symptom information according to an embodiment of the present application.

[0093] like Figure 8 As shown, the information association device provided in this embodiment includes a data acquisition module 10, a data preprocessing module 20, a model training module 30, a model processing module 40 and a classification execution module 50.

[0094] The data acquisition module is used to collect image samples.

[0095] The image samples include binocular eye images of diabetic patients and binocular eye images of normal patients, and the demographic information includes demographic information of diabetic patients and demographic information of normal patients. The binocular eye images include the upward view, downward view, left view and right view of the binocular eyes, such as Figure 2It is worth pointing out that since the image samples are of diabetic people and normal people, the image samples here need to be sufficient in number.

[0096] The data preprocessing module is used to preprocess the image samples to obtain the eye image.

[0097] The module includes an iris annotation unit, a foreground extraction unit, an image extraction unit, and an enhancement processing unit. In the specific processing process, the iris annotation unit is used to automatically annotate the center of the eyeball, that is, the iris center. The iris, sclera, and iris center are automatically segmented using image processing algorithms, and the image processing algorithms here include but are not limited to image enhancement algorithms, corrosion algorithms, expansion algorithms, edge detection algorithms, and division of connected areas.

[0098] The foreground extraction unit is used to perform foreground extraction processing, removing the skin, eyelashes, and the edge of the eye image acquisition device in the binocular eye image, and only retaining the sclera image and iris image. And under the premise of maintaining the image ratio, the image size is normalized to 512x512x3 pixels, such as Figure 3 The methods used include but are not limited to active contour models (SNAKE, level set method, etc.), graph segmentation methods (GraphCut, GrabCut and Random Walk, etc.), threshold segmentation methods (Otsu method), etc.

[0099] The image extraction unit is used to accurately extract the image obtained by the above operation. The image processing algorithm is used to first extract the approximate iris part, and then the polar coordinate transformation is used to calculate the iris radius of a single image. The iris part is masked and the sclera part is calculated at the same time. The result is as follows: Figure 4 Specifically, the image processing algorithm includes but is not limited to image enhancement, corrosion algorithm, expansion algorithm, edge detection algorithm, and division of connected areas.

[0100] The enhancement processing unit is used to enhance the obtained image to obtain an eye image. The eye image here may include only the image of the iris part or the image of the sclera part, or may include the image of the iris part and the image of the sclera part. The enhancement processing method includes but is not limited to image correction (histogram equalization, normalization, white balance, grayscale correction and transformation, image smoothing, etc.), illumination distortion (random changes in brightness, contrast, saturation, etc. of the image) and geometric distortion (random image scaling, cropping, flipping, rotation), local random occlusion of the image (block occlusion, grid occlusion), etc. The eye image here is finally scaled to a size of 224x224 pixels.

[0101] The model training module is used to train the deep residual neural network using the training sample set.

[0102] Specifically, the deep residual neural network is trained using multivariate, multi-perspective, and multi-stage transfer learning, that is, the deep residual neural network is trained using the training sample set obtained above to obtain the image-level classifier for diabetes. The module includes a data preparation unit, a training execution unit, and a model adjustment unit.

[0103] The data preparation unit is used to prepare data based on the training sample set. That is, the training set and the test set are randomly divided according to the ID of the sample. Taking the training set as an example, for each sample, the multivariate eye image corresponding to the sample is divided into G and L = {L p}, p∈[1, 2, ..., m]. That is, G is the set of all visual angle eye images, which is used for the model to find the global eye image features. Each subset of L contains a certain specific eye image of the relative position of the inner corner of the eye and the eyeball (or the eye image after horizontal mirroring), which is used to find the local eye image features. The data partitioning strategy includes but is not limited to K-fold cross validation, random sampling without replacement, stratified random sampling, stratified equal ratio random sampling, etc. The multivariate eye image includes the sclera image and iris image obtained by the above operation, as well as their enhanced images.

[0104] The training execution unit is used to input the training set into the above residual deep neural network for training to obtain the initial model. The model training process here is divided into T stages. In the first stage, the ImageNet pre-trained model is used to fine-tune the model with images in G and L respectively to obtain the model G 1 and in, Only images with viewpoint p are trained and predicted.

[0105] As in the first stage, in each subsequent stage, the model obtained in the previous stage is used as the initial model of the current stage, and the model is fine-tuned using the images in G and L respectively. In particular, in order to improve the practicality of the model, in the last few stages, only the data of the categories that the model in the previous stage classified more incorrectly can be used to fine-tune the model. For example, if the model in the T0 stage has a low classification accuracy for a certain type of diabetes and a certain type of healthy people, then in the subsequent T-T0 stages, only the data of these two subclasses in the training set can be used to fine-tune the model.

[0106] Specifically, if the model obtained in the previous stage consists of m sub-models, that is, Then in the i+1 stage, the m models are fine-tuned using the data in G, and the image L with a viewing angle of p is used p Fine-tuning the network For T stages, this step will yield 2 TEach group of models consists of 1 or m sub-models. Taking m = 8 and T = 4 as an example, the transfer learning process at each stage is as follows: Figure 5 shown.

[0107] Specifically, T is an integer greater than 0; batch size is an integer with factors of 2 and 5; the learning rate range is (0.0000001, 1); optimization methods include but are not limited to Ranger, Adam, SGD, RMSProp, AdaGrad, etc.; the deep residual network includes the following modules in sequence: convolution layer, normalization, activation layer, pooling layer, 8 submodules, random drop, softmax layer. Each submodule includes in sequence: convolution layer, normalization layer, activation layer, convolution layer, normalization layer, residual connection layer, activation layer. The convolution kernel size of the convolution layer includes but is not limited to integers such as 3, 5, 7, and the probability of random drop includes but is not limited to 0.3, 0.5, 0.7.

[0108] The model adjustment unit is used to test the obtained initial model using the validation set and sort out the samples with incorrect predictions. From this part of the samples, a relatively balanced data sample set is randomly selected to fine-tune the previous model, thereby finally obtaining the above-mentioned multi-view multi-stage image-level classifier.

[0109] The model processing module is used to obtain a diabetes sample-level classifier by extracting high-dimensional features of the image.

[0110] According to the high-dimensional features of the model of 2×m unenhanced eye images of each subject, a diabetes sample-level classifier is established based on the multi-view multi-stage image-level classifier to calculate the probability that the subject is predicted to be a diabetic patient.

[0111] Specifically, high-dimensional features include but are not limited to feature maps and feature vectors of the middle layer of the model, image prediction results of the last layer, etc., and classifiers include but are not limited to voting, logistic regression, random forest, support vector machine, decision tree, adaptive enhancement, extreme gradient boosting, etc. In addition, this step can also extract high-dimensional features from the iris part of the image, and model it together with other high-dimensional features to generate a diabetes sample-level classifier.

[0112] The classification execution module is used to input the specific eye image into the image level classifier to obtain the important reference area. The module includes a statistical analysis unit, a fusion processing unit, a first classification unit and a second classification unit.

[0113] The statistical analysis unit is used for generating vector sets according to the eye image of the left eye and the right eye respectively.

[0114] Taking the iris center as the origin, for the same coordinate position (x, y)∈[-224, 224] 2, respectively calculate the pixel value of the monocular eye image of diabetes and healthy people at this position in the R, G, and B channels to form a vector set in, Represents the pixel set vector of channel C at the coordinate (x, y) position of the left (right) eye image of all diabetic patients, Represents the pixel set vector of channel C at the coordinate (x, y) position of the left (right) eye image of all healthy people.

[0115] The statistical analysis unit is used to perform statistical analysis on the vector sets generated by the eye images of the left eye and the right eye respectively, generate a P value matrix, and fuse the P value matrices of the two eyes.

[0116] Specifically, a difference test method including but not limited to the Mann-whitney test and T test is used to test each set of vectors. Perform a two-sided test on the null hypothesis that the means of the two groups of vectors are equal and calculate its P value Where C∈[R,G,B],(x,y)∈[-224,224] 2 . Select the part with P value less than the specified significance level (α) as the P value matrix of the left eye / right eye. That is:

[0117]

[0118] When the P value matrix is ​​fused, the right eye P value matrix of the R, G, and B channels is horizontally mirrored and dot-multiplied with the right eye P value matrix to generate the P value matrix P0∈[0, α 2 ] 224*224*3 . Make a transformation The P values ​​were normalized to [0, 255] to visualize the P value heatmap, i.e., the significant regions associated with symptom information.

[0119] like Figure 6 As shown, the lighter the color, the more significant the difference between the diabetic and healthy people's eye images at the corresponding position, and the position where the color is not black indicates that there are significant differences in the left and right eye images of the diabetic and healthy people's eye images at the corresponding position.

[0120] The first classification unit is used for inputting a specific eye image X to be identified into the above-mentioned image-level classifier, and defining the importance of each pixel in X by calculating the derivative of the classifier at each pixel in X. The importance of all pixels in each eye image forms an importance matrix M. Specifically, when the given eye image is a right eye image, its importance matrix is ​​horizontally mirrored.

[0121] The second classification unit is used to obtain the importance matrix {M i}, and the fusion is performed with the iris center as the origin. Where i∈{1, 2, …, n} represents the i-th image. Specifically, for the pixel point (x, y)∈[-224, 224] 2 , average the elements of all importance matrices at this position to get the fusion matrix. Similarly, the fusion matrix is ​​normalized to [0, 255] and visualized to obtain the above important reference areas, such as Figure 7 shown.

[0122] The fusion method of the above-mentioned importance matrices includes but is not limited to first horizontally mirroring the importance matrix of each left eye image and then averaging all the importance matrices, first averaging the importance matrices of the left and right eye image images respectively, and then horizontally mirroring the left eye average fusion matrix and fusing it with the right eye average fusion matrix.

[0123] It can be seen from the above technical scheme that this embodiment provides an information association device for eye images and symptom information, which is specifically used to collect image samples; pre-process the image samples to obtain an eye image diagram; train the deep residual neural network using the training sample set to obtain a multi-view multi-stage image-level classifier; establish a diabetes sample-level classifier based on the image-level classifier by extracting high-dimensional features of the image; input a specific eye image into the diabetes image-level classifier to obtain an important reference area corresponding to the symptom information. Specifically, the technical scheme of the present application can provide doctors with an information processing scheme, through which the subject does not need to be invasively diagnosed, thereby not only avoiding physical trauma to the subject, but also avoiding infection.

[0124] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0125] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0126] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0127] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0129] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0130] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0131] The technical solution provided by the present invention is introduced in detail above. Specific examples are used in this article to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for associating eye images with symptom information, characterized in that: The information association method comprises the steps of: Collecting image samples, wherein the image samples include binocular eye images of diabetic people and binocular eye images of normal people; Preprocessing the image sample to obtain an eye image, wherein the eye image includes an iris portion and / or a sclera portion based on an iris center; The deep residual neural network is trained using the training sample set to obtain a multi-view multi-stage diabetes image-level classifier; a diabetes sample-level classifier is established based on the multi-view multi-stage diabetes image-level classifier by extracting high-dimensional features of the image; Performing statistical analysis on the image samples to obtain the important eye image area associated with the symptom information in each of the binocular eye image images, specifically generating a vector set according to the eye image diagram, performing statistical analysis on the vector set to generate a P value matrix, and performing fusion processing on the P value matrix to obtain the important eye image area associated with the symptom information; An image-level classifier is used to analyze the features and regions of disease-related eye images to obtain important reference regions corresponding to the symptom information, a specific eye image is input into the image-level classifier to obtain an importance matrix in the specific eye image, and the importance matrix is ​​fused with the iris center as the origin to obtain a visualized important reference region, wherein the importance of each pixel in the specific eye image is obtained by calculating the derivative of the image-level classifier for each pixel, and the importance matrix is ​​constructed based on all the importances, and the fusion process includes horizontally mirroring the importance matrix of each left eye image and then averaging and fusing all the importance matrices, or respectively averaging and fusing the importance matrix of the left eye image and the importance matrix of the right eye image, and then horizontally mirroring the obtained left eye average fusion matrix and fusing it with the obtained right eye average fusion matrix.

2. The information association method according to claim 1, characterized in that: The preprocessing of the image samples to obtain an eye image comprises the following steps: Marking the iris center of each of the binocular eye images; performing foreground extraction processing on the binocular eye image according to the iris center, so that only the sclera image and the iris image are retained in the binocular eye image; Performing extraction processing on the binocular eye image that has undergone foreground extraction processing to obtain the iris part and the sclera part; The iris part and the sclera part are enhanced to obtain the eye image.

3. The information association method according to claim 1, wherein: The method of training a deep residual neural network using a training sample set to obtain a multi-view multi-stage diabetes image-level classifier comprises the following steps: Performing data preparation based on the training sample set to obtain a training set and a test set; Inputting the training sample set into the deep residual neural network for model training to obtain an initial model; The initial model is tested and adjusted using the test set to obtain the multi-view multi-stage diabetes image-level classifier.

4. An information association device for eye images and symptom information, characterized in that: The information association device comprises: A data acquisition module, used for acquiring image samples, wherein the image samples include binocular eye images of diabetic patients and binocular eye images of normal patients; A data preprocessing module, used for preprocessing the image sample to obtain an eye image, wherein the eye image includes an iris part and / or a sclera part based on the iris center; A model training module is used to train a deep residual neural network using a training sample set to obtain a multi-view multi-stage diabetes image-level classifier; and to establish a diabetes sample-level classifier based on the multi-view multi-stage diabetes image-level classifier by extracting high-dimensional features of the image; A model processing module, used for performing statistical analysis on the image samples to obtain the important eye image area associated with the symptom information in each of the binocular eye image, specifically generating a vector set according to the eye image, performing statistical analysis on the vector set to generate a P value matrix, and performing fusion processing on the P value matrix to obtain the important eye image area associated with the symptom information; The correlation analysis module uses an image-level classifier to analyze the characteristics and areas of disease-related eye images, obtains important reference areas corresponding to the symptom information, inputs a specific eye image into the image-level classifier, obtains an importance matrix in the specific eye image, and fuses the importance matrix with the iris center as the origin to obtain a visualized important reference area, wherein the importance of each pixel in the specific eye image is obtained by calculating the derivative of the image-level classifier for each pixel, and the importance matrix is ​​constructed based on all the importances, and the fusion processing includes horizontally mirroring the importance matrix of each left eye image and then averaging all the importance matrices, or respectively averaging the importance matrices of the left eye image and the right eye image, and then horizontally mirroring the obtained left eye average fusion matrix and fusing it with the obtained right eye average fusion matrix.

5. The information association device according to claim 4, characterized in that: The data preprocessing module comprises: An iris marking unit, used for marking the iris center of each binocular eye image; a foreground extraction unit, configured to perform foreground extraction processing on the binocular eye image according to the iris center, so that only the sclera image and the iris image are retained in the binocular eye image; An image extraction unit, used for performing extraction processing on the binocular eye image after foreground extraction processing to obtain the iris part and the sclera part; The enhancement processing unit is used to perform enhancement processing on the iris part and the sclera part to obtain the eye image.

6. The information association device according to claim 4, characterized in that: The model training module includes: A data preparation unit, used to prepare data based on the training sample set to obtain a training set and a test set; A training execution unit, used for inputting the training sample set into the deep residual neural network for model training to obtain an initial model; A model adjustment unit is used to use the test set to test and adjust the initial model to obtain the multi-view multi-stage diabetes image-level classifier.