A recognition system for eye appearance images

Abstract

The present invention belongs to the field of graphic recognition technology, and discloses a recognition system for eye appearance images, the system comprising an input module, a preprocessing module and a recognition module; the input module is used to input the eye appearance image into the recognition system; the preprocessing module is used to perform data preprocessing on the input eye appearance image, and adjust the eye appearance image to a preset input specification; the recognition module is used to classify the eye appearance image according to a preset neural network model and give the image classification result and the confidence of the classification result. Beneficial effects: The recognition system analyzes and judges the eye appearance image, and can conveniently collect and detect; the neural network analyzes and judges the collected data, and can extract characteristic laws that are difficult for people to find and observe, thereby improving the accuracy of judgment.

Description

Technical Field

[0001] The present invention relates to the field of pattern recognition technology, and in particular to a recognition system for eye appearance images. Background Art

[0002] At present, thyroid-associated ophthalmopathy (TAO) is an eye disease closely related to autoimmune thyroid disease. Because of its close relationship with Graves' disease, it is also called Graves' ophthalmopathy. TAO is one of the most common orbital diseases in adults. Among patients with Graves' disease, about 30-45% of patients will develop TAO, and its incidence has been gradually increasing in recent years. The clinical manifestations of TAO range from mild to severe, and can be manifested as various eye signs, involvement of ocular soft tissues, proptosis, involvement of extraocular muscles, corneal involvement, and even vision loss.

[0003] The development of TAO is not completely reversible, that is, treatment can only partially alleviate the adverse effects of the disease and cannot completely restore to the state before the onset of the disease. If timely intervention is given in the early stages of the disease and close monitoring is carried out, the progression of the disease can be avoided and the impact of the disease on patients can be reduced; if the best treatment opportunity is missed and intervention is carried out in the late stages of the disease, it will lead to irreversible visual damage, seriously affecting the patient's appearance and affecting the patient's lifelong quality of life. Therefore, the best diagnosis and treatment strategy for TAO is early detection, early diagnosis, early intervention, and continuous observation and control of its progression to avoid the appearance of TAO ocular signs or maintain the course of the disease in an early stage.

[0004] At present, in clinical practice, the diagnosis of TAO can only be made by an ophthalmologist who is a subspecialist in orbital diseases after asking the patient about the medical history and completing relevant examinations. However, this diagnostic method not only requires the patient to pay attention to his or her own abnormalities, but also requires an experienced doctor to conduct relevant tests before making a judgment. The diagnostic efficiency is low, which is not conducive to the universal screening of potential patients, causing many patients to delay the best treatment time.

[0005] In recent years, with the advancement of technology, image recognition technology using neural networks has begun to be widely used in the medical field. The learning and analysis capabilities of neural networks are used to summarize features and patterns that are not easily observed and recognized by humans, helping doctors make diagnoses. The present invention combines TAO and neural network related technologies to propose an eye appearance image recognition system that can recognize eye appearance images of potential TAO patients and provide analysis results. Summary of the invention

[0006] The purpose of the present invention is to provide an eye appearance image recognition system to recognize the eye appearance images of potential TAO patients and provide analysis results.

[0007] In order to achieve the above-mentioned object, the present invention provides a system for recognizing eye appearance images, comprising: an input module, a preprocessing module and a recognition module.

[0008] The input module is used to input the eye appearance image into the recognition system.

[0009] The preprocessing module is used to perform data preprocessing on the input eye appearance image and adjust the eye appearance image to a preset input specification.

[0010] The recognition module is used to classify the eye appearance image according to a preset neural network model and give the image classification result and the confidence of the classification result. The training method of the neural network model includes: obtaining a first eye appearance image of a patient with eyelid retraction, a second eye appearance image of a healthy person, and a third eye appearance image obtained after data enhancement; inputting the first eye appearance image, the second eye appearance image, and the third eye appearance image into the neural network for training, and obtaining a binary classification neural network model based on the eye appearance images of patients with eyelid retraction and healthy people.

[0011] Furthermore, the data preprocessing of the input eye appearance image is specifically as follows:

[0012] Detect the eye region of the eye appearance image and segment the ROI image of the eye region.

[0013] Furthermore, the method of detecting the eye region of the eye appearance image and segmenting the ROI image of the eye region specifically includes:

[0014] Acquire a fourth eye appearance image and mark an eye region ROI for a portion of the fourth eye appearance image through a rectangular frame, and mark orbital key points in the rectangular frame; the fourth eye appearance image includes an eye appearance image of a healthy person and an eye appearance image of a patient with eyelid retraction, and the orbital key points include: inner canthus, upper eyelid, lower eyelid, and outer canthus;

[0015] The labeled image is input into the eye region ROI segmentation neural network for training to obtain the eye region ROI segmentation model.

[0016] Furthermore, the identification system further comprises: a patient information reading module;

[0017] The patient information reading module is used to read the medical condition information provided by the patient and determine whether the medical condition information includes proptosis, strabismus, diplopia, and decreased vision. If so, the medical condition information, image classification results, and confidence of the classification results are fed back to the doctor.

[0018] Furthermore, the recognition system also includes a cloud platform, which is used to receive uploaded eye appearance images and store the eye appearance images in the cloud. At the same time, the cloud platform is also used to send the eye appearance images to the image input module, receive the recognition result and the confidence of the recognition result of the recognition module, and feed back the recognition result and the confidence of the recognition result to the uploader.

[0019] Furthermore, the preset input specification is a three-channel RGB image.

[0020] Furthermore, the preset neural network model is Resnet101, which is composed of a Bottleneck structural unit module consisting of 1x1, 3x3, and 1x1 convolutional layers, and the convolutional layers are connected by Batchnorm and Relu activation layers; the Bottleneck structure includes a residual block, and the residual block is composed of 1x1 convolution.

[0021] Furthermore, the stride of the residual block is 2 and the padding is 1 when downsampling; the stride of the residual block other than downsampling is 1 and the padding is 1.

[0022] Furthermore, the image classification is performed by a preset neural network, specifically:

[0023] First, downsampling is performed through 64 convolutional layers with a kernel size of 7 and a stride of 2 to extract a 256x256x64 feature matrix. Then, the feature maps of each layer in the feature matrix are further downsampled to 128x128, 64x64, 32x32, and 16x16 through the four layers of ResNet. Finally, the fully connected layer is connected to output the classification result and the corresponding confidence.

[0024] Furthermore, the first layer first downsamples features through a maxpool operation with a stride of 3x3 and a stride of 2 to obtain a feature map of 128x128x64, and then uses BottleNeck repeated 3 times; the first Bottleneck in the second layer uses a residual block in a downsampling manner, and the 1x1, 3x3, and 1x1 convolutional layers of Bottleneck transform the feature map to 128x128x128, 64x64x128, and 64x64x512, respectively. The output of Bottleneck is added to the output of the residual block to obtain the first Bottleneck output of the second layer, and the feature size is 64x64x512, and then BottleNeck without downsampling is repeated 3 times to obtain the output 64x64x512 feature map of the second layer; the structure of the third layer is the same as that of the second layer, except that the number of convolution kernels for 3x3 downsampling is 256, and the residual The number of downsampling convolution kernels in the block is 1024; the structure of the fourth layer is the same as the second layer, the difference is that the number of 3x3 downsampling convolution kernels is 512, and the number of downsampling convolution kernels in the residual block is 2048.

[0025] Compared with the prior art, the advantageous effects of the eye appearance image recognition system of the embodiment of the present invention are as follows: the recognition system analyzes and judges through the eye appearance image, and the source of the eye appearance image is wide, and it can be conveniently collected for detection; the collected data is analyzed and judged by the neural network, and characteristic laws that are difficult for humans to find and observe can be extracted, thereby improving the accuracy of the judgment; the integrated system can be conveniently promoted and applied, and convenient for screening of a large number of potential patients, improving the diagnostic ability of doctors, and enabling patients to be diagnosed and treated early to avoid worsening of the disease. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a first structural schematic diagram of a system for recognizing eye appearance images according to the present invention;

[0027] Figure 2 It is a second structural schematic diagram of a system for recognizing eye appearance images according to the present invention. DETAILED DESCRIPTION

[0028] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0029] Embodiment 1:

[0030] Refer to the attached Figure 1This embodiment discloses a recognition system for eye appearance images, which is applied to recognition and judgment of eye appearance images, and includes: an input module 101, a preprocessing module 102 and a recognition module 103.

[0031] The input module 101 is used to input the eye appearance image into the recognition system.

[0032] The preprocessing module 102 is used to perform data preprocessing on the input eye appearance image and adjust the eye appearance image to a preset input specification.

[0033] The recognition module 103 is used to classify the eye appearance image according to a preset neural network model and give the image classification result and the confidence of the classification result. The training method of the neural network model includes: obtaining a first eye appearance image of a patient with eyelid retraction, a second eye appearance image of a healthy person, and a third eye appearance image obtained after data enhancement; inputting the first eye appearance image, the second eye appearance image, and the third eye appearance image into the neural network for training, and obtaining a binary classification neural network model based on the eye appearance images of patients with eyelid retraction and healthy people.

[0034] In this embodiment, data enhancement is applied before training to randomly flip and scale the input images, which can increase the number of samples and improve the generalization ability of the model.

[0035] In this embodiment, the present invention uses artificial intelligence technology based on a deep learning algorithm, and utilizes clinical data such as a large number of ocular appearance images, clinical diagnosis results, etc. that are collected in advance to establish an ocular appearance image recognition system, which can determine whether the ocular appearance image is an eyelid retraction image, so that the doctor can further make a diagnosis with reference to the judgment result, thereby improving the doctor's diagnostic ability.

[0036] In this embodiment, the recognition system analyzes and judges through the eye appearance image, which has a wide range of sources and can be easily collected for detection. In the hospital, a high-pixel camera can be equipped for shooting. After shooting, the photo can be directly input into the recognition system to obtain the recognition result, which is convenient and efficient.

[0037] In this embodiment, the data preprocessing of the input eye appearance image is specifically as follows:

[0038] Detect the eye region of the eye appearance image and segment the ROI image of the eye region.

[0039] In this embodiment, automatic image extraction can be achieved by detecting the eyelid region and segmenting the ROI image of the eye region, so that the features to be extracted in the input image are retained and unnecessary data input is reduced.

[0040] In this embodiment, the method of detecting the eye region of the eye appearance image and segmenting the ROI image of the eye region includes:

[0041] A fourth eye appearance image is obtained, and an eye area ROI is marked on a portion of the fourth eye appearance image through a rectangular frame, and orbital key points are marked in the rectangular frame; the fourth eye appearance image includes an eye appearance image of a healthy person and an eye appearance image of a patient with eyelid retraction, and the orbital key points include: inner canthus, upper eyelid, lower eyelid, and outer canthus.

[0042] The labeled image is input into the eye region ROI segmentation neural network for training to obtain the eye region ROI segmentation model.

[0043] In this embodiment, the ROI segmentation model of the eye region is established to realize automatic extraction of the eye ROI region, reduce human labor, and realize automation. At the same time, in order to ensure the quality of the extracted image, when training the model, the input image is annotated with key points of the eye socket, including features such as the inner canthus, upper eyelid, lower eyelid, and outer canthus, so that the obtained neural network model can better realize the segmentation of the input image.

[0044] In this embodiment, the identification system further includes: a patient information reading module;

[0045] The patient information reading module is used to read the medical condition information provided by the patient and determine whether the medical condition information includes proptosis, strabismus, diplopia, and decreased vision. If so, the medical condition information, image classification results, and confidence of the classification results are fed back to the doctor.

[0046] In this embodiment, since patients with thyroid disease may not have eyelid retraction symptoms, the neural network cannot distinguish them from healthy people. In order to better screen, a patient information reading module is set up to read the patient's medical records and other information, and determine whether the condition information includes specific features. If so, the information is fed back to the doctor for judgment to avoid omissions.

[0047] In this embodiment, the recognition system also includes a cloud platform, which is used to receive uploaded eye appearance images and store the eye appearance images in the cloud. At the same time, the cloud platform is also used to send the eye appearance images to the image input module, receive the recognition results of the recognition module and the confidence of the recognition results, and feed back the recognition results and the confidence of the recognition results to the uploader.

[0048] In this embodiment, in order to better collect and judge data, a cloud platform is set up, and ordinary people can also upload their own eye appearance images to the cloud platform. The cloud platform will save the image, send the eye appearance image to the recognition system for recognition, and feedback information to the uploader. This not only expands the data source, but also facilitates ordinary people to monitor their own health.

[0049] In this embodiment, the preset input specification is a three-channel RGB image.

[0050] In this embodiment, the preset neural network model is Resnet101 (101 residual network), and the model is composed of a Bottleneck (bottleneck-like structure) structural unit module composed of 1x1, 3x3, and 1x1 convolutional layers, and the convolutional layers are connected by Batchnorm (normalization operation) and Relu (truncation function) activation layers; the Bottleneck structure includes a residualblock (the residual block of the Resnet101 residual network described above), and the residual block is composed of a 1x1 convolution.

[0051] In this embodiment, before training, the model parameters pre-trained on the image prediction competition ImageNet (the name of the competition) can be used, and then the learned knowledge can be "migrated" to the neural network in the present invention to obtain faster convergence speed and better performance.

[0052] In this embodiment, the stride (the number of grids that the convolution kernel needs to jump when scanning the original image) of the residual block is 2 when downsampling, and the padding (to keep the feature image not too small and increase the number of pixels on each side) is 1; the stride of the residual block other than downsampling is 1, and the padding is 1. By setting the stride and padding, the size of the input feature map can be kept unchanged, and only a certain linear transformation and the number of channels of the expanded feature map can be completed.

[0053] In this embodiment, the image classification is performed by a preset neural network, specifically:

[0054] First, 64 convolution layers with a kernel size of 7 and a stride of 2 are used to downsample the feature matrix of 256x256x64. Then, the feature maps of each layer in the feature matrix are further downsampled to 128x128, 64x64, 32x32, and 16x16 through the four layers of ResNet. Finally, the fully connected layer is used to output the classification results and the corresponding confidence.

[0055] In this embodiment, by analyzing and judging the collected data through a neural network, characteristic patterns that are difficult for humans to discover and observe can be extracted, thereby improving the accuracy of judgment; the integrated system can be easily promoted and applied, facilitating the screening of a large number of potential patients, improving the diagnostic ability of doctors, and enabling patients to be diagnosed and treated early to avoid worsening of the disease.

[0056] In this embodiment, the first layer first downsamples features through a maxpool operation with a stride of 3x3 and 2 to obtain a feature map of 128x128x64, and then uses BottleNeck repeated 3 times; the first 1x1 convolution and 3x3 convolution of BottleNeck both use 64 convolution kernels, and the last 1x1 convolution uses 256 convolution kernels. The residual block uses a 1x1 convolution that retains the input feature size, that is, 256 1x1 convolution kernels with a stride of 1 and a padding of 1. After the convolution, they are added to the output of the last Bottleneck, and the output features obtained are (in order) 128x128x64, 128x128x64, and 128x128x256.

[0057] The first Bottleneck in the second layer uses a downsampling residual block (512 1x1 convolution kernels with a step size of 2). The 1x1, 3x3, and 1x1 convolution layers of Bottleneck transform the feature map to 128x128x128, 64x64x128, and 64x64x512, respectively. The output of Bottleneck is added to the output of the residual block to obtain the first Bottleneck output of the second layer with a feature size of 64x64x512. Then, the BottleNeck without downsampling is repeated 3 times to obtain the output 64x64x512 feature map of the second layer.

[0058] The structure of the third layer is the same as that of the second layer, except that the number of convolution kernels for 3x3 downsampling is 256, and the number of convolution kernels for downsampling in the residual block is 1024; the 1x1, 3x3, and 1x1 convolution layers of the first Bottleneck of the third layer transform the feature maps to (in order) 64x64x256, 32x32x256, and 32x32x1024, respectively, followed by 23 Bottleneck layers without downsampling operations, and the final output is 32x32x1024.

[0059] The structure of the fourth layer is the same as that of the second layer, except that the number of convolution kernels for 3x3 downsampling is 512, and the number of convolution kernels for downsampling in the residual block is 2048. The first 1x1, 3x3, and 1x1 convolution layers of the fourth layer's first Bottleneck transform the feature map to (in order) 32x32x512, 16x16x512, and 16x16x2048, respectively, followed by two Bottleneck layers without downsampling operations, and the final output is 16x16x2048. Finally, the feature matrix is ​​pooled and the final classification output of 2x1 is output through the fully connected layer to obtain the classification confidence of the recognition result of the image.

[0060] In summary, the embodiment of the present invention provides an eye appearance image recognition system, which has the following beneficial effects compared with the prior art: the recognition system analyzes and judges through eye appearance images, and the sources of eye appearance images are wide, and they can be easily collected for detection; the collected data is analyzed and judged by a neural network, and characteristic patterns that are difficult for humans to discover and observe can be extracted, thereby improving the accuracy of judgment; the integrated system can be easily promoted and applied, and a large number of potential patients can be screened, thereby improving the diagnostic ability of doctors, enabling patients to be diagnosed and treated early, and avoiding worsening of the disease.

[0061] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A system for recognizing eye appearance images, characterized in that: include: Input module, preprocessing module and recognition module; The input module is used to input the eye appearance image into the recognition system; The preprocessing module is used to perform data preprocessing on the input eye appearance image and adjust the eye appearance image to a preset input specification; The recognition module is used to classify the eye appearance image according to a preset neural network model and give the image classification result and the confidence of the classification result. The training method of the neural network model includes: obtaining a first eye appearance image of a patient with eyelid retraction, a second eye appearance image of a healthy person, and a third eye appearance image obtained through data enhancement; inputting the first eye appearance image, the second eye appearance image, and the third eye appearance image into the neural network for training, and obtaining a binary classification neural network model based on the eye appearance images of the patient with eyelid retraction and the healthy person; The data preprocessing of the input eye appearance image is specifically as follows: Detect the eye region of the eye appearance image and segment the ROI image of the eye region; The method of detecting the eye region of the eye appearance image and segmenting the ROI image of the eye region specifically includes: Acquire a fourth eye appearance image and mark an eye region ROI for a portion of the fourth eye appearance image through a rectangular frame, and mark orbital key points in the rectangular frame; the fourth eye appearance image includes an eye appearance image of a healthy person and an eye appearance image of a patient with eyelid retraction, and the orbital key points include: inner canthus, upper eyelid, lower eyelid, and outer canthus; The labeled image is input into the eye region ROI segmentation neural network for training to obtain the eye region ROI segmentation model; The preset neural network model is Resnet101, which is composed of a Bottleneck structural unit module consisting of 1x1, 3x3, and 1x1 convolutional layers, and the convolutional layers are connected by Batchnorm and Relu activation layers; the Bottleneck structure includes a residual block, and the residual block is composed of 1x1 convolution.

2. The eye appearance image recognition system according to claim 1, characterized in that: The identification system further includes: a patient information reading module; The patient information reading module is used to read the medical condition information provided by the patient and determine whether the medical condition information includes proptosis, strabismus, diplopia, and decreased vision. If so, the medical condition information, image classification results, and confidence of the classification results are fed back to the doctor.

3. The eye appearance image recognition system according to claim 1, characterized in that: The recognition system also includes a cloud platform, which is used to receive uploaded eye appearance images and store the eye appearance images in the cloud. At the same time, the cloud platform is also used to send the eye appearance images to the image input module, receive the recognition results and confidence levels of the recognition results of the recognition module, and feed back the recognition results and confidence levels of the recognition results to the uploader.

4. The system for recognizing eye appearance images according to claim 1, characterized in that: The preset input specification is a three-channel RGB image.

5. The eye appearance image recognition system according to claim 1, characterized in that: When the residual block is downsampled, the stride is 2 and the padding is 1; except for the downsampling, the stride of the residual block is 1 and the padding is 1.

6. The eye appearance image recognition system according to claim 1, characterized in that: The image classification is performed by using a preset neural network, specifically: First, 64 convolution layers with a kernel size of 7 and a stride of 2 are used to downsample the feature matrix of 256x256x64. Then, the feature maps of each layer in the feature matrix are further downsampled to 128x128, 64x64, 32x32, and 16x16 through the four layers of ResNet. Finally, the fully connected layer is connected to output the classification results and the corresponding confidence.

7. The system for recognizing eye appearance images according to claim 6, characterized in that: The first layer first downsamples the features through the maxpool operation of 3x3 and stride 2 to obtain a feature map of 128x128x64, and then uses BottleNeck repeated 3 times; the first Bottleneck in the second layer uses a residual block with downsampling method, and the 1x1, 3x3, and 1x1 convolution layers of Bottleneck transform the feature map to 128x128x128, 64x64x128, and 64x64x512 respectively. The output of Bottleneck is added to the output of the residual block to obtain the first Bottleneck output of the second layer, with a feature size of 64x64x512, and then BottleNeck without downsampling is repeated 3 times to obtain the output 64x64x512 feature map of the second layer; the structure of the third layer is the same as the second layer, except that the number of convolution kernels for 3x3 downsampling is 256, and the residual The number of downsampling convolution kernels in the block is 1024; the structure of the fourth layer is the same as the second layer, the difference is that the number of 3x3 downsampling convolution kernels is 512, and the number of downsampling convolution kernels in the residual block is 2048.

Images