Method and system for identifying target areas in video images based on ocular surface staining

Abstract

The present invention relates to an eye video image recognition and processing technology, specifically a method and system for identifying a target area in an ocular surface staining video image, the method comprising: collecting an ocular surface staining video image and ocular physical examination data of a target object, extracting key time points of the ocular surface staining video image; using the ocular physical examination data of the target object as a classification basis, mapping the image data corresponding to the target area, using the extracted key time points as the main feature labels, inputting the training set samples into an initialized neural network model for training, debugging the neural network model parameters to the optimum according to the training results, and obtaining a target area recognition model; inputting the acquired ocular surface staining video image to be recognized into the target area recognition model, and determining the final video features of the target area in the video image to be recognized. The present invention solves the problem of lack of using fluorescein-stained ocular surface video data as a recognition means to determine the user's eye health index data.

Description

Technical Field

[0001] The present invention relates to an eye video image recognition and processing technology, and in particular to a recognition method and system based on a target area in an eye surface dyed video image. Background Art

[0002] Dry eye is one of the most common eye diseases. With the popularization of video terminals, its incidence rate has exceeded 30%. Early screening and diagnosis can help early treatment and effectively improve prognosis. The tear film state of the ocular surface is an important indicator for determining whether dry eye exists; the tear film breakup time count under sodium fluorescein staining is one of the current gold standards for dry eye diagnosis, and the ocular surface video data after sodium fluorescein staining can clearly reflect the dynamic changes in the tear film state and can be used as a target area for identification and comparison. However, the existing technology lacks relevant technical means to determine the eye health indicator data of the user corresponding to the target area based on the comparison results.

[0003] In the prior art, when the subject keeps his eyes open, a Placido ring (concentric rings) is projected on the eye surface. By continuously taking pictures, the time when the projection of the Placido ring on the eyeball changes is obtained by analysis. This time point is the time when the tear film of the subject breaks up. Figure 1 The above prior art has the following defects:

[0004] (1) It is not a technology that can be used to directly perform intelligent screening for dry eyes. It can only obtain the time when the tear film breaks up in the subject, but cannot obtain information such as the morphology of the tear film breakup. Therefore, it cannot be directly used for dry eye diagnosis.

[0005] (2) Because there are gaps between the Placido rings of the observation object of this technology, they cannot completely cover the corneal surface, and they need to deform to a certain extent before they can be detected by the instrument. The tear film breakup time measured by this technology is longer than the gold standard method of clinical observation.

[0006] (3) The clinical application of this technology requires professional instruments, which are expensive and inconvenient to carry, making it difficult to use widely in primary medical institutions. Summary of the invention

[0007] In order to solve the technical problems existing in the prior art, the present invention provides a method and system for identifying target areas in ocular surface stained video images, which uses a pre-trained target area recognition model to locate the relevant target frames in the eye video image to be identified and identify the relevant target areas. After the eye-related target areas are associated with eye health examination data, specific targets and areas in the ocular surface video image can be temporally located and spatially identified, which solves the technical problem in the prior art of lacking the use of fluorescein-stained ocular surface video data as an identification means to determine the eye health indicator data of the corresponding user.

[0008] The recognition method of the present invention is implemented by the following technical solution: a recognition method based on a target area in an ocular surface staining video image, comprising the following steps:

[0009] S1. Collect the target object's ocular surface staining video images and eye examination data, and extract four key time points of the ocular surface staining video images: eye opening time point, tear film rupture time point, blink time point and video end time point;

[0010] S2. After extracting the key time points, the eye examination data of the target object is used as the classification basis to map the image data corresponding to the target area, determine the training set and test set of the target object, use the extracted key time points as the main feature labels, input the training set samples into the initialized neural network model for training, and adjust the neural network model parameters to the optimal value according to the training results to obtain the target area recognition model;

[0011] S3. Input the acquired video image of the ocular surface staining to be identified into the target area recognition model to determine the final video features of the target area in the video image to be identified.

[0012] The recognition system of the present invention is implemented by the following technical scheme: a recognition system based on a target area in an ocular surface staining video image, comprising:

[0013] The key point extraction module is used to collect the target object's ocular surface staining video image and eye examination data, and extract the four key time points of the ocular surface staining video image, namely, the eye opening time point, the tear film rupture time point, the blink time point and the video end time point;

[0014] The target area recognition model construction module uses the eye examination data of the target object as the classification basis, maps the image data corresponding to the target area, determines the training set and test set of the target object, uses the extracted key time points as the main feature labels, inputs the training set samples into the initialized neural network model for training, and adjusts the neural network model parameters to the optimal value according to the training results to obtain the target area recognition model;

[0015] An ocular surface staining video image acquisition module is used to acquire the ocular surface staining video image to be identified and input it into the recognition model application module;

[0016] The recognition model application module inputs the acquired video image of the ocular surface staining to be recognized into the target area recognition model to determine the final video features of the target area in the video image to be recognized.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0018] (1) The present invention uses a pre-trained target area recognition model to locate the relevant target frame number in the eye video image to be recognized, and recognize the relevant target area. After the eye-related target area is associated with the eye health examination data, the specific target and area in the ocular surface video image can be temporally located and spatially recognized, which solves the technical problem in the prior art of lacking the use of fluorescein-stained ocular surface video data as a recognition means to determine the eye health indicator data of the corresponding user. The recognition results of the target area recognition model of the present invention can also be used for dry eye screening referral.

[0019] (2) The clinical data analyzed by the identification method and system of the present invention is the gold standard for detecting tear film breakup time: tear film breakup time detection under sodium fluorescein staining, and there is no measurement error caused by different detection methods.

[0020] (3) The target area recognition model of the present invention only needs to be used in conjunction with a slit lamp or a mobile phone camera. The required instrument is relatively cheap, small in size, and easy to carry. It is more suitable for various screening application scenarios such as primary medical institutions and communities. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic diagram of ocular surface image data used in the prior art;

[0022] Figure 2 The four key time points of the ocular surface staining video image are shown, where (a) is a schematic diagram of the eye opening time point, (b) is a schematic diagram of the tear film breakup time point, (c) is a schematic diagram of the blinking time point, and (d) is a schematic diagram of the video end time point;

[0023] Figure 3 It is a flow chart of the identification method of the present invention. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0025] Example

[0026] This embodiment provides a method for identifying a target area in an ocular surface staining video image, comprising the following steps:

[0027] S1. Collect the target object's ocular surface staining video images and eye examination data, and extract the four key time points of the ocular surface staining video images: eye opening time point, tear film rupture time point, blink time point and video end time point, such as Figure 2 As shown, where:

[0028] 1. “Eye opening time” is defined as the time when the target subject starts to open his eyes continuously after blinking several times;

[0029] 2. The “tear film breakup time point” is defined as the time point when the tear film evenly covering the ocular surface area breaks when the target subject blinks, and the black breakup point first appears in the ocular surface area. The evenly covered tear film is a layer of green water film presented under cobalt blue light after sodium fluorescein staining;

[0030] 3. “Blink time” is defined as the time when the target subject starts blinking for the first time after the eye opening time, and a blink is defined as the position of the upper eyelid margin below the upper edge of the pupil;

[0031] 4. “Video end time” refers to the time 15 seconds from the time the eyes open.

[0032] S2. After extracting the key time points, the eye examination data of the target object is used as the classification basis, the image data corresponding to the target area is mapped, and the training set and test set of the target object are determined, and the extracted key time points are used as the main feature labels. The eye examination data used for target object classification include: Dry Eye Symptom Evaluation (SPEED) scale, Ocular Surface Disease Index (OSDI) scale score, eye redness score, ocular surface sodium fluorescein staining tear film breakup time, corneal injury severity score, meibomian gland opening score and meibomian gland secretion score; the target area includes the ocular surface area. Then the training set samples are input into the initialized ResNet152+LSTM neural network model for training, and the neural network model parameters are adjusted to the optimal according to the training results, so that the sensitivity and specificity of the model in the validation set samples of the target object are optimized, and the target area recognition model is obtained.

[0033] Since the main clinical sign of dry eyes is insufficient tear coverage of the ocular surface, and the main behavioral characteristic of the target object when there is too little tear on the ocular surface is blinking to help the tears re-apply evenly on the ocular surface, dry eye patients generally have the characteristics of a short interval between the tear film breakup time and the eye opening time, and a short interval between the blinking time and the eye opening time. The video end time point is used to control the consistency of the length of the detection video clips of different target objects to avoid the bias caused by the long video shooting time due to the continuous inability to observe the tear film breakup time point of normal target objects (those who do not suffer from dry eyes). Therefore, the key time points in the ocular surface staining video image contain the key image features and information required for prediction, which is the key basis for the target area recognition model to make accurate judgments.

[0034] ResNet152 in the neural network model is a variant of the residual network (ResNet), which mainly consists of 6 parts:

[0035] 1. First, the first convolution layer Conv1 with a convolution kernel size of 7x7 and a stride of 2;

[0036] 2. This is followed by a pooling layer of size 3x3 and stride 2;

[0037] 3. Then there are three second convolutional layers Conv2. The structure of each second convolutional layer Conv2 is: 64 1x1 convolution kernels connected to 64 3x3 convolution kernels, and then connected to 256 1x1 convolution kernels;

[0038] 4. Then there are 8 third convolutional layers Conv3. The structure of each third convolutional layer Conv3 is: 128 1x1 convolution kernels connected to 128 3x3 convolution kernels, and then connected to 512 1x1 convolution kernels;

[0039] 5. Then there are 36 fourth convolutional layers Conv4. The structure of each fourth convolutional layer Conv4 is: 256 1x1 convolution kernels, connected to 256 3x3 convolution kernels, and then connected to 1024 1x1 convolution kernels;

[0040] 6. Finally, there are three fifth convolutional layers Conv5. The structure of each fifth convolutional layer Conv5 is: 512 1x1 convolution kernels, connected to 512 3x3 convolution kernels, and then connected to 2048 1x1 convolution kernels; the final output of the residual network represents the vector features of the input image.

[0041] Long Short Term Memory (LSTM) is a variant of Recurrent Neural Network (RNN). Recurrent Neural Network mainly analyzes continuous data, such as text and video. LSTM mainly consists of two inputs Ct-1 and ht-1, two outputs Ct and ht, and the following three components:

[0042] 1. The Forget Gate determines which information of the input Ct-1 should be forgotten;

[0043] 2. The input gate determines what information is added to the new Ct value;

[0044] 3. The output gate determines which information of the Ct value will be used as output.

[0045] That is to say, the ResNet152 model of the target area recognition model obtained in this step is connected to the LSTM model, wherein the ResNet152 model includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer, and the LSTM model includes a forget gate, an input gate and an output gate.

[0046] S3. Input the acquired video image of the ocular surface staining to be identified into the target area recognition model obtained in step 2; the model downsamples each frame of the identified ocular surface staining video image through the convolutional neural network ResNet152 model to obtain vector features, and then inputs the temporal relationship into the LSTM recurrent neural network to obtain the final video features.

[0047] After the video image of ocular surface staining to be identified is input into the first convolution layer of the target area recognition model, the first feature result is obtained by downsampling extraction; the target area recognition model downsamples each layer of feature maps in the first feature result through the second convolution layer to obtain the second feature result; downsamples each layer of feature maps in the second feature result through the third convolution layer to obtain the third feature result; downsamples each layer of feature maps in the third feature result through the fourth convolution layer to obtain the fourth feature result; downsamples each layer of feature maps in the fourth feature result through the fifth convolution layer to obtain the fifth feature result, that is, the vector feature result of the image; the fifth feature result is input into the LSTM model, and the final video features of the target area in the video image to be identified are determined through the forget gate, input gate and output gate.

[0048] The above-mentioned final video features identified by the target recognition model will be combined with the general information of other examined objects for dry eye diagnosis, and referral recommendations will be directly given based on the diagnosis results, solving the technical problem of the lack of using the ocular surface staining video images of the human eye as a recognition method to determine the user's eye health index data in the existing technology. The eye health index data include: ocular surface disease index scale score, tear film breakup time, corneal epithelial damage staining point count, Schirmer I tear secretion test, meibomian gland infrared imaging photos, etc.

[0049] Based on the same inventive concept, this embodiment also provides a recognition system based on a target area in an ocular surface staining video image, which includes:

[0050] The key point extraction module is used to collect the target object's ocular surface staining video image and eye examination data, and extract the four key time points of the ocular surface staining video image, namely, the eye opening time point, the tear film rupture time point, the blink time point and the video end time point;

[0051] The target area recognition model construction module uses the eye examination data of the target object as the classification basis, maps the image data corresponding to the target area, determines the training set and test set of the target object, uses the extracted key time points as the main feature labels, inputs the training set samples into the initialized ResNet152+LSTM neural network model for training, and adjusts the neural network model parameters to the optimal value according to the training results to obtain the target area recognition model;

[0052] An ocular surface staining video image acquisition module is used to acquire the ocular surface staining video image to be identified and input it into the recognition model application module; it can be a slit lamp or a mobile phone camera;

[0053] The recognition model application module downsamples each frame of the identified ocular surface staining video image through the convolutional neural network ResNet152 model to obtain vector features, and then inputs the temporal relationship into the LSTM recurrent neural network to obtain the final video features of the target area in the video image to be identified.

[0054] The recognition results of the present invention can be directly used for screening and diagnosis of dry eyes, rather than just evaluating a certain clinical manifestation of dry eyes; the ocular surface data used is the same as the data used for clinical diagnosis and is highly interpretable; the diagnosis method required is simple and time-saving, and the instruments used are inexpensive and relatively easy to carry, which is more suitable for application scenarios of dry eye screening in primary medical institutions and communities.

[0055] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for identifying a target area in an ocular surface staining video image, characterized in that: The following steps are involved: S1. Collect the target object's ocular surface staining video images and eye examination data, and extract four key time points of the ocular surface staining video images: eye opening time point, tear film rupture time point, blink time point and video end time point; S2. After extracting the key time points, the eye examination data of the target object is used as the classification basis to map the image data corresponding to the target area, determine the training set and test set of the target object, use the extracted key time points as the main feature labels, input the training set samples into the initialized neural network model for training, and adjust the neural network model parameters to the optimal value according to the training results to obtain the target area recognition model; S3. Input the acquired video image of the ocular surface staining to be identified into the target area recognition model to determine the final video features of the target area in the video image to be identified.

2. The identification method according to claim 1, characterized in that: In step S1, the eye-opening time point is defined as the time point when the target object starts to open its eyes continuously after blinking several times; the tear film breakup time point is defined as the time point when the tear film evenly covering the ocular surface area breaks when the target object blinks, and the black breakup point first appears in the ocular surface area; the blinking time point is defined as the time point when the target object starts to blink for the first time after the eye-opening time point, and the blinking is defined as the position of the upper eyelid edge below the upper edge of the pupil; the video end time point refers to the time point after a period of time from the eye-opening time point.

3. The identification method according to claim 2, characterized in that: The uniformly covered tear film refers to a layer of green water film under cobalt blue light after sodium fluorescein staining and four key time points.

4. The identification method according to claim 1, characterized in that: In step S2, the neural network model includes a connected ResNet152 model and an LSTM model, wherein the ResNet152 model includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer, and the LSTM model includes a forget gate, an input gate and an output gate.

5. The identification method according to claim 4, characterized in that: In step S3, after the video image of the ocular surface staining to be identified is input into the first convolutional layer of the target area recognition model, a first feature result is obtained by downsampling extraction; The target area recognition model downsamples each layer of feature maps in the first feature result through the second convolution layer to obtain a second feature result; downsamples each layer of feature maps in the second feature result through the third convolution layer to obtain a third feature result; downsamples each layer of feature maps in the third feature result through the fourth convolution layer to obtain a fourth feature result; Each layer of feature graph in the fourth feature result is downsampled through the fifth convolutional layer to obtain the fifth feature result, that is, the vector feature result of the image; The fifth feature result is input into the LSTM model, and the final video feature of the target area in the video image to be identified is determined through the forget gate, input gate and output gate.

6. A system for identifying target areas in ocular surface staining video images, characterized in that: include: The key point extraction module is used to collect the target object's ocular surface staining video image and eye examination data, and extract the four key time points of the ocular surface staining video image, namely, the eye opening time point, the tear film rupture time point, the blink time point and the video end time point; The target area recognition model construction module uses the eye examination data of the target object as the classification basis, maps the image data corresponding to the target area, determines the training set and test set of the target object, uses the extracted key time points as the main feature labels, inputs the training set samples into the initialized neural network model for training, and adjusts the neural network model parameters to the optimal value according to the training results to obtain the target area recognition model; An ocular surface staining video image acquisition module is used to acquire the ocular surface staining video image to be identified and input it into the recognition model application module; The recognition model application module inputs the acquired video image of the ocular surface staining to be recognized into the target area recognition model to determine the final video features of the target area in the video image to be recognized.

7. The identification system according to claim 6, characterized in that: The neural network model includes a connected convolutional neural network ResNet152 and an LSTM recurrent neural network; in the recognition model application module, each frame of the identified ocular surface staining video image is downsampled and extracted through the convolutional neural network ResNet152 model to obtain vector features, and then the temporal relationship is input into the LSTM recurrent neural network to obtain the final video features of the target area in the video image to be identified.

8. The identification system according to claim 6, characterized in that: The ocular surface staining video image acquisition module is a slit lamp or a mobile phone camera.

Images