Automatic detection system and method for eye lens and iris tremor

Abstract

The invention discloses an automatic detection system and method for eye lens and iris tremor, and relates to the technical field of image processing and pattern recognition, wherein the system comprises: an acquisition module, a segmentation module, a calculation module, a first discrimination module, a second discrimination module and a third discrimination module; the acquisition module is used for acquiring dynamic video of the eyeball of the subject; the segmentation module segments the lens and cornea areas in adjacent video frames; the calculation module is used for calculating the similarity index of the lens and cornea contours in adjacent video frames; the first discrimination module is used for constructing a lens displacement discrimination model to judge whether the lens is displaced relative to the iris in adjacent video frames; the second discrimination module is used for calculating the motion vector size of the lens contour centroid relative to the cornea contour centroid in adjacent video frames, and judging whether the lens is displaced relative to the cornea according to the motion vector size; the third discrimination module is used for calculating and judging whether the entire dynamic video has iris tremor or lens tremor.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing and pattern recognition, and in particular to an automatic detection system and method for eye lens and iris tremor. Background Art

[0002] Ocular lens tremor is an important reference for understanding ocular anatomy and guiding surgery, and is usually related to the relaxation of the lens suspensory ligaments. The lens may tremor with aging, eye injury, and other conditions. Iris tremor is manifested as involuntary tiny vibrations of the iris when the eye moves or is still. It may be caused by a variety of factors, including nerve damage, eye trauma, complications after eye surgery, or side effects of certain medications. In most cases, lens tremor and iris tremor themselves may not cause obvious symptoms and are usually only discovered through specialized eye examinations.

[0003] Currently, the detection of ocular lens tremor and iris tremor mainly relies on the doctor's experience and subjective judgment, which is not only time-consuming and labor-intensive, but may also lead to inaccurate judgment due to the doctor's fatigue or experience differences. With the development of computer vision and artificial intelligence technology, automated image analysis methods have made it possible to improve the accuracy and efficiency of lens tremor and iris tremor detection. Therefore, the development of a method and system that can accurately and quickly automatically detect ocular lens tremor and iris tremor from video data has important clinical significance and application value. Summary of the invention

[0004] In order to solve the technical problems in the above background, the present invention aims to provide an automatic detection system and method for eye lens and iris tremor. The present invention can automatically determine whether the eye has lens tremor and iris tremor by reading the dynamic video of the eyeball. The eye lens and iris are connected by suspensory ligaments. If the suspensory ligaments are loose, the lens will be displaced relative to the iris, resulting in lens tremor, which is manifested in the image as a change in the visible features of the lens area. If the suspensory ligaments are connected normally, the lens will not be displaced relative to the iris, and the two are fixed as a whole for movement. In this case, it is necessary to further determine whether the whole is displaced relative to the outer cornea. If it is displaced, it means that the eye has iris tremor and lens tremor at the same time, which is manifested in the image as the center of mass of the lens is displaced relative to the center of mass of the cornea.

[0005] To achieve the above-mentioned object, the present invention provides an automatic detection system for eye lens and iris tremor, comprising: an acquisition module, a segmentation module, a calculation module, a first discrimination module, a second discrimination module and a third discrimination module;

[0006] The acquisition module is used to acquire dynamic videos of the subject's eyeballs rotating upward, downward, nasal, and temporal, respectively, wherein the dynamic videos include multiple video frames;

[0007] The segmentation module is used to construct an automatic segmentation model, and use the automatic segmentation model to segment the lens and cornea areas in adjacent video frames. The structure of the automatic segmentation model includes: based on the Pytorch Detection2 framework, using the Mask-RCNN instance segmentation model, using the pyramid feature extraction network FPN as the backbone architecture, and using the residual network ResNeXt-101 as the convolution structure;

[0008] The calculation module is used to calculate the similarity index of the lens and cornea contours in adjacent video frames;

[0009] The first discrimination module is used to construct a lens displacement discrimination model to discriminate whether the lens is displaced relative to the iris in adjacent video frames, wherein the lens displacement discrimination model includes a Resnet50 backbone network, a fully connected layer and a sigmoid function, and uses binary cross entropy as a loss function;

[0010] The second determination module is used to calculate the size of the motion vector of the centroid of the lens contour relative to the centroid of the cornea contour in adjacent video frames, and determine whether the lens is displaced relative to the cornea according to the size of the motion vector;

[0011] The third determination module is used to calculate and determine whether iris tremor or lens tremor exists in the entire dynamic video.

[0012] Preferably, the workflow of the segmentation module includes: using the automatic segmentation model to segment the lens area and the cornea area on the video frame, and obtaining the lens contour map in two adjacent frames of images.

[0013] Preferably, the workflow of the calculation module includes: obtaining the similarity of two lens contour objects in two adjacent frames of images using a similarity index calculation formula:

[0014] ,

[0015] in, represents the similarity index; , ; Indicates the sign of the input value to be returned. If the input value is greater than 0, 1 is returned; if the input value is equal to 0, 0 is returned; if the input value is less than 0, -1 is returned. and Represents the i-th Hu moment of contour object A and object B respectively;

[0016] The same process is used to calculate the similarity of corneal contours in adjacent frames.

[0017] Preferably, when the lens contour similarity index of two adjacent frames of images is less than 0.1, further determining the difference in the lens global image features of the pair of adjacent frames;

[0018] When the corneal contour similarity index of two adjacent frames is less than 0.1, the magnitude of the motion vector of the centroid of the lens contour relative to the centroid of the corneal contour of the adjacent frames is further calculated.

[0019] Preferably, the workflow of the first identification module includes:

[0020] Preprocessing the lens contour images in two adjacent frames of images to obtain processed data;

[0021] Performing image annotation on the processed data to obtain a training sample set;

[0022] Constructing a lens displacement discrimination model, and using the training sample set to train the constructed lens displacement discrimination model;

[0023] The trained lens displacement discrimination model is used to discriminate whether the lens is displaced relative to the iris.

[0024] Preferably, the process of calculating the size of the motion vector by the second determination module includes:

[0025] ,

[0026] ,

[0027] ,

[0028] Among them, a and b represent two adjacent frames of images; Represents the coordinates of the lens centroid of frame a; Represents the coordinates of the corneal centroid of frame a; represents the coordinates of the centroid of the lens in the b-frame image; It is represented by the coordinates of the corneal centroid of frame b; dist(a) and dist(b) represent the Euclidean distances between the coordinates of the lens centroid and the corneal centroid of the two frames respectively; dist(ab) represents the motion vector size of the lens centroid relative to the corneal contour centroid.

[0029] The present invention also provides an automatic detection method for eye lens and iris tremor, which is applied to the above system and comprises the following steps:

[0030] Collecting dynamic videos of the subject's eyeballs moving upward, downward, nasal, and temporal, respectively, wherein the dynamic videos include multiple video frames;

[0031] An automatic segmentation model is constructed, and the automatic segmentation model is used to segment the lens and cornea regions in adjacent video frames. The structure of the automatic segmentation model includes: based on the Pytorch Detection2 framework, using the Mask-RCNN instance segmentation model, using the pyramid feature extraction network FPN as the backbone architecture, and using the residual network ResNeXt-101 as the convolution structure;

[0032] Calculate the similarity index of the lens and cornea contours in adjacent video frames;

[0033] A lens displacement discrimination model is constructed to discriminate whether the lens is displaced relative to the iris in adjacent video frames. The discrimination model includes a Resnet50 backbone network, a fully connected layer, and a sigmoid function, and uses binary cross entropy as a loss function;

[0034] Calculating the magnitude of the motion vector of the center of mass of the lens contour relative to the center of mass of the cornea contour in adjacent video frames, and judging whether the lens is displaced relative to the cornea according to the magnitude of the motion vector;

[0035] Calculate and determine whether iris tremor or lens tremor exists in the entire dynamic video.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention is based on the dynamic video of the slit lamp anterior segment system imaging technology, which can automatically and quickly detect whether the eye in the video has lens tremor and iris tremor, and has important clinical significance and application value for judging the state of the eye structure. The present invention also has the advantages of non-contact and non-destructive. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1 A schematic diagram of the system structure of an embodiment of the present invention;

[0040] Figure 2 is a system working flow diagram of an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a segmentation model according to an embodiment of the present invention;

[0042] Figure 4This is a structural diagram of a twin convolutional neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] Embodiment 1

[0046] like Figure 1 As shown in FIG. 1 , the system structure diagram of the present invention includes: a collection module, a segmentation module, a calculation module, a first discrimination module, a second discrimination module and a third discrimination module. Figure 2 , and detailed description of the working process of the system of the present invention and how to solve technical problems in actual work.

[0047] First, a collection module is used to collect dynamic videos of the subject's eyeballs rotating upward, downward, nasally, and temporally, wherein the dynamic video includes multiple video frames. Specifically, in this embodiment, the collection module is a slit lamp anterior segment imaging device. In this embodiment, a total of 200 dynamic videos are collected, of which 100 videos have eye lenses and (or) iris tremors, and 100 videos do not have eye lenses and iris tremors. The above video frame refers to a frame of two-dimensional image in the dynamic video.

[0048] After that, the segmentation module is used to build an automatic segmentation model, and the automatic segmentation model is used to segment the lens and cornea areas in adjacent video frames. The implementation process includes data annotation, image preprocessing, algorithm design, training process and prediction process. The specific process is as follows:

[0049] Data annotation: The dynamic video was split into several two-dimensional video frames. One frame was randomly selected from each video, for a total of 200 frames. The labelme software was used to annotate the lens area and cornea area of ​​each frame.

[0050] Image preprocessing: In order to adapt to the input of the segmentation model, the resolution of these 200 frames of two-dimensional images is adjusted to 1024*1024.

[0051] Automatic segmentation model design: Based on the Pytorch Detection2 framework, the Mask-RCNN instance segmentation model is adopted, the pyramid feature extraction network FPN is used as the backbone architecture, and the residual network ResNeXt-101 is used as the convolution structure. Among them, ResNeXt-101 has a 101-layer network structure, and its core component Residual Block is a bottleneck module (Bottleneck) structure. Each bottleneck module consists of three convolution layers of 1x1, 3x3, and 1x1. In the 3x3 convolution layer, 32 groups of group convolutions are used. FPN is based on the feature maps output by ResNeXt-101 at different stages, and applies a 3x3 convolution kernel to the output feature map of each layer to construct a multi-scale feature map pyramid. The pyramid feature map generates candidate regions through the region proposal network (RPN), and the candidate regions generated by RPN and the feature maps of FPN are spatially aligned through the ROI Align layer. Finally, the model outputs three branches through a fully convolutional network and a fully connected layer: a classification branch, a bounding box regression branch, and a mask branch. The loss function of the classification branch is the cross-entropy loss, the loss function of the bounding box regression branch is the smooth L1 loss, and the loss function of the mask branch is the cross-entropy loss of pixel-by-pixel binary classification. The structural diagram of the model design is shown in the figure. Figure 3 shown.

[0052] Training process: First load the COCO pre-trained model, and then perform full-layer fine-tuning training on the dataset with labeled lens and cornea regions. The initial learning rate is 0.02, and the learning rate is decayed to 1 / 10 of the original every 10 rounds; the optimizer is stochastic gradient descent (SGD), the momentum is 0.9, and the weight decay is 0.0001; the training process uses data enhancement methods such as random horizontal flipping (50% probability), scaling (0.8 to 1.2 times), and color jitter. A total of two segmentation models were trained, one for lens region segmentation and the other for cornea region segmentation.

[0053] Prediction process: In the prediction stage, if the confidence level output by the Mask R-CNN model is greater than or equal to 0.75, it means that the target area (lens area or cornea area) has been detected. Then, the contour data of the target area is obtained through findContours of OpenCV. The contour data is a set of continuous coordinate points on the boundary of the target area.

[0054] The calculation module is used to calculate the similarity index of the lens and corneal contours in adjacent video frames. The process includes: first, the automatic segmentation model is used to obtain the lens contour image data on two adjacent frames of images, and then the similarity of the two lens contour objects in the two adjacent frames of images is obtained using the similarity index calculation formula:

[0055] ,

[0056] in, represents the similarity index; , ; Indicates the sign of the input value to be returned. If the input value is greater than 0, 1 is returned; if the input value is equal to 0, 0 is returned; if the input value is less than 0, -1 is returned. and Represents the i-th Hu moment of contour object A and object B respectively; Hu moment is composed of these 7 invariant moments to form a set of feature quantities, which are invariant to rotation, scaling and translation. In actual operation, the matchshape function of opencv2 can be directly used to calculate the similarity index to determine the similarity of the two contour data. The method parameter of the matchshape function is set to CV_CONTOURS_MATCH_I1. The function returns a similarity index, which is a value between 0 and 1. The smaller the index, the better the match of the two contours, that is, the more similar the shape of the peripheral edge of the lens is. The same method is used to calculate the similarity of the corneal contours of adjacent frames.

[0057] The first discrimination module is used to construct a lens displacement discrimination model to determine whether the lens is displaced relative to the iris in adjacent video frames. First, it is necessary to determine whether the lens is displaced relative to the iris in adjacent video frames. The basis for determining whether the lens is displaced relative to the iris in adjacent frames is as follows: 1) The lens contours of adjacent frames are basically consistent; 2) There are large differences in the global image features of the lens in adjacent frames.

[0058] In this embodiment, only when the lens contour similarity index of two adjacent frames is less than 0.1, the difference in the global image features of the lens of the adjacent frames is further determined. This system uses a twin convolutional neural network to build and train a lens displacement discrimination model to determine the difference in the global image features of the lens of adjacent frames. This is a binary classification task, and the process includes image preprocessing, image annotation, model construction, and model prediction. The specific process includes:

[0059] Image preprocessing: Using the automatic segmentation model, the lens area of ​​all video frames is segmented to obtain the lens segmentation map, and then the resolution of the segmentation map is further adjusted to 224*224 to adapt to the input of the twin convolutional neural network.

[0060] Image annotation: From each dynamic video with lens tremor, 10 pairs of lens segmentation maps of adjacent frames are randomly selected. They satisfy the contour similarity index less than 0.1 and the global image feature difference is large (that is, the lens has shifted relative to the iris). Each pair of images is annotated as 1. From each dynamic video without lens tremor, 10 pairs of lens segmentation maps of adjacent frames are randomly selected. They satisfy the contour similarity index less than 0.1 and the global image feature difference is small (that is, the lens has not shifted relative to the iris). Each pair of images is annotated as 0. A total of 2000 pairs of lens segmentation map samples of adjacent frames, of which 1000 pairs are positive samples annotated as 1 and 1000 pairs are negative samples annotated as 0.

[0061] Construction of lens displacement discrimination model: The twin convolutional neural network, as a convolutional neural network with a special structure, has been proven to perform well in similarity measurement tasks by processing two input images simultaneously and comparing their similarities. This embodiment uses the Resnet50 backbone network to extract image features, and inputs two adjacent images in the training set into the backbone network respectively to obtain two feature vectors: Gw(X1) and Gw(X2). The two feature vectors are then subtracted to obtain a new vector, and then the new vector obtained by subtraction is input into a fully connected layer (FC) to obtain a scalar, and finally the sigmoid function is used to map the result to a labeled value of 0 or 1. Binary cross entropy is used as the loss function. The structure of the twin convolutional network is as follows Figure 4 shown.

[0062] Model prediction: During prediction, two lens segmentation images of adjacent frames are input. If the output result is 1, it means that the lens has shifted relative to the iris; otherwise, if the output result is 0, it means that the lens has not shifted relative to the iris.

[0063] The second discrimination module is used to calculate the motion vector size of the centroid of the lens contour relative to the centroid of the corneal contour in adjacent video frames, and to judge whether the lens is displaced relative to the cornea based on the motion vector size. The specific process includes: if the lens contours of adjacent frames are basically the same, but there is no significant difference in the global image features of the lens (the suspensory ligament is connected normally), it is necessary to further judge whether the iris and lens are displaced relative to the cornea in adjacent video frames, so as to determine whether the pair of adjacent frames has iris tremor and lens tremor at the same time. Since the lens and iris move as a whole in this case, if the lens is displaced relative to the cornea, the iris will also be displaced relative to the cornea, so it is only necessary to judge whether the centroid of the lens is displaced relative to the centroid of the cornea in terms of image features. The judgment basis for the displacement of the centroid of the lens relative to the centroid of the cornea in adjacent frames is as follows: 1) the corneal contours of adjacent frames are basically the same; 2) the motion vector size of the centroid of the lens contour relative to the centroid of the corneal contour in adjacent frames exceeds the preset threshold.

[0064] Only when the corneal contour similarity index of two adjacent frames is less than 0.1, the motion vector size of the lens contour center of mass relative to the corneal contour center of mass of the pair of adjacent frames is further calculated. For two adjacent frames of images, the lens contour image data and corneal contour image data obtained by the automatic segmentation model are first used to calculate the lens center of mass coordinates and corneal center of mass coordinates of one frame of the image, and their Euclidean distance is calculated through these two center of mass coordinates; then the Euclidean distance between the lens center of mass coordinates and the corneal center of mass coordinates of the other frame of the image is calculated using the same method. The absolute value of the difference between the two Euclidean distances is calculated, which is the motion vector size of the lens contour center of mass relative to the corneal contour center of mass. If the value is greater than the preset threshold (for example, 10 pixels), it means that the lens center of mass of the adjacent frames has shifted relative to the corneal center of mass, thereby judging that the two frames have iris tremor and lens tremor at the same time. The calculation formula of the motion vector size is as follows:

[0065] ,

[0066] ,

[0067] ,

[0068] Among them, a and b represent two adjacent frames of images; Represents the coordinates of the lens center of mass of frame a; Represents the coordinates of the corneal centroid of frame a; represents the coordinates of the centroid of the lens in the b-frame image; It is represented by the coordinates of the corneal centroid of frame b; dist(a) and dist(b) represent the Euclidean distances between the coordinates of the lens centroid and the corneal centroid of the two frames respectively; dist(ab) represents the motion vector size of the lens centroid relative to the corneal contour centroid.

[0069] The centroid coordinates of the contour can be obtained through the moments function of opencv. Assuming that the contour data object is contour, first calculate the contour moment Mu = cv2.moments(contour), and then obtain the centroid coordinates (Px, Py) of the contour through the following Python programming statement:

[0070] Px = int(Mu['m10'] / Mu['m00']),

[0071] Py = int(Mu['m01'] / Mu['m00']).

[0072] Finally, the third discrimination module is used to calculate and judge whether there is iris tremor or lens tremor in the entire dynamic video. The specific process includes:

[0073] The image of the lens area is extracted using the automatic segmentation model, and then the lens contour similarity index of each pair of adjacent frames is calculated. The lens segmentation images of adjacent frames with a similarity index less than 0.1 are input into the lens displacement discrimination model to determine whether the lens is displaced relative to the iris. If the model outputs a result of 1, it means that the lens of the pair of adjacent frames has shifted relative to the iris. If the model outputs a result of 0, it means that the lens of the pair of adjacent frames has not shifted relative to the iris. For the entire video, if the number of adjacent frames where the lens is displaced relative to the iris exceeds a preset threshold (for example, 10 pairs), it is determined that the eye corresponding to the video has lens tremor.

[0074] In the third discrimination module, if the similarity index of the lens contour of adjacent frames is less than 0.1, but the result output by the twin convolutional neural network model is 0, the lens has not shifted relative to the iris. In this case, it is necessary to further calculate the motion vector size of the lens center of mass relative to the corneal center of mass. If the corneal contour similarity index of adjacent frames is less than 0.1, and the motion vector size of the lens center of mass coordinates relative to the corneal center of mass coordinates exceeds a preset threshold (e.g., 10 pixels), it indicates that there is iris displacement combined with lens displacement in this pair of adjacent frames. For the entire video, if the number of adjacent frames with iris displacement combined with lens displacement exceeds a preset threshold (e.g., 10 pairs), it is determined that the eye corresponding to the video has iris tremor and lens tremor at the same time.

[0075] Embodiment 2

[0076] The present invention also provides an automatic detection method for eye lens and iris tremor, the steps comprising:

[0077] S1. Collect dynamic videos of the subject's eyeballs moving upward, downward, nasally, and temporally.

[0078] A slit lamp anterior segment imaging device is used to collect dynamic videos of the subject's eyeballs rotating upward, downward, nasally, and temporally, wherein the dynamic videos include multiple video frames. Specifically, in this embodiment, a total of 200 dynamic videos are collected, of which 100 videos have eye lenses and (or) iris tremors, and 100 videos do not have eye lenses and iris tremors. The above video frame refers to a frame of two-dimensional image in the dynamic video.

[0079] S2. Build an automatic segmentation model and use the automatic segmentation model to segment the lens and cornea regions in adjacent video frames.

[0080] An automatic segmentation model is constructed and used to segment the lens and cornea regions in adjacent video frames. The implementation process includes data annotation, image preprocessing, algorithm design, training process, and prediction process. The specific process is as follows:

[0081] Data annotation: The dynamic video was split into several two-dimensional video frames. One frame was randomly selected from each video, for a total of 200 frames. The labelme software was used to annotate the lens area and cornea area of ​​each frame.

[0082] Image preprocessing: In order to adapt to the input of the segmentation model, the resolution of these 200 frames of two-dimensional images is adjusted to 1024*1024.

[0083] Automatic segmentation model design: Based on the Pytorch Detection2 framework, the Mask-RCNN instance segmentation model is adopted, the pyramid feature extraction network FPN is used as the backbone architecture, and the residual network ResNeXt-101 is used as the convolutional structure.

[0084] Training process: First load the COCO pre-trained model, and then perform full-layer fine-tuning training on the dataset with labeled lens and cornea regions. A total of two segmentation models are trained, one for lens region segmentation and the other for cornea region segmentation.

[0085] Prediction process: In the prediction stage, if the confidence level output by the Mask R-CNN model is greater than or equal to 0.75, it means that the target area (lens area or cornea area) has been detected. Then, the contour data of the target area is obtained through findContours of OpenCV. The contour data is a set of continuous coordinate points on the boundary of the target area.

[0086] S3. Calculate the similarity index of the lens and cornea contours in adjacent video frames.

[0087] Calculate the similarity index of the lens and corneal contours in adjacent video frames. The process includes: first, use the automatic segmentation model to obtain the lens contour image data on two adjacent frames of images, and then use the similarity index calculation formula to obtain the similarity of the two lens contour objects in the two adjacent frames of images:

[0088] ,

[0089] in, represents the similarity index; , ; Indicates the sign of the input value to be returned. If the input value is greater than 0, 1 is returned; if the input value is equal to 0, 0 is returned; if the input value is less than 0, -1 is returned. and Represents the i-th Hu moment of contour object A and object B respectively; Hu moment is composed of these 7 invariant moments to form a set of feature quantities, which are invariant to rotation, scaling and translation. In actual operation, the matchshape function of opencv2 can be directly used to calculate the similarity index to determine the similarity of the two contour data. The method parameter of the matchshape function is set to CV_CONTOURS_MATCH_I1. The function returns a similarity index, which is a value between 0 and 1. The smaller the index, the better the match of the two contours, that is, the more similar the shape of the peripheral edge of the lens is. The same method is used to calculate the similarity of the corneal contours of adjacent frames.

[0090] S4. Construct a lens displacement discrimination model to determine whether the lens is displaced relative to the iris in adjacent video frames.

[0091] First, we need to determine whether the lens is displaced relative to the iris in adjacent video frames. The basis for determining whether the lens is displaced relative to the iris in adjacent frames is as follows: 1) the lens contours in adjacent frames are basically consistent; 2) there are large differences in the global image features of the lens in adjacent frames.

[0092] In this embodiment, only when the lens contour similarity index of two adjacent frames is less than 0.1, the difference in the global image features of the lens of the adjacent frames is further determined. This system uses a twin convolutional neural network to build and train a lens displacement discrimination model to determine the difference in the global image features of the lens of adjacent frames. This is a binary classification task, and the process includes image preprocessing, image annotation, model construction, and model prediction. The specific process includes:

[0093] Image preprocessing: Using the automatic segmentation model, the lens area of ​​all video frames is segmented to obtain the lens segmentation map, and then the resolution of the segmentation map is further adjusted to 224*224 to adapt to the input of the twin convolutional neural network.

[0094] Image annotation: From each dynamic video with lens tremor, 10 pairs of lens segmentation maps of adjacent frames are randomly selected. They satisfy the contour similarity index less than 0.1 and the global image feature difference is large (that is, the lens has shifted relative to the iris). Each pair of images is annotated as 1. From each dynamic video without lens tremor, 10 pairs of lens segmentation maps of adjacent frames are randomly selected. They satisfy the contour similarity index less than 0.1 and the global image feature difference is small (that is, the lens has not shifted relative to the iris). Each pair of images is annotated as 0. A total of 2000 pairs of lens segmentation map samples of adjacent frames, of which 1000 pairs are positive samples annotated as 1 and 1000 pairs are negative samples annotated as 0.

[0095] Construction of lens displacement discrimination model: As a convolutional neural network with a special structure, the twin convolutional neural network has been proven to perform well in similarity measurement tasks by processing two input images simultaneously and comparing their similarities. This embodiment uses the Resnet50 backbone network to extract image features, and inputs two adjacent images in the training set into the backbone network respectively to obtain two feature vectors, which are then subtracted to obtain a new vector. The subtracted new vector is then input into a fully connected layer (FC) to obtain a scalar, and finally the sigmoid function is used to map the result to a labeled value of 0 or 1. Binary cross entropy is used as the loss function. The structure of the twin convolutional network is as follows: Figure 3 shown.

[0096] Model prediction: During prediction, two lens segmentation images of adjacent frames are input. If the output result is 1, it means that the lens has shifted relative to the iris; otherwise, if the output result is 0, it means that the lens has not shifted relative to the iris.

[0097] S5. Calculate the size of the motion vector of the center of mass of the lens contour relative to the center of mass of the cornea contour in adjacent video frames, and determine whether the lens is displaced relative to the cornea based on the size of the motion vector.

[0098] Calculate the size of the motion vector of the center of mass of the lens contour relative to the center of mass of the corneal contour in adjacent video frames, and judge whether the lens is displaced relative to the cornea based on the size of the motion vector. The specific process includes: if the lens contours of adjacent frames are basically the same, but there is no significant difference in the global image features of the lens (the suspensory ligament is connected normally), it is necessary to further judge whether the iris and lens are displaced relative to the cornea in adjacent video frames, so as to determine whether the iris tremor and lens tremor have occurred in this pair of adjacent frames. Since the lens and iris move as a whole in this case, if the lens is displaced relative to the cornea, the iris will also be displaced relative to the cornea, so it is only necessary to judge whether the center of mass of the lens is displaced relative to the center of mass of the cornea in terms of image features. The basis for judging whether the center of mass of the lens relative to the center of mass of the cornea in adjacent frames is as follows: 1) The corneal contours of adjacent frames are basically the same; 2) The size of the motion vector of the center of mass of the lens contour relative to the center of mass of the corneal contour in adjacent frames exceeds the preset threshold.

[0099] Only when the corneal contour similarity index of two adjacent frames is less than 0.1, the motion vector size of the lens contour center of mass relative to the corneal contour center of mass of the pair of adjacent frames is further calculated. For two adjacent frames of images, the lens contour image data and corneal contour image data obtained by the automatic segmentation model are first used to calculate the lens center of mass coordinates and corneal center of mass coordinates of one frame of the image, and their Euclidean distance is calculated through these two center of mass coordinates; then the Euclidean distance between the lens center of mass coordinates and the corneal center of mass coordinates of the other frame of the image is calculated using the same method. The absolute value of the difference between the two Euclidean distances is calculated, which is the motion vector size of the lens contour center of mass relative to the corneal contour center of mass. If the value is greater than the preset threshold (for example, 10 pixels), it means that the lens center of mass of the adjacent frames has shifted relative to the corneal center of mass, thereby judging that the two frames have iris tremor and lens tremor at the same time. The calculation formula of the motion vector size is as follows:

[0100] ,

[0101] ,

[0102] ,

[0103] Among them, a and b represent two adjacent frames of images; Represents the coordinates of the lens center of mass of frame a; Represents the coordinates of the corneal centroid of frame a; represents the coordinates of the centroid of the lens in the b-frame image; It is represented by the coordinates of the corneal centroid of frame b; dist(a) and dist(b) represent the Euclidean distances between the coordinates of the lens centroid and the corneal centroid of the two frames respectively; dist(ab) represents the motion vector size of the lens centroid relative to the corneal contour centroid.

[0104] The centroid coordinates of the contour can be obtained through the moments function of opencv. Assuming that the contour data object is contour, first calculate the contour moment Mu = cv2.moments(contour), and then obtain the centroid coordinates (Px, Py) of the contour through the following Python programming statement:

[0105] Px = int(Mu['m10'] / Mu['m00']),

[0106] Py = int(Mu['m01'] / Mu['m00']).

[0107] S6. Calculate and determine whether there is iris tremor or lens tremor in the entire dynamic video.

[0108] The image of the lens area is extracted using the automatic segmentation model, and then the lens contour similarity index of each pair of adjacent frames is calculated. The lens segmentation images of adjacent frames with a similarity index less than 0.1 are input into the lens displacement discrimination model to determine whether the lens is displaced relative to the iris. If the model outputs a result of 1, it means that the lens of the pair of adjacent frames has shifted relative to the iris. If the model outputs a result of 0, it means that the lens of the pair of adjacent frames has not shifted relative to the iris. For the entire video, if the number of adjacent frames where the lens is displaced relative to the iris exceeds a preset threshold (for example, 10 pairs), it is determined that the eye corresponding to the video has lens tremor.

[0109] If the lens contour similarity index of adjacent frames is less than 0.1, but the result output by the twin convolutional neural network model is 0, the lens has not shifted relative to the iris. In this case, it is necessary to further calculate the motion vector size of the lens center of mass relative to the corneal center of mass. If the corneal contour similarity index of adjacent frames is less than 0.1, and the motion vector size of the lens center of mass coordinates relative to the corneal center of mass coordinates exceeds a preset threshold (e.g., 10 pixels), it means that there is iris displacement combined with lens displacement in this pair of adjacent frames. For the entire video, if the number of adjacent frames with iris displacement combined with lens displacement exceeds a preset threshold (e.g., 10 pairs), it is determined that the eye corresponding to the video has iris tremor and lens tremor at the same time.

[0110] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. An automatic detection system for eye lens and iris tremor, characterized in that: include: An acquisition module, a segmentation module, a calculation module, a first discrimination module, a second discrimination module and a third discrimination module; The acquisition module is used to acquire dynamic videos of the subject's eyeballs rotating upward, downward, nasal, and temporal, respectively, wherein the dynamic videos include multiple video frames; The segmentation module is used to construct an automatic segmentation model, and use the automatic segmentation model to segment the lens and cornea areas in adjacent video frames. The structure of the automatic segmentation model includes: based on the Pytorch Detection2 framework, using the Mask-RCNN instance segmentation model, using the pyramid feature extraction network FPN as the backbone architecture, and using the residual network ResNeXt-101 as the convolution structure; The calculation module is used to calculate the similarity index of the lens and cornea contours in adjacent video frames; The first discrimination module is used to construct a lens displacement discrimination model to discriminate whether the lens is displaced relative to the iris in adjacent video frames, wherein the lens displacement discrimination model includes a Resnet50 backbone network, a fully connected layer and a sigmoid function, and uses binary cross entropy as a loss function; The second determination module is used to calculate the size of the motion vector of the centroid of the lens contour relative to the centroid of the cornea contour in adjacent video frames, and determine whether the lens is displaced relative to the cornea according to the size of the motion vector; The third determination module is used to calculate and determine whether iris tremor or lens tremor exists in the entire dynamic video.

2. The automatic detection system for eye lens and iris tremor according to claim 1, characterized in that: The workflow of the segmentation module includes: using the automatic segmentation model to segment the lens area and the cornea area on the video frame, and obtaining the lens contour map in two adjacent frames of images.

3. The automatic detection system for eye lens and iris tremor according to claim 2, characterized in that: The workflow of the calculation module includes: using a similarity index calculation formula to obtain the similarity of two lens contour objects in two adjacent frames of images: ， in, represents the similarity index; , , Indicates the sign of the input value to be returned. If the input value is greater than 0, 1 is returned; if the input value is equal to 0, 0 is returned; if the input value is less than 0, -1 is returned. and Represents the i-th Hu moment of contour object A and object B respectively; The same process is used to calculate the similarity of corneal contours in adjacent frames.

4. The automatic detection system for eye lens and iris tremor according to claim 3, characterized in that: When the lens contour similarity index of two adjacent frames is less than 0.1, the difference of the lens global image features of the adjacent frames is further determined; When the corneal contour similarity index of two adjacent frames is less than 0.1, the magnitude of the motion vector of the centroid of the lens contour relative to the centroid of the corneal contour of the adjacent frames is further calculated.

5. The automatic detection system for eye lens and iris tremor according to claim 3, characterized in that: The workflow of the first discrimination module includes: Preprocessing the lens contour images in two adjacent frames of images to obtain processed data; Performing image annotation on the processed data to obtain a training sample set; Constructing a lens displacement discrimination model, and using the training sample set to train the constructed lens displacement discrimination model; The trained lens displacement discrimination model is used to discriminate whether the lens is displaced relative to the iris.

6. The automatic detection system for eye lens and iris tremor according to claim 4, characterized in that: The process of the second determination module calculating the size of the motion vector includes: , ， ， Among them, a and b represent two adjacent frames of images; Represents the coordinates of the lens center of mass of frame a; Represents the coordinates of the corneal centroid of frame a; represents the coordinates of the centroid of the lens in the b-frame image; It is represented by the coordinates of the corneal centroid of frame b; dist(a) and dist(b) represent the Euclidean distances between the coordinates of the lens centroid and the corneal centroid of the two frames respectively; dist(ab) represents the motion vector size of the lens centroid relative to the corneal contour centroid.

7. An automatic detection method for eye lens and iris tremor, the method being applied to the system according to any one of claims 1 to 6, characterized in that the steps include: Collecting dynamic videos of the subject's eyeballs moving upward, downward, nasal, and temporal, respectively, wherein the dynamic videos include multiple video frames; An automatic segmentation model is constructed, and the automatic segmentation model is used to segment the lens and cornea regions in adjacent video frames. The structure of the automatic segmentation model includes: based on the Pytorch Detection2 framework, using the Mask-RCNN instance segmentation model, using the pyramid feature extraction network FPN as the backbone architecture, and using the residual network ResNeXt-101 as the convolution structure; Calculate the similarity index of the lens and cornea contours in adjacent video frames; A lens displacement discrimination model is constructed to discriminate whether the lens is displaced relative to the iris in adjacent video frames. The discrimination model includes a Resnet50 backbone network, a fully connected layer, and a sigmoid function, and uses binary cross entropy as a loss function; Calculating the magnitude of the motion vector of the center of mass of the lens contour relative to the center of mass of the cornea contour in adjacent video frames, and judging whether the lens is displaced relative to the cornea according to the magnitude of the motion vector; Calculate and determine whether iris tremor or lens tremor exists in the entire dynamic video.

Images