Strabismus Intelligent Recognition Method, Device, Terminal and Medium Based on Eye Images

Abstract

The present invention discloses a method, device, terminal and medium for intelligent identification of strabismus based on eye images. Compared with the prior art, the information of the pupil area and the corneal reflection point is extracted through an eyeball segmentation network, so as to obtain the monocular corneal reflectivity; a random forest model makes a voting decision based on the monocular corneal reflectivity, the pupil area and the information of the corneal reflection point to obtain a first prediction result; a deep feature extraction network extracts deep features reflecting the strabismus state and combines with the monocular corneal reflectivity to obtain a second prediction result, and the above prediction results are weighted and averaged to obtain the human eye strabismus recognition result. Therefore, by analyzing the eye images, the state of the human eye can be simply and accurately identified, and it is found that there may be a strabismus problem in the eyes, so as to timely remind the user to go to the hospital for diagnosis and treatment.

Description

Technical Field

[0001] The present invention relates to the technical field of vision detection, and particularly to a method, device, terminal and medium for intelligent identification of strabismus based on eye images. Background Art

[0002] There are many reasons for eye strabismus. It may be caused by imperfect development in early childhood, or it may also be due to incorrect eye use habits. Because of incorrect eye use, it will cause the convergence of the eye movement center to be too strong or the abduction to be insufficient. As a result, the two eyeballs will either be overly convergent or overly separated, and it is very easy to have strabismus.

[0003] Due to the lack of simple and convenient detection methods and devices, currently, only when the strabismus degree is relatively high and abnormal is observed with the naked eye, will people go to the hospital for diagnosis and treatment, resulting in missing the opportunity for early intervention and treatment.

[0004] Therefore, the existing technology needs to be improved. Summary of the Invention

[0005] The main purpose of the present invention is to provide a method, device, intelligent terminal and storage medium for intelligent identification of strabismus based on eye images. Through the captured eye images, it can simply and conveniently identify whether there is strabismus in the eyes to remind users to go to the hospital for further diagnosis and treatment in time.

[0006] To achieve the above object, in the first aspect of the present invention, a method for intelligent identification of strabismus based on eye images is provided. The method includes:

[0007] Input the eye image into an eyeball segmentation network to obtain pupil region information and corneal reflection point information, where the eyeball segmentation network is a convolutional neural network for segmenting eye images;

[0008] Based on the pupil region information and the corneal reflection point information, obtain a single-eye corneal reflectivity reflecting the positional relationship between the pupil and the corneal reflection point;

[0009] Input the pupil region information, the corneal reflection point information and the single-eye corneal reflectivity into a random forest model to obtain a first strabismus prediction result;

[0010] Input the eye image into a high-dimensional feature extraction network to obtain pupil features reflecting the strabismus state;

[0011] Fuse the single-eye corneal reflectivity and the pupil features to obtain a second strabismus prediction result;

[0012] Perform weighted averaging on the first strabismus prediction result and the second strabismus prediction result to obtain a strabismus recognition result.

[0013] Optionally, obtaining a monocular corneal reflectance that reflects the positional relationship between the pupil and the corneal light reflex point based on the pupil region information and the corneal light reflex point information includes:

[0014] Obtaining edge data of the pupil according to the pupil region information, where the edge data includes left edge data and right edge data;

[0015] Fitting the pupil according to the pupil region information and the edge data by the least squares method to obtain a fitted circle;

[0016] Obtaining a monocular corneal reflectance that reflects the positional relationship between the pupil and the corneal light reflex point based on the fitted circle and the corneal light reflex point information.

[0017] Optionally, obtaining a monocular corneal reflectance that reflects the positional relationship between the pupil and the corneal light reflex point based on the fitted circle and the corneal light reflex point information includes:

[0018] Obtaining the radius and center coordinates of the fitted circle;

[0019] Obtaining the coordinates of the corneal light reflex point based on the corneal light reflex point information;

[0020] Calculating the difference between the center coordinates and the coordinates of the corneal light reflex point and calculating the ratio of the difference to the radius of the fitted circle to obtain the monocular corneal reflectance.

[0021] Optionally, fusing the monocular corneal reflectance and the pupil feature to obtain a second strabismus prediction result includes:

[0022] Performing a vector multiplication of the monocular corneal reflectance and the pupil feature to obtain a fused feature;

[0023] Inputting the fused feature into a fully connected layer to obtain the second strabismus prediction result.

[0024] Optionally, after obtaining a monocular corneal reflectance that reflects the positional relationship between the pupil and the corneal light reflex point based on the pupil region information and the corneal light reflex point information, it further includes:

[0025] Obtaining a pupil deformation index;

[0026] Obtaining an estimated result of the strabismus degree based on the monocular corneal reflectance and the pupil deformation index according to a pre-established non-linear regression equation.

[0027] Optionally, inputting the pupil region information, the corneal light reflex point information, and the monocular corneal reflectance into a random forest model to obtain a first strabismus prediction result includes:

[0028] Based on the pupil region information, obtain the pupil coordinates and the pupil radius;

[0029] Based on the corneal reflection point information, obtain the coordinates of the corneal reflection point;

[0030] Input the pupil coordinates, the pupil radius, the coordinates of the corneal reflection point, and the monocular corneal reflectivity into a random forest model, and obtain the first strabismus prediction result according to the voting mechanism.

[0031] A second aspect of the present invention provides a strabismus intelligent recognition device based on an eye image. Among them, the above device includes:

[0032] An image segmentation module, configured to input an eye image into an eyeball segmentation network to obtain pupil region information and corneal reflection point information, where the eyeball segmentation network is a convolutional neural network for segmenting an eye image;

[0033] A monocular corneal reflectivity module, configured to obtain a monocular corneal reflectivity reflecting the positional relationship between the pupil and the corneal reflection point based on the pupil region information and the corneal reflection point information;

[0034] A random forest module, configured to input the pupil region information, the corneal reflection point information, and the monocular corneal reflectivity into a random forest model to obtain a first strabismus prediction result;

[0035] A high-dimensional feature extraction module, configured to input an eye image into a high-dimensional feature extraction network to obtain pupil features reflecting the strabismus state;

[0036] A fusion module, configured to fuse the monocular corneal reflectivity and the pupil features to obtain a second strabismus prediction result;

[0037] A determination module, configured to perform weighted averaging on the first strabismus prediction result and the second strabismus prediction result to obtain a strabismus recognition result.

[0038] Optionally, it further includes a strabismus degree module, configured to obtain a pupil deformation index, and based on the monocular corneal reflectivity and the pupil deformation index, obtain an estimated result of the strabismus degree according to a pre-established non-linear regression equation.

[0039] A third aspect of the present invention provides an intelligent terminal. The above intelligent terminal includes a memory, a processor, and a strabismus intelligent recognition program based on an eye image stored on the above memory and executable on the above processor. When the above strabismus intelligent recognition program based on an eye image is executed by the above processor, it implements the steps of any one of the above strabismus intelligent recognition methods based on an eye image.

[0040] A fourth aspect of the present invention provides a computer-readable storage medium, on which an intelligent strabismus recognition program based on an eye image is stored. When the intelligent strabismus recognition program based on the eye image is executed by a processor, the steps of any one of the above-mentioned intelligent strabismus recognition methods based on the eye image are implemented.

[0041] As can be seen from the above, compared with the prior art, the information of the pupil area and the corneal reflection point is extracted through the eyeball segmentation network, so as to obtain the corneal reflectivity of a single eye; the random forest model makes a voting decision based on the corneal reflectivity of a single eye, the pupil area and the information of the corneal reflection point to obtain a first prediction result; the high-dimensional feature extraction network extracts the depth features reflecting the strabismus state and combines them with the corneal reflectivity of a single eye to obtain a second prediction result, and the above prediction results are weighted and averaged to obtain the strabismus recognition result. Therefore, by analyzing the captured eye image, it is possible to simply and accurately identify whether there is strabismus in the eye, and timely remind the user to go to the hospital for further diagnosis and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0043] Figure 1 is a schematic flowchart of an intelligent strabismus recognition method based on an eye image provided by an embodiment of the present invention;

[0044] Figure 2 is Figure 1 a schematic diagram of the model in the embodiment;

[0045] Figure 3 is Figure 1 a specific flowchart of step S200 in the embodiment;

[0046] Figure 4 is Figure 1 a schematic diagram of the corneal reflection point and the pupil center in the embodiment;

[0047] Figure 5 is a schematic structural diagram of an intelligent strabismus recognition device based on an eye image provided by an embodiment of the present invention;

[0048] Figure 6 is a schematic block diagram of the internal structure principle of an intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures, technologies, etc. are presented to provide a thorough understanding of the embodiments of the present invention. However, those skilled in the art should understand that the present invention can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from obscuring the description of the present invention.

[0050] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their combinations.

[0051] It should also be understood that the terms used in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0052] It should be further understood that the term "and / or" used in the specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0053] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "once" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]" depending on the context.

[0054] The following describes the technical solutions in the embodiments of the present invention clearly and completely in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0055] In the following description, many specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0056] Classified according to the direction of strabismus, strabismus includes esotropia, exotropia, hypertropia, and hypotropia. If classified according to the nature of strabismus, it can be further divided into monocular strabismus and alternating strabismus.

[0057] Among them, esotropia and exotropia are both abnormal eye positions, caused by abnormal binocular visual function. When one eye (the dominant eye) fixates on a target straight ahead, if the other eye deviates medially, it is esotropia; if it deviates laterally, it is exotropia.

[0058] Since strabismus is usually difficult to detect, especially intermittent strabismus, it can only be detected by the naked eye when the strabismus degree is relatively high. Due to the failure to detect it in time, the best period for timely intervention and treatment is missed. Therefore, a detection method is needed to conveniently detect whether the eye state is abnormal usually, so as to remind users to go to the hospital for diagnosis and treatment in time.

[0059] The present invention provides an intelligent method for identifying strabismus based on eye images. By inputting the captured eye images, it can simply and accurately detect the state of the human eye and timely discover the existing problems of the eyes.

[0060] Exemplary method

[0061] As Figure 1 shown, the embodiment of the present invention provides an intelligent method for identifying strabismus based on eye images, which can run on various terminals with image capture functions, such as computer terminals, tablet computers, smart phones, etc.; it can also be captured by a dedicated shooting device and then process the captured eye images to obtain the recognition result.

[0062] Specifically, the above method includes the following steps:

[0063] Step S100: Input the eye image into the eye segmentation network to obtain pupil region information and corneal reflection point information, where the eye segmentation network is a convolutional neural network for segmenting eye images;

[0064] Specifically, the shooting device in this embodiment includes a flexible light source and a high-resolution camera, and the flexible light source is fixed directly above the high-resolution camera. A red marker is arranged below the flexible light source, and the position of the red marker is close to the lens to guide the person being detected to fixate on the red marker. Using this device can better obtain eye images. In this embodiment, for the convenience of children's detection, the camera is fixed on a tripod, and the height of the camera from the ground can be reduced or increased by adjusting the angle of the tripod.

[0065] Since the model architecture of the convolutional neural network U-Net combines context information, has a fast training speed, and uses a small amount of data, it meets the requirements of eye image segmentation. U-Net is a symmetric network architecture that includes an encoder and a decoder. The image will first be processed by the encoder and then by the decoder to finally achieve image segmentation. Therefore, the eyeball segmentation network in the present invention is preferably a U-Net convolutional neural network.

[0066] Among them, the encoder part can also choose VGG (deep convolutional network structure), ResNet (residual convolutional network structure), etc. As the convolutional layer deepens, the length and width of the feature map decrease, and the number of channels increases. Although the encoder extracts the high-level features of the image, it discards the position information of the image. However, when encoding the eye image, only the content information of the eye image needs to be extracted without the position information. Each layer of the decoder performs deconvolution and combines with the corresponding primary features of the encoder to gradually restore the position information of the image. In the decoder, as the convolutional layer deepens, the length and width of the feature map increase, and the number of channels decreases. The expression of the loss function of U-Net is:

[0067]

[0068] where y is the independent variable, is the expected value, N is the number of input images in a single batch, and L1 is the total loss value of the eyeball segmentation network in a single batch.

[0069] The U-Net segmentation network automatically segments the eyeball area and the corneal reflection point area using deep learning features to obtain pupil area information and corneal reflection point information. Among them, the pupil area information includes the boundary of the pupil, the position of the pupil, the pixels of the pupil area, etc.; the corneal reflection point information mainly includes the position coordinates where the corneal reflection point is located.

[0070] Step S200: Based on the pupil area information and the corneal reflection point information, obtain the monocular corneal reflectivity that reflects the positional relationship between the pupil and the corneal reflection point;

[0071] Specifically, according to the pupil area information and the corneal reflection point information, the positions of the pupil and the corneal reflection point are obtained respectively, and by analyzing the relationship between the two positions, the monocular corneal reflectivity that reflects the positional relationship between the pupil and the corneal reflection point can be obtained, such as the corneal reflection point is located to the left of the pupil center, the corneal reflection point is located to the right of the pupil center; the corneal reflection point is located above the pupil center, etc.

[0072] Furthermore, since the relationship between the strabismus estimation degree Y and the monocular corneal reflectivity (R o ) is a non-linear relationship, a non-linear regression equation with the strabismus estimation degree Y as the dependent variable and the monocular corneal reflectivity (R o ) as the independent variable is established in advance: where k is a coefficient constant, b is an offset, and r represents the pupil deformation index.

[0073] After obtaining the monocular corneal reflectance, the pupil deformation index can also be obtained. That is, the non-linear regression in the statistics of the pupil deformation index is transformed into a linear regression scheme for solution. Let u = ln(Y - b), c = ln(k), and v = ln R0. Then the original non-linear regression problem Y = kR0 r +b can be transformed into a linear regression problem u = rv + c. Finally, the pupil deformation index can be obtained by the least squares method of the linear regression method, and the monocular corneal reflectance and the pupil deformation index are input into the pre-established non-linear regression equation to obtain the estimated result of the strabismus degree.

[0074] Different from the existing corneal reflection method for measuring the strabismus degree, the monocular corneal reflectance (SCLRDR) according to the present invention can measure the monocular strabismus degree. At the same time, compared with the current characteristic corneal reflection deviation ratio (CCLRDR), it only needs one eye to calculate, and the accuracy is the same.

[0075] Step S300: Input the pupil region information, corneal reflection point information, and monocular corneal reflectance into the random forest model to obtain the first strabismus prediction result;

[0076] Specifically, the random forest is a classifier containing multiple decision trees. Voting decisions are made based on the position information (such as pupil coordinates, pupil radius, and coordinates of the corneal reflection point) obtained from the input pupil region information, corneal reflection point information, etc. of the eye segmentation network and the monocular corneal reflectance to obtain the first strabismus prediction result. In this embodiment, the normal state, esotropia state, and exotropia state are mainly detected, and the first strabismus prediction result is the probability generated by voting on the three classifications.

[0077] First, the random forest model needs to be pre-trained: input the pupil radius, pupil coordinates, reflection point coordinates, and monocular corneal reflectance obtained from the eye segmentation network. The classification labels of the eye images are esotropia, exotropia, and normal. Set the number of trees in the random forest to 1000 and train through the training samples.

[0078] Step S400: Input the eye image into the high-dimensional feature extraction network to obtain the pupil features reflecting the strabismus state;

[0079] Specifically, the high-dimensional feature extraction network in this embodiment is the GoogLeNet-V3 neural network. This network structure contains a large number of layer structures, uses 1*1, 3*3, 5*5, 1*7, and 7*1 filters, and integrates the final results as the input of the next layer. It can reduce the number of network parameters while retaining high efficiency in extraction, and can optimize the calculation time. This high-dimensional feature extraction network based on a convolutional neural network realizes data-driven automatic feature setting and optimization through deep learning methods, and obtains pupil features reflecting the strabismus state.

[0080] Since GoogLeNet-v3 requires the input image dimension to be (3×299×299), it is necessary to scale the eye image before inputting it into the high-dimensional feature extraction network.

[0081] Among them, the loss function of the GoogLeNet-V3 neural network is defined as:

[0082]

[0083] y is the independent variable, is the expected value, N is the number of input images in a single batch, and L2 is the total loss value of the high-dimensional feature extraction network in a single batch.

[0084] Step S500: Fuse the single-eye corneal reflectance and pupil features to obtain the second strabismus prediction result;

[0085] Specifically, input the single-eye corneal reflectance into the high-dimensional feature extraction network to fuse with the pupil features extracted by the high-dimensional feature extraction network, and obtain the second strabismus prediction result. In this embodiment, the normal state, esotropia state, and exotropia state are mainly detected, and the second strabismus prediction result is the probability of three classifications.

[0086] In this embodiment, first multiply the single-eye corneal reflectance and the pupil features vectorially to obtain the fused feature, and then input the fused feature into the fully connected layer to obtain the second strabismus prediction result. That is: namely, the pupil features and the single-eye corneal reflectance are fused between the convolutional layer and the fully connected layer. Considering that the single-eye corneal reflectance (SCLRDR) has only one dimension, the feature fusion can be simply completed in the form of a tensor outer product, and no more computing resources are required. Suppose the high-dimensional feature of the image is: (x1, x2,..., x 1000 ), the fused feature is (x1, x2,..., x 1000 )gSCLRDR, the dimension of the fully connected layer is (1000×3), and the output generated by the final three-classification is denoted as (V1 g , V2 g , V3 g ).

[0087] Step S600: Perform weighted averaging on the first strabismus prediction result and the second strabismus prediction result to obtain a strabismus recognition result.

[0088] Specifically, after obtaining the first strabismus prediction result and the second strabismus prediction result, perform weighted averaging on the two prediction results according to the set weight to obtain the final strabismus recognition result. This strabismus recognition result is the probability of three classifications.

[0089] In this embodiment, the weighted averaging expression is:

[0090] (V1, V2, V3) = u(V1 g , V2 g , V3 g ) + (1 - u)(V1 r , V2 r , V3 r )

[0091] where u represents the weight, which is set to 0.7 in this embodiment, (V1 g , V2 g , V3 g ) is the first strabismus prediction result output by the high-dimensional feature extraction network, and (V1 r , V2 r , V3 r ) is the second strabismus prediction result output by the random forest model. Finally, obtain the prediction probability of each classification through normalization.

[0092] In summary, as Figure 2 shown, by segmenting and recognizing the pupil area and the corneal reflection point through the eye segmentation network, the eye target area can be more effectively extracted, and the pupil point can be more accurately located. Through the random forest model for voting decision-making, the first prediction result is obtained; through the high-dimensional feature extraction network, the second prediction result is obtained, and finally the above prediction results are weighted averaged to obtain the strabismus recognition result. Therefore, by processing the eye image, obtaining the prediction results of multiple models, and fusing the results, the strabismus state can be simply and accurately detected, and whether the eyes are abnormal can be timely discovered.

[0093] In one embodiment, as Figure 3 shown, the above step S200 specifically includes the following steps:

[0094] Step S210: Obtain edge data according to the pupil information, where the edge data includes left edge data and right edge data;

[0095] Step S220: Fit the pupil according to the pupil area information and the edge data by the least squares method to obtain a fitted circle;

[0096] Specifically, referring toFigure 4 As shown, based on the boundary information in the pupil information, the left edge data and the right edge data are obtained. The pupil center is fitted by the least squares method. Let the center be (A, B) and the radius be R. Any point in the set T of the left edge or the right edge points of the selected feature region ο can be expressed as {(x i , y i ) | (x i , y i ) ∈ T}. Finally, the least squares optimization function is:

[0097]

[0098] where a = -2A, b = -2B, c = A 2 + B 2 - R 2 . After solving, the center coordinates are obtained, that is, the pupil center position.

[0099] The present invention takes into account that the eyelid or eyelid blocks the pupil, thus interfering with the edge points in the upper and lower positions of the pupil region. Therefore, the edge points in the left and right positions are selected.

[0100] Step S230: Based on the fitted circle and the corneal reflection point information, obtain the single-eye corneal reflectivity reflecting the positional relationship between the pupil and the corneal reflection point.

[0101] Specifically, first calculate the radius and center coordinates of the fitted circle; obtain the coordinates of the corneal reflection point according to the corneal reflection point information; then calculate the difference between the center coordinates and the coordinates of the corneal reflection point, and calculate the ratio of the difference to the radius of the fitted circle to obtain the single-eye corneal reflectivity. That is: The single-eye corneal reflectivity (SCLRDR) feature is the proportional relationship between the horizontal distance between the reflection point and the pupil center point and the eyeball radius. Taking the right eye as an example, the expression of the single-eye corneal reflectivity is:

[0102]

[0103] where L lc is the horizontal distance between the reflection point and the pupil center point, R r is the right eyeball radius, d r is the coordinate of the pupil center point, and x r is the coordinate of the reflection point.

[0104] As described above, the single-eye corneal reflectivity is the proportional relationship between the horizontal distance between the reflection point and the pupil center point and the eyeball radius. Compared with the existing corneal reflection method, the degree of strabismus abnormality can be more accurately reflected by measuring the ratio of the reflection point and the pupil radius.

[0105] The performance of the system of the present invention is measured by the accuracy rate. The accuracy rate index is: Among them, True indicates that the strabismus classification prediction is correct, and Total indicates the total number of samples. After practical tests using approximately 200 prediction samples, the accuracy rate can reach 96.6%. Moreover, the present invention also has a certain anti-interference ability for eye occlusion. That is, when a child wears glasses, it will interfere with the reflected light points, and it is difficult for traditional algorithms based solely on reflected light points to distinguish. However, the present invention combines a high-dimensional feature extraction network of deep learning and can well eliminate interference and accurately identify.

[0106] Exemplary device

[0107] As Figure 5 shown, corresponding to the above-mentioned intelligent strabismus recognition method based on eye images, an embodiment of the present invention further provides an intelligent strabismus recognition device based on eye images. The above-mentioned intelligent strabismus recognition device based on eye images includes:

[0108] An image segmentation module 600, configured to input an eye image into an eyeball segmentation network to obtain pupil region information and corneal reflection point information. The eyeball segmentation network is a convolutional neural network for segmenting eye images;

[0109] A single-eye corneal reflectivity module 610, configured to obtain a single-eye corneal reflectivity reflecting the positional relationship between the pupil and the corneal reflection point based on the pupil region information and the corneal reflection point information;

[0110] A random forest module 620, configured to input the pupil region information, the corneal reflection point information, and the single-eye corneal reflectivity into a random forest model to obtain a first strabismus prediction result;

[0111] A high-dimensional feature extraction module 630, configured to input an eye image into a high-dimensional feature extraction network to obtain pupil features;

[0112] A fusion module 640, configured to fuse the single-eye corneal reflectivity and the pupil features to obtain a second strabismus prediction result;

[0113] A determination module 650, configured to perform weighted averaging on the first strabismus prediction result and the second strabismus prediction result to obtain a strabismus recognition result.

[0114] Optionally, it further includes a strabismus degree module, configured to obtain a pupil deformation index, and based on the single-eye corneal reflectivity and the pupil deformation index, obtain an estimated result of the strabismus degree according to a pre-established non-linear regression equation.

[0115] Specifically, in this embodiment, the specific functions of the above-mentioned modules of the intelligent strabismus recognition device based on eye images can refer to the corresponding descriptions in the above-mentioned intelligent strabismus recognition method based on eye images, and will not be elaborated here.

[0116] Based on the above embodiments, the present invention further provides an intelligent terminal, and its principle block diagram can be as Figure 6 shown. The above intelligent terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a strabismus intelligent recognition program based on eye images. The internal memory provides an environment for the operation of the operating system and the strabismus intelligent recognition program based on eye images in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the strabismus intelligent recognition program based on eye images is executed by the processor, the steps of any one of the above-mentioned strabismus intelligent recognition methods based on eye images are implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0117] Those skilled in the art can understand that Figure 6 the principle block diagram shown in

[0118] is only a block diagram of some structures related to the solution of the present invention, and does not constitute a limitation on the intelligent terminal to which the solution of the present invention is applied. The specific intelligent terminal may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0118] In one embodiment, an intelligent terminal is provided. The above intelligent terminal includes a memory, a processor, and a strabismus intelligent recognition program based on eye images stored on the above memory and executable on the above processor. When the strabismus intelligent recognition program based on eye images is executed by the above processor, the following operation instructions are performed:

[0119] Input the eye image into the eyeball segmentation network to obtain pupil region information and corneal reflection point information. The eyeball segmentation network is a convolutional neural network for segmenting eye images;

[0120] Based on the pupil region information and the corneal reflection point information, obtain a single-eye corneal reflectance reflecting the positional relationship between the pupil and the corneal reflection point;

[0121] Input the pupil region information, the corneal reflection point information, and the single-eye corneal reflectance into a random forest model to obtain a first strabismus prediction result;

[0122] Input the eye image into the high-dimensional feature extraction network to obtain pupil features reflecting the strabismus state;

[0123] Fuse the single-eye corneal reflectance and the pupil features to obtain a second strabismus prediction result;

[0124] Perform weighted averaging on the first strabismus prediction result and the second strabismus prediction result to obtain a strabismus recognition result.

[0125] Optionally, obtaining a single-eye corneal reflectance reflecting the positional relationship between the pupil and the corneal light reflex point based on the pupil region information and the corneal light reflex point information includes:

[0126] Obtain the edge data of the pupil according to the pupil region information, where the edge data includes left edge data and right edge data;

[0127] Fit the pupil according to the pupil region information and the edge data using the least squares method to obtain a fitted circle;

[0128] Based on the fitted circle and the corneal light reflex point information, obtain a single-eye corneal reflectance reflecting the positional relationship between the pupil and the corneal light reflex point.

[0129] Optionally, obtaining a single-eye corneal reflectance reflecting the positional relationship between the pupil and the corneal light reflex point based on the fitted circle and the corneal light reflex point information includes:

[0130] Obtain the radius and center coordinates of the fitted circle;

[0131] Based on the corneal light reflex point information, obtain the coordinates of the corneal light reflex point;

[0132] Calculate the difference between the center coordinates and the coordinates of the corneal light reflex point and calculate the ratio of the difference to the radius of the fitted circle to obtain the single-eye corneal reflectance.

[0133] Optionally, fusing the single-eye corneal reflectance and the pupil feature to obtain a second strabismus prediction result includes:

[0134] Perform vector multiplication on the single-eye corneal reflectance and the pupil feature to obtain a fused feature;

[0135] Input the fused feature into a fully connected layer to obtain the second strabismus prediction result.

[0136] Optionally, after obtaining a single-eye corneal reflectance reflecting the positional relationship between the pupil and the corneal light reflex point based on the pupil region information and the corneal light reflex point information, it further includes:

[0137] Obtain a pupil deformation index;

[0138] Based on the single-eye corneal reflectance and the pupil deformation index, obtain an estimated result of the strabismus degree according to a pre-established non-linear regression equation.

[0139] Optionally, inputting the pupil area information, the corneal reflection point information, and the single-eye corneal reflectivity into the random forest model to obtain a first strabismus prediction result includes:

[0140] Based on the pupil area information, obtaining the pupil coordinates and the pupil radius;

[0141] Based on the corneal reflection point information, obtaining the coordinates of the corneal reflection point;

[0142] Inputting the pupil coordinates, the pupil radius, the coordinates of the corneal reflection point, and the single-eye corneal reflectivity into the random forest model, and obtaining the first strabismus prediction result according to the voting mechanism.

[0143] An embodiment of the present invention further provides a computer-readable storage medium. A strabismus intelligent recognition program based on an eye image is stored on the computer-readable storage medium. When the strabismus intelligent recognition program based on the eye image is executed by a processor, the steps of any one of the strabismus intelligent recognition methods provided by the embodiments of the present invention are implemented.

[0144] It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the above device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of mutual distinction and do not limit the protection scope of the present invention. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0146] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0147] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0148] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal device and method can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the above-mentioned division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0149] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-mentioned method embodiments of the present invention, it can also be completed by a computer program instructing the relevant hardware. The above-mentioned computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The above-mentioned computer-readable medium can include: any entity or device that can carry the above-mentioned computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0150] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An intelligent strabismus recognition method based on eye images, characterized in that, The method includes: Inputting the eye image into an eyeball segmentation network to obtain pupil area information and corneal reflection point information, where the eyeball segmentation network is a convolutional neural network for segmenting eye images; Based on the pupil area information and the corneal reflection point information, obtaining a single-eye corneal reflectivity that reflects the positional relationship between the pupil and the corneal reflection point; Inputting the pupil area information, the corneal reflection point information, and the single-eye corneal reflectivity into a random forest model to obtain a first strabismus prediction result; Inputting the eye image into a high-dimensional feature extraction network to obtain pupil features that reflect the strabismus state; Fusing the single-eye corneal reflectivity and the pupil features to obtain a second strabismus prediction result; Performing weighted averaging on the first strabismus prediction result and the second strabismus prediction result to obtain a strabismus recognition result; The fusing the single-eye corneal reflectivity and the pupil features to obtain a second strabismus prediction result includes: Performing vector multiplication on the single-eye corneal reflectivity and the pupil features to obtain a fused feature; Inputting the fused feature into a fully connected layer to obtain the second strabismus prediction result; The inputting the pupil area information, the corneal reflection point information, and the single-eye corneal reflectivity into a random forest model to obtain a first strabismus prediction result includes: Based on the pupil area information, obtaining pupil coordinates and a pupil radius; Based on the corneal reflection point information, obtaining the coordinates of the corneal reflection point; Inputting the pupil coordinates, the pupil radius, the coordinates of the corneal reflection point, and the single-eye corneal reflectivity into a random forest model, and obtaining the first strabismus prediction result according to the voting mechanism.

2. The intelligent strabismus recognition method based on eye images according to claim 1, characterized in that, The obtaining a single-eye corneal reflectivity that reflects the positional relationship between the pupil and the corneal reflection point based on the pupil area information and the corneal reflection point information includes: According to the pupil area information, obtaining edge data of the pupil, where the edge data includes left edge data and right edge data; According to the pupil area information and the edge data, fitting the pupil by the least squares method to obtain a fitted circle; Based on the fitted circle and the corneal reflection point information, obtaining a single-eye corneal reflectivity that reflects the positional relationship between the pupil and the corneal reflection point.

3. The strabismus intelligent recognition method based on eye images according to claim 2, wherein, The obtaining a single-eye corneal reflectivity that reflects the positional relationship between the pupil and the corneal reflection point based on the fitted circle and the corneal reflection point information includes: Obtaining the radius and center coordinates of the fitted circle; Based on the corneal reflection point information, obtaining the coordinates of the corneal reflection point; Calculating the difference between the center coordinates and the coordinates of the corneal reflection point and calculating the ratio of the difference to the radius of the fitted circle to obtain the single-eye corneal reflectivity.

4. The strabismus intelligent recognition method based on eye images according to claim 1, characterized in that, After obtaining a single-eye corneal reflectivity that reflects the positional relationship between the pupil and the corneal reflection point based on the pupil area information and the corneal reflection point information, it further includes: Obtaining a pupil deformation index; Based on the single-eye corneal reflectivity and the pupil deformation index, obtaining an estimated result of the strabismus degree according to a pre-established non-linear regression equation.

5. The intelligent strabismus recognition device based on eye images is characterized in that The device is used to implement the steps of the method for intelligent strabismus recognition based on eye images according to any one of claims 1-4, and the device includes: An image segmentation module, configured to input an eye image into an eyeball segmentation network to obtain pupil region information and corneal reflection point information, where the eyeball segmentation network is a convolutional neural network for segmenting eye images; A single-eye corneal reflectivity module, configured to obtain a single-eye corneal reflectivity reflecting the positional relationship between the pupil and the corneal reflection point based on the pupil region information and the corneal reflection point information; A random forest module, configured to input the pupil region information, the corneal reflection point information, and the single-eye corneal reflectivity into a random forest model to obtain a first strabismus prediction result; A high-dimensional feature extraction module, configured to input an eye image into a high-dimensional feature extraction network to obtain pupil features reflecting the strabismus state; A fusion module, configured to fuse the single-eye corneal reflectivity and the pupil features to obtain a second strabismus prediction result; A determination module, configured to perform weighted averaging on the first strabismus prediction result and the second strabismus prediction result to obtain a strabismus recognition result.

6. The strabismus intelligent recognition device based on an eye image according to claim 5, wherein, It further includes a strabismus degree module, configured to obtain a pupil deformation index, and based on the single-eye corneal reflectivity and the pupil deformation index, obtain an estimated result of the strabismus degree according to a pre-established non-linear regression equation.

7. Smart terminal, characterized in that, The intelligent terminal includes a memory, a processor, and a strabismus intelligent recognition program based on eye images stored on the memory and executable on the processor. When the strabismus intelligent recognition program based on eye images is executed by the processor, it implements the steps of the strabismus intelligent recognition method based on eye images according to any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a strabismus intelligent recognition program based on eye images. When the strabismus intelligent recognition program based on eye images is executed by a processor, it implements the steps of the strabismus intelligent recognition method based on eye images according to any one of claims 1-4.

Images