Method, device, and program product for representing an eye topology based on an eye image

Abstract

The present invention belongs to the field of intelligent medicine, and particularly relates to a method, device and program product for characterizing an eye topological structure based on an eye image. The method includes obtaining an eye image of a person to be tested; extracting eye key points based on the eye image; calculating a characterization factor of the eye topological structure based on the eye key points, where the characterization factor includes the angle between the horizontal line of the lower eyelid margin and the line connecting the inner canthi, the ratio of the inner canthal distance to the pupil distance, the palpebral fissure inclination angle, and the inclination angle of the upper eyelid eyelashes. In the present invention, the eye topological structure is quantitatively characterized by the eye key points, so that the prediction of childhood astigmatism is made more rapid and effective.

Description

Technical Field

[0001] The present invention relates to the field of intelligent healthcare, and more particularly, to a method, device, medium, and program product for characterizing an eye topological structure based on an eye image. Background Art

[0002] Astigmatism is one of the most common types of refractive errors in children, which means that the image formed after the parallel light rays are refracted by the eye optical system is not a single focus, but two focal lines at different positions in space, resulting in blurred vision. For preschool children, persistent uncorrected astigmatism is a high-risk factor for amblyopia, seriously affecting the quality of life of children. Therefore, active clinical intervention is needed for astigmatism that may progress to amblyopia in preschool children.

[0003] However, it is not easy to intervene in astigmatism in preschool children clinically, manifested as missing the best treatment opportunity due to late detection. In addition, the inconvenience of the examination method is also a restrictive factor. Traditional methods rely on professional optometry instrument equipment, which is not conducive to the rapid screening of a large population. Therefore, for preschool children, finding a convenient and fast method for early identifying possible astigmatism is of great significance.

[0004] Currently, artificial intelligence technology has achieved remarkable results in the diagnosis of anterior segment diseases, demonstrating its effectiveness and broad development prospects in the intelligent diagnosis of ophthalmic diseases. However, there are still many challenges in the field of intelligent astigmatism prediction. Research shows that the occurrence of astigmatism is closely related to factors such as eyelid pressure and shape. The cornea is affected by the combined action of the zonal pressure from the upper eyelid and the surface rigidity of the cornea, resulting in the generation of with-the-rule astigmatism. However, it is difficult to comprehensively depict these complex spatial distribution characteristics only relying on traditional image analysis methods. To solve this problem, introducing a topological structure analysis method has significant advantages. The topological structure can model and characterize the spatial relationship of eye features from both global and local perspectives, helping to capture feature information closely related to the occurrence of astigmatism, such as eyelid pressure distribution and corneal morphological changes. Based on the eye images taken by a conventional camera, by constructing and analyzing the topological structure of eye features, the complex associations of astigmatism-related factors can be comprehensively characterized, providing a new technical path for realizing convenient and fast intelligent astigmatism prediction for preschool children, and having important research value.

[0005] Current eye structure analysis technologies have many challenges in the diagnosis and treatment of astigmatism eye diseases, such as difficult extraction of external eye structure characterization factors and being greatly affected by the imaging effect of the captured eye image data. Existing methods lack an accurate eye appearance structure characterization model for astigmatism diagnosis and have poor adaptability to clinical shooting data with large individual-scale morphological changes and diverse shooting environments. Therefore, how to establish an eye structure characterization model for astigmatism diagnosis is still a problem that needs to be studied. Summary of the Invention

[0006] In view of the above problems, the present invention provides a method for characterizing the eye topological structure based on eye images, which uses key points extracted by machine vision technology to characterize the eye topological structure, facilitating the prediction and rapid screening of children's astigmatism.

[0007] The present application (in the first aspect) discloses a method for characterizing the eye topological structure based on eye images, including: A method for characterizing the eye topological structure based on eye images, the method including:

[0008] Obtain an eye image of the person to be tested;

[0009] Extract eye key points and corneal area based on the eye image;

[0010] Calculate a characterization factor of the eye topological structure based on the eye key points and the corneal area, where the characterization factor includes the ratio of the vertical diameter to the horizontal diameter of the cornea and / or the ratio of the area of the upper eyelid covering the cornea.

[0011] Further, the ratio of the vertical diameter to the horizontal diameter of the cornea is expressed as:

[0012]

[0013] where, represents the vertical radius or vertical diameter of the cornea, and represents the horizontal radius or horizontal diameter of the cornea;

[0014] Optionally, the ratio of the vertical diameter to the horizontal diameter of the cornea can also be expressed as:

[0015]

[0016] where, y c represents the ordinate of the center point of the pupil of the left / right eye,

[0017] y l represents the ordinate of the end point of the vertical diameter of the pupil of the left / right eye,

[0018] x c represents the abscissa of the center point of the pupil of the left / right eye,

[0019] x d represents the abscissa of the end point of the horizontal diameter of the pupil of the left / right eye.

[0020] Further, the ratio of the area of the upper eyelid covering the cornea is the ratio of the area covered by the upper eyelid to the exposed corneal area;

[0021] Optionally, the calculation method of the ratio of the area of the upper eyelid covering the cornea is:

[0022]

[0023] Among them, E a represents the horizontal radius of the left / right eye pupil, and E b represents the vertical radius of the left / right eye pupil, and S out represents the exposed area of the cornea;

[0024] Optionally, the calculation formula for the upper eyelid covering area ratio of the cornea is expressed as:

[0025]

[0026] Among them, x d represents the abscissa of any endpoint of the horizontal diameter of the left / right eye pupil,

[0027] x c represents the abscissa of the center point of the left / right eye pupil,

[0028] y c represents the ordinate of the center point of the left / right eye pupil,

[0029] y l represents the ordinate of the endpoint of the vertical diameter of the left / right eye pupil.

[0030] Furthermore, the characterization factor further includes one or more of the following: the angle between the lower eyelid margin horizontal line and the inner canthus connection line, the ratio of the inner canthus distance to the pupil distance, the palpebral fissure inclination angle, and the upper eyelid lash inclination angle.

[0031] Furthermore, the characterization factor further includes the upper eyelid lash inclination angle, and the upper eyelid lash inclination angle is calculated based on the lash tip coordinates and the highest point coordinates of the upper eyelid;

[0032] Optionally, the calculation of the upper eyelid lash inclination angle is expressed as:

[0033]

[0034] Among them, the coordinates of the left / right eye lash tip are expressed as: (x e , y e ), and the coordinates of the highest point of the left / right eye upper eyelid are expressed as (x u , y u ).

[0035] Furthermore, the characterization factor further includes the ratio of the inner canthus distance to the pupil distance:

[0036]

[0037] Among them, represents the abscissa of the right eye inner canthus point, represents the abscissa of the left eye inner canthus point,

[0038] represents the abscissa of the center point of the right eye pupil, represents the abscissa of the center point of the left eye pupil, || represents taking the absolute value;

[0039] Optionally, the characterization factor further includes: the tilt angle of the palpebral fissure, and the tilt angle of the palpebral fissure is expressed as:

[0040]

[0041] where y o represents the ordinate of the outer canthus point of the left / right eye, and y i represents the ordinate of the inner canthus point of the left / right eye;

[0042] x o represents the abscissa of the outer canthus point of the left / right eye, and x i represents the abscissa of the inner canthus point of the left / right eye;

[0043] Optionally, the characterization factor further includes the angle between the horizontal line of the lower eyelid margin and the connecting line of the inner canthi, and the angle between the horizontal line of the lower eyelid margin and the connecting line of the inner canthi is expressed as:

[0044]

[0045] where y l represents the ordinate of the lowest point of the lower eyelid of the left / right eye, and y i represents the ordinate of the inner canthus point of the left / right eye; x l represents the abscissa of the lowest point of the lower eyelid of the left / right eye, and x i represents the abscissa of the inner canthus point of the left / right eye;

[0046] Optionally, the eye key points are obtained based on computer vision recognition and the coordinates of the eye key points are obtained; the characterization factor is calculated based on the coordinates of the eye key points.

[0047] The present application also discloses a method for predicting astigmatism based on eye characterization factors, and the method includes:

[0048] Obtain an eye image of the person to be tested;

[0049] Use the method for characterizing the eye topological structure based on the eye image to obtain the characterization factor of the eye topological structure of the person to be tested;

[0050] Input the characterization factor into a classifier, and predict whether the person to be tested has astigmatism according to the output of the classifier.

[0051] The second aspect of the present application discloses a system for characterizing the eye topological structure based on an eye image, including:

[0052] Acquisition module 201: used to acquire an eye image of a person to be tested;

[0053] Extraction module 202: used to extract eye key points based on the eye image;

[0054] Topological characterization module 203: used to obtain a characterization factor of the eye topological structure based on the eye key points, and the characterization factor includes the ratio of the vertical diameter to the horizontal diameter of the cornea.

[0055] A third aspect of the present application discloses a computer device, which includes: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions, and when the program instructions are executed, it is used to execute the steps of the above method.

[0056] A fourth aspect of the present application discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the above method.

[0057] A fifth aspect of the present application discloses a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the steps of the above method.

[0058] The present application has the following beneficial effects:

[0059] 1. The present invention provides a method for characterizing the eye topological structure based on an eye image, and uses eye key points and corneal area to characterize the eye topological structure, which is convenient for realizing the prediction and rapid screening of children's astigmatism.

[0060] 2. The present invention quantifies and characterizes for the first time the ratio of the area of the upper eyelid covering the cornea to the exposed corneal area. In previous studies, this index only had a vague distinction of size. In the present invention, this index is quantified and characterized through eye key points, thereby making the prediction of children's astigmatism faster and more effective. Description of the Drawings

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those skilled in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0062] Figure 1 It is a schematic flowchart of the method provided in the first aspect of the embodiment of the present invention;

[0063] Figure 2 It is a schematic diagram of the program product provided in the second aspect of the embodiment of the present invention;

[0064] Figure 3It is a schematic diagram of a computer device provided by an embodiment of the present invention;

[0065] Figure 4 It is a schematic diagram of the architecture of an exemplary computing device provided by an embodiment of the present invention;

[0066] Figure 5 It is a schematic diagram of a storage medium provided by an embodiment of the present invention;

[0067] Figure 6A It is a flowchart of a method for extracting external eye structure characterization factors provided by an embodiment of the present invention;

[0068] Figure 6B It is a flowchart of another method for extracting external eye structure characterization factors provided by an embodiment of the present invention;

[0069] Figure 7 It is a diagram showing the definition of key eye points provided by an embodiment of the present invention; where: 0 represents the outer canthus point of the right eye; 1 represents the 1 / 4 equal division point of the upper eyelid of the right eye; 2 represents the highest point of the upper eyelid of the right eye; 3 represents the 3 / 4 equal division point of the upper eyelid of the right eye; 4 represents the inner canthus point of the right eye; 5 represents the 1 / 4 equal division point of the lower eyelid of the right eye; 6 represents the lowest point of the lower eyelid of the right eye; 7 represents the 3 / 4 equal division point of the lower eyelid of the right eye; 8 represents the left end point of the horizontal diameter of the right cornea; 9 represents the right end point of the horizontal diameter of the right cornea; 10 represents the center point of the right cornea; 11 represents the end point of the eyelashes of the right eye; 12 represents the outer canthus point of the left eye; 13 represents the 1 / 4 equal division point of the upper eyelid of the left eye; 14 represents the highest point of the upper eyelid of the left eye; 15 represents the 3 / 4 equal division point of the upper eyelid of the left eye; 16 represents the inner canthus point of the left eye; 17 represents the 1 / 4 equal division point of the lower eyelid of the left eye; 18 represents the lowest point of the lower eyelid of the left eye; 19 represents the 3 / 4 equal division point of the lower eyelid of the left eye; 20 represents the left end point of the horizontal diameter of the left cornea; 21 represents the right end point of the horizontal diameter of the left cornea; 22 represents the center point of the left cornea; 23 represents the end point of the eyelashes of the left eye;

[0070] Figure 8 It is a schematic diagram of the calculation of structure parameter characterization factors provided by an embodiment of the present invention;

[0071] Figure 9 It is a structural diagram of an eye key point detection network combining topological structure characteristics provided by an embodiment of the present invention;

[0072] Figure 10 It is a schematic diagram of the design of image preprocessing provided by an embodiment of the present invention;

[0073] Figure 11 It is a schematic diagram of a topological structure constraint (taking rotation as an example) provided by an embodiment of the present invention;

[0074] Figure 12It is a schematic diagram of learning the correlation of multi-scale context features of eye key points provided by an embodiment of the present invention;

[0075] Figure 13 It is a schematic diagram of a Hourglass module structure provided by an embodiment of the present invention; Detailed implementation manners

[0076] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0077] In some processes described in the specification, claims and the above-mentioned drawings of the present invention, a plurality of operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear herein or may be executed in parallel. The serial numbers of the operations, such as S101, S102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit that "first" and "second" are of different types.

[0078] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0079] Figure 1 It is a schematic diagram of a method for representing the eye topology structure based on an eye image evaluation provided by an embodiment of the present invention. Specifically, the method includes the following steps:

[0080] S101: Obtain an eye image of the person to be tested;

[0081] In some embodiments, taking a photo of the eyes of the person to be tested with a camera is the eye image, and the eye photo includes both eyes completely.

[0082] In some embodiments, take a front photo of the person to be tested, and the front photo includes a complete binocular region. Intercepting the binocular region is the eye image.

[0083] S102: Locate the eye key points based on the eye image;

[0084] Clinically, the relatively consistent experience of ophthalmologists shows that children's ocular topological structure information such as the inclination angle of the upper eyelid lashes, the ratio of the vertical diameter to the horizontal diameter of the cornea, the ratio of the intercanthal distance to the pupil distance, the inclination angle of the palpebral fissure, and the angle between the horizontal line of the lower eyelid margin and the connecting line of the inner canthi are potential leading factors for assisting in the diagnosis of astigmatism. However, the degree of eyelid coverage and the ratio of the vertical diameter to the horizontal diameter of the cornea used in previous studies are only vaguely qualitative magnitudes, and in the present invention, this index is quantified for the first time.

[0085] In some embodiments, such as Figure 6A shows the overall process of extracting the external eye topological structure characterization factors. For the multi-perspective eye images of children, first, a preprocessing filtering operation is performed to enhance the image quality. Then, the input is branched into two parts to respectively extract key element information. The first branch is the eye key point extraction network. The image undergoes multi-scale feature extraction through the backbone network and the feature pyramid, then point and box regression, as well as key point classification, and finally, a non-maximum suppression post-processing operation is performed to output the position information of the eye key points of interest. The other branch is the corneal segmentation network. The input is a coding-decoding architecture that fuses multi-scale features, and finally, the area of the exposed cornea is obtained. The output extraction results of the two branches are input into the structure parameter characterization model, and through mathematical relationship calculations, several typical external eye topological structure characterization factors are finally obtained. The technical route mainly includes ① the definition of the external eye topological structure model, ② the eye key point detection technology combined with topological structure characteristics, and ③ the multi-form adaptive corneal segmentation method.

[0086] (1) Definition of the external eye structure characterization model

[0087] Such as Figure 7 As shown, in order to more fully and detailedly characterize the topological structure of the eye, 24 eye key points are determined, with 12 key points corresponding to each of the left and right eyes. The 12 key points represent, in sequence, the outer canthus point, the 1 / 4 equal division point of the upper eyelid, the highest point of the upper eyelid, the 3 / 4 equal division point of the upper eyelid, the inner canthus point, the 3 / 4 equal division point of the lower eyelid, the lowest point of the lower eyelid, the 1 / 4 equal division point of the lower eyelid, the left endpoint of the corneal transverse diameter, the right endpoint of the corneal transverse diameter, the corneal center point, and the end point of the eyelashes.

[0088] Table 1 Correspondence diagram of key point names

[0089]

[0090]

[0091] Assume that the positions of several main key points obtained by key point extraction in the two-dimensional coordinate system are respectively represented as:

[0092] 1. The inner canthus points of the left and right eyes are respectively represented as:

[0093] 2. The outer canthus points of the left and right eyes are respectively represented as:

[0094] 3. The highest points of the upper eyelids of the left and right eyes are respectively represented as:

[0095] 4. The lowest points of the lower eyelids of the left and right eyes are respectively represented as:

[0096] 6. The center points of the pupils of the left and right eyes are respectively represented as:

[0097] 7. The ends of the eyelashes of the left and right eyes are respectively represented as:

[0098] 8. The endpoints of the horizontal diameters of the pupils of the left and right eyes are respectively represented as:

[0099] Taking the left eye as an example, according to the mathematical expressions of geometric relationships, it can be calculated that

[0100] The horizontal diameter of the cornea is The vertical diameter of the cornea is

[0101] In the two-dimensional coordinate system, the direction of the eyelashes is measured. Taking the vertical downward direction as 0°, the angle gradually increases in the counterclockwise direction.

[0102] In the two-dimensional coordinate system, the inclination angle of the palpebral fissure is measured. Taking the horizontal rightward direction as 0°, the angle gradually increases in the counterclockwise direction. Similarly, when measuring the angle between the horizontal line of the lower eyelid margin and the inner canthus connection line in the two-dimensional coordinate system, taking the horizontal rightward direction as 0°, the angle gradually increases in the counterclockwise direction.

[0103] S103: Calculate the characterization factors of the eye topological structure based on the key eye points. The characterization factors include one or more of the following: the ratio of the area of the upper eyelid covering the cornea, the angle between the horizontal line of the lower eyelid margin and the inner canthus connection line, the ratio of the inner canthus distance to the pupil distance, the palpebral fissure inclination angle, the ratio of the vertical diameter to the horizontal diameter of the cornea, and the inclination angle of the upper eyelid eyelashes.

[0104] The calculation expressions for the final 6 parameter characterization factors are:

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111] When calculating the corresponding characterization factor of the right eye in the above calculation formula, the corresponding subscript is replaced by r.

[0112] Use the eye key point detection technology combined with topological structure characteristics to extract the key points of the eye image. The proposed eye key point detection network combined with topological structure characteristics adopts a multi-task architecture. First, perform an affine transformation on the input eye image, then perform multi-scale feature extraction, and perform an affine transformation on the output heat map and offset map to achieve the constraint of the topological structure. The overall process is as Figure 9 shown.

[0113] 1) Image preprocessing based on Gabor affine transformation

[0114] Under the shooting conditions of the data collected in this study, there is a situation where the facial lighting environment is complex, which will affect the detection effect of the algorithm. In traditional image data processing methods, the Gabor filter extracts relevant features at different scales and different directions in the frequency domain, and is often used for texture recognition and has achieved good results. To a certain extent, it can highlight the eye features. On the other hand, the Gabor filter processes less data and is fast. Therefore, the Gabor preprocessing process of the original image in this project is to extract the feature maps in three directions: 0°, 45°, and 90° respectively, obtain a feature map in each direction, synthesize a new image with each feature map as a channel, and finally perform grayscale processing ( Figure 10 ).

[0115] 2) Topological structure constraint

[0116] Inspired by the equivalent landmark transformation, this study proposes a topological structure constraint method based on affine transformation, constructs additional constraints according to the overall properties of the topological structure between landmark points, so that the network can learn more robust landmark point features.

[0117] Given an image X and an affine transformation T, X′ = T(X, θ) represents the result of the image X and the result after the affine transformation T, where θ is the affine transformation parameter. At this time, the landmark point coordinates on the image X′ will also undergo the same affine transformation. Therefore, the result obtained by inputting the image X′ into the deep neural network f should follow the same affine transformation as the result obtained by inputting the original image X into the neural network f, that is, there is f(T(X, θ)) = T(f(X, θ)), as Figure 11 shown.

[0118] Based on this, by leveraging the self-similar property of the topological structure formed by the eye key points in the image after affine transformation, an additional constraint can be constructed by comparing the similarity degrees between the heat maps and offset maps output by the network before and after the image transformation respectively. This enables the network to capture the structural information contained between the eye key points, thereby extracting more robust features. In this study, the mean squared error loss function is used to penalize the prediction differences of the network before and after the image transformation to construct the structure invariance loss.

[0119] L stucture = ||f(T(X,θ)) - T(f(X),θ)|| 2 (1)

[0120] Among them, L stucture represents the structure invariance loss, X represents the eye image, T() represents the affine transformation, θ represents the parameters of the affine transformation, f() represents the output of the multi-scale feature extraction network, and || || 2 represents the mean squared error.

[0121] 3) Network structure design

[0122] The FPN feature map pyramid structure is for better detection effect on small-sized human eyes. However, in the context of the corresponding human eye pictures taken in this project, the proportion of the human eye area in the collected image in the original image is moderate. Therefore, the P2 and P6 layer structures of the feature pyramid network are removed, which can ensure the detection results of medium-sized human eyes while reducing unnecessary network inference calculation processes.

[0123] Meanwhile, in the eye key point detection, the combination of the deep network and the key point topological structure can also be considered. In the network design, the effective information transmission between feature extraction and key point positions is fully considered. The deep network is used to extract image features and the key point topological structure, such as the fully connected layer or similar structures, is introduced in the last few layers to capture more high-level semantic information and longer-distance associations, and the accurate positioning of the key points is achieved through joint optimization of network parameters and topological structure.

[0124] In this section, a multi-branch dilated convolution structure suitable for eye key point localization is designed and applied to the decoder part of the model. Each branch uses dilated convolution layers with different dilation rates, and different branches extract context information at different scales, enabling the decoder to capture the correlation between pixel features to a greater extent during the process of fusing low-level features and high-level features, and thus better restoring information such as the spatial position of the image.

[0125] Define the parameters of the multi-branch dilated convolution structure as (k, R), where k represents the convolution kernel size and R represents the combination of dilation rates of each branch, that is, R = (r1, r2,..., r n), where r1, r2, …, r n represent n different expansion rates on n branches respectively.

[0126] The feature maps of different scales extracted by dilated convolutions with different expansion rates are respectively passed through batch normalization and ReLU activation layers, then concatenated in channels, and then the number of channels of the new concatenated features is reduced using 1×1 convolution, and the result is input into the subsequent upsampling layer. In this way, not only the problem of lack of utilization of context information in the feature fusion process is solved, but also the network can better focus on multi-scale features without increasing the number of parameters and computational complexity, thereby improving the accuracy of the network in predicting eye key points.

[0127] 4) Loss function design

[0128] For human eye key point detection, since there is no need to perform dense point regression on the face and the computational complexity of dense point regression is relatively large, the dense point regression loss function is removed from the loss function. The optimized human eye key point loss function includes human eye classification, human eye bounding box position regression, and human eye key point regression loss functions.

[0129]

[0130] In the formula, p i in is the probability that the i-th anchor box predicted by the network contains a human eye, and i , t y , t w , t h , are the coordinates of the predicted anchor box and the coordinates of the data label respectively, including 4 positioning data of the human eye bounding box: t x , t y , t w , t h . In , for the prediction of 22 key points of the human eye, the regression loss function for the human eye key points, for the 22 key points of the human eye is:

[0131]

[0132] represent the predicted coordinates of 22 key points and the true coordinates of 22 points in the label respectively.

[0133] Use the multi-modal adaptive corneal segmentation method to extract the corneal exposure area

[0134] For the illumination and shadow changes caused by shooting angles and other factors in children's eye images, which result in blurred corneal edges, and the significant differences in the exposed corneal shapes under different resolutions and different degrees of eyelid occlusion of images captured by different devices, such as severe occlusion of eyelashes and eyelids, ineffective off-axis rotation, motion blur, and irregular reflections in the eye region. To address these issues, this section proposes a corneal segmentation scheme that is robust to noise and designs an improved stacked Hourglass network ( Figure 13 ), to

[0135] ① parameterize the boundary of the cornea;

[0136] ② simultaneously design the dense connection method between network layers to enable better transmission of the gradient flow containing real corneal boundary information;

[0137] ③ fuse the deep network and the shallow network together to form a multi-scale network to better capture rough and fine details;

[0138] ④ for the target region, change its size and input images of different sizes into the designed network respectively to obtain the global and regional features of the cornea;

[0139] ⑤ finally connect a SegNet decoder and a pixel classification layer to independently classify each pixel into the corneal and background regions, realizing the precise segmentation of children's corneas.

[0140] 1) Improved stacked Hourglass feature extraction network

[0141] The Hourglass network is a symmetric and fully convolutional architecture, similar to the U-Net, where the feature map is downsampled through pooling operations and then its size is restored using nearest-neighbor upsampling. At each scale level, a residual connection is applied to connect the corresponding layers between the two sides of the Hourglass. This bottom-up and top-down architecture is stacked multiple times to capture long-range context information. Therefore, even the occluded part of the eyeball may be predicted by capturing the spatial relationship between the corneal boundaries.

[0142] To keep the size of the input unchanged throughout the network and make the network lightweight, the original Hourglass network is adjusted by making four improvements:

[0143] (1) The stride of the initial convolutional layer with a kernel size of 7×7 is changed from 2 to 1,

[0144] (2) Remove the initial max-pooling layer,

[0145] (3) The number of channels in each layer of the network is reduced from 256 to 64,

[0146] (4) Each Hourglass module processes at 4 different image scales and is stacked 3 times. At the end of each Hourglass module, 64 fine feature maps are combined through a 1×1 convolutional layer to generate 6 maps: 2 for the segmentation mask, 2 for the pupil, and the remaining 2 for the cornea. Each pair consists of a foreground map and a background map, which are further processed through a softmax layer to obtain a binary mask. The stacked Hourglass network structure is as Figure 13 shown.

[0147] Although the original stacked structure of Hourglass can better fuse multi-scale features and enhance detail representation, there may still be a problem of basic feature loss during the module cascading process. To address this, the improved stacked Hourglass structure adopts a dense connection method. Compared with the original Hourglass network, the improved network adds connections between the key layers at the input and output of the hourglass structure and all the previous key layers of the hourglass structures (the key layers refer to the input and output feature layers of each hourglass structure). In this way, each layer can directly access the features of all the previous layers, which helps the network utilize information more effectively and improve the feature reuse rate. At the same time, the enhanced dense connection ensures that information can circulate more widely in the network, thereby improving the quality of feature fusion and the overall representation ability of the network.

[0148] The hourglass structure of the Hourglass network is a deep learning module that uses symmetric downsampling and upsampling paths to achieve multi-scale feature fusion. This structure consists of a series of repeated hourglass modules. In the contraction path of each module, the spatial size of the feature map is reduced through pooling or strided convolution, while the feature depth is increased to capture more abstract features; in the expansion path, the spatial size of the feature map is gradually restored through upsampling to refine and enhance local details. At the bottleneck of the hourglass, the low-resolution feature map is fused with the upsampled feature map, and the high-resolution detail information is directly passed to the deep network using skip connections, so as to perform accurate keypoint detection at different scales. This design of the Hourglass network is particularly suitable for capturing small-scale structures and details in images and is widely used in visual tasks that require fine spatial localization.

[0149] For the task of eye key-point detection and extraction, it involves precisely locating specific parts of the eyes in a face image, such as the corneal edge, points on the eyelids, etc. These small-scale features of the eyes are very subtle. The eye region contains multiple key points, and the positional relationship between them is complex and requires precise modeling. The high structural similarity between the left and right eyes may also cause confusion. The improved stacked Hourglass network enhances dense connections, which is beneficial for capturing both the small-scale details of the eyes and the overall facial features, ensuring that information can flow and be reused fully from the shallow to the deep layers of the network, helping to retain and strengthen the key information of the eyes in the deep network. The upsampling path of the Hourglass network helps to refine features while maintaining a high resolution, which is crucial for precisely locating the key points of the eyes.

[0150] The Fully Convolutional Network (FCN) reduces its output dimension through downsampling, and its output features are coarser and more global, which is very useful for capturing global structures or context information. However, this makes it difficult to segment the fine details of the eye image, including the corneal boundary and non-corneal pixel regions occluded by eyelashes or reflections. To solve the above problems, an improved densely connected Hourglass network structure is proposed, which combines shallow, fine, deep, and global layers. It consists of multiple convolutional layers and pooling layers, fusing the network layers from shallow to deep to form a multi-scale network to better capture both rough and fine details. Multiple network layers upsample the features to the same size as the input image and perform a summation operation on their outputs.

[0151] Assume the size of the input image is 256×256. After a convolution with a size of 7*7 and a stride of 2, as well as a max pooling operation, the size of the feature map of the final input module is 64×64. Before each downsampling, the upper half is separated to retain the original scale information; after each upsampling, it is added to the data of the previous scale; between two downsamplings, three Residual modules are used to extract features; between two additions, one Residual module is used to extract features. The Residual module has a skip connection structure. The first row is the convolutional branch, which is composed of three convolutional layers with different kernel scales in series, with BatchNormalization and ReLU interspersed; the second row is the skip connection, which only contains one convolutional layer with a kernel scale of 1. The stride of all convolutional layers is 1, and the padding is 1, which does not change the data size, but only changes the data depth (number of channels). In order to prevent problems such as the loss of original feature information and bias during the multi-layer forward propagation of the network and enhance the reuse of basic features, the inputs and outputs of multiple hourglass structures are connected. The feature map F2 is the sum of the feature map F1 and the output feature map of the first hourglass structure, and the feature map F3 is the sum of the feature map F1, the feature map F2, and the output feature map of the second hourglass structure, and so on.

[0152] 1) Multi-scale information fusion

[0153] In addition to multi-scale fusion at the feature level, the input image is also designed to be segmented into blocks to generate different-sized inputs centered on the target corneal region. The features obtained by multiple inputs through the same network are fused through a fully connected layer and sent to the subsequent decoder for per-pixel prediction.

[0154] For each pixel in the image, three different-scale blocks are cropped from small to large to 56×56, 112×112, and 224×224 respectively. Each pixel is separately trained or tested, and finally the predicted labels of each pixel are merged into the corneal mask.

[0155] To ensure the reliability and accuracy of the network in corneal segmentation, the encoder of the stacked densely connected Hourglass network with enhanced feature propagation designed and the decoder of the SegNet deep convolutional network are combined. SegNet is a practical fully convolutional network for per-pixel semantic segmentation. It uses 13 layers of encoders with the same structure as the VGG16 core network, and at the same time removes the fully connected layers. The decoder of SegNet upsamples the low-resolution input feature map with pooling indices. The stacked densely connected Hourglass network has a better information gradient flow due to its densely connected structure. Each convolutional layer is connected to all convolutional layers in a feed-forward manner, which is very useful for strengthening the feature propagation of subsequent layers in the dense block.

[0156] Through this connection, the designed corneal segmentation network develops the network potential by reusing the features of previous layers, improving the efficiency of the model. The direct connection of the network is achieved by cascading layers with multiple inputs and one output to cascade the features. A dense block includes a basic convolutional layer (Conv), batch normalization (BN), and a rectified linear unit (ReLU), which are connected in a cascaded manner. In the transition layer of the decoder, the max pooling after each dense block provides pooling indices, which are used for upsampling and upsampling for per-pixel semantic segmentation of the cornea and corneal boundary.

[0157] In the overall network, dense blocks are used to improve the feature extraction performance of the network, and the SegNet decoder is used to create a densely connected convolutional network for corneal segmentation. The bottleneck layer in the transition layer after each dense block in the dense encoder helps reduce the number of input feature maps and improve the computational efficiency. The transition layer is basically a combination of Conv1×1 and max pooling, separating two adjacent dense blocks.

[0158] 2) Loss function for fusion parameter fitting

[0159] For the mask output by corneal segmentation, further contour parameter fitting is performed. Guided by the prior information of the corneal shape, it better learns the boundary information of the corneal region and realizes refined segmentation of the boundary. The elliptical fitting of the contour means approximating the shape of the contour with an ellipse. When the major axis and minor axis of this ellipse are equal, it is a circle. The basic idea of elliptical fitting is: for a given set of sample points on a plane, find an ellipse that is as close as possible to these sample points. That is, a set of data in the image is fitted with an ellipse equation as a model, so that a certain ellipse equation can satisfy these data as much as possible, and the parameters of this ellipse equation are calculated.

[0160] The total loss function of the final model consists of three parts: inner iris segmentation loss, outer corneal segmentation loss, and boundary parameter fitting loss.

[0161] L = λ1L fit + λ2L inner + λ3L outter

[0162] where L fit 、L inner and L outter represent the parameter fitting loss, the segmentation losses of the inner iris and outer cornea respectively. The coefficients λ1, λ2, λ3 are set to 1 respectively to make the ranges of these loss values comparable. The loss function form of the segmentation part is the cross-entropy loss, and the parameter fitting part is the mean squared error loss. For the two binary segmentation losses of L inner and L outter the cross-entropy loss function is as follows:

[0163]

[0164] where y and represent the true value and the predicted value respectively. The form of the parameter fitting loss is as follows:

[0165]

[0166] where n is the number of samples, x i represents the true value of the i-th sample, and y i is the predicted value of the i-th sample by the model. The parameters finally output by the parameter fitting are the center position, the major axis diameter, and the minor axis diameter of the ellipse.

[0167] The present invention also discloses a method for predicting astigmatism based on eye characterization factors, and the method includes:

[0168] Obtaining an eye image of the person to be tested;

[0169] Using the above method for characterizing the eye topological structure based on the eye image to obtain the characterization factors of the eye topological structure of the person to be tested;

[0170] Inputting the characterization factors into a classifier, and predicting whether the person to be tested has astigmatism according to the output of the classifier.

[0171] In some embodiments, subjects in a test set are collected, including children with astigmatism and children with normal vision. Eye images of the subjects in the test set are obtained. The method for characterizing the eye topological structure based on the eye image is used to obtain the characterization factors of the eye topological structure of the person to be tested. The columns of the feature set include the ratio of the vertical diameter to the horizontal diameter of the cornea and the ratio of the area covered by the upper eyelid to the corneal area; the feature set and the corresponding labels (astigmatism or normal vision) are input into a support vector machine model for training, and the trained support vector machine is the classifier used for prediction. The classifier can be used to predict whether the person to be tested has astigmatism according to the ratio of the vertical diameter to the horizontal diameter of the cornea and the ratio of the area covered by the upper eyelid to the corneal area of the new person to be tested input.

[0172] In some embodiments, the columns of the feature set include the ratio of the area covered by the upper eyelid to the exposed corneal area, the inclination angle of the upper eyelid cilia, the ratio of the vertical diameter to the horizontal diameter of the cornea, the ratio of the inner canthal distance to the pupil distance, the inclination angle of the palpebral fissure, and the angle between the horizontal line of the lower eyelid margin and the inner canthal connection line; the feature set and the corresponding labels (astigmatism or normal vision) are input into a logistic regression model for training, and the trained model is the classifier used for prediction.

[0173] When it is necessary to predict whether a person to be tested has astigmatism, the ratio of the area of the upper eyelid covering the cornea to the exposed cornea area, the inclination angle of the upper eyelid cilia, the ratio of the vertical diameter to the horizontal diameter of the cornea, the ratio of the inner canthal distance to the pupil distance, the inclination angle of the palpebral fissure, and the included angle between the horizontal line of the lower eyelid margin and the inner canthal connection line are obtained from the eye image of the person to be tested. These features are input into a classifier, and based on the output of the classifier, it is predicted whether the person to be tested has astigmatism.

[0174] Figure 3 It is a schematic diagram of a computer device provided by an embodiment of the present invention. As Figure 3 shown, the device may include: one or more processors and one or more memories; wherein, computer-readable code is stored in the memory, and when the computer-readable code is run by the one or more processors, the above-described method can be executed.

[0175] The processor in this embodiment may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, operations and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc., and may be of the X86 architecture or the ARM architecture.

[0176] Generally speaking, the various example embodiments of the present disclosure may be implemented in hardware or special circuits, software, firmware, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that can be executed by a controller, a microprocessor, or other computing devices. When aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented as non-limiting examples in hardware, software, firmware, special circuits or logic, general hardware or controllers or other computing devices, or some combination thereof.

[0177] For example, the method or device according to the embodiments of the present disclosure may also be implemented by means of Figure 4 the architecture of the computing device 3000 shown. As Figure 4As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, input / output components 3060, a hard disk 3070, and so on. Storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the methods provided by this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 4 the architecture shown is merely exemplary, and when implementing different devices, one or more components in the computing device shown may be omitted according to actual needs. Figure 4

[0178] An embodiment of the present invention also provides a computer-readable storage medium, such as Figure 5 shown, which is a schematic diagram of the storage medium provided by an embodiment of the present invention. Computer-readable instructions 4010 are stored on the computer storage medium 4020. When the computer-readable instructions 4010 are run by a processor, the methods according to the embodiments of the present disclosure described with reference to the above figures can be executed. The computer-readable storage medium in the embodiments of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memories for the methods described herein are intended to include, but are not limited to, these and any other suitable types of memories. It should be noted that the memories for the methods described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0179] An embodiment of the present disclosure also provides a computer program product or a computer program, which, when executed by a processor, implements the steps of the above method, such as Figure 2 shown, the computer program product or the computer program includes:

[0180] Acquisition module 201: configured to acquire an eye image of a person to be tested;

[0181] Extraction module 202: configured to extract eye key points based on the eye image;

[0182] Topological characterization module 203: configured to calculate a characterization factor of an eye topological structure based on the eye key points, where the characterization factor includes a ratio of a vertical diameter to a horizontal diameter of a cornea.

[0183] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, the program segment, or the part of code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0184] Generally speaking, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that can be executed by a controller, a microprocessor, or other computing devices. When aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, devices, systems, technologies, or methods described herein may be implemented as non-limiting examples in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof.

[0185] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above may refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0186] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0187] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional unit in various embodiments of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0189] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art should understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for characterizing the eye's topological structure based on ocular surface images, characterized in that, The method includes: S1: Obtain an eye image of the person to be tested; S2: Extract eye key points and the exposed corneal area based on the eye image; S3: Calculate a characterization factor of the eye topology structure based on the eye key points and the exposed corneal area. The characterization factor includes: the ratio of the vertical diameter to the horizontal diameter of the cornea or the ratio of the upper eyelid covering area of the cornea. The ratio of the upper eyelid covering area of the cornea is the ratio of the covered area by the upper eyelid to the exposed corneal area. The calculation formula of the ratio of the upper eyelid covering area of the cornea is expressed as: Among them, represents the abscissa of any endpoint of the horizontal diameter of the pupil of the left / right eye, represents the abscissa of the center point of the pupil of the left / right eye, represents the ordinate of the center point of the pupil of the left / right eye, The ordinate of the endpoint representing the vertical diameter of the pupil of the left / right eye, represents the exposed corneal area.

2. The method for characterizing the eye topology based on the ocular surface image according to claim 1, wherein The characterization factor includes the ratio of the vertical diameter to the horizontal diameter of the cornea and the ratio of the upper eyelid covering area of the cornea; The ratio of the vertical diameter to the horizontal diameter of the cornea is expressed as: Among them, represents the vertical radius or vertical diameter of the cornea, represents the horizontal radius or horizontal diameter of the cornea; Or, the ratio of the vertical diameter to the horizontal diameter of the cornea can also be expressed as: Ratio of corneal vertical diameter to horizontal diameter= Among them, represents the ordinate of the center point of the pupil of the left / right eye, The ordinate of the endpoint representing the vertical diameter of the pupil of the left / right eye, represents the abscissa of the center point of the pupil of the left / right eye, The abscissa representing the endpoints of the horizontal diameter of the pupil of the left / right eye.

3. The method for characterizing the eye topology based on the ocular surface image according to claim 1, wherein The calculation method of the ratio of the upper eyelid covering area of the cornea can also be expressed as: Among them, represents the horizontal radius of the left / right eye pupil, represents the vertical radius of the left / right eye pupil, represents the exposed corneal area.

4. The method for characterizing the eye topology based on the ocular surface image according to claim 1, wherein, The characterization factor further includes one or more of the following: the angle between the lower eyelid margin horizontal line and the inner canthus connection line, the ratio of the inner canthal distance to the pupil distance, the palpebral fissure inclination angle, and the inclination angle of the upper eyelid eyelashes.

5. The method for characterizing the eye topology based on the ocular surface image according to claim 1, wherein The characterization factor further includes the inclination angle of the upper eyelid eyelashes, and the inclination angle of the upper eyelid eyelashes is calculated based on the coordinates of the eyelash tip and the highest point coordinate of the upper eyelid; The calculation of the inclination angle of the upper eyelid eyelashes is expressed as: Upper eyelid lash tilt angle = Among them, the coordinates of the ends of the left / right eyelashes are expressed as: , and the coordinates of the highest points of the left / right upper eyelids are expressed as ; Or the characterization factor further includes the ratio of the inner canthal distance to the pupil distance: Ratio of inner canthal distance to interpupillary distance= Among them, represents the abscissa of the inner canthus point of the right eye, represents the abscissa of the inner canthus point of the left eye. represents the abscissa of the center point of the right eye pupil, represents the abscissa of the center point of the left eye pupil, represents taking the absolute value; Or, the characterization factor further includes: the palpebral fissure inclination angle, and the palpebral fissure inclination angle is expressed as: Palpebral fissure tilt angle= Among them, represents the ordinate of the outer canthus point of the left / right eye, represents the ordinate of the inner canthus point of the left / right eye; represents the abscissa of the outer canthus point of the left / right eye, represents the abscissa of the inner canthus point of the left / right eye; Or, the characterization factor further includes the angle between the lower eyelid margin horizontal line and the inner canthus connection line, and the angle between the lower eyelid margin horizontal line and the inner canthus connection line is expressed as: The angle between the horizontal line of the lower eyelid margin and the connection line of the inner canthus = Among them, represents the ordinate of the lowest point of the left / right lower eyelid, represents the ordinate of the medial canthus point of the left / right eye; represents the abscissa of the lowest point of the left / right lower eyelid, represents the abscissa of the medial canthus point of the left / right eye; Further, the eye key points are obtained based on computer vision recognition and the coordinates of the eye key points are obtained; the characterization factor is calculated based on the coordinates of the eye key points.

6. A method for predicting astigmatism based on eye characterization factors, characterized in that, The method includes: Obtain an eye image of the person to be tested; Use the method described in any one of claims 1-5 to obtain the characterization factor of the eye topology structure of the person to be tested; Input the characterization factor into a classifier, and predict whether the person to be tested has astigmatism according to the output of the classifier.

7. A computer device, characterized in that, The device includes: a memory and a processor; the memory is used to store a computer program; the processor executes the computer program to implement the steps of the method described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and when the computer program is executed by the processor, the steps of the method described in any one of claims 1-6 are implemented.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method described in any one of claims 1-6 are implemented.

Images