Method, apparatus and program product for characterizing eye topology based on eye image

Abstract

The invention belongs to the field of intelligent medical treatment, and particularly relates to a method, device and program product for representing an eye topological structure based on an eye image, and the method comprises the steps: obtaining the eye image of a to-be-tested person; extracting eye key points based on the eye image; and calculating representation factors of an eye topological structure based on the eye key points, wherein the representation factors comprise an included angle between a lower eyelid margin horizontal line and an inner canthus connecting line, a ratio of an inner canthus distance to a pupil distance, an angle of inclination of the palpebral fissure and an angle of inclination of upper eyelid eyelashes. According to the method, the eye topological structure is quantitatively represented through the eye key points, so that the prediction of the child astigmatism is quicker and more effective.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical treatment, and more specifically, to a method, device, medium and program product for characterizing eye topology based on eye images. Background Art

[0002] Astigmatism is one of the most common types of refractive errors in children. It means that the image formed by the refraction of parallel light through the optical system of the eye is not a focal point, but two focal points in different positions in space, causing blurred vision. For preschool children, persistent uncorrected astigmatism is a high-risk factor for amblyopia, which seriously affects the quality of life of children. Therefore, active clinical intervention is required for astigmatism that may progress to amblyopia in preschool children.

[0003] However, it is not easy to intervene in preschool children's astigmatism in a timely manner in clinical practice, as late detection may easily miss the best time for treatment. In addition, the inconvenience of the examination method is also a limiting factor. The traditional method relies on professional optometry instruments and equipment, which is not conducive to rapid screening of large-scale populations. Therefore, for preschool children, it is of great significance to find a convenient and fast method to identify possible astigmatism at an early stage.

[0004] At present, artificial intelligence technology has achieved remarkable results in the diagnosis of anterior segment diseases, demonstrating its effectiveness and broad development prospects in the intelligent diagnosis of ophthalmic diseases. However, there are still many challenges in the field of intelligent prediction of astigmatism. Studies have shown that the occurrence of astigmatism is closely related to factors such as eyelid pressure and shape. The cornea is affected by the band pressure from the upper eyelid and the rigidity of the corneal surface, resulting in the generation of regular astigmatism. However, it is difficult to fully characterize these complex spatial distribution characteristics relying solely on traditional image analysis methods. To solve this problem, the introduction of topological structure analysis methods has significant advantages. The topological structure can model and characterize the spatial relationship of eye features from a global and local perspective, helping to capture feature information closely related to the occurrence of astigmatism, such as eyelid pressure distribution and corneal morphological changes. Based on eye images taken by conventional cameras, by constructing and analyzing the topological structure of eye features, the complex association of astigmatism-related factors can be fully characterized, providing a new technical path for realizing convenient and fast intelligent prediction of astigmatism in preschool children, which has important research value.

[0005] The current eye structure analysis technology faces many challenges in the diagnosis and treatment of astigmatism eye diseases, such as the difficulty in extracting eye external structure representation factors and the great influence of the imaging effect of the eye image data. The existing methods lack an accurate eye appearance structure representation model for astigmatism diagnosis, and have poor adaptability to differentiated clinical shooting data with large individual scale morphology changes and diverse shooting environments. Therefore, how to establish an eye structure representation model for astigmatism diagnosis is still a problem that needs to be studied. Summary of the invention

[0006] In view of the above problems, the present invention provides a method for characterizing the eye topology based on eye images, which uses key points extracted by machine vision technology to characterize the eye topology, thereby facilitating the prediction and rapid screening of children's astigmatism.

[0007] The present application (first aspect) discloses a method for characterizing an eye topology structure based on an eye image, comprising: a method for characterizing an eye topology structure based on an eye image, the method comprising:

[0008] Acquiring an eye image of a subject;

[0009] Extracting eye key points and corneal area based on the eye image;

[0010] The characterization factor of the eye topology is calculated based on the eye key points and the corneal area, and the characterization factor includes the ratio of the vertical diameter to the horizontal diameter of the cornea and / or the ratio of the upper eyelid covering the corneal area.

[0011] Further, the ratio of the vertical diameter to the horizontal diameter of the cornea is expressed as:

[0012]

[0013] Among them, represents the vertical radius or vertical diameter of the cornea, represents the horizontal radius or horizontal diameter of the cornea;

[0014] Optionally, the ratio of the vertical diameter to the horizontal diameter of the cornea can also be expressed as:

[0015]

[0016] Among them, y c The vertical coordinate of the center point of the left / right eye pupil.

[0017] y l The ordinate of the end point of the vertical diameter of the left / right eye pupil,

[0018] x c The horizontal coordinate of the center point of the left / right eye pupil.

[0019] x d Indicates the abscissa of the end points of the horizontal diameter of the left / right eye pupil.

[0020] Furthermore, the upper eyelid covered cornea area ratio is the ratio of the area covered by the upper eyelid to the exposed cornea area;

[0021] Optionally, the upper eyelid covered cornea area ratio is calculated as follows:

[0022]

[0023] Among them, E a Indicates the horizontal radius of the left / right eye pupil, E b Indicates the vertical radius of the left / right eye pupil, S out Indicates the exposed area of ​​the cornea;

[0024] Optionally, the upper eyelid covered cornea area ratio is calculated as:

[0025]

[0026] Among them, x d The horizontal coordinate of either end point of the horizontal diameter of the left / right eye pupil,

[0027] x c The horizontal coordinate of the center point of the left / right eye pupil.

[0028] y c The vertical coordinate of the center point of the left / right eye pupil.

[0029] y l Indicates the ordinate of the end point of the vertical diameter of the left / right eye pupil.

[0030] Furthermore, the characterization factors also include one or more of the following: the angle between the lower eyelid margin horizontal line and the inner canthus line, the ratio of the inner canthus distance to the pupil distance, the inclination angle of the palpebral fissure, and the inclination angle of the upper eyelid eyelashes.

[0031] Furthermore, the characterization factor also includes an upper eyelid eyelash tilt angle, which is calculated based on the eyelash tip coordinates and the upper eyelid highest point coordinates;

[0032] Optionally, the calculation of the upper eyelid eyelash tilt angle is expressed as:

[0033]

[0034] The coordinates of the left / right eyelash ends are expressed as: (x e ,y e ), the coordinates of the highest point of the upper eyelid of the left / right eye are expressed as (x u ,y u ).

[0035] Furthermore, the characterization factor also includes the ratio of the inner canthus distance to the pupil distance:

[0036]

[0037] in, Represents the horizontal coordinate of the inner canthus point of the right eye, Represents the horizontal coordinate of the inner canthus point of the left eye,

[0038] Represents the horizontal coordinate of the center point of the right pupil, Represents the horizontal coordinate of the center point of the left pupil, || represents the absolute value;

[0039] Optionally, the characterization factor further includes: a palpebral fissure inclination angle, and the palpebral fissure inclination angle is expressed as:

[0040]

[0041] Among them, y o Indicates the ordinate of the left / right eye outer canthus point, y i Indicates the ordinate of the inner canthus point of the left / right eye;

[0042] x o Indicates the horizontal coordinate of the left / right eye outer canthus point, x i Represents the horizontal coordinate of the inner canthus point of the left / right eye;

[0043] Optionally, the characterization factor further includes an angle between a lower eyelid margin horizontal line and a line connecting the inner canthus, and the angle between the lower eyelid margin horizontal line and the line connecting the inner canthus is expressed as:

[0044]

[0045] Among them, y l Indicates the ordinate of the lowest point of the lower eyelid of the left / right eye, y i Indicates the ordinate of the inner canthus point of the left / right eye; x l Indicates the horizontal coordinate of the lowest point of the lower eyelid of the left / right eye, x i Represents the horizontal coordinate of the inner canthus point of the left / right eye;

[0046] Optionally, the eye key points are obtained based on computer vision recognition and the coordinates of the eye key points are obtained; and the characterization factors are calculated based on the coordinates of the eye key points.

[0047] The present application also discloses a method for predicting astigmatism based on eye characterization factors, the method comprising:

[0048] Acquiring an eye image of a subject;

[0049] Using the method for characterizing the eye topology structure based on the eye image to obtain a characterization factor of the eye topology structure of the subject to be tested;

[0050] The characterization factors are input into a classifier, and whether the subject suffers from astigmatism is predicted based on the output of the classifier.

[0051] The second aspect of the present application discloses a system for representing eye topology based on eye images, comprising:

[0052] Acquisition module 201: used to acquire eye images of the subject;

[0053] Extraction module 202: used for extracting eye key points based on the eye image;

[0054] The topological characterization module 203 is used to obtain a characterization factor of the eye topological structure based on the eye key points, wherein the characterization factor includes a ratio of a vertical diameter to a horizontal diameter of the cornea.

[0055] The third aspect of the present application discloses a computer device, which includes: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions, and when the program instructions are executed, it is used to execute the steps of the above method.

[0056] A fourth aspect of the present application discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0057] A fifth aspect of the present application discloses a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0058] This application has the following beneficial effects:

[0059] 1. The present invention provides a method for characterizing the eye topology based on eye images, which uses eye key points and corneal area to characterize the eye topology, thereby facilitating the prediction and rapid screening of children's astigmatism.

[0060] 2. The present invention quantitatively characterizes for the first time the ratio of the area of ​​the cornea covered by the upper eyelid to the area of ​​the cornea exposed. This indicator used in previous studies only had a vague difference in size. The present invention quantitatively characterizes this indicator through key points of the eye, thereby making the prediction of astigmatism in children faster and more effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 is a schematic diagram of a method flow chart provided by the first aspect of an embodiment of the present invention;

[0063] Figure 2 is a schematic diagram of a program product provided by the second aspect of an embodiment of the present invention;

[0064] Figure 3is a schematic diagram of a computer device provided by an embodiment of the present invention;

[0065] Figure 4 is a schematic diagram of the architecture of an exemplary computing device provided by an embodiment of the present invention;

[0066] Figure 5 is a schematic diagram of a storage medium provided by an embodiment of the present invention;

[0067] Fig. 6A This is a flow chart of a method for extracting factors representing external eye structures provided by an embodiment of the present invention;

[0068] Figure 6B is a flow chart of another method for extracting factors representing external eye structures provided by an embodiment of the present invention;

[0069] Figure 7 is a diagram of the definition of key points of the eye provided by an embodiment of the present invention; wherein: 0 represents the outer canthus point of the right eye; 1 represents the 1 / 4 point of the upper eyelid of the right eye; 2 represents the highest point of the upper eyelid of the right eye; 3 represents the 3 / 4 point of the upper eyelid of the right eye; 4 represents the inner canthus point of the right eye; 5 represents the 1 / 4 point of the lower eyelid of the right eye; 6 represents the lowest point of the lower eyelid of the right eye; 7 represents the 3 / 4 point of the lower eyelid of the right eye; 8 represents the left end point of the transverse diameter of the right cornea; 9 represents the right end point of the transverse diameter of the right cornea; 10 represents the center point of the right cornea; 11 represents The end point of the right eyelash; 12 represents the outer canthus point of the left eye; 13 represents the 1 / 4 point of the upper eyelid of the left eye; 14 represents the highest point of the upper eyelid of the left eye; 15 represents the 3 / 4 point of the upper eyelid of the left eye; 16 represents the inner canthus point of the left eye; 17 represents the 1 / 4 point of the lower eyelid of the left eye; 18 represents the lowest point of the lower eyelid of the left eye; 19 represents the 3 / 4 point of the lower eyelid of the left eye; 20 represents the left end point of the transverse diameter of the left cornea; 21 represents the right end point of the transverse diameter of the left cornea; 22 represents the center point of the left cornea; 23 represents the end point of the left eyelash;

[0070] Figure 8 It is a schematic diagram of calculating a structural parameter characterization factor provided by an embodiment of the present invention;

[0071] Fig. 9 It is a network structure diagram of eye key point detection combined with topological structure characteristics provided by an embodiment of the present invention;

[0072] Fig.10 is a schematic diagram of an image preprocessing design provided by an embodiment of the present invention;

[0073] Fig.11 is a schematic diagram of a topological structure constraint (taking rotation as an example) provided in an embodiment of the present invention;

[0074] Fig.12is a schematic diagram of multi-scale context feature correlation learning of eye key points provided by an embodiment of the present invention;

[0075] Fig.13 is a schematic diagram of an Hourglass module structure provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0077] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The sequence numbers of the operations, such as S101, S102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0078] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0079] Figure 1 1 is a flow chart of a method for evaluating eye topology based on eye images provided by an embodiment of the present invention. Specifically, the method comprises the following steps:

[0080] S101: Acquire an eye image of a subject;

[0081] In some embodiments, a photograph of the eye of the subject taken with a camera is an eye image, and the eye photograph includes both complete eyes.

[0082] In some embodiments, a frontal photo of the subject is taken, the frontal photo includes the complete eye area, and the eye area is captured to obtain the eye image.

[0083] S102: Locating eye key points based on the eye image;

[0084] Clinical ophthalmologists have a relatively consistent experience that the topological structure of children's eyes, such as the tilt angle of upper eyelid eyelashes, the ratio of corneal vertical diameter / horizontal diameter, the ratio of inner canthus distance / pupillary distance, the tilt angle of palpebral fissure, and the angle between the lower eyelid horizontal line and the inner canthus line, are potential leading factors in assisting the diagnosis of astigmatism. However, the degree of eyelid coverage and the ratio of corneal vertical diameter / horizontal diameter used in previous studies are only vague qualitative measures, and this indicator is quantified for the first time in this invention.

[0085] In some embodiments, Fig. 6A The overall process of extracting the external ocular topological structure representation factors is shown. For the multi-view eye images of children, the pre-processing filtering operation is first performed to enhance the image quality. Then two branches are input to extract the key element information respectively. The first branch is the eye key point extraction network. The image is subjected to multi-scale feature extraction through the backbone network and feature pyramid, followed by point and box regression, and key point classification. Finally, the non-maximum suppression post-processing operation is performed to output the location information of the eye key points of interest. The other branch is the corneal segmentation network, which inputs the encoding and decoding architecture that integrates multi-scale features, and finally obtains the area of ​​exposed cornea. The output extraction results of the two branches are input into the structural parameter representation model. After mathematical relationship calculation, several typical external ocular topological structure representation factors are finally obtained. The technical route mainly includes ① the definition of the external ocular topological structure model, ② the eye key point detection technology combined with the topological structure characteristics, and ③ the multi-modal adaptive corneal segmentation method.

[0086] (1) Definition of the model for representing the external structure of the eye

[0087] like Figure 7 As shown in the figure, in order to fully and detailedly characterize the topological structure of the eye, 24 eye key points are determined, 12 key points for each eye. The 12 key points represent the outer canthus point, the 1 / 4 point of the upper eyelid, the highest point of the upper eyelid, the 3 / 4 point of the upper eyelid, the inner canthus point, the 3 / 4 point of the lower eyelid, the lowest point of the lower eyelid, the 1 / 4 point of the lower eyelid, the left end point of the corneal transverse diameter, the right end point of the corneal transverse diameter, the center point of the cornea, and the end point of the eyelash.

[0088] Table 1 Correspondence diagram of key point names

[0089]

[0090]

[0091] Assume that the positions of several key points extracted from key points in the two-dimensional coordinate system are expressed as:

[0092] 1. The inner canthus points of the left and right eyes are represented as:

[0093] 2. The outer canthus points of the left and right eyes are represented as:

[0094] 3. The highest points of the upper eyelids of the left and right eyes are expressed as:

[0095] 4. The lowest points of the left and right eyelids are:

[0096] 6. The pupil center points of the left and right eyes are represented as:

[0097] 7. The ends of the left and right eyelashes are represented as:

[0098] 8. The endpoints of the horizontal diameter of the left and right pupils are expressed as:

[0099] Taking the left eye as an example, according to the mathematical expression of geometric relationship, it can be calculated that:

[0100] The horizontal diameter of the cornea is The vertical diameter of the cornea is

[0101] The direction of the eyelashes is measured in a two-dimensional coordinate system, with vertical downwards being 0° and the angle gradually increasing in the counterclockwise direction.

[0102] When measuring the inclination angle of the palpebral fissure in a two-dimensional coordinate system, the horizontal right is 0°, and the angle gradually increases in the counterclockwise direction. Similarly, when measuring the angle between the lower eyelid horizontal line and the inner canthus line in a two-dimensional coordinate system, the horizontal right is 0°, and the angle gradually increases in the counterclockwise direction.

[0103] S103: Calculating the characterization factors of the eye topology based on the eye key points, the characterization factors include one or more of the following: the ratio of the upper eyelid covering the corneal area, the angle between the lower eyelid horizontal line and the inner canthus line, the ratio of the inner canthus distance to the pupil distance, the inclination angle of the palpebral fissure, the ratio of the vertical diameter of the cornea to the horizontal diameter, and the inclination angle of the upper eyelid eyelashes.

[0104] The calculation expressions of the final 6 parameter characterization factors are:

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111] When calculating the corresponding characterization factor of the right eye in the above calculation formula, the corresponding subscript is replaced by r.

[0112] The key points of the eye image are extracted using the eye key point detection technology combined with topological structure characteristics. The proposed eye key point detection network combined with topological structure characteristics adopts a multi-task architecture. First, the input eye image is affine transformed, and then multi-scale feature extraction is performed. The output heat map and offset map are affine transformed to realize the topological structure constraint. The overall process is as follows: Fig. 9 shown.

[0113] 1) Image preprocessing based on Gabor affine transformation

[0114] Under the shooting conditions of the data collected in this study, there are complex facial lighting environments, which will affect the detection effect of the algorithm. In traditional image data processing methods, Gabor filters extract relevant features at different scales and directions in the frequency domain, and are often used for texture recognition, and have achieved good results. To a certain extent, they can highlight eye features. On the other hand, Gabor filters process a small amount of data and are fast. Therefore, the Gabor preprocessing process of the original image in this project is to extract feature maps in three directions: 0°, 45°, and 90°, and obtain a feature map in each direction. Each feature map is used as a channel to synthesize a new image, and finally grayscale processing is performed ( Fig.10 ).

[0115] 2) Topological constraints

[0116] Inspired by equivalent landmark point transformation, this study proposes a topological structure constraint method based on affine transformation, which constructs additional constraints according to the overall properties of the topological structure between landmark points, so that the network can learn more robust landmark point features.

[0117] Given an image X and an affine transformation T, X′=T(X,θ) represents the image X and the result after affine transformation T, θ is the affine transformation parameter, and the coordinates of the landmarks on the image X′ will also undergo the same affine transformation. Therefore, the result obtained by inputting the image X′ into the deep neural network f should follow the same affine transformation as the result obtained by inputting the original image X into the neural network f, that is, there exists f(T(x,θ))=T(f(X,θ)), such as Fig.11 shown.

[0118] Based on this, by utilizing the self-similar nature of the topological structure formed by the eye key points in the image after affine transformation, an additional constraint can be formed by comparing the similarities between the heat map and the offset map output by the network before and after the image transformation, so that the network can capture the structural information contained in the eye key points and then extract more robust features. In this study, the mean square error loss function is used to penalize the prediction differences of the network before and after the image transformation to construct the structural invariance loss.

[0119] L structure =‖f(T(X,θ))-T(f(X),θ)‖ 2 (1)

[0120] Among them, L structure represents the structural invariance loss, X represents the eye image, T() represents the affine transformation, θ represents the parameters of the affine transformation, f() represents the output of the multi-scale feature extraction network, ‖‖ 2 represents the mean square error.

[0121] 3) Network structure design

[0122] The FPN feature map pyramid structure is designed to achieve better detection results for small-sized human eyes. However, in the context of taking pictures of human eyes in this project, the area of ​​human eyes in the captured images accounts for a moderate proportion of the original images. Therefore, the P2 and P6 layers of the feature pyramid network are removed to ensure the detection results of medium-sized human eyes in a targeted manner while reducing unnecessary network reasoning calculation processes.

[0123] At the same time, in eye key point detection, the combination of deep network and key point topology structure can also be considered. In the network design, full consideration should be given to the effective information transfer between feature extraction and key point position. The deep network should be used to extract image features and the key point topology structure should be introduced in the last few layers, such as fully connected layers or similar structures, to capture higher-level semantic information and longer-distance associations. The accurate positioning of key points can be achieved by jointly optimizing the network parameters and topology structure.

[0124] In this section, a multi-branch dilated convolution structure suitable for eye key point localization is designed and applied to the decoder part of the model. Each branch uses a dilated convolution layer with a different dilation rate. Different branches extract contextual information of different scales, so that the decoder can capture the correlation between pixel features to a greater extent in the process of fusing low-level features with high-level features, thereby better restoring information such as the spatial position of the image.

[0125] The parameters defining the multi-branch dilated convolution structure are (k, R), where k represents the size of the convolution kernel and R represents the combination of the expansion rates of each branch, i.e., R = (r1, r2, ..., r n ), where r1, r2, …, rn They represent n different expansion rates on n branches respectively.

[0126] The feature maps of different scales extracted by the dilated convolution with different expansion rates are passed through batch normalization and ReLU activation layers, and then the channels are concatenated. Then, 1×1 convolution is used to reduce the number of channels of the concatenated new features and input them into the next upsampling layer. In this way, not only the problem of lack of use of contextual information in the process of feature fusion is solved, but also the network can better focus on multi-scale features without increasing the number of parameters and calculations, thereby improving the accuracy of the network in predicting eye key points.

[0127] 4) Loss function design

[0128] For eye key point detection, since dense point regression of the face is not required and the amount of calculation of dense point regression is relatively large, the dense point regression loss function is removed from the loss function. The optimized eye key point loss function includes eye classification, eye frame position regression and eye key point regression loss functions.

[0129]

[0130] In the formula, p i is the probability that the i-th anchor box contains a human eye, predicted by the network. Data labels for human eyes; middle, They are the coordinates of the predicted anchor box and the coordinates of the data label, including 4 positioning data of the human eye box: x , t y , t w , t h ,exist In , the 22 key points of the human eye are predicted, and the regression loss function of the key points of the human eye is:

[0131]

[0132] They represent the predicted coordinates of the 22 key points and the true coordinates of the 22 points in the label.

[0133] Extracting corneal exposed area using multi-modality adaptive corneal segmentation method

[0134] In order to solve the problems of blurred corneal edges caused by changes in illumination and shadows due to shooting angles, as well as different resolutions of images taken by different devices and different degrees of eyelid occlusion, the exposed corneal shapes vary greatly, such as severe occlusion of eyelashes and eyelids, invalid off-axis rotation, motion blur, and irregular reflections in the eye area, etc., this section proposes a noise-robust corneal segmentation scheme and designs an improved stacked Hourglass network ( Fig.13 ),by

[0135] ① Parameterize the corneal boundary;

[0136] ② At the same time, design a dense connection method between network layers so that the gradient flow containing the real corneal boundary information can be better transmitted;

[0137] ③ Fuse the deep network and the shallow network together to form a multi-scale network to better capture coarse and fine details;

[0138] ④ For the target area, change its size and input images of different sizes into the designed network to obtain the global and regional features of the cornea;

[0139] ⑤Finally, connect a SegNet decoder and pixel classification layer to independently classify each pixel into cornea and background areas to achieve accurate segmentation of children's cornea.

[0140] 1) Improved stacked hourglass feature extraction network

[0141] The Hourglass network is a symmetric and fully convolutional architecture, similar to U-Net, where feature maps are downsampled by pooling operations and then restored to their dimensions using nearest neighbor upsampling. At each scale level, a residual connection is applied to connect the corresponding layers between the two sides of the Hourglass. This bottom-up, top-down architecture is stacked multiple times to capture long-range contextual information, so even occluded parts of the eyeball may be predicted by capturing the spatial relationship between the corneal boundaries.

[0142] In order to keep the input size constant throughout the network and make the network lightweight, the original Hourglass network is adapted by making four improvements:

[0143] (1) The stride of the initial convolutional layer with a kernel size of 7×7 is changed from 2 to 1,

[0144] (2) Remove the initial maximum pooling layer,

[0145] (3) The number of channels per layer of the network is reduced from 256 to 64,

[0146] (4) Each Hourglass module processes at 4 different image scales and is stacked 3 times. At the end of each Hourglass module, 64 fine feature maps are combined through 1×1 convolutional layers to generate 6 maps: 2 for segmentation masks, 2 for pupils, and the remaining 2 for corneas. Each pair consists of a foreground map and a background map, which are further processed through a softmax layer to obtain a binary mask. The stacked Hourglass network structure is shown in Figure 2. Fig.13 shown.

[0147] Although the original stacked structure of Hourglass can better fuse multi-scale features and enhance detail representation, there may still be a problem of basic feature loss during the module cascading process. To address this, the improved stacked Hourglass structure adopts a dense connection method. Compared with the original Hourglass network, the improved network adds connections between the key layers of the input and output parts of the hourglass structure and the key layers of all previous hourglass structures (the key layer refers to the input and output feature layer of each hourglass structure). In this way, each layer can directly access the features of all previous layers, which helps the network to use information more effectively and improve the reuse rate of features. At the same time, the enhanced dense connection ensures that information can circulate more widely in the network, thereby improving the quality of feature fusion and the overall representation ability of the network.

[0148] The hourglass structure of the Hourglass network is a deep learning module that uses symmetrical downsampling and upsampling paths to achieve multi-scale feature fusion. This structure consists of a series of repeated hourglass modules, each of which reduces the spatial size of the feature map through pooling or strided convolution in the contraction path, while increasing the feature depth to capture more abstract features; in the magnification path, the spatial size of the feature map is gradually restored through upsampling to refine and enhance local details. At the bottleneck of the hourglass, the low-resolution feature map is fused with the upsampled feature map, and the high-resolution detail information is directly passed to the deep network using jump connections, thereby performing accurate key point detection at different scales. This design of the Hourglass network is particularly suitable for capturing small-scale structures and details in images, and is widely used in visual tasks that require fine spatial positioning.

[0149] The task of detecting and extracting eye key points involves accurately locating specific parts of the eye from the face image, such as the edge of the cornea, points on the eyelids, etc. These small-scale features of the eye are very subtle, and the eye area contains multiple key points. The positional relationship between them is complex and needs to be accurately modeled. The high similarity in the structure of the left and right eyes may also cause confusion. The improved stacked Hourglass network enhances dense connections, which is beneficial for capturing small-scale details of the eye and overall facial features, ensuring that information can flow and reuse fully from the shallow to the deep layers of the network, which helps to retain and strengthen the key information of the eye in the deep network. The upsampling path of the Hourglass network helps to refine features while maintaining high resolution, which is crucial for accurately locating the key points of the eye.

[0150] The Fully Convolutional Network (FCN) reduces its output dimension by downsampling, and its output features are coarser and more global, which is very useful for capturing global structure or contextual information. However, this makes it difficult to segment the fine details of the eye image, including the corneal boundary and non-corneal pixel areas occluded by eyelashes or reflections. To address the above problems, an improved densely connected Hourglass network structure is proposed, combining shallow layers, fine layers, deep layers and global layers. It consists of multiple convolutional layers and pooling layers, which fuse network layers from shallow to deep to form a multi-scale network to better capture coarse and fine details. Multiple network layers upsample the features to the same size as the input image and sum their outputs.

[0151] Assuming that the size of the input image is 256×256, after a 7*7 convolution with a stride of 2 and a maximum pooling operation, the feature map size of the final input module is 64×64. Before each downsampling, the upper half is separated to retain the original scale information; after each upsampling, it is added to the data of the previous scale; between two downsamplings, three Residual modules are used to extract features; between two additions, one Residual module is used to extract features. The Residual module is a skip connection structure. The first line is a convolution branch, which is composed of three convolutional layers with different kernel scales in series, interspersed with BatchNormalization and ReLU; the second line is a skip connection, which contains only one convolutional layer with a kernel scale of 1. The stride of all convolutional layers is 1, and the padding is 1. The data size is not changed, and only the data depth (number of channels) is changed. In order to prevent the loss of original feature information and deviation during the multi-layer forward propagation of the network, and to enhance the reuse of basic features, the input and output of multiple hourglass structures are connected. Feature map F2 is the sum of feature map F1 and the output feature map of the first hourglass structure, feature map F3 is the sum of feature map F1, feature map F2, and the output feature map of the second hourglass structure, and so on.

[0152] 1) Multi-scale information fusion

[0153] In addition to multi-scale fusion at the feature level, the input image is also designed to be divided into blocks to generate inputs of different sizes centered on the target corneal area. The features obtained by multiple inputs through the same network are fused through a fully connected layer and sent to the subsequent decoder for pixel-by-pixel prediction.

[0154] For each pixel in the image, three patches of different scales are cropped from small to large to 56 × 56, 112 × 112, and 224 × 224. Each pixel is trained or tested separately, and the predicted label of each pixel is finally merged into the corneal mask.

[0155] To ensure the reliability and accuracy of the network in corneal segmentation, the encoder of the designed stacked densely connected Hourglass network with enhanced feature propagation is combined with the decoder of the SegNet deep convolutional network. SegNet is a practical fully convolutional network for pixel-by-pixel semantic segmentation, which uses a 13-layer encoder with the same structure as the VGG16 core network, while removing the fully connected layers. The decoder of SegNet upsamples the low-resolution input feature map with pooling indices. The stacked densely connected Hourglass network has better information gradient flow due to its densely connected structure, and each of its convolutional layers is connected to all convolutional layers in a feed-forward manner, which is very useful for strengthening the feature propagation of subsequent layers in dense blocks.

[0156] Through this connection, the designed cornea segmentation network exploits the network potential and improves the efficiency of the model by reusing the features of the previous layers. The direct connection of the network is to achieve the cascading of features by cascading layers with multiple inputs and one output. In a dense block, the basic convolution layer (Conv), batch normalization (BN) and rectified linear unit (ReLU) are included, which are connected in a cascade manner. In the transition layer of the decoder, the maximum pooling after each dense block provides pooling indices, which are used for up-pooling and up-sampling for pixel-by-pixel semantic segmentation of the cornea and corneal boundaries.

[0157] In the overall network, dense blocks are used to improve the feature extraction performance of the network, and the SegNet decoder is used to create a densely connected convolutional network for corneal segmentation. The bottleneck layer in the transition layer after each dense block in the dense encoder helps to reduce the number of input feature maps and improve computational efficiency. The transition layer is basically a combination of Conv1×1 and max pooling to separate two adjacent dense blocks.

[0158] 2) Loss function of fusion parameter fitting

[0159] For the mask output by corneal segmentation, contour parameter fitting is further performed. Under the guidance of prior information of corneal shape, the boundary information of the corneal area is better learned to achieve refined boundary segmentation. Ellipse fitting of contour refers to using an ellipse to approximate the shape of the contour. When the major axis and minor axis of the ellipse are equal, it is a circle. The basic idea of ​​ellipse fitting is: for a set of sample points on a given plane, find an ellipse that is as close to these sample points as possible. In other words, a set of data in the image is fitted with an ellipse equation as a model, so that a certain ellipse equation can satisfy these data as much as possible, and the various parameters of the ellipse equation are calculated.

[0160] The total loss function of the final model consists of three parts: inner iris segmentation loss, outer cornea segmentation loss and boundary parameter fitting loss.

[0161] L=λ1L fit +λ2L inner +λ3L outter

[0162] Among them, L fit , L inner and L outter They represent the parameter fitting loss, the segmentation loss of the inner iris and the outer cornea respectively. The coefficients λ1, λ2, and λ3 are set to 1 to make the ranges of these loss values ​​comparable. The loss function of the segmentation part is the cross entropy loss, and the parameter fitting part is the mean square error loss. inner and L outter Two binary segmentation losses, the cross entropy loss function is as follows:

[0163]

[0164] where y and Represent the true value and predicted value respectively. The parameter fitting loss is as follows:

[0165]

[0166] Where n is the number of samples, x i represents the true value of the i-th sample, y i is the model's predicted value for the i-th sample. The parameters output by parameter fitting are the center position, major axis diameter, and minor axis diameter of the ellipse.

[0167] The present invention also discloses a method for predicting astigmatism based on eye characterization factors, the method comprising:

[0168] Acquiring an eye image of a subject;

[0169] Using the above-mentioned method for representing the eye topology structure based on the eye image to obtain the representation factor of the eye topology structure of the subject;

[0170] The characterization factors are input into a classifier, and whether the subject suffers from astigmatism is predicted based on the output of the classifier.

[0171] In some embodiments, test set subjects are collected, including children with astigmatism and children with normal vision, eye images of the test set subjects are obtained, and a method for characterizing the eye topology based on eye images is used to obtain characterization factors of the eye topology of the subject, and the columns of the feature set include the ratio of the vertical diameter of the cornea to the horizontal diameter and the ratio of the cornea area covered by the upper eyelid; the feature set and the corresponding labels (astigmatism or normal vision) are input into a support vector machine model for training, and the trained support vector machine is the classifier used for prediction, and the classifier can be used to predict whether the subject suffers from astigmatism based on the ratio of the vertical diameter of the cornea to the horizontal diameter and the ratio of the cornea area covered by the upper eyelid.

[0172] In some embodiments, the columns of the feature set include the ratio of the area of ​​the cornea covered by the upper eyelid to the area of ​​the cornea exposed, the tilt angle of the upper eyelid eyelashes, the ratio of the vertical diameter of the cornea to the horizontal diameter, the ratio of the inner canthus distance to the pupil distance, the tilt angle of the palpebral fissure, and the angle between the lower eyelid horizontal line and the inner canthus line; the feature set and the corresponding labels (astigmatism or normal vision) are input into the logistic regression model training, and the trained model is the classifier used for prediction.

[0173] When it is necessary to predict whether the subject suffers from astigmatism, the ratio of the area of ​​the cornea covered by the upper eyelid to the area of ​​the cornea exposed, the inclination angle of the upper eyelid eyelashes, the ratio of the vertical diameter of the cornea to the horizontal diameter, the ratio of the inner canthus distance to the pupil distance, the inclination angle of the palpebral fissure, and the angle between the horizontal line of the lower eyelid margin and the inner canthus line are obtained based on the eye image of the subject. These features are input into the classifier, and the output of the classifier is used to predict whether the subject suffers from astigmatism.

[0174] Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present invention, such as Figure 3 As shown, the device may include: one or more processors, and one or more memories; wherein the memories store computer-readable codes, and when the computer-readable codes are run by the one or more processors, the method described above may be executed.

[0175] The processor in this embodiment can be an integrated circuit chip with signal processing capabilities. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, operations and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can be an X86 architecture or an ARM architecture.

[0176] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0177] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Figure 4 The architecture of the computing device 3000 shown in FIG. Figure 4As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as ROM 3030 or hard disk 3070, may store various data or files used for processing and / or communication of the method provided by the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 4 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 4 One or more components of a computing device are shown.

[0178] The embodiment of the present invention also provides a computer-readable storage medium, such as Figure 5 As shown, it is a schematic diagram of a storage medium provided by an embodiment of the present invention, and a computer readable instruction 4010 is stored on the computer storage medium 4020. When the computer readable instruction 4010 is executed by the processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0179] The present disclosure also provides a computer program product or a computer program, which implements the steps of the above method when executed by a processor, such as Figure 2 As shown, the computer program product or computer program comprises:

[0180] Acquisition module 201: used to acquire eye images of the subject;

[0181] Extraction module 202: used for extracting eye key points based on the eye image;

[0182] The topological characterization module 203 is used to calculate the characterization factor of the eye topological structure based on the eye key points, and the characterization factor includes the ratio of the vertical diameter to the horizontal diameter of the cornea.

[0183] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0184] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0186] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0189] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for characterizing eye topology based on ocular surface images, characterized in that: The method comprises: S1: Acquire the eye image of the subject; S2: extracting eye key points based on the eye image; S3: Calculate the characterization factor of the eye topology based on the eye key points, and the characterization factor includes: the ratio of the vertical diameter to the horizontal diameter of the cornea.

2. The method for characterizing eye topology based on ocular surface images according to claim 1, characterized in that: S2 is replaced by: extracting eye key points and corneal area based on the eye image, and S3 is replaced by: calculating the characterization factors of the eye topological structure based on the eye key points and corneal area, and the characterization factors include: the ratio of the upper eyelid covering the corneal area.

3. The method for characterizing eye topology based on ocular surface images according to claim 1, characterized in that: The characterization factors include the ratio of the vertical diameter to the horizontal diameter of the cornea and the ratio of the cornea area covered by the upper eyelid; Optionally, the ratio of the vertical diameter to the horizontal diameter of the cornea is expressed as: Among them, represents the vertical radius or vertical diameter of the cornea, represents the horizontal radius or horizontal diameter of the cornea; Optionally, the ratio of the vertical diameter to the horizontal diameter of the cornea can also be expressed as: Among them, y c The vertical coordinate of the center point of the left / right eye pupil. y l The ordinate of the end point of the vertical diameter of the left / right eye pupil, x c The horizontal coordinate of the center point of the left / right eye pupil. x d Indicates the abscissa of the end points of the horizontal diameter of the left / right eye pupil.

4. The method for characterizing eye topology based on ocular surface images according to claim 1, characterized in that: The upper eyelid covered cornea area ratio is the ratio of the area covered by the upper eyelid to the exposed cornea area; Optionally, the upper eyelid covered cornea area ratio is calculated as follows: Among them, E a Indicates the horizontal radius of the left / right eye pupil, E b Indicates the vertical radius of the left / right eye pupil, S out Indicates the exposed area of ​​the cornea; Optionally, the upper eyelid covered cornea area ratio is calculated as: Among them, x d The horizontal coordinate of either end point of the horizontal diameter of the left / right eye pupil, x c The horizontal coordinate of the center point of the left / right eye pupil. y c The vertical coordinate of the center point of the left / right eye pupil. y l Indicates the ordinate of the end point of the vertical diameter of the left / right eye pupil.

5. The method for characterizing eye topology based on ocular surface images according to claim 1, characterized in that: The characterization factors also include one or more of the following: the angle between the lower eyelid margin horizontal line and the inner canthus line, the ratio of the inner canthus distance to the pupil distance, the inclination angle of the palpebral fissure, and the inclination angle of the upper eyelid eyelashes.

6. The method for characterizing eye topology based on ocular surface images according to claim 1, characterized in that: The characterization factor also includes an upper eyelid eyelash tilt angle, which is calculated based on the eyelash tip coordinates and the upper eyelid highest point coordinates; Optionally, the calculation of the upper eyelid eyelash tilt angle is expressed as: The coordinates of the left / right eyelash ends are expressed as: (x e ,y e ), the coordinates of the highest point of the upper eyelid of the left / right eye are expressed as (x u ,y u ). Optionally, the characterization factor also includes the ratio of inner canthal distance to pupil distance: in, Represents the horizontal coordinate of the inner canthus point of the right eye, Represents the horizontal coordinate of the inner canthus point of the left eye, Represents the horizontal coordinate of the center point of the right pupil, Represents the horizontal coordinate of the center point of the left pupil, || represents the absolute value; Optionally, the characterization factor further includes: a palpebral fissure inclination angle, and the palpebral fissure inclination angle is expressed as: Among them, y o Indicates the ordinate of the left / right eye outer canthus point, y i Indicates the ordinate of the inner canthus point of the left / right eye; x o Indicates the horizontal coordinate of the left / right eye outer canthus point, x i Represents the horizontal coordinate of the inner canthus point of the left / right eye; Optionally, the characterization factor further includes an angle between a lower eyelid margin horizontal line and a line connecting the inner canthus, and the angle between the lower eyelid margin horizontal line and the line connecting the inner canthus is expressed as: Among them, y l Indicates the ordinate of the lowest point of the lower eyelid of the left / right eye, y i Indicates the ordinate of the inner canthus point of the left / right eye; x l Indicates the horizontal coordinate of the lowest point of the lower eyelid of the left / right eye, x i Represents the horizontal coordinate of the inner canthus point of the left / right eye; Optionally, the eye key points are obtained based on computer vision recognition and the coordinates of the eye key points are obtained; The characterization factor is calculated based on the coordinates of the eye key points.

7. A method for predicting astigmatism based on eye characterization factors, characterized in that: The method comprises: Acquiring an eye image of a subject; Using the method described in claims 1 to 6 to obtain a characterization factor of the eye topology of the subject; The characterization factors are input into a classifier, and whether the subject suffers from astigmatism is predicted based on the output of the classifier.

8. A computer device, characterized in that: The device comprises: a memory and a processor; the memory is used to store a computer program; the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images