Method, device and program product for extracting eye topology based on eye image

Abstract

The present invention belongs to the field of intelligent medical treatment, and specifically relates to a method, device and program product for extracting eye topology based on eye images. The method comprises obtaining an eye image of a subject to be tested; inputting the eye image into an eye key point extraction network to obtain eye key points; inputting the eye image into a corneal segmentation network to obtain a corneal segmentation area; inputting the eye key points and corneal segmentation area into an eye topology structure characterization model to obtain a characterization factor of the eye topology structure. The present invention uses eye key points and corneal area to characterize the eye topology structure. The eye key point extraction network introduces a classification head, a frame regression head and a key point regression head during feature extraction, so that the key point extraction is more accurate. The corneal segmentation network first improves the stacked Hourglass structure to be densely connected, and then performs elliptical fitting on the cornea after the cornea is segmented, so that the corneal segmentation is more accurate, so that the topological factors obtained by the present invention can be used for disease prediction.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical treatment, and more specifically, to a method, device, medium and program product for extracting eye topological structure based on eye images. Background Art

[0002] Astigmatism is one of the most common types of refractive errors in children. It means that the image formed by the refraction of parallel light through the optical system of the eye is not a focal point, but two focal points in different positions in space, causing blurred vision. The prevalence of astigmatism in children is high, and it is a social public health problem that needs to be solved urgently. For preschool children, persistent uncorrected astigmatism is a high-risk factor for amblyopia, which seriously affects the quality of life of children. Therefore, active clinical intervention is required for astigmatism that may progress to amblyopia in preschool children.

[0003] However, it is not easy to intervene in preschool children's astigmatism in a timely manner in clinical practice, as late detection may easily miss the best time for treatment. In addition, the inconvenience of the examination method is also a limiting factor. The traditional method relies on professional optometry instruments and equipment, which is not conducive to rapid screening of large-scale populations. Therefore, for preschool children, it is of great significance to find a convenient and fast method to identify possible astigmatism at an early stage.

[0004] At present, artificial intelligence technology has been widely used in the diagnosis of various anterior segment diseases with outstanding results, proving its effectiveness and development potential in the intelligent diagnosis of ophthalmic diseases. However, there are still challenges in the intelligent prediction of astigmatism. Previous studies have shown that the occurrence of astigmatism is closely related to factors such as eyelid pressure and shape. The cornea is subjected to the interaction of band pressure from the upper eyelid and the rigidity of the corneal surface, resulting in regular astigmatism in the eye. Therefore, based on eye images taken by conventional cameras, exploring new artificial intelligence methods to extract and characterize eye features related to the occurrence of astigmatism has important research value for realizing convenient and fast intelligent prediction and diagnosis of astigmatism in preschool children.

[0005] The current eye structure analysis technology faces many challenges in the diagnosis and treatment of astigmatism eye diseases, such as the difficulty in extracting eye external structure representation factors and the great influence of the imaging effect of the eye image data. The existing methods lack an accurate eye appearance structure representation model for astigmatism diagnosis, and have poor adaptability to clinical shooting data with large individual scale morphology changes and diverse shooting environments. Therefore, how to predict astigmatism after extracting eye features based on image data is still a problem that needs to be studied. Summary of the invention

[0006] In view of the above problems, the present invention provides a method for extracting eye topology based on eye images, which uses key points extracted by machine vision technology to characterize the eye topology, thereby facilitating the prediction and rapid screening of children's astigmatism.

[0007] The present application (first aspect) discloses a method for extracting eye topology structure based on an eye image, comprising:

[0008] Acquiring an eye image of a subject to be tested;

[0009] Inputting the eye image into an eye key point extraction network to obtain eye key points;

[0010] Inputting the eye image into a corneal segmentation network to obtain a corneal segmentation area;

[0011] The eye key points and corneal segmentation area are input into the eye topology structure representation model to obtain the representation factor of the eye topology structure.

[0012] Furthermore, the eye key point extraction network includes a multi-scale feature extraction network and a key point prediction network; the eye image is passed through the multi-scale feature extraction network to obtain extracted features (i.e., the first features), and the extracted features are passed through the key point prediction network to obtain the eye key points.

[0013] Furthermore, the initial eye key point extraction network includes an initial multi-scale feature extraction network, an initial key point prediction network, and a topological constraint layer; the training steps of the eye key point extraction network include:

[0014] Obtaining a training set of eye images, wherein the eye image training set includes annotated key points;

[0015] The training set of eye images is input into the initial multi-scale feature extraction network to obtain extracted features;

[0016] The extracted features are input into an initial key point prediction network to obtain a predicted key point heat map; a labeled key point heat map is obtained based on the labeled key points, and a prediction loss is obtained based on the predicted key point heat map and the labeled key point heat map;

[0017] The training set of eye images is affine transformed and then input into the initial multi-scale feature extraction network to obtain the extracted features of the affine transformation map (i.e., the second features); the extracted features are subjected to the same affine transformation to obtain the affine transformation of the extracted features (i.e., the third features); the structural invariance loss is the difference between the affine transformation of the extracted features and the extracted features of the affine transformation map;

[0018] The objective function of the initial eye key point extraction network includes the prediction loss and the structural invariance loss, and the multi-scale feature extraction network and the key point prediction network are obtained after the objective function is gradually optimized through iteration to reach the stopping condition; the multi-scale feature extraction network and the key point prediction network constitute the eye key point extraction network;

[0019] Optionally, the objective function of the initial eye key point extraction network is expressed as:

[0020]

[0021] Among them, L represents the objective function of the initial eye key point extraction network, represents the prediction loss, represents the loss of structural invariance;

[0022] Optionally, the structural invariance loss is expressed as:

[0023]

[0024] in, represents the structural invariance loss, X represents the eye image, T() represents the affine transformation, represents the parameters of the affine transformation, represents the output of the multi-scale feature extraction network, represents the mean square error.

[0025] Furthermore, the initial multi-scale feature extraction network includes an initial multi-scale feature pyramid and an initial multi-head attention layer; the training set of eye images is input into the initial multi-scale feature pyramid to obtain pyramid features, and the pyramid features are extracted through multi-head attention to obtain the extracted features, and the multi-head attention includes classification head attention, box regression head attention and key point regression head attention; the pyramid features are subjected to classification head attention to obtain classification head attention features; the pyramid features are subjected to box regression head attention to obtain eye socket attention features; the pyramid features are subjected to key point regression head attention to obtain key point attention features; the extracted features include the classification head attention features, the eye socket attention features and the key point attention features.

[0026] Furthermore, the eye image training set also includes a labeled classification head and a labeled eye socket; the prediction loss includes a predicted classification head loss, a predicted eye socket loss, and a predicted key point loss;

[0027] Optionally, the prediction loss is expressed as:

[0028]

[0029] in, represents the prediction loss,

[0030] represents the predicted classification head loss, where is the network's prediction of The probability that an anchor box is classified as containing a human eye, Is the data label of whether the anchor box contains the human eye;

[0031] represents the predicted orbital loss, They are the vertex coordinates of the four predicted anchor boxes and the vertex coordinates of the annotated eye sockets;

[0032] represents the predicted key point loss, Respectively represent the coordinates of the predicted key points of the human eye and the coordinates of the marked key points;

[0033] Optionally, a heat map and an offset map are obtained based on the extracted features, a real heat map and an offset map are obtained based on the marked key points, and the prediction loss is obtained based on the difference between the heat map and the offset map and the real heat map and the offset map.

[0034] Further, the eye image of the subject is input into a corneal segmentation network to obtain a corneal mask and an iris mask, wherein the corneal segmentation network includes a stacked Hourglass encoder and a SegNet decoder, wherein the stacked Hourglass includes three layers of Hourglass, the input of the second layer of Hourglass includes the output of the first layer of Hourglass, and the input of the third layer of Hourglass includes the output of the first layer of Hourglass and the output of the second layer of Hourglass;

[0035] Optionally, the input of the third-layer Hourglass is the sum of the output of the first-layer Hourglass and the output of the second-layer Hourglass;

[0036] Optionally, each layer of the hourglass includes an initial maximum pooling layer, a basic convolution layer, a batch normalization layer, a rectified linear unit, an upsampling layer, and an output layer. The first layer of the hourglass includes a basic convolution layer, a batch normalization layer, a rectified linear unit, an upsampling layer, and an output layer.

[0037] Optionally, the eye image is converted into eye images of different scales, and the eye images of different scales are passed through a cornea segmentation network, and the predicted label of each pixel is merged into a cornea mask and an iris mask;

[0038] Furthermore, the training steps of the cornea segmentation network include:

[0039] Get a training set of eye images;

[0040] The training set of eye images is input into the initial cornea segmentation network to obtain a cornea mask and an iris mask;

[0041] Performing contour fitting on the corneal mask to obtain contour parameters, wherein the contour fitting is to fit the corneal mask using an ellipse, and the contour parameters include a major axis diameter and a minor axis diameter of the ellipse;

[0042] The total loss function of the initial cornea segmentation network consists of three parts, including iris segmentation loss, cornea segmentation loss and boundary parameter fitting loss; the cornea segmentation network is obtained after iteratively optimizing the total loss function;

[0043] Optionally, the total loss function of the initial cornea segmentation network is expressed as follows:

[0044]

[0045] in, , and Represent contour fitting loss, iris segmentation loss and cornea segmentation loss respectively, and the coefficient Represents the weight coefficients of contour fitting loss, iris segmentation loss and cornea segmentation loss;

[0046] Optionally, the iris segmentation loss and the cornea segmentation loss are cross entropy losses, and the contour fitting loss is mean square error loss;

[0047] Optionally, the contour fitting loss is in the following form:

[0048]

[0049] Where n is the number of samples, represents the true value of the i-th sample, is the model's predicted value for the i-th sample.

[0050] A method for predicting astigmatism based on eye characterization factors, the method comprising:

[0051] Acquiring an eye image of a subject to be tested;

[0052] Using the method for extracting eye topology based on eye images to obtain a representation factor of the eye topology of the subject;

[0053] The characterization factors are input into a classifier, and whether the subject suffers from astigmatism is predicted based on the output of the classifier.

[0054] The second aspect of the present application discloses a system for extracting eye topology based on eye images, comprising:

[0055] Acquisition module 201: used to acquire eye images of the subject;

[0056] Key point extraction module 202: used for inputting the eye image into the eye key point extraction network to obtain eye key points;

[0057] Corneal area extraction module 203: used for inputting the eye image into the corneal segmentation network to obtain the corneal segmentation area;

[0058] Topological characterization module 204: used to calculate the characterization factors of the eye topological structure based on the eye key points and corneal area, the characterization factors including one or more of the following: the ratio of the upper eyelid covering the corneal area, the angle between the lower eyelid horizontal line and the inner canthus line, the ratio of the inner canthus distance to the pupil distance, the inclination angle of the palpebral fissure, the ratio of the vertical diameter of the cornea to the horizontal diameter, and the inclination angle of the upper eyelid eyelashes.

[0059] The third aspect of the present application discloses a computer device, which includes: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions, and when the program instructions are executed, it is used to execute the steps of the above method.

[0060] A fourth aspect of the present application discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0061] A fifth aspect of the present application discloses a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0062] This application has the following beneficial effects:

[0063] 1. The present invention provides a method for extracting eye topology based on eye images, using eye key points and corneal area to characterize the eye topology. The present invention provides an eye key point extraction network, which introduces a classification head, a frame regression head and a key point regression head during feature extraction to make key point extraction more accurate, thereby making the calculated topology more accurate.

[0064] 2. The topological structure to be calculated by the present invention regards the human eye as an elliptical structure. The corneal segmentation network of the present invention first improves the stacked Hourglass structure to be densely connected, and then performs elliptical fitting on the cornea after the cornea is segmented, so that the corneal segmentation is more accurate and the corneal area obtained based on the corneal segmentation network is more suitable for the characterization calculation of the eye topological structure in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0066] Figure 1 It is a schematic diagram of the network flow of eye key point extraction used in the method flow provided by the first aspect of the embodiment of the present invention;

[0067] Figure 2 is a schematic diagram of a program product provided by the second aspect of an embodiment of the present invention;

[0068] Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present invention;

[0069] Figure 4 is a schematic diagram of the architecture of an exemplary computing device provided by an embodiment of the present invention;

[0070] Figure 5 is a schematic diagram of a storage medium provided by an embodiment of the present invention;

[0071] Figure 6 This is a flow chart of a method for extracting factors representing external eye structures provided by an embodiment of the present invention;

[0072] Figure 7 : is a diagram of the definition of eye key points provided by an embodiment of the present invention; the eye key points represented in the figure are represented as follows; wherein: 0 represents the outer canthus point of the right eye; 1 represents the 1 / 4 dividing point of the upper eyelid of the right eye; 2 represents the highest point of the upper eyelid of the right eye; 3 represents the 3 / 4 dividing point of the upper eyelid of the right eye; 4 represents the inner canthus point of the right eye; 5 represents the 1 / 4 dividing point of the lower eyelid of the right eye; 6 represents the lowest point of the lower eyelid of the right eye; 7 represents the 3 / 4 dividing point of the lower eyelid of the right eye; 8 represents the left end point of the transverse diameter of the right cornea; 9 represents the right end point of the transverse diameter of the right cornea; 1 0 represents the center point of the right cornea; 11 represents the end point of the right eyelash; 12 represents the outer canthus point of the left eye; 13 represents the 1 / 4 point of the left upper eyelid; 14 represents the highest point of the left upper eyelid; 15 represents the 3 / 4 point of the left upper eyelid; 16 represents the inner canthus point of the left eye; 17 represents the 1 / 4 point of the left lower eyelid; 18 represents the lowest point of the left lower eyelid; 19 represents the 3 / 4 point of the left lower eyelid; 20 represents the left end point of the left corneal transverse diameter; 21 represents the right end point of the left corneal transverse diameter; 22 represents the center point of the left cornea; 23 represents the end point of the left eyelash;

[0073] Figure 8 It is a schematic diagram of calculating a structural parameter characterization factor provided by an embodiment of the present invention;

[0074] Fig. 9It is a network structure diagram of eye key point detection combined with topological structure characteristics provided by an embodiment of the present invention;

[0075] Fig.10 is a schematic diagram of an image preprocessing design provided by an embodiment of the present invention;

[0076] Fig.11 is a schematic diagram of a topological structure constraint (taking rotation as an example) provided by an embodiment of the present invention;

[0077] Fig.12 is a schematic diagram of multi-scale context feature correlation learning of eye key points provided by an embodiment of the present invention;

[0078] Fig.13 It is a schematic diagram of a flow chart of a corneal area extraction network provided by an embodiment of the present invention;

[0079] Fig.14 is a schematic diagram of an improved stacked Hourglass network structure provided by an embodiment of the present invention;

[0080] Fig.15 is a schematic diagram of an Hourglass module structure provided by an embodiment of the present invention;

[0081] Fig.16 is a schematic diagram of multi-size input of a corneal area extraction network provided by an embodiment of the present invention;

[0082] Fig.17 It is a schematic diagram of the overall network structure of eye key point detection provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0083] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0084] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The sequence numbers of the operations, such as S101, S102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0085] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0086] Figure 1 1 is a schematic diagram of a network flow of eye key point extraction used in a method for extracting eye topology structure based on an eye image provided by an embodiment of the present invention. Specifically, the method comprises the following steps:

[0087] S1: Acquire the eye image of the subject;

[0088] S2: Inputting the eye image into an eye key point extraction network to obtain eye key points, and calculating the eye topology based on the eye key points; the method for constructing the eye key point extraction network includes:

[0089] S101: Acquire a training set of eye images, wherein the training set of eye images includes annotated key points;

[0090] S102: extracting features from a training set of eye images to obtain a first feature;

[0091] S103: performing key point prediction based on the first feature to obtain a predicted key point heat map; obtaining a labeled key point heat map based on the labeled key points, and obtaining a prediction loss based on the difference between the predicted key point heat map and the labeled key point heat map;

[0092] S104: extracting features from the eye image training set after affine transformation to obtain a second feature; extracting the first feature after the same affine transformation to obtain a third feature; and obtaining a structural invariance loss based on the difference between the third feature and the second feature;

[0093] S105: The eye key point extraction network is obtained after the objective function is gradually optimized through iteration and a stopping condition is reached, wherein the objective function includes the prediction loss and the structural invariance loss.

[0094] Step 1: Obtain an eye image of the subject;

[0095] In some embodiments, a photograph of the eye of the subject taken with a camera is an eye image, and the eye photograph includes both complete eyes.

[0096] In some embodiments, a frontal photo of the subject is taken, the frontal photo includes the complete eye area, and the eye area is captured to obtain the eye image.

[0097] Step 2: Input the eye image into an eye key point extraction network to obtain eye key points;

[0098] Step 3: inputting the eye image into a corneal segmentation network to obtain a corneal segmentation area;

[0099] Step 4: Input the eye key points and corneal segmentation area into the eye topology structure representation model to obtain the representation factor of the eye topology structure.

[0100] The relatively consistent experience of clinical ophthalmologists shows that the topological structure of children's eyes, such as the ratio of the area of ​​cornea covered by the upper eyelid to the area of ​​cornea exposed, the inclination angle of the upper eyelid eyelashes, the ratio of the vertical diameter / horizontal diameter of the cornea, the ratio of the inner canthus distance / pupillary distance, the inclination angle of the palpebral fissure, and the angle between the horizontal line of the lower eyelid margin and the inner canthus line, are potential leading factors in assisting the diagnosis of astigmatism.

[0101] However, the upper eyelash coverage area and corneal area ratio used in previous studies are only vague qualitative measures. This invention quantifies this indicator for the first time.

[0102] Figure 6 The overall process of extracting the external ocular topological structure representation factors is shown. For the multi-view eye images of children, the pre-processing filtering operation is first performed to enhance the image quality. Then two branches are input to extract the key element information respectively. The first branch is the eye key point extraction network. The image is subjected to multi-scale feature extraction through the backbone network and feature pyramid, followed by point and box regression, and key point classification. Finally, the non-maximum suppression post-processing operation is performed to output the location information of the eye key points of interest. The other branch is the corneal segmentation network, which inputs the encoding and decoding architecture that integrates multi-scale features, and finally obtains the area of ​​exposed cornea. The output extraction results of the two branches are input into the structural parameter representation model. After mathematical relationship calculation, several typical external ocular topological structure representation factors are finally obtained. The technical route mainly includes ① the definition of the external ocular topological structure model, ② the eye key point detection technology combined with the topological structure characteristics, and ③ the multi-modal adaptive corneal segmentation method.

[0103] 1. Definition of the model for representing the external structure of the eye

[0104] As shown in Table 1 and Figure 7 As shown in the figure, in order to more fully and more detailedly characterize the topological structure of the eye, 24 eye key points are determined, 12 key points for each eye. The 12 key points represent the outer canthus point, the 1 / 4 point of the upper eyelid, the highest point of the upper eyelid, the 3 / 4 point of the upper eyelid, the inner canthus point, the 3 / 4 point of the lower eyelid, the lowest point of the lower eyelid, the 1 / 4 point of the lower eyelid, the left end point of the corneal transverse diameter, the right end point of the corneal transverse diameter, the center point of the cornea, and the end point of the eyelashes.

[0105] Table 1 Correspondence diagram of key point names

[0106]

[0107] Assume that the positions of several key points extracted from key points in the two-dimensional coordinate system are expressed as:

[0108] 1. The inner canthus points of the left and right eyes ;

[0109] 2. Left and right outer canthus points ;

[0110] 3. The highest point of the upper eyelids of both eyes ;

[0111] 4. The lowest point of the left and right eyelids ;

[0112] 6. The center of the pupils of the left and right eyes ;

[0113] 7. The ends of the left and right eyelashes ;

[0114] 8. End points of the horizontal diameter of the left and right pupils .

[0115] The corneal exposure area can be obtained according to the corneal segmentation results. .

[0116] Taking the left eye as an example, according to the mathematical expression of geometric relationship, it can be calculated that:

[0117] The horizontal diameter of the cornea is The vertical diameter of the cornea is ;

[0118] The direction of the eyelashes is measured in a two-dimensional coordinate system, with vertical downward being 0° and the angle gradually increasing in the counterclockwise direction.

[0119] When measuring the inclination angle of the palpebral fissure in a two-dimensional coordinate system, the horizontal right is 0°, and the angle gradually increases in the counterclockwise direction. Similarly, when measuring the angle between the lower eyelid horizontal line and the inner canthus line in a two-dimensional coordinate system, the horizontal right is 0°, and the angle gradually increases in the counterclockwise direction.

[0120] The characterization factor of the eye topology is calculated based on the extracted eye key points and corneal area ( Figure 8 As shown), the characterization factors include one or more of the following: the ratio of the upper eyelid covering the corneal area, the angle between the lower eyelid horizontal line and the inner canthus line, the ratio of the inner canthus distance to the pupil distance, the inclination angle of the palpebral fissure, the ratio of the vertical diameter of the cornea to the horizontal diameter, and the inclination angle of the upper eyelid eyelashes.

[0121] The calculation expressions of the final 6 parameter characterization factors are:

[0122] The ratio of the area of ​​cornea covered by the upper eyelid to the area of ​​cornea exposed =

[0123] Upper eyelid eyelash tilt angle =

[0124] The ratio of the vertical diameter to the horizontal diameter of the cornea =

[0125] Ratio of inner canthus distance to pupil distance =

[0126] Angle of palpebral fissure tilt =

[0127] The angle between the lower eyelid margin horizontal line and the inner canthus line =

[0128] 2. Extract the key points of the eye image using the eye key point detection technology combined with topological structure characteristics

[0129] The proposed eye key point detection network combined with topological structure characteristics adopts a multi-task architecture. First, the input eye image is affine transformed, then multi-scale feature extraction is performed, and the output heat map and offset map are affine transformed to realize the topological structure constraint. The overall process is as follows: Fig. 9 shown.

[0130] 2.1 Image preprocessing based on Gabor affine transformation

[0131] Under the shooting conditions of the data collected in this study, there are complex facial lighting environments, which will affect the detection effect of the algorithm. In traditional image data processing methods, Gabor filters extract relevant features at different scales and directions in the frequency domain, and are often used for texture recognition, and have achieved good results. To a certain extent, they can highlight eye features. On the other hand, Gabor filters process a small amount of data and are fast. Therefore, the Gabor preprocessing process of the original image in this project is to extract feature maps in three directions: 0°, 45°, and 90°, and obtain a feature map in each direction. Each feature map is used as a channel to synthesize a new image, and finally grayscale processing is performed ( Fig.10 ).

[0132] 2.2 Topological constraints

[0133] Inspired by equivalent landmark point transformation, this study proposes a topological structure constraint method based on affine transformation, which constructs additional constraints according to the overall properties of the topological structure between landmark points, so that the network can learn more robust landmark point features.

[0134] Given an image and affine transformation , X′= Representing images and affine transformed The result after is the affine transformation parameter, and the coordinates of the landmark points on the image X′ will also undergo the same affine transformation. Therefore, the image X′ is input into the deep neural network The result should be consistent with the original image. Input to the neural network The result obtained follows the same affine transformation, that is, there exists ,like Fig.11 shown.

[0135] Based on this, by utilizing the self-similar nature of the topological structure formed by the eye key points in the image after affine transformation, an additional constraint can be formed by comparing the similarities between the heat map and the offset map output by the network before and after the image transformation, so that the network can capture the structural information contained in the eye key points and then extract more robust features. In this study, the mean square error loss function is used to penalize the prediction differences of the network before and after the image transformation to construct the structural invariance loss.

[0136]

[0137] 2.3 Multi-scale contextual semantic learning network structure design

[0138] The FPN feature map pyramid structure is designed to achieve better detection results for small-sized human eyes. However, in the context of taking pictures of human eyes in this project, the area of ​​human eyes in the captured images accounts for a moderate proportion of the original images. Therefore, the P2 and P6 layers of the feature pyramid network are removed to ensure the detection results of medium-sized human eyes in a targeted manner while reducing unnecessary network reasoning calculation processes.

[0139] At the same time, in eye key point detection, the combination of deep network and key point topology structure can also be considered. In the network design, full consideration should be given to the effective information transfer between feature extraction and key point position. The deep network should be used to extract image features and the key point topology structure should be introduced in the last few layers, such as fully connected layers or similar structures, to capture higher-level semantic information and longer-distance associations. The accurate positioning of key points can be achieved by jointly optimizing the network parameters and topology structure.

[0140] 2.3.1 In this section, a multi-branch dilated convolution structure suitable for eye key point localization is designed and applied to the decoder part of the model. Each branch uses a dilated convolution layer with a different dilation rate. Different branches extract contextual information of different scales, so that the decoder can capture the correlation between pixel features to a greater extent in the process of fusing low-level features with high-level features, thereby better restoring information such as the spatial position of the image.

[0141] The parameters defining the multi-branch dilated convolution structure are , in represents the convolution kernel size, represents the combination of the expansion rates of each branch, that is , in, They represent n different expansion rates on n branches respectively.

[0142] like Fig.12 As shown in the figure, the feature maps of different scales extracted by the hole convolution with different expansion rates are respectively subjected to batch normalization and ReLU activation layers for channel splicing, and then the 1×1 convolution is used to reduce the number of channels of the spliced ​​new features and input them into the next upsampling layer. In this way, not only the problem of lack of use of contextual information in the process of feature fusion is solved, but also the network can better focus on multi-scale features without increasing the number of parameters and calculations, thereby improving the accuracy of the network in predicting eye key points.

[0143] 2.4 Introduction to the overall network structure

[0144] The overall framework of the eye key point detection technology proposed in this invention ( Fig.17 (as shown) is an eye key point detection method based on adaptive point set, which uses the center point of the eye area and several eye key points to represent the eye structure. In the feature extraction stage, this method performs feature extraction based on the Hourglass network. By introducing the multi-scale hole convolution technology in 2.3.1, multi-scale contextual semantic learning is realized, which effectively improves the accuracy of key point detection in the eye area of ​​children of different sizes. In addition, the present invention innovatively adds the target box center (i.e., the human eye classification head) and the boundary regression head (i.e., the orbital regression head) to the network output head part, realizing the parallel learning of the anchor-based target detection task and the eye key point detection task. Through this multi-task learning method, more prior knowledge is introduced, the generalization and robustness of the network are improved, and the performance of eye key point detection is further improved.

[0145] Specifically, the eye key point detection method of the present invention includes the following key steps:

[0146] Step 1. Using the Hourglass network and multi-scale dilated convolution technology, the fusion of multi-scale features is realized in the feature extraction part, the learning of multi-scale context feature correlation is realized, and the network's detection ability for targets of different sizes is enhanced.

[0147] Step 2. In the network output head part, in addition to the key point regression task, the target box center and boundary regression heads are also integrated to introduce more prior knowledge, which can effectively predict the precise position and range of the eye target area and key points.

[0148] Step 3. Through this improved network structure, the present invention can quickly and accurately complete the detection of key points of children's eyes in a single forward propagation without complicated post-processing.

[0149] In the key point detection task, generating the true value label of the heatmap is an important step. A heatmap is a two-dimensional graph that represents the probability of key point locations, where the value of each pixel represents the probability that the pixel is a key point location. The following are the general steps to generate the true value of the heatmap:

[0150] Step 1: Annotate key points: First, you need to manually or use automated tools to annotate the real locations of key points on the image. These annotations usually include the categories and coordinates of the key points.

[0151] Step 2, define the heat map size: Determine the resolution of the heat map, which is usually proportional to the resolution of the input image, for example, it can be 1 / 4 of the input image resolution.

[0152] Step 3, Generate Gaussian distribution: For each annotated keypoint, generate a Gaussian distribution on the heat map with the keypoint as the center. The peak of the Gaussian distribution is located at the actual position of the keypoint and decreases rapidly as the distance from the center of the keypoint increases. The standard deviation (σ) of the Gaussian distribution is a hyperparameter that controls the degree of diffusion of the Gaussian kernel.

[0153] Step 4, accumulate Gaussian maps: If there are multiple key points, accumulate the Gaussian distribution generated by each key point to the same heat map. In this way, each pixel value on the heat map reflects the probability that the pixel is the key point location.

[0154] Step 5, handling occluded and invisible points: For occluded or invisible key points, they can be represented by adjusting the value of the Gaussian distribution or using specific markers.

[0155] Step 6, Convert and Save: Convert the generated heat map into a format suitable for neural network training and save it as an image file or embed it directly into the dataset.

[0156] In practice, different tools and libraries can be used to generate heat maps, such as using Python's OpenCV library or MATLAB's related functions. Some key point detection frameworks, such as the annotation tool for the COCO dataset, have built-in functions for heat map generation. In addition, some deep learning libraries, such as TensorFlow and PyTorch, provide tools and functions for manipulating and processing heat maps. Through these steps, high-quality true value labels can be generated for the key point detection network, thereby improving the training effect and prediction accuracy of the model.

[0157] 2.5 Loss Function Design

[0158] For eye key point detection, since dense point regression of the face is not required and the amount of calculation of dense point regression is relatively large, the dense point regression loss function is removed from the loss function. The optimized eye key point loss function includes eye classification, eye socket position regression and eye key point regression loss functions.

[0159]

[0160] In the formula, In is the network's prediction of The probability that the anchor box contains a human eye, Data labels for human eyes; middle, They are the coordinates of the predicted anchor box and the coordinates of the data label, including 4 positioning data of the human eye socket: , , , ,exist In , the 22 key points of the human eye are predicted, and the regression loss function of the key points of the human eye is:

[0161]

[0162] They represent the predicted coordinates of the 22 key points and the true coordinates of the 22 points in the label.

[0163] 3. Extracting corneal exposed area using multi-modality adaptive corneal segmentation method

[0164] In order to solve the problems of blurred corneal edges caused by changes in illumination and shadows due to shooting angles, as well as different resolutions of images taken by different devices and different degrees of eyelid occlusion, the exposed corneal shapes vary greatly, such as severe occlusion of eyelashes and eyelids, invalid off-axis rotation, motion blur, and irregular reflections in the eye area, etc., this section proposes a noise-robust corneal segmentation scheme and designs an improved stacked Hourglass network ( Fig.13 ),by

[0165] ① Parameterize the corneal boundary;

[0166] ② At the same time, design a dense connection method between network layers so that the gradient flow containing the real corneal boundary information can be better transmitted;

[0167] ③ Fuse the deep network and the shallow network together to form a multi-scale network to better capture coarse and fine details;

[0168] ④ For the target area, change its size and input images of different sizes into the designed network to obtain the global and regional features of the cornea;

[0169] ⑤Finally, connect a SegNet decoder and pixel classification layer to independently classify each pixel into cornea and background areas to achieve accurate segmentation of children's cornea.

[0170] 3.1 Improved Stacked Hourglass Feature Extraction Network

[0171] The Hourglass network is a symmetric and fully convolutional architecture, similar to U-Net, where feature maps are downsampled by pooling operations and then restored to their dimensions using nearest neighbor upsampling. At each scale level, a residual connection is applied to connect the corresponding layers between the two sides of the Hourglass. This bottom-up, top-down architecture is stacked multiple times to capture long-range contextual information, so even occluded parts of the eyeball may be predicted by capturing the spatial relationship between the corneal boundaries.

[0172] In order to keep the input size constant throughout the network and make the network lightweight, the original Hourglass network is adapted by making four improvements:

[0173] (1) The stride of the initial convolutional layer with a kernel size of 7×7 is changed from 2 to 1,

[0174] (2) Remove the initial maximum pooling layer,

[0175] (3) The number of channels per layer of the network is reduced from 256 to 64,

[0176] (4) Each Hourglass module processes at 4 different image scales and stacks them 3 times.

[0177] The four improvements here are to ensure that the input size of the entire network remains unchanged while being lightweight. Removing the maximum pooling layer does not have the effect of lightweighting, but to ensure that the input size remains unchanged. At the end of each Hourglass module, 64 fine feature maps are combined through 1×1 convolutional layers to generate 6 maps: 2 for segmentation masks, 2 for pupils, and the remaining 2 for corneas. Each pair consists of a foreground map and a background map, which are further processed by a softmax layer to obtain a binary mask. The stacked Hourglass network structure is shown in Figure 13.

[0178] Although the original stacked structure of Hourglass can better fuse multi-scale features and enhance detail representation, there may still be a problem of basic feature loss during the module cascading process. To address this, the improved stacked Hourglass structure adopts a dense connection method. Compared with the original Hourglass network, the improved network adds connections between the key layers of the input and output parts of the hourglass structure and the key layers of all previous hourglass structures (the key layer refers to the input and output feature layer of each hourglass structure). In this way, each layer can directly access the features of all previous layers, which helps the network to use information more effectively and improve the reuse rate of features. At the same time, the enhanced dense connection ensures that information can circulate more widely in the network, thereby improving the quality of feature fusion and the overall representation ability of the network.

[0179] The hourglass structure of the Hourglass network is a deep learning module that uses symmetrical downsampling and upsampling paths to achieve multi-scale feature fusion. This structure consists of a series of repeated hourglass modules, each of which reduces the spatial size of the feature map through pooling or strided convolution in the contraction path, while increasing the feature depth to capture more abstract features; in the magnification path, the spatial size of the feature map is gradually restored through upsampling to refine and enhance local details. At the bottleneck of the hourglass, the low-resolution feature map is fused with the upsampled feature map, and the high-resolution detail information is directly passed to the deep network using jump connections, thereby performing accurate key point detection at different scales. This design of the Hourglass network is particularly suitable for capturing small-scale structures and details in images, and is widely used in visual tasks that require fine spatial positioning.

[0180] The task of detecting and extracting eye key points involves accurately locating specific parts of the eye from the face image, such as the edge of the cornea, points on the eyelids, etc. These small-scale features of the eye are very subtle, and the eye area contains multiple key points. The positional relationship between them is complex and needs to be accurately modeled. The high similarity in the structure of the left and right eyes may also cause confusion. The improved stacked Hourglass network enhances dense connections, which is beneficial for capturing small-scale details of the eye and overall facial features, ensuring that information can flow and reuse fully from the shallow to the deep layers of the network, which helps to retain and strengthen the key information of the eye in the deep network. The upsampling path of the Hourglass network helps to refine features while maintaining high resolution, which is crucial for accurately locating the key points of the eye.

[0181] The Fully Convolutional Network (FCN) reduces its output dimension by downsampling, and its output features are coarser and more global, which is very useful for capturing global structure or contextual information. However, this makes it difficult to segment the fine details of the eye image, including the corneal boundary and non-corneal pixel areas occluded by eyelashes or reflections. To address the above problems, an improved densely connected Hourglass network structure is proposed, combining shallow layers, fine layers, deep layers, and global layers. It consists of multiple convolutional layers and pooling layers, which fuse network layers from shallow to deep to form a multi-scale network to better capture coarse and fine details. Multiple network layers upsample the features to the same size as the input image and sum their outputs.

[0182] like Fig.15 As shown in the figure, assuming that the size of the input image is 256×256, after a 7*7 size, stride 2 convolution and maximum pooling operation, the feature map size of the final input module is 64×64. Before each downsampling, the upper half (i.e., residual network) is separated to retain the original scale information; after each upsampling, it is added to the data of the previous scale; between two downsamplings, three Residual modules are used to extract features; between two additions, one Residual module is used to extract features. The Residual module is a jump connection structure. The first line is a convolution branch, which is composed of three convolution layers with different kernel scales in series, interspersed with Batch Normalization and ReLU; the second line is a jump connection, which only contains one convolution layer with a kernel scale of 1. The stride of all convolution layers is 1, and the padding is 1. The data size is not changed, and only the data depth (number of channels) is changed. In order to prevent the loss of original feature information and deviation during the multi-layer forward propagation of the network, and to enhance the reuse of basic features, the input and output of multiple hourglass structures are connected. As shown Fig.14As shown, feature map F2 is the sum of feature map F1 and the output feature map of the first hourglass structure, feature map F3 is the sum of feature map F1, feature map F2, and the output feature map of the second hourglass structure, and so on.

[0183] Multi-scale information fusion: In addition to multi-scale fusion at the feature level, the input image is also divided into blocks to generate inputs of different sizes centered on the target corneal area. The features obtained by multiple inputs through the same network are fused through a fully connected layer and sent to the subsequent decoder for pixel-by-pixel prediction.

[0184] As shown in Figure 16, for each pixel in the image, three blocks of different scales are cropped from small to large to 56×56, 112×112, and 224×224, respectively. Each pixel is trained or tested separately, and the predicted label of each pixel is finally merged into the corneal mask. The merge here means that each pixel will get a predicted label, and the pixels with the predicted label of cornea are combined to form the final corneal mask. Training and fusion of different scales are automatically completed in the data preprocessing and network output stages. Processing eye images of different scales can be used as a means of data enhancement. Fusion refers to the fusion of multi-scale features output by images of different scales, which is a feature-level fusion, and then the final segmentation result is output. Fig.16 The emphasis is on the multi-size characteristics of "input". The detailed operations and network structures of CNN and fusion are in the Hourglass above.

[0185] To ensure the reliability and accuracy of the network in corneal segmentation, the encoder of the designed stacked densely connected Hourglass network with enhanced feature propagation is combined with the decoder of the SegNet deep convolutional network. SegNet is a practical fully convolutional network for pixel-by-pixel semantic segmentation, which uses a 13-layer encoder with the same structure as the VGG16 core network, while removing the fully connected layers. The decoder of SegNet upsamples the low-resolution input feature map with pooling indices. The stacked densely connected Hourglass network has better information gradient flow due to its densely connected structure, and each of its convolutional layers is connected to all convolutional layers in a feed-forward manner, which is very useful for strengthening the feature propagation of subsequent layers in dense blocks.

[0186] Through this connection, the designed cornea segmentation network exploits the network potential and improves the efficiency of the model by reusing the features of the previous layers. The direct connection of the network is to achieve the cascading of features by cascading layers with multiple inputs and one output. In a dense block, the basic convolution layer (Conv), batch normalization (BN) and rectified linear unit (ReLU) are included, which are connected in a cascade manner. In the transition layer of the decoder, the maximum pooling after each dense block provides pooling indices, which are used for up-pooling and up-sampling for pixel-by-pixel semantic segmentation of the cornea and corneal boundaries.

[0187] In the overall network, dense blocks are used to improve the feature extraction performance of the network, and the SegNet decoder is used to create a densely connected convolutional network for corneal segmentation. The bottleneck layer in the transition layer after each dense block in the dense encoder helps to reduce the number of input feature maps and improve computational efficiency. The transition layer is basically a combination of Conv 1×1 and max pooling to separate two adjacent dense blocks.

[0188] Loss function of fusion parameter fitting: For the mask output by corneal segmentation, further contour parameter fitting is performed. Under the guidance of prior information of corneal shape, the boundary information of the corneal area is better learned to achieve refined boundary segmentation. Ellipse fitting of contour refers to using an ellipse to approximate the shape of the contour. When the major axis and minor axis of this ellipse are equal, it is a circle. The basic idea of ​​ellipse fitting is: for a set of sample points on a given plane, find an ellipse that is as close to these sample points as possible. In other words, a set of data in the image is fitted with an ellipse equation as a model, so that a certain ellipse equation can satisfy these data as much as possible, and the various parameters of the ellipse equation are calculated.

[0189] The total loss function of the final model consists of three parts: inner iris segmentation loss, outer cornea segmentation loss and boundary parameter fitting loss.

[0190]

[0191] in, , and Represent parameter fitting loss, segmentation loss of inner iris and outer cornea, coefficient are set to 1 respectively to make the ranges of these loss values ​​comparable. The loss function form of the segmentation part is cross entropy loss, and the parameter fitting part is mean square error loss. and Two binary segmentation losses, the cross entropy loss function is as follows:

[0192]

[0193] in and Represent the true value and the predicted value respectively. The parameter fitting loss is as follows:

[0194]

[0195] Where n is the number of samples, represents the true value of the i-th sample, is the model's predicted value for the i-th sample. The parameters output by parameter fitting are the center position, major axis diameter, and minor axis diameter of the ellipse.

[0196] The present invention also discloses a method for predicting astigmatism based on eye characterization factors, the method comprising:

[0197] Acquiring an eye image of a subject to be tested;

[0198] Using the above-mentioned method of extracting eye topology based on eye image to obtain the representation factor of the eye topology of the subject;

[0199] The characterization factors are input into a classifier, and whether the subject suffers from astigmatism is predicted based on the output of the classifier.

[0200] In some embodiments, test set subjects are collected, including children with astigmatism and children with normal vision, eye images of the test set subjects are obtained, and a method for characterizing the eye topology based on eye images is used to obtain characterization factors of the eye topology of the subject, and the columns of the feature set include the ratio of the vertical diameter of the cornea to the horizontal diameter and the ratio of the area of ​​the cornea covered by the upper eyelid; the feature set and the corresponding labels (astigmatism or normal vision) are input into a support vector machine model for training, and the trained support vector machine is the classifier used for prediction, and the classifier can be used to predict whether the subject suffers from astigmatism based on the ratio of the vertical diameter of the cornea to the horizontal diameter and the ratio of the area of ​​the cornea covered by the upper eyelid.

[0201] In some embodiments, the columns of the feature set include the ratio of the area of ​​the cornea covered by the upper eyelid to the area of ​​the cornea exposed, the tilt angle of the upper eyelid eyelashes, the ratio of the vertical diameter of the cornea to the horizontal diameter, the ratio of the inner canthus distance to the pupil distance, the tilt angle of the palpebral fissure, and the angle between the lower eyelid horizontal line and the inner canthus line; the feature set and the corresponding labels (astigmatism or normal vision) are input into the logistic regression model training, and the trained model is the classifier used for prediction.

[0202] When it is necessary to predict whether the subject suffers from astigmatism, the ratio of the area of ​​the cornea covered by the upper eyelid to the area of ​​the cornea exposed, the inclination angle of the upper eyelid eyelashes, the ratio of the vertical diameter of the cornea to the horizontal diameter, the ratio of the inner canthus distance to the pupil distance, the inclination angle of the palpebral fissure, and the angle between the horizontal line of the lower eyelid margin and the inner canthus line are obtained based on the eye image of the subject. These features are input into the classifier, and the output of the classifier is used to predict whether the subject suffers from astigmatism.

[0203] Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present invention, such as Figure 3 As shown, the device 2000 may include: one or more processors 2010, and one or more memories 2020; wherein the memories store computer-readable codes, and when the computer-readable codes are run by the one or more processors, the method described above may be executed.

[0204] The processor in this embodiment can be an integrated circuit chip with signal processing capabilities. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, operations and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can be an X86 architecture or an ARM architecture.

[0205] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0206] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Figure 4 The architecture of the computing device 3000 shown in FIG. Figure 4 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as ROM 3030 or hard disk 3070, may store various data or files used for processing and / or communication of the method provided by the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 4 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 4 One or more components of a computing device are shown.

[0207] The embodiment of the present invention also provides a computer-readable storage medium, such as Figure 5 As shown, it is a schematic diagram of a storage medium 4000 provided in an embodiment of the present invention, and a computer readable instruction 4010 is stored on the computer storage medium 4020. When the computer readable instruction 4010 is executed by a processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0208] The present disclosure also provides a computer program product or a computer program, which implements the steps of the above method when executed by a processor, such as Figure 2 As shown, the computer program product or computer program comprises:

[0209] Acquisition module 201: used to acquire eye images of the subject;

[0210] Key point extraction module 202: used for inputting the eye image into the eye key point extraction network to obtain eye key points;

[0211] Corneal area extraction module 203: used for inputting the eye image into the corneal segmentation network to obtain the corneal segmentation area;

[0212] Topological characterization module 204: used to calculate the characterization factors of the eye topological structure based on the eye key points and corneal area, the characterization factors including one or more of the following: the ratio of the upper eyelid covering the corneal area, the angle between the lower eyelid horizontal line and the inner canthus line, the ratio of the inner canthus distance to the pupil distance, the inclination angle of the palpebral fissure, the ratio of the vertical diameter of the cornea to the horizontal diameter, and the inclination angle of the upper eyelid eyelashes.

[0213] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0214] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0215] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0216] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0217] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0218] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0219] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for extracting eye topology based on eye images, characterized in that: The method comprises: S1: Acquire the eye image of the subject; S2: Inputting the eye image into an eye key point extraction network to obtain eye key points, and calculating the eye topology based on the eye key points; the method for constructing the eye key point extraction network includes: Obtaining a training set of eye images, wherein the eye image training set includes annotated key points; Extracting features from a training set of eye images through a feature extraction network to obtain first features, wherein the first features include a first classification head attention feature, a first orbital attention feature, and a first key point attention feature; Based on the first feature, key point prediction is performed to obtain a predicted key point heat map; based on the labeled key points, a labeled key point heat map is obtained, and based on the difference between the predicted key point heat map and the labeled key point heat map, a prediction loss is obtained; The training set of eye images is subjected to affine transformation and feature extraction to obtain a second feature, wherein the second feature includes a second classification head attention feature, a second eye socket attention feature, and a second key point attention feature; the first feature is subjected to the same affine transformation to obtain a third feature, wherein the third feature includes a third classification head attention feature, a third eye socket attention feature, and a third key point attention feature; and a structural invariance loss is obtained based on the difference between the third feature and the second feature; The eye key point extraction network is obtained after the objective function is gradually optimized by iteration and the stopping condition is reached, and the objective function includes the prediction loss and the structural invariance loss.

2. The method for extracting eye topology based on eye image according to claim 1, characterized in that: The structural invariance loss is expressed as: in, represents the structural invariance loss, X represents the eye image, T() represents the affine transformation, represents the parameters of the affine transformation, represents feature extraction, represents the mean square error.

3. The method for extracting eye topology based on eye image according to claim 1, characterized in that: The feature extraction network includes an initial multi-scale feature pyramid and an initial multi-head attention layer; the training set of eye images is input into the initial multi-scale feature pyramid to obtain pyramid features, and the pyramid features are extracted through multi-head attention to obtain the first features, and the multi-head attention includes classification head attention, box regression head attention and key point regression head attention; the pyramid features are subjected to classification head attention to obtain classification head attention features; the pyramid features are subjected to box regression head attention to obtain orbital attention features; the pyramid features are subjected to key point regression head attention to obtain key point attention features.

4. The method for extracting eye topology based on eye image according to claim 1, characterized in that: The eye image training set also includes a labeled classification head and a labeled eye socket; the prediction loss includes a predicted classification head loss, a predicted eye socket loss, and a predicted key point loss; the prediction loss is expressed as: in, represents the prediction loss, represents the predicted classification head loss, where is the network's prediction of The probability that an anchor box is classified as containing a human eye, Is the data label of whether the anchor box contains the human eye; represents the predicted orbital loss, They are the vertex coordinates of the four predicted anchor boxes and the vertex coordinates of the annotated eye sockets; represents the predicted key point loss, They respectively represent the coordinates of the predicted human eye key points and the coordinates of the marked key points.

5. The method for extracting eye topology based on eye image according to claim 1, characterized in that: The method further comprises inputting the eye image into a corneal segmentation network to obtain a corneal segmentation area; Calculating the eye topology based on the eye key points and the corneal segmentation area; The method for constructing the cornea segmentation network comprises: Get a training set of eye images; The training set of eye images is input into the initial cornea segmentation network to obtain a cornea mask and an iris mask; Performing contour fitting on the corneal mask to obtain contour parameters, wherein the contour fitting is to fit the corneal mask using an ellipse, and the contour parameters include a major axis diameter and a minor axis diameter of the ellipse; The total loss function of the initial cornea segmentation network consists of three parts, including iris segmentation loss, cornea segmentation loss and boundary parameter fitting loss; the cornea segmentation network is obtained after iteratively optimizing the total loss function.

6. The method for extracting eye topology based on eye images according to claim 5, characterized in that: The method also includes that the total loss function of the initial cornea segmentation network is expressed as follows: in, Represent contour fitting loss, iris segmentation loss and cornea segmentation loss respectively, and the coefficient Represents the weight coefficient of contour fitting loss, iris segmentation loss and cornea segmentation loss.

7. A method for predicting astigmatism based on eye characterization factors, characterized in that: The method comprises: Acquiring an eye image of a subject to be tested; Using the method according to claim 1 to obtain a characterization factor of the eye topology of the subject; The characterization factors are input into a classifier, and whether the subject suffers from astigmatism is predicted based on the output of the classifier.

8. A computer device, characterized in that: The device comprises: a memory and a processor; the memory is used to store a computer program; the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images