Line of sight estimation method, device, computer equipment and storage medium
Through the capsule network-based line of sight estimation method, the encoder and decoder obtain the position information of the key points of the human eye, solving the problem that line of sight estimation in the prior art is susceptible to environmental interference, and achieving higher accuracy and robustness.
Patent Information
- Application Number
- CN202110703027.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-06-24
AI Technical Summary
Existing estimation methods are susceptible to environmental factors, resulting in inaccurate estimation. Especially under the influence of changes in light, face occlusion and blurred jitter in natural environments, the accuracy of estimation cannot be guaranteed.
Using the capsule network-based line of sight estimation method, by acquiring face images and positioning feature points, using encoder and decoder to feature code and decode the eye area image, obtaining position information of key points of the human eye, and using the dynamics of the capsule network The routing mechanism retains spatial feature information, enhances the robustness of the model and spatial generalization capabilities, and thus improves the accuracy of line of sight estimation.
It effectively improves the accuracy of line of sight estimation, enhances the robustness of the model and spatial generalization ability, can better resist the influence of environmental factors, and improves the reliability of the estimation results.
Smart Images

Figure CN113269148B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a line of sight estimation method, device, computer equipment and storage medium. Background Art
[0002] Human line of sight is an important non-verbal information in the process of human communication. It has been applied to scenarios such as medicine, games, assisted driving, and interactive applications. Early traditional line of sight estimation methods mainly used geometric features of the human eye such as the pupil, cornea, or Purkinje spots formed by infrared light reflection, and achieved estimation by simulating the two-dimensional regression function between eye feature changes and the focus position. However, current traditional algorithms usually require a certain amount of prior experience, so they are limited by environmental factors. For example, in natural environments, they are affected by different environmental factors such as lighting changes, face occlusion, jitter blur, etc., resulting in differences in the accuracy of line of sight estimation, and the accuracy of line of sight estimation cannot be guaranteed. Summary of the invention
[0003] The embodiments of the present invention provide a line of sight estimation method, apparatus, computer equipment and storage medium to solve the problem that the existing line of sight estimation is easily disturbed by environmental factors, resulting in inaccurate estimation.
[0004] A line of sight estimation method, comprising:
[0005] Acquire a face image and a sight line estimation model trained based on a capsule network; wherein the sight line estimation model includes an encoder and a decoder;
[0006] Positioning feature points of the face image to obtain an eye region image;
[0007] Performing feature encoding on the eye region image by the encoder to obtain a human eye feature map; wherein the human eye feature map includes a plurality of human eye key points;
[0008] Decoding the eye feature map through the decoder to obtain a heat map corresponding to the eye region image; wherein the heat map includes position information of each eye key point;
[0009] The line of sight is estimated according to the position information of the key points of the human eye to obtain a line of sight estimation result.
[0010] A sight line estimation device, comprising:
[0011] A data acquisition module, used to acquire a face image and a sight line estimation model trained based on a capsule network; wherein the sight line estimation model includes an encoder and a decoder;
[0012] A feature point positioning module, used to locate feature points of the face image to obtain an eye area image;
[0013] An encoding module, configured to perform feature encoding on the eye region image through the encoder to obtain a human eye feature map; wherein the human eye feature map includes a plurality of human eye key points;
[0014] A decoding module, used for decoding the human eye feature map through the decoder to obtain a heat map corresponding to the eye area image; wherein the heat map includes the position information of each key point of the human eye;
[0015] The sight line estimation module is used to perform sight line estimation according to the position information of the key points of the human eye to obtain a sight line estimation result.
[0016] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned sight line estimation method when executing the computer program.
[0017] A computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned sight line estimation method are implemented.
[0018] In the above sight estimation method, device, computer equipment and storage medium, by obtaining a face image and a sight estimation model based on capsule network training, the feature points of the face image are located to obtain an eye region image, and the eye region image is feature encoded by an encoder to obtain a human eye feature map containing multiple human eye key points corresponding to the eye region image, so that the capsule network is applied to the regression task for sight estimation, thereby feature encoding the key points of the human eye with the help of the dynamic routing mechanism of the capsule network, retaining spatial feature information, and effectively obtaining eye-related feature information, so that the sight estimation model is more robust. Then, the human eye feature map is feature regressed by a decoder to obtain a heat map corresponding to the eye region image, and the local features in space are introduced by heat map regression to enhance the spatial generalization ability of the sight estimation model, which is more conducive to regressing the key points of the human eye. Finally, the sight estimation is performed according to the position information of the key points of the human eye to obtain the sight estimation result, which can effectively improve the accuracy of the estimation. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0020] Figure 1is a schematic diagram of an application environment of a sight line estimation method according to an embodiment of the present invention;
[0021] Figure 2 is a flow chart of a sight line estimation method according to an embodiment of the present invention;
[0022] Figure 3 is a flow chart of a sight line estimation method according to an embodiment of the present invention;
[0023] Figure 4 is a flow chart of a sight line estimation method according to an embodiment of the present invention;
[0024] Figure 5 yes Figure 2 A specific flow chart of step S203;
[0025] Figure 6 yes Figure 2 A specific flow chart of step S205;
[0026] Figure 7 yes Figure 6 A specific flow chart of step S601;
[0027] Figure 8 yes Figure 6 A specific flow chart of step S602;
[0028] Fig. 9 is a schematic diagram of a sight line estimation device according to an embodiment of the present invention;
[0029] Fig.10 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION
[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0031] The line of sight estimation method can be applied in Figure 1 In an application environment, a computer device communicates with a server via a network. The computer device may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented as a standalone server.
[0032] In one embodiment, if Figure 2 As shown, a line of sight estimation method is provided, and the method is applied in Figure 1The server in the example is used as an example to illustrate the process, including the following steps:
[0033] S201: Acquire a face image and a gaze estimation model trained based on a capsule network; wherein the gaze estimation model includes an encoder and a decoder.
[0034] Among them, this method can be applied in VR, video surveillance or other eye tracking fields to track the line of sight of the target person and achieve accurate line of sight estimation. In this embodiment, the face image can be captured frame by frame from the real-time video stream recorded in real time or an image containing a face captured from the uploaded offline video, which is not limited here. Furthermore, in order to ensure the effectiveness of subsequent line of sight estimation, it is possible to detect whether the frame image captured from the video contains a face. If it does, the frame image can be used as a face image to perform subsequent line of sight estimation. If it does not contain, the image can be directly discarded and the next frame can be selected, or the corresponding frame image can be selected from the video at a set time interval and the above face detection steps can be repeated until an image containing a face is obtained.
[0035] Specifically, the line of sight estimation model is obtained based on capsule network training. Understandably, due to the pooling operation, such as max pooling, in the current convolutional neural network, the data is reduced in dimension, so that only the most active neurons are retained and passed to the next layer, resulting in the loss of valuable spatial information and low spatial resolution. Therefore, when the input changes slightly, the output remains basically unchanged, so additional steps are required to restore the image details. For the capsule network, each neuron in the capsule network is a vector neuron carrying a direction, and the traditional pooling operation is abandoned. Through the dynamic routing mechanism, the low-level features are output to the high-level with higher similarity, so that detailed sample information (such as precise target position, rotation, size, etc.) is saved, rather than restored after being lost, which can retain more image details and has stronger interpretability.
[0036] Furthermore, the lower-level features in the capsule network will only be passed to the high-level features that match them. For example, low-level features such as eyes and mouth will be passed to the high-level features of "face"; low-level features such as fingers and palms will be passed to the high-level features of "hand", so as to construct the spatial relationship between low-level features and high-level features, making the trained line of sight estimation model more robust.
[0037] S202: Locate feature points of the face image to obtain an eye region image.
[0038] Among them, the facial feature points are the feature coordinate points obtained by inputting the standard facial image into the feature point detection model for identification. The facial feature points include five feature points, namely, the left eye, the right eye, the nose tip, the left corner of the mouth, and the right corner of the mouth. Specifically, the standard facial image is input into the feature point detection model for identification, and the feature point detection model will obtain the positioning coordinates of the above five feature points. Furthermore, the server crops the facial image based on the positioning coordinates of the facial feature points to obtain the eye area image.
[0039] Among them, the facial feature point detection model can be implemented by training the images with the facial feature point positions marked using the DCNN network. DCNN (Deep Convolutional Neural Network) is a deep convolutional neural network.
[0040] S203: Perform feature encoding on the eye region image through an encoder to obtain a human eye feature map; wherein the human eye feature map includes a plurality of human eye key points.
[0041] S204: Decode the human eye feature map through a decoder to obtain a heat map corresponding to the eye area image; wherein the heat map includes position information of each key point of the human eye.
[0042] Among them, the line of sight estimation model includes an encoder and a decoder, the encoder is used to encode the low-level features and high-level features of the network to obtain eye feature point information, and the decoder performs feature regression on the eye feature point information obtained above to obtain relevant feature point position information. Specifically, the line of sight estimation model can be trained using eye images and pre-made heat maps corresponding to the eye images as training samples. The heat map includes multiple annotated key points of the human eye. Furthermore, the key points of the human eye include multiple (such as 8) boundary points corresponding to the limbus, multiple (such as 8) boundary points of the iris edge, the center point of the iris, and the center point of the cornea.
[0043] In this embodiment, the heat map and the original image have the same size. In the heat map, the closer the pixel value is to the target key point, the larger it is, and vice versa. It can be understood that compared with the direct regression coordinate method, the heat map regression method can introduce local features in space, has strong spatial generalization ability, and is more conducive to regressing key points.
[0044] In this embodiment, the basic features are extracted by using a traditional convolutional network, and then the basic features are input into the capsule layer of the capsule network, so that the above basic features are combined through the main capsule layer using a dynamic routing mechanism to output an 18D vector, so as to focus on the eye feature point information. Then the 18D vector output by the main capsule layer is input into the decoder to decode the relevant feature information of the eyes, so as to obtain the position information of 18 key points of the human eye.
[0045] S205: Performing sight line estimation according to the position information of the key points of the human eye to obtain a sight line estimation result.
[0046] Among them, the sight estimation results include but are not limited to pitch angle and yaw angle. The pitch angle (pitch) refers to the rotation of the target object around the X-axis. The position information includes the positioning coordinates of each key point of the human eye. The yaw angle (yaw) refers to the rotation of the target object around the Y-axis. Specifically, based on the position information of multiple key points of the human eye, that is, the position coordinates, coordinate calculation is performed to obtain the pitch angle and yaw angle, and the pitch angle and yaw angle can be used to determine the gaze direction or sight direction of the human eye.
[0047] Furthermore, by judging the pitch angle and the yaw angle against a preset threshold, the gaze direction of the human eye can be determined, such as left, right, upper left, lower right, etc.
[0048] In this embodiment, by obtaining a face image and a sight estimation model based on capsule network training, and then locating the feature points of the face image, an eye region image is obtained, and the eye region image is feature encoded by an encoder to obtain a human eye feature map containing multiple human eye key points corresponding to the eye region image, so that the capsule network is applied to the regression task for sight estimation, thereby feature encoding the key points of the human eye with the help of the dynamic routing mechanism of the capsule network, retaining the spatial feature information, and effectively obtaining the eye-related feature information, so that the sight estimation model is more robust. Then, the human eye feature map is feature regressed by the decoder to obtain a heat map corresponding to the eye region image, and the local features in space are introduced by adopting the heat map regression method to enhance the spatial generalization ability of the sight estimation model, which is more conducive to regressing the key points of the human eye. Finally, the sight estimation is performed according to the position information of the key points of the human eye to obtain the sight estimation result, which can effectively improve the accuracy of the estimation.
[0049] In one embodiment, if Figure 3 As shown, before step S203, the following steps are also included:
[0050] S301: Performing eye-opening detection on the eye region image to obtain a detection result.
[0051] S302: If the detection result is that the eyes are open, execute the step of encoding the features of the eye area image by the encoder to obtain a human eye feature map.
[0052] In this embodiment, in order to further ensure the effectiveness of subsequent line of sight estimation, eye-opening detection can be performed on the face image to detect the closed state of the human eye. If the eye is detected as open, the eye area image is considered valid and can be input into the line of sight estimation model for line of sight estimation; when the eye is detected as closed, the eye area image is considered invalid, that is, line of sight estimation cannot be performed, and the image can be directly discarded to select the next frame, or the corresponding frame image can be selected from the video at a set time interval and the above-mentioned eye-opening detection steps are repeated until a valid eye area image is obtained. Among them, the eye-opening detection can be implemented by a pre-trained eye-opening detection model, and the eye-opening detection model can be trained by annotated eye-opening images and eye-closed images.
[0053] In one embodiment, if Figure 4 As shown, the sight estimation model corresponds to a target recognition area; wherein the target recognition area is used to indicate the left eye or the right eye; before step S203, the following steps are also included:
[0054] S401: Acquire a face image and a gaze estimation model trained based on a capsule network; wherein the gaze estimation model includes an encoder and a decoder.
[0055] Specifically, the specific execution steps of step S401 are consistent with those of step S201, and are not repeated here to avoid repetition.
[0056] S402: Locate feature points of the face image to obtain an eye region image.
[0057] Specifically, the specific execution steps of step S402 are consistent with those of step S202, and are not repeated here to avoid repetition.
[0058] S403: If the eye region corresponding to the eye region image is different from the target recognition region, mirror symmetry processing is performed on the eye region image to obtain a symmetrical image corresponding to the eye region image.
[0059] The eye region corresponding to the eye region image, that is, the left eye or the right eye, can be directly obtained through step S202, which will not be described in detail here.
[0060] In this embodiment, the line of sight estimation model is mainly used for estimation of monocular images. The human eye distinguishes between the left and right eyes. Therefore, during training, the monocular images uniformly use the left eye or the right eye as training samples for training. In actual applications, it is necessary to further judge the eye area image to distinguish between the left and right eyes. If the left eye is used as the training sample during training, that is, the specific area of the eye is the left eye, then in actual applications, it is necessary to mirror the right eye image to obtain a symmetrical image, and then input the symmetrical image into the line of sight estimation model for estimation; conversely, if the right eye is used as the training sample during training, that is, the specific area of the eye is the right eye, then in actual applications, it is necessary to mirror the left eye image to obtain a symmetrical image, and then input the symmetrical image into the line of sight estimation model for line of sight estimation, so as to ensure the effectiveness and accuracy of the line of sight estimation.
[0061] S404: Perform feature encoding on the symmetrical image through an encoder to obtain a human eye feature map.
[0062] Specifically, the feature encoding step of step S404 is consistent with step S203 and will not be described again here to avoid repetition.
[0063] S405: Decode the human eye feature map through a decoder to obtain a heat map corresponding to the eye area image; wherein the heat map includes position information of each key point of the human eye.
[0064] Specifically, the execution steps of step S405 are consistent with step S204, and are not repeated here to avoid repetition.
[0065] S406: Performing sight line estimation according to the position information of the key points of the human eye to obtain a sight line estimation result.
[0066] Specifically, the execution steps of step S406 are consistent with step S205, and are not repeated here to avoid repetition.
[0067] In one embodiment, if Figure 5 As shown, the encoder includes a convolutional layer, a main capsule layer, and a sight estimation capsule layer connected in sequence; in step S203, the encoder performs feature encoding on the eye region image to obtain a human eye feature map, which specifically includes the following steps:
[0068] S501: extracting features of the eye region image through a convolution layer to obtain a first feature output by the convolution layer.
[0069] S502: Perform feature combination on the output of the convolution layer through the main capsule layer to obtain the second feature of the output of the main capsule layer.
[0070] S503: Perform feature transformation on the second feature through the line of sight estimation capsule layer to obtain a human eye feature map.
[0071] Specifically, when encoding the features of the eye area image, multiple basic feature vectors, i.e., the first feature, can be obtained by extracting features through the convolution layer in sequence, and then the basic feature combinations extracted by the convolution layer are combined through the main capsule layer through a dynamic routing mechanism to obtain the second feature output by the main capsule layer, so as to output the current low-level capsule to which high-level capsule, replacing the pooling operation in the existing convolution network and retaining more spatial information. Finally, the second feature is transformed through the line of sight estimation capsule layer, i.e., a fully connected layer, to obtain a human eye feature map of a preset dimension (e.g., 18D).
[0072] In this embodiment, the line of sight estimation capsule layer is constructed through the actual application scenario of this case to output a feature vector of a preset dimension. In this embodiment, since 18 key points of the human eye need to be regressed in the end (i.e., 8 boundary points of the corneal limbus (edge of the white of the eye) , 8 boundary points of the iris edge, the corneal center, and the iris center), the line of sight estimation capsule layer performs a feature transformation on the second feature to obtain a human eye feature map with 18 feature channels, and each feature channel corresponds to a key point of the human eye.
[0073] In one embodiment, if Figure 6 As shown, the key points of the human eye include the center point of the iris, the center point of the cornea, and a plurality of limbal boundary points; in step S205, the line of sight is estimated according to the position information of the key points of the human eye to obtain the line of sight estimation result, which specifically includes the following steps:
[0074] S601: Calculate the yaw angle based on the abscissa corresponding to the iris center point, the abscissa corresponding to the cornea center point, and the abscissa corresponding to the limbus boundary point.
[0075] S602: Calculate the pitch angle based on the ordinate corresponding to the iris center point, the ordinate corresponding to the cornea center point, and the ordinate corresponding to the limbus boundary point.
[0076] In this embodiment, the pitch angle can be calculated by the horizontal coordinate corresponding to the iris center, the horizontal coordinate of the cornea center and the horizontal coordinate corresponding to the boundary point, and the yaw angle can be calculated by the vertical coordinate corresponding to the iris center, the vertical coordinate of the cornea center and the vertical coordinate corresponding to the boundary point.
[0077] In one embodiment, if Figure 7 As shown, in step S601, the yaw angle is calculated based on the horizontal coordinate corresponding to the iris center point, the horizontal coordinate of the cornea center point and the horizontal coordinate corresponding to the corneal edge boundary point, which specifically includes the following steps:
[0078] S701: If the horizontal coordinate corresponding to the iris center point is not greater than the horizontal coordinate of the cornea center point, obtain the minimum horizontal coordinate from the horizontal coordinates corresponding to the plurality of corneal limbus boundary points.
[0079] S702: Calculate the yaw angle based on the minimum horizontal coordinate, the horizontal coordinate corresponding to the cornea center point, and the horizontal coordinate corresponding to the iris center point.
[0080] S703: If the horizontal coordinate corresponding to the iris center point is not greater than the horizontal coordinate of the cornea center point, obtain the maximum horizontal coordinate from the horizontal coordinates corresponding to the plurality of corneal limbus boundary points.
[0081] S704: Calculate the yaw angle based on the maximum horizontal coordinate, the horizontal coordinate corresponding to the cornea center point, and the horizontal coordinate corresponding to the iris center point.
[0082] Specifically, the calculation process of the above pitch angle is expressed by the following formula: Among them, θ represents the yaw angle, and the coordinates of the corneal center are The coordinates of the iris center are Multiple boundary points of the limbus
[0083] In one embodiment, if Figure 8 As shown, in step S601, the yaw angle is calculated based on the horizontal coordinate corresponding to the iris center point, the horizontal coordinate of the cornea center point and the horizontal coordinate corresponding to the corneal edge boundary point, which specifically includes the following steps:
[0084] S801: If the ordinate corresponding to the iris center point is not greater than the ordinate of the cornea center point, obtaining the minimum ordinate from the ordinates corresponding to the plurality of limbal boundary points;
[0085] S802: Calculate the pitch angle based on the minimum ordinate, the ordinate corresponding to the cornea center point, and the ordinate corresponding to the iris center point;
[0086] S803: If the ordinate corresponding to the iris center point is greater than the abscissa of the cornea center point, obtaining the maximum ordinate from the ordinates corresponding to the plurality of limbal boundary points;
[0087] S804: Calculate the pitch angle based on the maximum ordinate, the ordinate corresponding to the cornea center point, and the ordinate corresponding to the iris center point.
[0088] Specifically, the calculation process of the above pitch angle is expressed by the following formula: Where φ represents the pitch angle, and the coordinates of the corneal center are The coordinates of the iris center are Multiple boundary points of the limbus
[0089] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0090] In one embodiment, a sight line estimation device is provided, which corresponds one-to-one to the sight line estimation method in the above embodiment. Fig. 9 As shown, the sight line estimation device includes a data acquisition module 10 , a feature point positioning module 20 , an encoding module 30 , a decoding module 40 and a sight line estimation module 50 .
[0091] The detailed description of each functional module is as follows:
[0092] The data acquisition module 10 is used to acquire a face image and a sight line estimation model trained based on a capsule network; wherein the sight line estimation model includes an encoder and a decoder.
[0093] The feature point positioning module 20 is used to locate feature points of the face image to obtain an eye region image.
[0094] The encoding module 30 is used to perform feature encoding on the eye region image through the encoder to obtain a human eye feature map; wherein the human eye feature map includes a plurality of human eye key points.
[0095] The decoding module 40 is used to decode the human eye feature map through the decoder to obtain a heat map corresponding to the eye area image; wherein the heat map includes the position information of each of the key points of the human eye.
[0096] The sight line estimation module 50 is used to perform sight line estimation according to the position information of the key points of the human eye to obtain a sight line estimation result.
[0097] Specifically, the sight line estimation device also includes an eye-opening detection module and a detection result processing module.
[0098] The eye-opening detection module is used to perform eye-opening detection on the eye region image to obtain a detection result.
[0099] The detection result processing module is used to execute the step of encoding the eye area image by the encoder to obtain a human eye feature map if the detection result is open eyes.
[0100] Specifically, the sight line estimation model corresponds to a target recognition area; wherein the target recognition area is used to indicate a left eye or a right eye; the sight line estimation device also includes a symmetry processing module and a coding module.
[0101] The symmetry processing module is used to perform mirror symmetry processing on the eye area image if the eye area corresponding to the eye area image is different from the target recognition area, so as to obtain a symmetrical image corresponding to the eye area image.
[0102] The encoding module is used to perform feature encoding on the symmetrical image through the encoder to obtain a human eye feature map.
[0103] Specifically, the encoder includes a convolution layer, a main capsule layer and a line of sight estimation capsule layer connected in sequence; the encoding module 30 includes a convolution unit, a feature combination unit and a feature change unit.
[0104] A convolution unit is used to extract features of the eye region image through the convolution layer to obtain a first feature output by the convolution layer.
[0105] A feature combination unit is used to perform feature combination on the output of the convolution layer through the main capsule layer to obtain a second feature of the output of the main capsule layer.
[0106] A feature transformation unit is used to perform feature transformation on the second feature through the line of sight estimation capsule layer to obtain the human eye feature map.
[0107] Specifically, the key points of the human eye include the center point of the iris, the center point of the cornea, and multiple corneal edge boundary points; the line of sight estimation result includes a pitch angle and a yaw angle; and the line of sight estimation module includes a yaw angle calculation unit and a pitch angle calculation unit.
[0108] The yaw angle calculation unit is used to calculate the yaw angle based on the horizontal coordinate corresponding to the iris center point, the horizontal coordinate of the cornea center point and the horizontal coordinate corresponding to the corneal limbus boundary point.
[0109] The pitch angle calculation unit is used to calculate the pitch angle based on the ordinate corresponding to the iris center point, the ordinate corresponding to the cornea center point, and the ordinate corresponding to the corneal limbus boundary point.
[0110] Specifically, the yaw angle calculation unit includes a minimum horizontal coordinate determining subunit, a first calculation subunit, a maximum horizontal coordinate determining subunit and a second calculation subunit.
[0111] The minimum horizontal coordinate determining subunit is used to obtain the minimum horizontal coordinate from the horizontal coordinates corresponding to the multiple corneal limbus boundary points if the horizontal coordinate corresponding to the iris center point is not greater than the horizontal coordinate of the cornea center point.
[0112] The first calculation subunit is used to calculate the yaw angle based on the minimum horizontal coordinate, the horizontal coordinate corresponding to the cornea center point, and the horizontal coordinate corresponding to the iris center point.
[0113] The maximum horizontal coordinate determining subunit is used to obtain the maximum horizontal coordinate from the horizontal coordinates corresponding to the multiple corneal limbus boundary points if the horizontal coordinate corresponding to the iris center point is not greater than the horizontal coordinate of the cornea center point.
[0114] The second calculation subunit is used to calculate the yaw angle based on the maximum horizontal coordinate, the horizontal coordinate corresponding to the cornea center point and the horizontal coordinate corresponding to the iris center point. Specifically, the first linear normalization processing unit includes a linear transformation processing unit and a normalization processing unit.
[0115] Specifically, the pitch angle calculation unit includes a minimum ordinate determination subunit, a first calculation subunit, a maximum ordinate determination subunit and a second calculation subunit.
[0116] The minimum ordinate determination subunit is used to obtain the minimum ordinate from the ordinates corresponding to the plurality of corneal limbus boundary points if the ordinate corresponding to the iris center point is not greater than the ordinate of the cornea center point.
[0117] The first calculation subunit is used to calculate the pitch angle based on the minimum vertical coordinate, the vertical coordinate corresponding to the cornea center point, and the vertical coordinate corresponding to the iris center point.
[0118] The maximum vertical coordinate determining subunit is used to obtain the maximum vertical coordinate from the vertical coordinates corresponding to the plurality of corneal limbus boundary points if the vertical coordinate corresponding to the iris center point is greater than the horizontal coordinate of the cornea center point.
[0119] The second calculation subunit is used to calculate the pitch angle based on the maximum vertical coordinate, the vertical coordinate corresponding to the cornea center point, and the vertical coordinate corresponding to the iris center point.
[0120] For the specific definition of the sight line estimation device, please refer to the definition of the sight line estimation method above, which will not be repeated here. Each module in the above sight line estimation device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0121] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a computer storage medium and an internal memory. The computer storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the computer storage medium. The database of the computer device is used to store data generated or obtained during the execution of the line of sight estimation method, such as a line of sight estimation model. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a line of sight estimation method is implemented.
[0122] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the sight line estimation method in the above embodiment are implemented, such as Figure 2 Steps S201-S205 shown, or Figures 3 to 8 Alternatively, when the processor executes the computer program, the functions of each module / unit in the embodiment of the sight line estimation device are realized, for example Fig. 9 The functions of the modules / units shown are not described here in detail to avoid repetition.
[0123] In one embodiment, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the sight line estimation method in the above embodiment are implemented, for example: Figure 2 Steps S201-S205 shown, or Figures 3 to 8 Alternatively, when the computer program is executed by a processor, the functions of each module / unit in the above-mentioned sight line estimation device embodiment are realized, for example Fig. 9 The functions of the modules / units shown are not described here in detail to avoid repetition.
[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0125] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0126] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. A line of sight estimation method, characterized in that: include: Acquire a face image and a sight line estimation model based on capsule network training; wherein the sight line estimation model includes an encoder and a decoder; Positioning feature points of the face image to obtain an eye region image; The encoder performs feature encoding on the eye region image to obtain a human eye feature map; wherein the human eye feature map includes a plurality of human eye key points; the human eye key points include an iris center point, a cornea center point, and a plurality of corneal limbus boundary points; Decoding the eye feature map through the decoder to obtain a heat map corresponding to the eye region image; wherein the heat map includes position information of each eye key point; Performing line of sight estimation according to the position information of the key points of the human eye to obtain a line of sight estimation result; the line of sight estimation result includes a pitch angle and a yaw angle; The line of sight is estimated based on the position information of the key points of the human eye to obtain a line of sight estimation result, including calculating the yaw angle based on the horizontal coordinate corresponding to the center point of the iris, the horizontal coordinate of the center point of the cornea, and the horizontal coordinate corresponding to the limbus boundary point; and calculating the pitch angle based on the vertical coordinate corresponding to the center point of the iris, the vertical coordinate of the center point of the cornea, and the vertical coordinate corresponding to the limbus boundary point.
2. The line of sight estimation method according to claim 1, characterized in that: Before the encoder performs feature encoding on the eye region image to obtain a human eye feature map, the sight line estimation method further includes: Performing eye-opening detection on the eye region image to obtain a detection result; If the detection result is that the eyes are open, the step of performing feature encoding on the eye area image by the encoder to obtain a human eye feature map is performed.
3. The line of sight estimation method according to claim 1, wherein: The sight estimation model corresponds to a target recognition area; wherein the target recognition area is used to indicate a left eye or a right eye; before the encoder performs feature encoding on the eye area image to obtain a human eye feature map, the method further includes: If the eye area corresponding to the eye area image is different from the target recognition area, performing mirror symmetry processing on the eye area image to obtain a symmetrical image corresponding to the eye area image; The step of encoding the eye region image by the encoder to obtain a human eye feature map includes: The encoder performs feature encoding on the symmetrical image to obtain a human eye feature map.
4. The line of sight estimation method according to claim 1, wherein: The encoder includes a convolutional layer, a main capsule layer, and a sight estimation capsule layer connected in sequence; the encoder performs feature encoding on the eye region image to obtain a human eye feature map, including: Performing feature extraction on the eye region image through the convolution layer to obtain a first feature output by the convolution layer; Performing feature combination on the output of the convolution layer through the main capsule layer to obtain a second feature of the output of the main capsule layer; The second feature is transformed by the line of sight estimation capsule layer to obtain the human eye feature map.
5. The line of sight estimation method according to claim 1, wherein: The calculating the yaw angle based on the abscissa corresponding to the iris center point, the abscissa corresponding to the cornea center point, and the abscissa corresponding to the limbus boundary point comprises: If the horizontal coordinate corresponding to the iris center point is not greater than the horizontal coordinate of the cornea center point, obtaining the minimum horizontal coordinate from the horizontal coordinates corresponding to the plurality of corneal limbus boundary points; Calculating the yaw angle based on the minimum horizontal coordinate, the horizontal coordinate corresponding to the cornea center point, and the horizontal coordinate corresponding to the iris center point; If the horizontal coordinate corresponding to the iris center point is not greater than the horizontal coordinate of the cornea center point, obtaining the maximum horizontal coordinate from the horizontal coordinates corresponding to the plurality of corneal limbus boundary points; The yaw angle is calculated based on the maximum horizontal coordinate, the horizontal coordinate corresponding to the cornea center point, and the horizontal coordinate corresponding to the iris center point.
6. The line of sight estimation method according to claim 1, wherein: The calculating the pitch angle based on the ordinate corresponding to the iris center point, the ordinate corresponding to the cornea center point, and the ordinate corresponding to the limbus boundary point comprises: If the ordinate corresponding to the iris center point is not greater than the ordinate of the cornea center point, obtaining the minimum ordinate from the ordinates corresponding to the plurality of corneal limbus boundary points; Calculating the pitch angle based on the minimum ordinate, the ordinate corresponding to the cornea center point, and the ordinate corresponding to the iris center point; If the ordinate corresponding to the iris center point is greater than the abscissa of the cornea center point, obtaining the maximum ordinate from the ordinates corresponding to the plurality of corneal limbus boundary points; The pitch angle is calculated based on the maximum ordinate, the ordinate corresponding to the cornea center point, and the ordinate corresponding to the iris center point.
7. A sight line estimation device, characterized in that: include: A data acquisition module, used to acquire a face image and a sight line estimation model trained based on a capsule network; wherein the sight line estimation model includes an encoder and a decoder; A feature point positioning module, used to locate feature points of the face image to obtain an eye area image; An encoding module, used for performing feature encoding on the eye region image through the encoder to obtain a human eye feature map; wherein the human eye feature map includes a plurality of human eye key points; the human eye key points include an iris center point, a cornea center point, and a plurality of corneal limbus boundary points; A decoding module, used for decoding the human eye feature map through the decoder to obtain a heat map corresponding to the eye area image; wherein the heat map includes the position information of each key point of the human eye; A sight line estimation module, used for performing sight line estimation according to the position information of the key points of the human eye to obtain a sight line estimation result; the sight line estimation result includes a pitch angle and a yaw angle; the sight line estimation module includes a yaw angle calculation unit and a pitch angle calculation unit; The yaw angle calculation unit is used to calculate the yaw angle based on the horizontal coordinate corresponding to the iris center point, the horizontal coordinate of the cornea center point and the horizontal coordinate corresponding to the corneal limbus boundary point; The pitch angle calculation unit is used to calculate the pitch angle based on the ordinate corresponding to the iris center point, the ordinate corresponding to the cornea center point, and the ordinate corresponding to the corneal limbus boundary point.
8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the sight line estimation method according to any one of claims 1 to 6 are implemented.
9. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the sight line estimation method according to any one of claims 1 to 6 are implemented.
