A sight line detection method, device, equipment and medium

Abstract

The present application discloses a line of sight detection method, device, equipment and medium. The present application obtains a facial image of a target object, determines the head posture of the target object according to the facial image of the target object, and if the yaw angle and the pitch angle in the head posture meet preset conditions, performs three-dimensional line of sight analysis processing on the facial image to determine the line of sight of the target object. When the yaw angle and the pitch angle are within a preset angle range, the accuracy of line of sight detection can be guaranteed and the efficiency of line of sight detection can be improved. If any one of the yaw angle and the pitch angle does not meet the preset conditions, performs line of sight analysis processing on the facial image using a preset line of sight recognition model to determine the line of sight of the target object. This can solve the noise problem caused by sample imbalance when training the preset line of sight recognition model and improve the accuracy of line of sight detection.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a line of sight detection method, device, equipment and medium. Background Art

[0002] In applications such as human-computer interaction, autonomous driving, and security monitoring, gaze detection plays an increasingly important role. Gaze detection is a technology that monitors the direction of human gaze in three-dimensional space. In related technologies, gaze detection is generally performed using either a 3D gaze detection method or a deep learning-based gaze detection method. However, these detection methods cannot maintain high accuracy and efficiency continuously. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a line of sight detection method, device, equipment and medium, which can improve the accuracy and efficiency of line of sight detection under all working conditions.

[0004] In order to achieve the purpose of the above application, the present application provides a line of sight detection method, the method comprising:

[0005] Obtain a facial image of the target object;

[0006] determining a head posture of the target object based on the facial image of the target object;

[0007] If the yaw angle and pitch angle in the head posture meet the preset conditions, the facial image is subjected to three-dimensional line of sight analysis and processing to determine the line of sight of the target object;

[0008] If any one of the yaw angle and the pitch angle does not meet the preset conditions, a preset line of sight recognition model is used to perform line of sight analysis on the facial image to determine the line of sight of the target object.

[0009] In a possible implementation, before acquiring the facial image of the target object, the method may further include:

[0010] collecting image data of a target object using a target camera;

[0011] Obtaining the target object's facial image includes:

[0012] A face image of the target object is extracted from the image data of the target object.

[0013] In a possible embodiment, before collecting image data of the target object using the target camera, the method may further include:

[0014] Calibrate the intrinsic parameters and distortion coefficients of the target camera;

[0015] Determining the head posture of the target object according to the facial image of the target object includes:

[0016] Extract facial feature points from the target object’s facial image;

[0017] Determine the two-dimensional coordinates of facial feature points;

[0018] Determine the three-dimensional coordinates of facial feature points based on a preset three-dimensional face model;

[0019] The head posture of the target object is determined based on the internal parameters, distortion coefficients, two-dimensional coordinates of facial feature points, and three-dimensional coordinates of facial feature points.

[0020] In a possible embodiment, performing three-dimensional sight line analysis processing on the facial image to determine the sight line of the target object may include:

[0021] Determine a first target point, a second target point, and a third target point from the facial feature points, wherein the first target point and the second target point are symmetrical points of the eyes of the target object, and the third target point is the center point of the left eyelid or the right eyelid of the target object;

[0022] Determine the three-dimensional coordinates of the first target point, the three-dimensional coordinates of the second target point, and the three-dimensional coordinates of the third target point;

[0023] Determining a target plane according to the three-dimensional coordinates of the first target point, the three-dimensional coordinates of the second target point, and the three-dimensional coordinates of the third target point;

[0024] Determine the three-dimensional coordinates of a fourth target point according to the three-dimensional coordinates of the first target point and the three-dimensional coordinates of the second target point, where the fourth target point is located on a line connecting the first target point and the second target point and on an axis of symmetry of the target object's face;

[0025] Determine the normal of the target plane;

[0026] A straight line passing through the three-dimensional coordinates of the fourth target point and parallel to the normal line is determined as the sight line of the target object.

[0027] In a possible implementation, performing gaze analysis on a facial image using a preset gaze recognition model to determine the gaze of a target object includes:

[0028] The convolution layer in the preset gaze recognition model is used to perform feature extraction processing on the facial image to obtain feature information corresponding to the facial image;

[0029] The fully connected layer in the preset gaze recognition model is used to perform gaze regression processing on the feature information corresponding to the facial image to determine the gaze of the target object.

[0030] In a possible implementation, the preset condition includes a preset yaw angle range and a preset pitch angle range. The preset yaw angle range is -45 degrees to 45 degrees, and the preset pitch angle range is -20 degrees to 20 degrees.

[0031] On the other hand, the present application also provides a sight line detection device, the device comprising:

[0032] A facial image acquisition module, used to acquire a facial image of a target object;

[0033] A head posture determination module, used to determine the head posture of the target object according to the facial image of the target object;

[0034] A first sight line analysis module, configured to perform three-dimensional sight line analysis processing on the facial image to determine the sight line of the target object when the yaw angle and the pitch angle in the head posture meet preset conditions;

[0035] The second sight line analysis module is used to perform sight line analysis processing on the facial image using a preset sight line recognition model to determine the sight line of the target object when any one of the yaw angle and the pitch angle in the head posture does not meet the preset conditions.

[0036] On the other hand, the present application also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the above-mentioned line of sight detection method.

[0037] On the other hand, the present application also provides a computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned line of sight detection method.

[0038] On the other hand, the present application also provides a computer-readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the above-mentioned line of sight detection method.

[0039] On the other hand, the present application also provides a computer program product, including computer instructions, characterized in that when the computer instructions are executed by a processor, the above-mentioned line of sight detection method is implemented.

[0040] The implementation of this application has the following beneficial effects:

[0041] The present application obtains a facial image of a target object, determines the head posture of the target object based on the facial image of the target object, and if the yaw angle and the pitch angle in the head posture meet preset conditions, performs three-dimensional line of sight analysis processing on the facial image to determine the line of sight of the target object. When the yaw angle and the pitch angle are within a preset angle range, the accuracy of line of sight detection can be ensured and the efficiency of line of sight detection can be improved. If any one of the yaw angle and the pitch angle does not meet the preset conditions, a preset line of sight recognition model is used to perform line of sight analysis processing on the facial image to determine the line of sight of the target object. This can solve the noise problem caused by sample imbalance when training the preset line of sight recognition model and improve the accuracy of line of sight detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A flow chart of a sight line detection method provided in an embodiment of the present application;

[0044] Figure 2 A flowchart of a sight line detection method provided in another embodiment of the present application;

[0045] Figure 3 A flowchart of a sight line detection method provided in another embodiment of the present application;

[0046] Figure 4 A schematic diagram of a process for performing three-dimensional line of sight analysis on a facial image to determine the line of sight of a target object provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of a flow chart of performing line of sight analysis and processing on a facial image using a preset line of sight recognition model to determine the line of sight of a target object provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of the structure of a sight line detection device provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of the structure of an electronic device for line of sight detection provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0051] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0052] In order to implement the technical solution of the present application and enable more engineering and technical workers to easily understand and apply the present application, the working principle of the present application will be further explained in combination with specific embodiments.

[0053] The following describes an embodiment of a sight line detection method of the present application. Figure 1 This is a flowchart of a line of sight detection method provided in an embodiment of the present application. This specification provides method operation steps as described in the embodiment or flowchart, but based on conventional or non-creative labor, more or fewer operation steps may be included. The order of steps listed in the embodiment is only one way of executing the steps among many, and does not represent the only execution order. Specifically, Figure 1 As shown, the method may include:

[0054] S101: Acquire a facial image of a target object.

[0055] In the embodiments of this specification, the target object may refer to an object that needs to be sighted, for example, a person, a dog, etc., which is not limited in this disclosure. It should be noted that the following description of this disclosure takes a person as the target object.

[0056] In one example, the facial image of the target object can be obtained from the multimedia data collected by the image acquisition device. The multimedia data may include video stream data and image data. When the multimedia data collected by the image acquisition device is video stream data, the video stream data can be firstly processed by video frame processing to obtain single-frame image data. The facial image of the target object is extracted from the collected image data or the single-frame image data obtained by video frame processing. The facial image of the target object can also be obtained from a background server, which is not limited in the present disclosure.

[0057] S103: Determine head posture information of the target object according to the facial image of the target object.

[0058] In the embodiments of this specification, the head posture of the target object may refer to the head orientation of the target object, and the head orientation may be represented by the head posture angle. The head posture angle may include a pitch angle and a yaw angle, wherein the pitch angle represents the angle at which the target object raises or lowers its head, and the yaw angle represents the angle at which the target object turns its head.

[0059] In practical applications, the facial feature points of the target object can be determined based on the facial image of the target object, and then the pitch angle and yaw angle of the target object's head can be determined based on the facial feature points.

[0060] S105: Determine whether the yaw angle and pitch angle in the head posture meet preset conditions.

[0061] In the embodiment of the present specification, the preset condition may include a preset yaw angle range and a preset pitch angle range. The yaw angle and pitch angle in the head posture satisfy the preset condition, which means that the yaw angle satisfies the preset yaw angle range and the pitch angle satisfies the preset range. The middle value of the preset yaw angle range and the preset pitch angle range may be 0 degrees, and the difference between the two end values ​​of the preset yaw angle range and 0 degrees may not be greater than the first preset value, for example, the first preset value is 45 degrees, the preset yaw angle range is -45 degrees to 45 degrees, or the first preset value is 45 degrees, and the preset yaw angle range is -44 degrees to 45 degrees, and the present disclosure is not limited to this. The difference between the two end values ​​of the preset pitch angle and 0 degrees may not be greater than the second preset value, for example, the second preset value is 20 degrees, the preset pitch angle range may be -20 degrees to 20 degrees, or the second preset value is 20 degrees, and the preset pitch angle range is -20 degrees to 19 degrees, and the present disclosure is not limited to this.

[0062] In a preferred embodiment, the preset yaw angle range may be from -45 degrees to 45 degrees, and the preset pitch angle range may be from -20 degrees to 20 degrees.

[0063] S107: If the yaw angle and the pitch angle meet the preset conditions, perform three-dimensional line of sight analysis on the facial image to determine the line of sight of the target object.

[0064] In the embodiments of this specification, performing three-dimensional line of sight analysis and processing on a facial image refers to using the target point coordinates of the facial image in three-dimensional space to perform line of sight analysis and processing. The target point coordinates refer to the coordinates of points at preset locations of the facial image, for example, the target point coordinates may be the coordinates of the left and right corners of the mouth, the coordinates of eight position points on the eye contour line, etc. This disclosure does not limit the line of sight analysis and processing method.

[0065] S109: If any one of the yaw angle and the pitch angle does not meet the preset condition, a preset sight line recognition model is used to perform sight line analysis processing on the facial image to determine the sight line of the target object.

[0066] In the embodiments of this specification, the preset sight line recognition model refers to a sight line recognition model obtained by pre-training a preset neural network model using training data and sight line labels corresponding to the training data. The preset neural network model may be a residual neural network ResNet, a convolutional neural network LeNet, etc., which is not limited in this disclosure.

[0067] In practical applications, the training data and the sight line labels corresponding to the training data may have a corresponding relationship with the head posture, and multimedia data in which the yaw angle and pitch angle in the head posture do not meet the preset conditions are used as training data. Any one of the yaw angle and the pitch angle that does not meet the preset conditions may mean that the yaw angle does not meet the preset yaw angle range, and / or the pitch angle does not meet the preset pitch angle range, for example, when the preset yaw angle range is -45 degrees to 45 degrees and the preset pitch angle range is -20 degrees to 20 degrees. When the yaw angle is 60 degrees and the pitch angle is 10 degrees, it can be determined that any one of the yaw angle and the pitch angle does not meet the preset conditions.

[0068] In the above-mentioned embodiment, by acquiring a facial image of the target object, the head posture of the target object is determined according to the facial image of the target object. If the yaw angle and the pitch angle in the head posture meet the preset conditions, the facial image is subjected to three-dimensional line of sight analysis and processing to determine the line of sight of the target object. When the yaw angle and the pitch angle in the head posture meet the preset conditions, the line of sight of the target object can be determined by using a three-dimensional line of sight analysis and processing method with a small amount of data processing. While improving the data processing efficiency, the accuracy of the line of sight result of the target object can be guaranteed. When any one of the yaw angle and the pitch angle in the head posture does not meet the preset conditions, the facial image is subjected to line of sight analysis and processing by using a preset line of sight recognition model. When any one of the yaw angle and the pitch angle does not meet the preset conditions, the line of sight of the target object can be accurately analyzed. The preset line of sight recognition model only needs to perform line of sight analysis and processing on the facial image whose yaw angle and the pitch angle do not meet the preset conditions. While ensuring the accuracy, the data processing efficiency can be greatly improved.

[0069] Figure 2FIG. 1 is a flow chart of a sight line detection method according to an exemplary embodiment. In a possible implementation, before step S101, obtaining a facial image of a target object, the method may further include:

[0070] S201: Collect image data of a target object using a target camera.

[0071] In the embodiment of this specification, the image data of the target object at least includes a facial image of the target object.

[0072] Step S101, obtaining a facial image of a target object may include:

[0073] S203: Extracting the face image of the target object from the image data of the target object.

[0074] In the embodiments of the present specification, the face area in the image data can be located by performing region of interest detection on the image data of the target object, and then the facial feature points in the face area can be detected. The facial feature points refer to the key feature points of the face area, and the facial image of the target object can be determined based on the key feature points.

[0075] In this embodiment, the target camera is used to collect image data of the target object, and the facial image of the target object is extracted from the image data of the target object, so that the facial image of the target object can be determined.

[0076] Figure 3 FIG. 1 is a flow chart of a line of sight detection method according to an exemplary embodiment. In a possible implementation, before collecting image data of a target object using a target camera, the method may further include:

[0077] S301: Calibrate the intrinsic parameters and distortion coefficients of the target camera.

[0078] In actual use, the internal parameters of the target camera can be fx, fy, u0 and v0, respectively, where fx and fy are the projections of the focus of the target camera on the horizontal and vertical axes of the image, respectively, and u0 and v0 represent the horizontal and vertical pixel numbers of the difference between the center pixel coordinates of the image and the pixel coordinates of the image origin, respectively. The distortion coefficient can include radial distortion coefficient and tangential distortion coefficient. Radial distortion occurs in the process of converting the camera coordinate system to the image coordinate system, and tangential distortion occurs in the process of camera manufacturing. Since the plane of the photosensitive element is not parallel to the lens, tangential distortion will occur.

[0079] In this embodiment, step S103, determining the head posture of the target object according to the facial image of the target object may include:

[0080] S1031: Extract facial feature points from the facial image of the target object.

[0081] In the embodiments of this specification, facial feature points refer to key points of various parts in the facial image, which may be contour points or corner points. The extracted facial feature points may be a preset number. The facial feature points in the facial image may be extracted using a preset feature point extraction model, for example, the facial feature points may be extracted using a dlib facial key point detector.

[0082] In actual applications, the number of facial feature points can be 14, 21, 29, 68, etc. The facial feature points can be points located on 13 contour lines, and the 13 contour lines can be the outer contour, left eyebrow, right eyebrow, nose bridge, nose boundary, left upper eyelid, right upper eyelid, right lower eyelid, upper upper lip, and upper lower lip. This application does not limit this.

[0083] S1033: Determine the two-dimensional coordinates of facial feature points.

[0084] In the embodiments of this specification, the two-dimensional coordinates of the facial feature points refer to the coordinates of the facial feature points in a two-dimensional coordinate system, which may include an image coordinate system.

[0085] In practical applications, the two-dimensional coordinates of the facial feature points can be determined based on u0 and v0 in the intrinsic parameters of the target camera.

[0086] S1035: Determine the three-dimensional coordinates of facial feature points based on a preset three-dimensional face model.

[0087] In the embodiments of this specification, the three-dimensional coordinates of the facial feature points refer to the coordinates of the facial feature points in a three-dimensional coordinate system, which may include a world coordinate system. The preset three-dimensional face model may be a three-dimensional universal head rigid body model, that is, a head model is established in the world coordinate system, and the coordinates of each key point of the model in the world coordinate system are fixed and known.

[0088] In practical applications, the number of facial feature points is extracted from the facial image of the target object based on the number of key points on the preset three-dimensional face model. For example, if there are 68 key points on the face of the preset three-dimensional face model, when extracting facial feature points, the number of facial feature points to be extracted is 68, and the positions of the 68 facial feature points on the face are consistent with the positions of the key points on the face in the preset three-dimensional face model. According to the correspondence between the facial feature points and the key points in the preset three-dimensional face model, and the two-dimensional coordinates of the facial feature points, coordinate transformation is performed to determine the three-dimensional coordinates of the facial feature points in the world coordinate system.

[0089] S1037: Determine the head posture of the target object according to the internal parameters, the distortion coefficients, the two-dimensional coordinates of the facial feature points, and the three-dimensional coordinates of the facial feature points.

[0090] In practical applications, the intrinsic parameters fx, fy, u0 and v0 of the target camera can be used, where fx and fy are the projections of the focus of the target camera on the horizontal and vertical axes of the image, respectively, and u0 and v0 represent the horizontal and vertical pixel numbers of the difference between the center pixel coordinates of the image and the pixel coordinates of the image origin, respectively.

[0091] In the embodiments of the present specification, the intrinsic parameters fx, fy, u0 and v0 of the target camera and the distortion coefficient can be combined to solve the geometric transformation relationship from the coordinates of the facial feature points in the image coordinate system to the coordinates in the world coordinate system. When solving the geometric transformation relationship, a polynomial equation is constructed using the coordinates of multiple points in the image coordinate system and the corresponding point coordinates in the world coordinate system to obtain the head posture of the target object.

[0092] In this embodiment, by acquiring the intrinsic parameters and distortion coefficients of the target camera, facial feature points are extracted from the facial image of the target object, the two-dimensional coordinates of the facial feature points are determined, and the three-dimensional coordinates of the facial feature points are determined based on a preset three-dimensional face model. The head posture of the target object is determined according to the intrinsic parameters, distortion coefficients, the two-dimensional coordinates of the facial feature points and the three-dimensional coordinates of the facial feature points, thereby realizing head posture analysis and processing of the target object.

[0093] Figure 4 The present invention is a flow chart showing a method of performing three-dimensional line of sight analysis on a facial image to determine the line of sight of a target object according to an exemplary embodiment. In a possible implementation, step S107, performing three-dimensional line of sight analysis on a facial image to determine the line of sight of a target object may include:

[0094] S1071: Determine a first target point, a second target point, and a third target point from facial feature points.

[0095] In the embodiment of this specification, the first target point and the second target point may be symmetrical points of the eyes of the target object. The third target point may be the center point of the left eyelid or the right eyelid of the target object. Determining the first target point, the second target point, and the third target point in the eye area may improve the accuracy of subsequent sight line detection.

[0096] S1073: Determine the three-dimensional coordinates of the first target point, the three-dimensional coordinates of the second target point, and the three-dimensional coordinates of the third target point.

[0097] By determining the three-dimensional coordinates of the facial feature points based on the preset three-dimensional face model in step S1035, the three-dimensional coordinates of all facial feature points can be determined. After determining the first target point, the second target point, and the third target point, the three-dimensional coordinates of the first target point, the second target point, and the third target point can be extracted from the three-dimensional coordinates of the full facial feature points.

[0098] S1075: Determine a target plane according to the three-dimensional coordinates of the first target point, the three-dimensional coordinates of the second target point, and the three-dimensional coordinates of the third target point.

[0099] According to the principle of determining the same plane by three points that are not on the same straight line, the target plane is determined according to the three-dimensional coordinates of the first target point, the second target point and the third target point. The target plane is perpendicular to the line of sight of the target object.

[0100] S1077: Determine the three-dimensional coordinates of the fourth target point according to the three-dimensional coordinates of the first target point and the three-dimensional coordinates of the second target point.

[0101] In the embodiment of this specification, the fourth target point is located on the line connecting the first target point and the second target point, for example, the fourth target point is the midpoint of the line connecting the first target point and the second target point. The three-dimensional coordinates of the fourth target point are determined according to the positional relationship between the fourth target point and the first target point and the second target point, the three-dimensional coordinates of the first target point and the three-dimensional coordinates of the second target point.

[0102] The fourth target point is located on the line connecting the first target point and the second target point, and is located on the symmetry axis of the target object's face.

[0103] S1079: Determine the normal of the target plane.

[0104] In the embodiments of this specification, the target plane refers to a plane parallel to the face of the person, the target plane is perpendicular to the sight line of the target object, and the normal of the target plane is parallel to or coincides with the sight line of the target plane.

[0105] S1081: Determine that a straight line passing through the fourth target point and parallel to the normal line is the sight line of the target object.

[0106] In the embodiment of this specification, the fourth target point is on the line of sight of the target object, the normal of the target plane is parallel to or coincides with the line of sight of the target plane, and the line of sight of the target object can be determined based on the three-dimensional coordinates of the fourth target point and the position of the target plane.

[0107] In the above embodiment, the first target point, the second target point and the third target point are determined from the facial feature points, the coordinates of the first target point, the second target point and the third target point are determined, the target plane is determined according to the coordinates of the first target point, the second target point and the third target point, the three-dimensional coordinates of the fourth target point are determined according to the three-dimensional coordinates of the first target point and the second target point, and then the normal of the target plane is determined, and the straight line passing through the three-dimensional coordinates of the fourth target point and parallel to the normal is determined as the line of sight of the target object. Simple mathematical principles can be used to determine the line of sight of the target object, and the accuracy of the obtained line of sight results can be ensured while greatly improving the processing efficiency of the target object line of sight analysis.

[0108] Figure 5 The present invention is a flow chart showing a method of performing a sight line analysis on a facial image using a preset sight line recognition model to determine the sight line of a target object according to an exemplary embodiment. In a possible implementation, performing a sight line analysis on a facial image using a preset sight line recognition model to determine the sight line of a target object in step S109 may include:

[0109] S1091: Perform feature extraction processing on the facial image using the convolution layer in the preset gaze recognition model to obtain feature information corresponding to the facial image.

[0110] In the embodiment of this specification, the number of convolutional layers in the preset sight line recognition model is at least one.

[0111] S1093: Performing gaze regression processing on feature information corresponding to the facial image using the fully connected layer in the preset gaze recognition model to determine the gaze of the target object.

[0112] In the embodiments of this specification, the preset sight line recognition model may be a ResNet network model or a LeNet-5 network model, which may include at least one convolution layer, at least one pooling layer, and at least one fully connected layer. The convolution kernel in the convolution layer may be used to extract features from the facial image, and the fully connected layer may be used to perform sight line regression processing on the feature information corresponding to the facial image.

[0113] In practical applications, before using a preset sight line recognition model to perform sight line analysis and processing on a target object, it is necessary to first train the training data and the sight line labels corresponding to the training data to obtain a preset sight line recognition model. When obtaining training data, a large amount of image data can be first classified. The first type of image data is image data in which the yaw angle and pitch angle in the head posture meet the preset conditions, and the second type of image data is image data in which any one of the yaw angle and pitch angle in the head posture does not meet the preset conditions. The second type of image data is used as training data, and the sight line labels corresponding to the second type of image data are obtained. The second type of image data and the corresponding sight line labels are trained to obtain a preset sight line recognition model, which can greatly improve the training efficiency of the preset sight line recognition model.

[0114] Figure 6 FIG. 1 is a structural diagram of a sight line detection device according to an exemplary embodiment. Figure 6 , the device may include:

[0115] The facial image acquisition module 401 is used to acquire the facial image of the target object.

[0116] The head posture determination module 403 is used to determine the head posture of the target object according to the facial image of the target object.

[0117] The judgment module 405 is used to judge whether the yaw angle and pitch angle in the head posture meet the preset conditions.

[0118] The first sight line analysis module 407 is used to perform three-dimensional sight line analysis processing on the facial image to determine the sight line of the target object when the yaw angle and the pitch angle in the head posture meet preset conditions.

[0119] The second sight line analysis module 409 is used to perform sight line analysis processing on the facial image using a preset sight line recognition model to determine the sight line of the target object when any one of the yaw angle and the pitch angle in the head posture does not meet the preset condition.

[0120] By acquiring a facial image of a target object, the head posture of the target object is determined according to the facial image of the target object. If the yaw angle and the pitch angle in the head posture meet the preset conditions, the facial image is subjected to three-dimensional line of sight analysis and processing to determine the line of sight of the target object. When the yaw angle and the pitch angle in the head posture meet the preset conditions, the line of sight of the target object can be determined by using a three-dimensional line of sight analysis and processing method with a small amount of data processing. While improving the data processing efficiency, the accuracy of the line of sight result of the target object can be guaranteed. When any one of the yaw angle and the pitch angle in the head posture does not meet the preset conditions, the facial image is subjected to line of sight analysis and processing by using a preset line of sight recognition model. When any one of the yaw angle and the pitch angle does not meet the preset conditions, the line of sight of the target object can be accurately analyzed. The preset line of sight recognition model only needs to perform line of sight analysis and processing on the facial image whose yaw angle and the pitch angle do not meet the preset conditions. While ensuring the accuracy, the data processing efficiency can be greatly improved.

[0121] In some embodiments, the apparatus may further include:

[0122] The image data acquisition module is used to acquire image data of a target object using a target camera.

[0123] Correspondingly, the facial image acquisition module 401 can be used to extract the facial image of the target object from the image data of the target object.

[0124] In this embodiment, the facial image of the target object can be determined by using the image data acquisition module and the facial image acquisition module.

[0125] In some embodiments, the apparatus may further include:

[0126] The parameter calibration module is used to calibrate the intrinsic parameters and distortion coefficients of the target camera.

[0127] The head posture determination module 403 may include:

[0128] The feature point extraction unit is used to extract facial feature points from the facial image of the target object.

[0129] The coordinate determination unit is used to determine the two-dimensional coordinates of the facial feature points.

[0130] The three-dimensional coordinate determination unit is used to determine the three-dimensional coordinates of facial feature points based on a preset three-dimensional face model.

[0131] The head posture determination unit is used to determine the head posture of the target object based on the internal parameters, the distortion coefficient, the two-dimensional coordinates of the facial feature points and the three-dimensional coordinates of the facial feature points.

[0132] In some embodiments, the first sight line analysis module 405 may include:

[0133] The target point determination unit is used to determine the first target point, the second target point and the third target point from the facial feature points, wherein the first target point and the second target point are symmetrical points of the target object's eyes, and the third target point is the center point of the target object's left eyelid or right eyelid.

[0134] The coordinate determination unit is used to determine the three-dimensional coordinates of the first target point, the three-dimensional coordinates of the second target point, and the three-dimensional coordinates of the third target point.

[0135] The target plane determining unit is used to determine the target plane according to the three-dimensional coordinates of the first target point, the three-dimensional coordinates of the second target point and the three-dimensional coordinates of the third target point.

[0136] The fourth target point determination unit is used to determine a fourth target point according to the three-dimensional coordinates of the first target point and the three-dimensional coordinates of the second target point. The fourth target point is located on the line connecting the first target point and the second target point and on the symmetry axis of the target object's face.

[0137] A determination unit is found to determine the normal of the target plane.

[0138] The sight line determination unit is used to determine that a straight line passing through the fourth target point and parallel to the normal line is the sight line of the target object.

[0139] In some embodiments, the second sight line analysis module 407 may include:

[0140] The first processing unit is used to perform feature extraction processing on the facial image by using the convolution layer in the preset sight line recognition model to obtain feature information corresponding to the facial image.

[0141] The second processing unit is used to perform line of sight regression processing on feature information corresponding to the facial image by using a fully connected layer in a preset line of sight recognition model to determine the line of sight of the target object.

[0142] On the other hand, the present application also provides an embodiment of an electronic device, Figure 7 A structural diagram of an electronic device for line of sight detection is shown, the electronic device may be a server, and the electronic device may include a processor, a memory and a network interface connected to a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for line of sight detection is implemented.

[0143] On the other hand, the present application also provides a computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can perform the line of sight detection method in the embodiment of the present disclosure. The computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0144] On the other hand, the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the line of sight detection method in the embodiment of the present disclosure.

[0145] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0146] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims of the present invention, the inventive aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0147] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0148] In addition, those skilled in the art will appreciate that, although the embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims of the present invention, any one of the claimed embodiments may be used in any combination.

[0149] The present invention may also be implemented as a device or system program (such as a computer program and a computer program product) for performing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or may be provided on a carrier signal, or in any other form.

[0150] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps, etc., that are not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim that lists several systems, several of these systems may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order, and these words may be interpreted as names.

Claims

1. A line of sight detection method, It is characterized in that The method comprises: Obtain a facial image of the target object; Determining the head posture of the target object according to the facial image of the target object; the step of determining the head posture of the target object according to the facial image of the target object comprises: Extracting facial feature points from the facial image of the target object; Determine the two-dimensional coordinates of the facial feature points; Determine the three-dimensional coordinates of the facial feature points based on a preset three-dimensional facial model; Determine the head posture of the target object according to the internal parameters of the target camera, the distortion coefficient of the target camera, the two-dimensional coordinates of the facial feature points, and the three-dimensional coordinates of the facial feature points; If the yaw angle and the pitch angle in the head posture meet preset conditions, three-dimensional line of sight analysis processing is performed on the facial image to determine the line of sight of the target object; the preset conditions include a preset yaw angle range and a preset pitch angle range; If any one of the yaw angle and the pitch angle does not satisfy the preset condition, performing line of sight analysis processing on the facial image using a preset line of sight recognition model to determine the line of sight of the target object; any one of the yaw angle and the pitch angle not satisfying the preset condition means that the yaw angle does not satisfy the preset yaw angle range, and / or the pitch angle does not satisfy the preset pitch angle range; The performing three-dimensional line of sight analysis and processing on the facial image refers to performing line of sight analysis and processing using the target point coordinates of the facial image in three-dimensional space, wherein the target point coordinates refer to the coordinates of a preset part point of the facial image, and determining the line of sight of the target object includes: Determine a first target point, a second target point, and a third target point from the facial feature points, wherein the first target point and the second target point are symmetrical points of the eyes of the target object, and the third target point is the center point of the left eyelid or the right eyelid of the target object; Determine the three-dimensional coordinates of the first target point, the three-dimensional coordinates of the second target point, and the three-dimensional coordinates of the third target point; determining a target plane according to the three-dimensional coordinates of the first target point, the three-dimensional coordinates of the second target point, and the three-dimensional coordinates of the third target point; Determine the three-dimensional coordinates of a fourth target point according to the three-dimensional coordinates of the first target point and the three-dimensional coordinates of the second target point, wherein the fourth target point is located on a line connecting the first target point and the second target point and on an axis of symmetry of the target object's face; determining a normal to the target plane; A straight line passing through the three-dimensional coordinates of the fourth target point and parallel to the normal line is determined as the sight line of the target object.

2. The method according to claim 1, It is characterized in that Before acquiring the facial image of the target object, the method further includes: Using a target camera to collect image data of the target object; The step of acquiring the facial image of the target object comprises: A facial image of the target object is extracted from the image data of the target object.

3. The method according to claim 2, It is characterized in that Before collecting the image data of the target object by using the target camera, the method further includes: Calibrate the intrinsic parameters and distortion coefficients of the target camera.

4. The method according to claim 1, It is characterized in that The performing line of sight analysis processing on the facial image using a preset line of sight recognition model to determine the line of sight of the target object comprises: Performing feature extraction processing on the facial image using the convolution layer in the preset sight line recognition model to obtain feature information corresponding to the facial image; The fully connected layer in the preset sight line recognition model is used to perform sight line regression processing on the feature information corresponding to the facial image to determine the sight line of the target object.

5. The method according to claim 1, It is characterized in that The preset yaw angle range is from -45 degrees to 45 degrees, and the preset pitch angle range is from -20 degrees to 20 degrees.

6. A sight line detection device, It is characterized in that The device comprises: A facial image acquisition module, used to acquire a facial image of a target object; A head posture determination module is used to determine the head posture of the target object according to the facial image of the target object; the step of determining the head posture of the target object according to the facial image of the target object includes: Extracting facial feature points from the facial image of the target object; Determine the two-dimensional coordinates of the facial feature points; Determine the three-dimensional coordinates of the facial feature points based on a preset three-dimensional facial model; Determine the head posture of the target object according to the internal parameters of the target camera, the distortion coefficient of the target camera, the two-dimensional coordinates of the facial feature points, and the three-dimensional coordinates of the facial feature points; A first sight line analysis module, configured to perform three-dimensional sight line analysis processing on the facial image to determine the sight line of the target object when the yaw angle and the pitch angle in the head posture meet preset conditions; the preset conditions include a preset yaw angle range and a preset pitch angle range; A second sight line analysis module is used to perform sight line analysis processing on the facial image using a preset sight line recognition model to determine the sight line of the target object when any one of the yaw angle and the pitch angle in the head posture does not meet the preset condition; any one of the yaw angle and the pitch angle does not meet the preset condition means that the yaw angle does not meet the preset yaw angle range, and / or the pitch angle does not meet the preset pitch angle range; The first sight line analysis module performs three-dimensional sight line analysis on the facial image, which refers to using the target point coordinates of the facial image in three-dimensional space to perform sight line analysis, wherein the target point coordinates refer to the coordinates of a preset part point of the facial image, and determining the sight line of the target object includes: Determine a first target point, a second target point, and a third target point from the facial feature points, wherein the first target point and the second target point are symmetrical points of the eyes of the target object, and the third target point is the center point of the left eyelid or the right eyelid of the target object; Determine the three-dimensional coordinates of the first target point, the three-dimensional coordinates of the second target point, and the three-dimensional coordinates of the third target point; determining a target plane according to the three-dimensional coordinates of the first target point, the three-dimensional coordinates of the second target point, and the three-dimensional coordinates of the third target point; Determine the three-dimensional coordinates of a fourth target point according to the three-dimensional coordinates of the first target point and the three-dimensional coordinates of the second target point, wherein the fourth target point is located on a line connecting the first target point and the second target point and on an axis of symmetry of the target object's face; determining a normal to the target plane; A straight line passing through the three-dimensional coordinates of the fourth target point and parallel to the normal line is determined as the sight line of the target object.

7. An electronic device, It is characterized in that The device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the line of sight detection method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, It is characterized in that The storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the line of sight detection method as described in any one of claims 1 to 5.

9. A computer program product comprising computer instructions, It is characterized in that When the computer instructions are executed by a processor, the line of sight detection method described in any one of claims 1 to 5 is implemented.

Images