Method for improving eye movement control precision of multi-screen system

Abstract

The application provides a multi-screen system eye movement control precision improvement method, a face image of an operator is used to predict a screen currently interested or needed to be operated by the operator, the system automatically performs interest screen switching, and a camera corresponding to the screen is used to perform in-screen gaze tracking.The technology improves eye movement control precision, simplifies a cross-screen operation process, reduces system operation links and time, speeds up a human-computer interaction cycle, provides a new thought and reference for future human-computer interaction mode design, improves gaze tracking precision and robustness, can be used as a first rough estimation of wide-range cascade gaze estimation, and can be used as a basis for human-computer interaction process optimization.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction in multi-screen electronic information systems, and specifically relates to a method for improving the accuracy of eye-tracking control in multi-screen systems. Background Technology

[0002] With the development of modern information technology, the information collection and processing capabilities of various electronic information systems have become increasingly powerful. In a complex electronic information system, the human-computer interaction interface is often composed of multiple screens because the operator needs to process information from various sources and in various forms, and make judgments and operations. The way of human-computer interaction directly restricts the overall information processing efficiency of the human-machine interface.

[0003] Because operators face multiple screens, they need to select the screen of interest when acquiring information or performing system operations, and this screen of interest may switch frequently. Current mainstream human-computer interaction methods still rely on traditional hardware such as keyboards, mice, or gamepads. In these traditional hardware-based interaction methods, when there are numerous, rapid, and frequent cross-screen operations, the response time for a single manual operation is relatively long, requiring sufficient hand-eye coordination from the operator.

[0004] Currently, emerging eye-tracking-based eye-interaction technologies involve switching between interest screens. This is reflected in the interaction flow as the cursor, the currently selected control, or the gaze point moves across the screen. Switching interest screens means the operator's head turns, and the gaze point shifts significantly, inevitably leading to difficulties in eye tracking, including drift and errors in the gaze point and delays in fixation estimation. Because such operations occur frequently and repeatedly in actual interaction flows, the resulting difficulties in eye tracking affect the accuracy of eye-tracking control.

[0005] In existing eye-tracking control systems, monocular gaze tracking is often used, which involves tracking the gaze using a single camera. However, cross-screen operations are accompanied by large-scale and rapid movements of the gaze point, causing the gaze point to drift or jitter instantly after the large-scale shift, resulting in significant errors. This makes it difficult to accurately estimate the gaze point, and a certain amount of time is required to stabilize the gaze point after the shift. At the same time, with cross-screen operations, the operator's head posture will change to some extent. Due to the limitations of the neural network model that implements gaze tracking, when the facial image input to the network model has multiple head postures, the accuracy of gaze point estimation will drop rapidly.

[0006] In summary, existing technologies, such as conventional monocular gaze tracking, are insufficient to meet the multi-screen human-computer interaction needs of modern and future information systems. Further research is needed to make the human-computer interaction process more natural and reduce the operator's operational pressure and error rate. Summary of the Invention

[0007] The technical problem solved by this invention is: to address the difficulty of cross-screen eye tracking when operating a multi-screen system.

[0008] This invention proposes a method to improve the accuracy of eye-tracking control in a multi-screen system, which improves the smoothness of eye-tracking control and enables operators to switch screens naturally simply by turning their heads without manually operating the equipment, thus meeting the needs of natural human-computer interaction in highly complex information environments.

[0009] A method for improving eye-tracking control accuracy in a multi-screen system includes the following steps:

[0010] Step 1: Face detection and facial landmark recognition in the operator's head image, including the following sub-steps:

[0011] Step 1.1: Define a camera as an interest screen judgment camera, acquire the image to be detected through the interest screen judgment camera, and reduce the image to be detected by the same factor until the image size is greater than or equal to the set minimum face size;

[0012] Step 1.2: Input the series of images obtained in Step 1.1 into the fully convolutional network P-Net structure. After training, the final output face detection bounding box and the coordinates of five facial feature points, namely the left eye, right eye, nose, left corner of mouth and right corner of mouth, are obtained.

[0013] Step 2: Recognize the operator's head pose. Then, transform the coordinate system obtained in Step 1.2 with the camera coordinate system to obtain the final head pose.

[0014] Step 3: Predict interest screens and switch cameras based on head Euler angles, including the following steps:

[0015] Step 3.1: Set the Euler angle range for the field of view of each screen: when [θ x ,θ y ,θ z ]∈[{θ αn- ,θ βn- ,θ χn-},{θ αn+ ,θ βn+ ,θ χn+ When [θ] is selected, the system determines that the operator is interested in the nth screen and uses the camera on the nth screen to estimate the gaze point within the screen, where [θ] x ,θ y ,θ z [θ] represents the Euler angle of the operator's head. αn- ,θ βn- ,θ χn-},{θ αn+ ,θ βn+ ,θχn+}] represents the Euler angle interval where the nth screen is located. The specific upper and lower limits of the Euler angle interval are determined based on the relative positions of the camera and the screen relative to the operator in actual implementation.

[0016] Step 3.2: When the head's Euler angle enters the Euler angle range of the corresponding screen's field of view during actual operation, it is determined that the screen is of interest to the operator; at the same time, a threshold for the dwell time of the head's Euler angle is set. When the Euler angle stays within the Euler angle range of the corresponding screen's field of view for a longer time than the preset threshold, the system switches the image input source of the eye-tracking control to the camera corresponding to the current screen of interest, maintaining the natural and smooth eye-tracking control process.

[0017] Furthermore, in step 1.1, the operator's head image input from the camera is reduced by a specific factor. The reasonable reduction factor for the image side length is approximately in the range of 0.7 to 0.8. The image side length is continuously reduced by the reduction factor of the original side length until the image size is greater than or equal to the set minimum face size.

[0018] Furthermore, step 1.2 includes the following sub-steps:

[0019] Step 1.2.1: The initial P-Net network structure consists of 4 convolutional layers and 1 max pooling layer, using PreLU and Softmax activation functions; bounding box regression and non-maximum suppression are applied to the results obtained from the P-Net network structure to obtain face candidate boxes;

[0020] Step 1.2.2: Transform the above candidate face windows to an nxn size and input them into the next level convolutional network R-Net. The R-Net convolutional network structure includes 3 convolutional layers, 2 max pooling layers, and 3 fully connected layers. PreLU and Softmax activation functions are used to eliminate a large number of erroneous candidate face windows, and bounding box regression and non-maximum suppression are used again.

[0021] Step 1.2.3: Transform the candidate face window obtained above to a size of 2n x 2n, and input it into the next level convolutional network O-Net. The O-Net convolutional network structure includes 4 convolutional layers, 3 max pooling layers, and 4 fully connected layers. PreLU and Softmax activation functions are used to finally output the face detection bounding box and the coordinates of five facial feature points.

[0022] Furthermore, in step 1.2.1, the non-maximum suppression process is as follows: sort the scores of all candidate face windows and select the window with the highest score; traverse the remaining windows, and if the overlap area between a certain remaining window and the current highest-scoring window is greater than a certain threshold, then delete the window; after the traversal is completed, select the highest-scoring window from the unprocessed windows and iterate the above process.

[0023] Furthermore, step 2 includes the following sub-steps:

[0024] Step 2.1: Define o as the optical center of the camera, and p as the two-dimensional projection point of the three-dimensional point P onto the image captured by the camera. Assume that the point P[X] in the world coordinate system is known. w ,Y w Z w If the position is determined, a rotation matrix R can be used. 3×3 Translation matrix T 3×1 Indicates the transformation relationship between the world coordinate system and the camera coordinate system:

[0025]

[0026] Step 2.2: Introduce the transformation relationship from camera coordinates to image coordinates:

[0027]

[0028] In the above formula, f x and f x It is the focal length of the camera in the x and y directions, (c x ,c y ) is the optical center, and it is assumed that there is no radial distortion, and s is an unknown scaling factor in the equation;

[0029] Step 2.3: Combine the formulas from the above two steps to obtain the transformation relationship between the image coordinate system and the world coordinate system:

[0030]

[0031] Step 2.4: Iteratively solve the above equation using a linear transformation algorithm combined with least squares, rotating sequentially according to the yaw-pitch-roll pattern to obtain the rotation matrix R. 3×3 ,

[0032]

[0033] Euler angles can be solved using the above formula:

[0034]

[0035] The head pose can be obtained by solving the transformation relation. θ is one of the values ​​in the equation. x θ y θ z These correspond to the angles of pitch, roll, and yaw, respectively.

[0036] Invention Effects

[0037] The technical advantages of this invention are as follows: This invention uses the operator's facial image to predict the screen they are currently interested in or need to operate on. The system automatically switches between screens of interest and uses the camera corresponding to that screen for in-screen gaze tracking. This technology improves the accuracy of eye-tracking control, simplifies the process of cross-screen operation, reduces system operation steps and time, accelerates the human-computer interaction cycle, and provides a new approach and reference for the design of future human-computer interaction methods.

[0038] This method provides a screen selection assistance method for multi-screen system operation, and selects the camera corresponding to the screen for in-screen gaze tracking, which improves the accuracy and robustness of gaze tracking. It can be used as the first step of rough estimation in large-scale cascaded gaze estimation, or it can be used as a basis for optimizing the human-computer interaction process. Attached Figure Description

[0039] Figure 1 : Schematic diagram for calculating intersection-union ratio;

[0040] Figure 2 : A schematic diagram illustrating the transformation relationship between the camera coordinate system, image coordinate system, and world coordinate system;

[0041] Figure 3 Three-screen industrial PC;

[0042] Figure 4 ,: Output results of the first stage of face detection and facial landmark recognition;

[0043] Figure 5 : Output results of the second stage of face detection and facial landmark recognition;

[0044] Figure 6 : Output results of the third stage of face detection and facial landmark recognition;

[0045] Figure 7 Head pose recognition results;

[0046] Figure 8 Interest screen is screen 1 (right screen);

[0047] Figure 9 Interest screen is screen 2 (middle screen);

[0048] Figure 10 The interest screen is screen 3 (left screen);

[0049] Figure 11 The gaze point (red dot) falls on the left side of the screen;

[0050] Figure 12 The gaze point (red dot) falls on the right side of the screen; Detailed Implementation

[0051] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0052] See Figures 1-12 The main content of this method is as follows: In a multi-target screen system, a camera is set up for each screen. The operator's head image is used to predict the screen of interest, and the camera corresponding to the screen of interest that the operator is facing is used to track gaze within that screen. Further explanation: Multiple cameras are used to acquire frontal head images of the operator when facing the screen of interest. The face in the images is detected, the head pose is identified, and the Euler angle of the head orientation is determined. This predicts the operator's current screen of interest. The camera corresponding to that screen of interest is used to estimate the gaze point within the screen. When the screen of interest changes, the operator's head turns to the next screen of interest, and the system uses the camera of the next screen of interest to estimate the gaze point.

[0053] The technical solution of this invention includes the following steps:

[0054] Step 1: Face detection and facial landmark recognition in the operator's head image:

[0055] This invention utilizes multiple cameras for eye-tracking control. When determining the screen of interest, only one specific camera, or screen of interest determination camera, is used to acquire the operator's head image. A pre-trained convolutional neural network (MTCNN) is employed for face detection and facial landmark recognition, as follows:

[0056] The first stage involves reducing the size of the image to be detected, obtained from the camera at the interest screen, until it is just greater than or equal to the minimum face detection size. This generates a series of images with identical content but progressively smaller sizes—an image pyramid. Specifically, to detect faces of different sizes within the image at a uniform scale, the operator's head image input from the camera is reduced by a specific factor. A reasonable reduction factor for the image side length is approximately between 0.7 and 0.8. The selection of the reduction factor is related to the distribution of face sizes in the dataset and affects the overall program runtime. Too large a reduction factor results in a longer program runtime; too small a reduction factor makes it easy to miss or ignore smaller faces. In this invention, the reduction factor of 0.710 was selected through the following example experiments, resulting in the best face detection performance. The image side length is continuously reduced by 0.710 times the original side length until the image size is greater than or equal to the set minimum face size. The minimum face detection size determines the smallest face size that the convolutional neural network can recognize; faces smaller than this size will not be detected. If the minimum face detection size is too large, smaller faces may be missed or ignored; if the minimum face detection size is too small, the program will run for a long time. In this invention, the minimum face detection size selected through the following example experiments is 12. The generated series of images are input into a fully convolutional network P-Net, the network structure of which is shown in the table below.

[0057] Table 1 P-Net Network Structure

[0058]

[0059] Boundary regression and non-maximum suppression (NMS) are applied to the results obtained from the fully convolutional network P-Net to obtain candidate face boxes. The NMS process involves: sorting all candidate face windows by score and selecting the highest-scoring window; iterating through the remaining windows, and deleting any window whose overlap area with the highest-scoring window exceeds a certain threshold; after iteration, selecting the highest-scoring window from the unprocessed windows and iterating the above process. The overlap area between two windows is calculated using the Intersection over Union (IoU) or Intersection over Minimum (IoM) ratio. P-Net and R-Net use IoU, as illustrated in the diagram below. Figure 1 As shown in the figure, the ratio of the shaded areas is the intersection-union ratio (IU). In this invention, the IU threshold for P-Net decision window deletion, selected through the following example experiments, is 0.85.

[0060] Second stage: Transform the candidate face window to a size of 24x24 and input it into the next level convolutional network R-Net. Its network structure is shown in the table below.

[0061] Table 2 R-Net Network Structure

[0062]

[0063]

[0064] This step eliminates a large number of erroneous candidate face windows, and bounding box regression and non-maximum suppression are used again. In this invention, the cross-union threshold for P-Net decision window deletion, selected through the following example experiments, is 0.90.

[0065] The third stage involves transforming the candidate face windows obtained above to a size of 48x48 and inputting them into the next level convolutional network O-Net, whose network structure is shown in the table below.

[0066] Table 3 O-Net Network Structure

[0067]

[0068]

[0069] The final output includes the bounding box of the face detection and the coordinates of five facial feature points (left eye, right eye, nose, left corner of mouth, and right corner of mouth).

[0070] Step 2: Identify operator head position:

[0071] The coordinates of the 2D facial key points in the image are obtained from the aforementioned steps. To determine the coordinate correspondence between the 3D geometric position of a point on the surface of a spatial object and the corresponding point in the 2D image, it is necessary to solve the transformation relationship between the 2D and 3D points, i.e., the pose of the 3D object relative to the camera, which can be represented by translation and rotation matrices. This invention mainly utilizes rotation matrices to solve for head motion pose, which involves the transformation relationship between the camera coordinate system, image coordinate system, and world coordinate system, such as... Figure 2 As shown. Here, o is the center of the camera, and p is the two-dimensional projection of the three-dimensional point P onto the image captured by the camera. Assume that the point P[X] in the world coordinate system is known. w ,Y w Z w If the position is determined, a rotation matrix R can be used. 3×3 Translation matrix T 3×1 Indicates the transformation relationship between the world coordinate system and the camera coordinate system:

[0072]

[0073] Facial feature point detection can yield the coordinates [x, y] in the image coordinate system, but it cannot obtain the coordinates [x, y] of a point in 3D space in the camera coordinate system. c ,Y c Z c Therefore, it is necessary to introduce the transformation relationship from camera coordinates to image coordinates:

[0074]

[0075] In the above formula, f x and f x It is the focal length of the camera in the x and y directions, (c x ,c y The image coordinate system is assumed to be the optical center, with no radial distortion. 's' is an unknown scaling factor in the equation. Combining the two equations above, we can obtain the transformation relationship between the image coordinate system and the world coordinate system:

[0076]

[0077] The above equation is solved iteratively using a linear transformation algorithm combined with least squares, rotating sequentially according to the yaw-pitch-roll pattern, to obtain the rotation matrix R. 3×3 Its expression is as follows. Where R x For pitch rotating about the X-axis, R y R is the yaw angle about the Y-axis. z For roll, which rotates around the Z-axis.

[0078]

[0079]

[0080]

[0081]

[0082] R 3×3 Recorded as:

[0083]

[0084] Euler angles can be solved using the above formula:

[0085]

[0086] The head pose can be obtained by solving the transformation relation. θ is one of the values ​​in the equation. x θ y θ z These correspond to the angles of pitch, roll, and yaw, respectively.

[0087] Step 3: Predict interest screens and switch cameras based on head Euler angles:

[0088] The head pose obtained from the aforementioned steps includes Euler angle information for the head rotation around the X, Y, and Z axes. Considering the relatively fixed positional relationship between the operator's head and the interest screen, the camera, and each screen in actual use, this invention experimentally determines the Euler angle range for the head corresponding to each screen, thereby setting the Euler angle range for the field of view of each screen. When [θ x ,θ y ,θ z ]∈[{θ αn- ,θ βn- ,θ χn-},{θ αn+ ,θ βn+ ,θ χn+ When [θ] is selected, the system determines that the operator is interested in the nth screen and uses the camera on the nth screen to estimate the gaze point within the screen, where [θ] x ,θ y ,θ z [θ] represents the Euler angle of the operator's head. αn- ,θ βn- ,θ χn-},{θ αn+ ,θ βn+ ,θ χn+}] represents the Euler angle range of the nth screen. During actual operation, if the head's Euler angle falls within the Euler angle range of the corresponding screen's field of view, it is determined that the screen is of interest to the operator. To avoid misjudgments of changes in the interest screen due to head Euler angle disturbances caused by external factors, a head Euler angle dwell time threshold is set. When the Euler angle remains within the Euler angle range of the corresponding screen's field of view for more than the preset time threshold, the system switches the image input source for eye-tracking control to the camera corresponding to the current interest screen, maintaining a natural and smooth eye-tracking control process.

[0089] The embodiments of the present invention are described in detail below. These embodiments are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0090] This embodiment further illustrates the invention by using eye-tracking control to operate a drone swarm in simulated combat on a multi-screen industrial control computer.

[0091] Step 1: Face detection and facial landmark recognition of operator's head image

[0092] This embodiment uses, as follows Figure 3 The three-screen industrial control computer shown uses a camera on the middle screen to capture an image of the operator's head. It then uses the aforementioned convolutional neural network for face detection and facial landmark recognition. The first-stage face candidate box results are as follows: Figure 4 As shown, the results of the second-stage face candidate boxes are as follows: Figure 5 As shown, the face candidate boxes in the third stage are as follows: Figure 6 As shown.

[0093] Step 2: Identify the operator's head position

[0094] Based on the coordinates of the face candidate bounding box and facial key points in the 2D image coordinate system obtained in Step 1, the operator's head orientation information can be obtained by solving the transformation relationship. Figure 7 As shown in the figure, the short green line indicates the direction the operator's head is facing.

[0095] Step 3: Predict interest screens and switch cameras based on head Euler angles.

[0096] The system determines the screen of interest by setting a threshold for the operator's head Euler angle when operating each screen. Through preliminary experiments, a reasonable Euler angle range is established. When the operator's head Euler angle falls within the Euler angle range corresponding to a screen, the system outputs that screen as the screen of interest. Figure 8 , 9 As shown in Figure 10, the camera is switched according to the interest screen to stably estimate the gaze point. In the simulation experiment, the operator performs cross-screen operations, with their head facing the screen above (the left screen of the industrial control computer, such as...). Figure 11 (The red dot represents the estimated gaze point) Shift to the next screen of interest (the right screen of the industrial control computer, such as...) Figure 12 If the head stays within the Euler angle range corresponding to the interest screen for a period of time exceeding a set threshold, the system will switch the camera used for eye tracking control and use the camera corresponding to the screen as the image source for eye tracking control.

[0097] In this embodiment, eye-tracking control is used, allowing the operator to control the system by only moving their head and eyes. Multiple cameras are used to track the gaze of each eye on each screen. The camera is switched according to changes in head posture to ensure that the operator's frontal facial image is captured. This eliminates the gaze point estimation error caused by head rotation and avoids gaze point drift caused by large-scale gaze shifts. It provides natural and smooth eye-tracking control and provides a basis for improving human-computer collaboration efficiency.

[0098] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A method for improving eye-tracking control accuracy in a multi-screen system, characterized in that, Includes the following steps: Step 1: In the multi-target screen system, set up a camera for each screen to perform face detection and facial landmark recognition on the operator's head image, including the following sub-steps: Step 1.1: Define a camera as an interest screen judgment camera, acquire the image to be detected through the interest screen judgment camera, and reduce the image to be detected by the same factor until the image size is greater than or equal to the set minimum face size; Step 1.2: Input the series of images obtained in Step 1.1 into the fully convolutional network P-Net structure. After training, the final output face detection bounding box and the coordinates of five facial feature points, namely the left eye, right eye, nose, left corner of mouth and right corner of mouth, are obtained. Step 2: Recognize the operator's head pose by transforming the coordinate system obtained in Step 1.2 with the camera coordinate system to obtain the final head pose; Step 3: Predict interest screens and switch cameras based on head Euler angles, including the following steps: Step 3.1: Set the Euler angle range for the field of view of each screen: when At that time, the system determines the operator's actions regarding the first... We are interested in a block of screens and use the camera on the nth screen to estimate the gaze points within that screen. Euler angle for operator's head, The Euler angle interval where the nth screen is located is defined by the upper and lower limits of the Euler angle interval, which are determined based on the relative positions of the camera and the screen relative to the operator in actual implementation. Step 3.2: When the head's Euler angle enters the Euler angle range of the corresponding screen's field of view during actual operation, it is determined that the screen is of interest to the operator; at the same time, a threshold for the dwell time of the head's Euler angle is set. When the Euler angle stays within the Euler angle range of the corresponding screen's field of view for a longer time than the preset threshold, the system switches the image input source of the eye-tracking control to the camera corresponding to the current screen of interest, maintaining the natural and smooth eye-tracking control process.

2. The method for improving eye-tracking control accuracy in a multi-screen system as described in claim 1, characterized in that, In step 1.1, the operator's head image input from the camera is reduced by a specific factor. The reasonable reduction factor for the image side length is in the range of 0.7 to 0.

8. The image side length is continuously reduced by the reduction factor of the original side length until the image size is greater than or equal to the set minimum face size.

3. The method for improving eye-tracking control accuracy in a multi-screen system as described in claim 1, characterized in that, Step 1.2 includes the following sub-steps: Step 1.2.1: The initial P-Net network structure consists of 4 convolutional layers and 1 max pooling layer, using PreLU and Softmax activation functions; bounding box regression and non-maximum suppression are applied to the results obtained from the P-Net network structure to obtain candidate face windows; Step 1.2.2: Transform the above candidate face windows to an NxN size and input them into the next level convolutional network R-Net. The R-Net convolutional network structure includes 3 convolutional layers, 2 max pooling layers, and 3 fully connected layers. PreLU and Softmax activation functions are used to eliminate a large number of erroneous candidate face windows, and bounding box regression and non-maximum suppression are used again. Step 1.2.3: Transform the candidate face window in the above results to a size of 2N x 2N, and input it into the next level convolutional network O-Net. The O-Net convolutional network structure includes 4 convolutional layers, 3 max pooling layers, and 4 fully connected layers. PreLU and Softmax activation functions are used to finally output the face detection bounding box and the coordinates of five facial feature points.

4. The method for improving eye-tracking control accuracy in a multi-screen system as described in claim 3, characterized in that, In step 1.2.1, the non-maximum suppression process is as follows: sort the scores of all candidate face windows and select the window with the highest score; traverse the remaining windows, and if the overlap area between a certain remaining window and the current highest-scoring window is greater than a certain threshold, then delete the window; after the traversal is completed, select the highest-scoring window from the unprocessed windows and iterate the above process.

5. The method for improving eye-tracking control accuracy in a multi-screen system as described in claim 1, characterized in that, Step 2 includes the following sub-steps: Step 2.1: Define o as the optical center of the camera, and p as the two-dimensional projection point of point P in three-dimensional space onto the image captured by the camera. Assume that point P in the world coordinate system is known. Position can be determined using a rotation matrix. Translation matrix Indicates the transformation relationship between the world coordinate system and the camera coordinate system: in The coordinates of a point in three-dimensional space in the camera coordinate system; Step 2.2: Introduce the transformation relationship from camera coordinates to image coordinates: ； In the above formula, and It is the focal length of the camera in the x and y directions. It is the optical center, and it is assumed that there is no radial distortion. In the equation, it is an unknown scaling factor; Step 2.3: Combine the formulas from the above two steps to obtain the transformation relationship between the image coordinate system and the world coordinate system: ； Step 2.4: Iteratively solve the above equation using a linear transformation algorithm combined with least squares, rotating sequentially according to the yaw-pitch-roll sequence to obtain the rotation matrix. , Euler angles can be solved using the above formula: The head pose can be obtained by solving the transformation relation; among which... , , These correspond to the angles of pitch, roll, and yaw, respectively.

Images