Method for obtaining head pose ground truth using gyroscopes combined with deep learning

Abstract

The application discloses a method for obtaining a head posture true value by combining a gyroscope and deep learning, and has the characteristics that the method comprises the following steps: 1) preparing a camera and a nine-axis Bluetooth gyroscope, wherein the camera is horizontally arranged at the height of a human head, is placed in front of the human face, and the nine-axis Bluetooth gyroscope is fixed on the top of the head of the collected person and the Z-axis faces upward; 2) calibrating the head posture, the human face faces the camera, and the deviation R between the nine-axis Bluetooth gyroscope and the actual head posture is updated t2p ; 3) reading the output value R of the nine-axis Bluetooth gyroscope corresponding to each frame of image t ; 4) calculating the head posture R of the frame according to the read output value R of the nine-axis Bluetooth gyroscope t and the deviation R t2p ; 5) judging whether the calculated head posture is close to the initial head posture, if yes, repeating the steps 2) to 4), if not, repeating the steps 3) to 4), and saving the image and simultaneously obtaining the head posture R of the collected person in the camera coordinate system in real time.

Description

Technical fields:

[0001] This invention belongs to the field of supervised deep learning training image acquisition technology, and specifically relates to a method for obtaining the true value of head posture by combining gyroscope and deep learning. Background technology:

[0002] Supervised deep learning techniques require that the training samples have corresponding ground truth values ​​as supervision. For example, training a head pose model requires a head image and corresponding head pose information. Current methods for obtaining head pose generally use the Proof-of-Nature (PNP) method, which matches a generic 3D facial landmark map with the corresponding 2D facial landmarks in the actual image to calculate the relative pose of the face in the image. However, this method has the following shortcomings: the 2D facial landmarks in the image contain errors; at large angles, especially when the head is turned more than 60°, the 2D facial landmarks themselves suffer from self-occlusion, and pixel-level errors also have a significant impact on the PNP calculation results. In addition, there are other methods for obtaining head posture using gyroscopes, but these all require a helmet to fix the equipment. These methods also have the following drawbacks: In real-world applications, helmets are not worn. For example, deep learning algorithms trained on samples collected while wearing a helmet may have significant errors when used without a helmet, especially at large angles; helmets and other equipment are heavy, and may loosen during movement, making the helmet unable to accurately represent the head posture; individual helmet-wearing habits differ, and even the same person may have different fixed angles when wearing a helmet twice. The same head posture may yield two significantly different ground truth values ​​due to helmet wearing issues, making it difficult to train an accurate model from such samples. Therefore, it is necessary to improve the above methods to enhance the accuracy of head posture ground truth values.

[0003] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention:

[0004] The purpose of this invention is to provide a method for obtaining the true value of head pose by combining a gyroscope with deep learning, thereby overcoming the defects in the prior art.

[0005] To achieve the above objectives, this invention provides a method for obtaining the true value of head pose using a gyroscope combined with deep learning, comprising the following steps:

[0006] 1) Prepare a camera and a nine-axis Bluetooth gyroscope. The camera should be an RGB or IR camera. The camera should use PC-side code to read and store images in real time. The camera should be positioned at the same height as person M's head, in front of their face. The nine-axis Bluetooth gyroscope should be fixed above person M's head with its Z-axis pointing upwards. The nine-axis Bluetooth gyroscope should use PC-side code to obtain the gyroscope angles pitch, yaw, and roll in real time, and the corresponding rotation matrix R should be calculated based on these angles. t The specific conversion formula is as follows:

[0007]

[0008]

[0009]

[0010] R t =R Z ·R x ·R y ;

[0011] 2) Calibrate head posture and update the deviation R between the nine-axis Bluetooth gyroscope and the actual head posture. t2p The specific calibration should be performed according to the following steps:

[0012] The face is facing the camera. The PC acquires the face image at time T and inputs it into the face key point detection algorithm to obtain the pixel coordinates of 68 key points of the face in the image. The face key point detection algorithm uses the existing algorithm of dlib.

[0013] Based on the known standard face 3D key point coordinates, combined with the PNP algorithm, the head pose R0 of the human head at time T is obtained. The standard face 3D key point coordinates are obtained by using open source coordinates or by reconstructing a dense face coordinate point using 3DMM technology and then selecting the 3D point coordinates corresponding to 68 points from it.

[0014] At time T, the output value R of the nine-axis Bluetooth gyroscope is read simultaneously. t The transformation relationship from the nine-axis gyroscope attitude to R0 was calculated, R t2p =R0·Rt -1 The deviation between the nine-axis gyroscope and the actual head posture was obtained;

[0015] 3) Read the output value R of the nine-axis Bluetooth gyroscope corresponding to each frame of the image. t ;

[0016] 4) Based on the read output value R of the nine-axis Bluetooth gyroscope t Combined bias R t2p The head pose R for this frame is calculated using the following formula: R = R t2p ·R t;

[0017] 5) Determine if the calculated head pose is close to the initial head pose. If close, repeat steps 2)-4). If not close, repeat steps 3)-4), and save the image while simultaneously acquiring the subject's head pose R in the camera coordinate system in real time. Specifically, when the head pose R and the initial head pose R0 (i.e., the head pose at time T) are within 10° in both the pitch and yaw directions, it is considered close, satisfying the correction R. t2p conditions.

[0018] Compared with the prior art, one aspect of the present invention has the following beneficial effects:

[0019] (1) This invention obtains the head posture angle through a nine-axis Bluetooth gyroscope. By combining the deviation between the nine-axis Bluetooth gyroscope and the actual head posture, the true head posture R can be calculated. It is applicable to the acquisition of head posture at any angle, avoiding the problem of not being able to obtain accurate data due to the excessive head deflection angle.

[0020] (2) This invention does not require helmets or other large-area head attachments, thus avoiding the impact of the scene on the collected data; and it is simple to install and convenient to collect data, which can solve the problem of needing a large amount of training data in deep learning. Attached image description:

[0021] Figure 1 This is a flowchart illustrating the method of obtaining the true value of head pose using a gyroscope combined with deep learning according to the present invention. Detailed implementation method:

[0022] The specific embodiments of the present invention will be described in detail below, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0023] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form to prepare for the more detailed descriptions that follow.

[0024] like Figure 1 As shown, the method for obtaining the true value of head pose using a gyroscope combined with deep learning includes the following steps:

[0025] 1) Prepare a camera and a nine-axis Bluetooth gyroscope. The camera should be either an RGB or IR camera; here, a standard RGB camera with at least 1 megapixel resolution is used. The camera's intrinsic parameters K are known. Fix the camera position, with the lens at eye level with person M's head, approximately 80cm in front of the face. Images can be read and stored in real-time using PC-side code. The nine-axis Bluetooth gyroscope is secured to the top of person M's head with the Z-axis pointing upwards using a thin string or transparent strap. Ensure the gyroscope does not move relative to the head during any head movement. After charging, use PC-side code to obtain the gyroscope's pitch, yaw, and roll angles in real-time, and calculate the corresponding rotation matrix R based on these angles. t The specific conversion formula is as follows:

[0026]

[0027]

[0028] R t =R Z ·R x ·R y ;

[0029] 2) Calibrate head posture and update the deviation R between the nine-axis Bluetooth gyroscope and the actual head posture. t2p The specific calibration should be performed according to the following steps:

[0030] The face is facing the camera. The PC acquires the face image at time T and inputs it into the face key point detection algorithm to obtain the pixel coordinates of 68 key points of the face in the image. The face key point detection algorithm here uses the existing algorithm of dlib.

[0031] Based on the known standard face 3D key point coordinates, combined with the PNP algorithm, the head pose R0 of the human head at time T is obtained. The standard face 3D key point coordinates are obtained by using open source coordinates or by reconstructing a dense face coordinate point using 3DMM technology and then selecting the 3D point coordinates corresponding to 68 points. Since PNP has relatively small and stable error when the human face is frontal, R0 can represent the head pose at time T.

[0032] At time T, the output value R of the nine-axis Bluetooth gyroscope is read simultaneously. t The transformation relationship from the nine-axis gyroscope attitude to R0 was calculated, R t2p =R0·Rt -1 The deviation between the nine-axis gyroscope and the actual head posture was obtained;

[0033] 3) Read the output value R of the nine-axis Bluetooth gyroscope corresponding to each frame of the image. tAfter calibration, the subject can be commanded to move their head and assume various head postures, and then the R value can be read. t value;

[0034] 4) Based on the read output value R of the nine-axis Bluetooth gyroscope t The calculated deviation R between the nine-axis Bluetooth gyroscope and the actual head posture. t2p Calculate the head pose R for this frame using the following formula: R = R t2p ·R t ;

[0035] 5) Determine if the calculated head pose is close to the initial head pose. If close, repeat steps 2)-4). If not close, repeat steps 3)-4), and save the image while simultaneously acquiring the subject's head pose R in the camera coordinate system in real time. Given R and R0, the deviation between the current head pose and the initial head pose is R0. err =R·R0, convert to Euler angles to obtain pitch and yaw respectively. If the absolute values ​​of pitch and yaw are both within 10°, it is determined to be close to the initial head posture state, which satisfies the calibration gyroscope deviation R. t2p If the conditions are met, otherwise it is considered not close enough and no calibration is performed; specifically, due to the drift problem of the gyroscope, the person being photographed needs to face the camera once every certain period of time (1 minute). When the head posture R calculated by the PC is close to the initial head posture R0, a calibration process can be triggered and R is updated. t2p If the subject is not close, images are continuously acquired, and the head pose R of the subject in the camera coordinate system is obtained in real time. R is decomposed into three basis vectors: v1(x1,y1,z1), v2(x2,y2,z2), and v3(x3,y3,z3), which are then converted into the corresponding angles:

[0036]

[0037] This invention uses a nine-axis Bluetooth gyroscope to obtain the head posture angle. By combining the deviation between the nine-axis Bluetooth gyroscope and the actual head posture, the true head posture R can be calculated. It is applicable to head posture acquisition at any angle, avoiding the problem of not being able to obtain accurate data due to excessive head deflection angle. Moreover, it does not require helmets or other large head attachments, thus avoiding the influence of the scene on the collected data. It is also simple to install and convenient for data acquisition, which can solve the problem of needing a large amount of training data in deep learning.

[0038] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for obtaining true head pose values ​​using a gyroscope combined with deep learning, characterized in that, Includes the following steps: 1) Prepare a camera and a nine-axis Bluetooth gyroscope. The camera should be positioned at the same height as person M's head and placed in front of the person's face. The nine-axis Bluetooth gyroscope should be fixed on top of the person being photographed, with the Z-axis facing upwards. 2) Calibrate head pose and update the bias R of the 9-axis Bluetooth gyroscope from the actual head pose t2p ; 3) read the nine-axis Bluetooth gyroscope output value R corresponding to each frame of image t ; 4) Calculate the frame header pose R from the read nine-axis Bluetooth gyroscope output values R t Combined bias R t2p Calculate the frame header pose R; 5) Determine whether the calculated head pose is close to the initial head pose. If they are close, repeat steps 2)-4) above. If they are not close, repeat steps 3)-4) above. Save the image and simultaneously acquire the head pose R of the person being captured in the camera coordinate system in real time. In the step 1), the nine-axis Bluetooth gyroscope uses PC end code to acquire the gyroscope angles pitch, yaw, and roll in real time, and a corresponding rotation matrix R of the angles is calculated t The specific conversion formula is as follows: ； ； When calibrating the head posture in step 2), the following steps should be followed: The face is facing the camera. The PC acquires the face image at time T and inputs it into the face key point detection algorithm to obtain the pixel coordinates of 68 key points of the face in the image. Based on the known coordinates of standard 3D face key points, and combined with the PNP algorithm, the head pose R0 of the human head at time T is obtained. At time T, the output value R of the nine-axis Bluetooth gyroscope is read simultaneously. t The conversion relationship from the nine-axis gyroscope attitude to R0 was calculated. The deviation between the nine-axis gyroscope and the actual head posture was obtained; According to the read nine-axis Bluetooth gyroscope output value R t The following formula is used to calculate the frame header attitude: ; The deviation between the current head pose and the initial head pose is The pitch and yaw are obtained by converting the values ​​to Euler angles. If the absolute values ​​of pitch and yaw are both within 10°, the head is considered to be close to the initial head posture state, which satisfies the calibration gyroscope deviation R. t2p If the condition is met, otherwise it is considered not close and no calibration is performed; when the head pose R calculated by the PC is close to the initial head pose R0, a calibration process is triggered and R is updated. t2p If the subject is not close, images will be continuously acquired, and the head pose R of the subject in the camera coordinate system will be obtained in real time.

2. The method of claim 1, wherein the head pose ground truth is obtained using a gyroscope in combination with deep learning. In step 1), the camera is an RGB camera or an IR camera, and the camera uses PC-side code to read and store images in real time.

3. The method of claim 1, wherein the head pose ground truth is obtained using a gyroscope in combination with deep learning. The facial landmark detection algorithm uses the dlib algorithm.

4. The method of claim 3, wherein the gyroscope is used in conjunction with deep learning to obtain the head pose ground truth. The standard 3D key point coordinates of the face are obtained by using open-source coordinates or by reconstructing a dense face coordinate point using 3DMM technology and then selecting 68 points corresponding to the 3D point coordinates.

Images