Head posture estimation method and system, electronic equipment and storage medium

Abstract

The invention provides a head posture estimation method and system, electronic equipment and a storage medium. The method comprises the following steps: acquiring image data acquired by a camera and measurement data of an inertial measurement unit; performing time synchronization processing on the image data and the measurement data; obtaining correction information based on the image data and the measurement data after the time synchronization processing; correcting real-time measurement data of the inertial measurement unit based on the correction information to obtain a head correction posture; and determining a real-time head posture according to the head correction posture. According to the method, the correction information is determined by using the image data obtained by the camera and the measurement data of the inertial measurement unit, and the error of the real-time head attitude estimated by the inertial measurement unit is corrected by using the correction information, so that the accuracy of the real-time head attitude estimated by the inertial measurement unit is improved.

Description

Technical Field

[0001] This application relates to the field of near-eye display technology, and in particular to a method and system for estimating head posture, an electronic device, and a storage medium. Background Technology

[0002] Many driving schools currently use simulation technology for driver training to enhance the driving experience. In simulated driving environments, to provide users with a superimposed virtual and real experience, a large screen is usually installed in front of the user for augmented reality (AR) display. However, this method is costly due to issues such as the large screen footprint. Therefore, head-mounted near-eye devices (such as AR glasses) have become the best choice to improve the simulated driving experience and reduce costs.

[0003] When using head-mounted near-eye devices as driving simulation display devices, using attitude estimation algorithms to estimate head posture can provide a more realistic sense of immersion and a more efficient training experience.

[0004] When estimating head pose, one approach is to capture facial images using a camera to obtain the head pose. However, this method suffers from poor stability due to the low frame rate of the camera, making it impossible to acquire head pose in real time. Another approach is to use pose estimation algorithms based on Inertial Measurement Units (IMUs). However, when using IMUs for pose estimation, the IMU's own drift can cause significant pose deviations and low accuracy in motion estimations during certain educational and training scenarios. Summary of the Invention

[0005] This application provides a method and system for estimating head posture, an electronic device, and a storage medium. It uses image data obtained from a camera and measurement data from an inertial measurement unit (IMU) to obtain correction information for head posture estimation by the IMU, and uses this correction information to correct the error of head posture estimation by the IMU. This avoids the shortcomings of using a camera or IMU alone to estimate head posture and improves the accuracy of head posture estimation by the IMU.

[0006] In a first aspect, this application provides a method for estimating head posture, applied to a head posture estimation system. The estimation system includes a camera and a head-mounted near-eye device, the head-mounted near-eye device including an inertial measurement unit (IMU). The estimation method includes: acquiring image data collected by the camera and measurement data from the IMU; performing time synchronization processing on the image data and the measurement data; obtaining correction information based on the time-synchronized image data and the measurement data; correcting the real-time measurement data of the IMU based on the correction information to obtain a corrected head posture; and determining the real-time head posture based on the corrected head posture.

[0007] In this embodiment, correction information for head posture estimation by the inertial measurement unit (IMU) is obtained by using image data from the camera and measurement data from the IMU. This correction information is then used to correct the error in head posture estimation by the IMU, resulting in a corrected head posture. Based on the corrected head posture, the real-time head posture is determined. This reduces the error caused by temperature drift of the IMU, improves the accuracy of the real-time head posture estimation by the IMU, avoids the shortcomings of using either the camera or the IMU alone to estimate head posture, and reduces the error caused by temperature drift of the IMU.

[0008] In some embodiments, the time synchronization processing of the image data and the measurement data includes: marking the image data with a first timestamp and the measurement data with a second timestamp according to the system clock of the estimation system; processing the image data marked with the first timestamp and the measurement data marked with the second timestamp to obtain a first head state corresponding to each first timestamp of the image data and a second head state corresponding to each second timestamp of the measurement data; and synchronizing the time of the image data and the measurement data using the first timestamp corresponding to the first head state being in a first head stationary state and the second timestamp corresponding to the second head state being in a second head stationary state.

[0009] In this embodiment, time synchronization is achieved by using the timestamp when the head is in a static state, which reduces the synchronization time and difficulty, consumes less computing power, and can save costs and reduce production complexity.

[0010] In some embodiments, the time synchronization processing of the image data and the measurement data includes: marking the image data with a first timestamp and the measurement data with a second timestamp according to the system clock of the estimation system; processing the image data marked with the first timestamp and the measurement data marked with the second timestamp to obtain a first yaw angle of the image data at each first timestamp and a second yaw angle of the measurement data at each second timestamp; and synchronizing the time of the image data and the measurement data using the first yaw angle corresponding to each first timestamp and the second yaw angle corresponding to each second timestamp.

[0011] In this embodiment, the accuracy of time synchronization can be improved by using the yaw angle to dynamically synchronize the time of image data and measurement data.

[0012] In some embodiments, obtaining correction information based on the time-synchronized image data and the measurement data includes: processing the time-synchronized image data and the measurement data to obtain a first yaw angle corresponding to the image data at a first time moment, a first yaw angle corresponding to the image data at a second time moment, a second yaw angle corresponding to the measurement data at a first time moment, and a second yaw angle corresponding to the measurement data at a second time moment; determining the difference between the first yaw angle corresponding to the image data at the first time moment and the second yaw angle corresponding to the measurement data at the first time moment to obtain a first yaw angle difference; determining the difference between the first yaw angle corresponding to the image data at the second time moment and the second yaw angle corresponding to the measurement data at the second time moment to obtain a second yaw angle difference; and obtaining correction information based on the first yaw angle difference and the second yaw angle difference.

[0013] In this embodiment, the error between the estimated yaw angles of the image data and measurement data at two different times can be calculated to determine the correction information. Subsequently, the inertial measurement unit is calibrated using this correction information, which can improve the accuracy of the real-time head attitude estimated by the inertial measurement unit.

[0014] In some embodiments, the step of correcting the real-time measurement data of the inertial measurement unit based on the correction information to obtain the head correction posture includes: acquiring a preset number of correction steps; obtaining step-by-step correction information based on the correction information and the number of correction steps; and correcting the real-time measurement data based on the step-by-step correction information to obtain the head correction posture.

[0015] In this embodiment, by performing step-by-step correction on the real-time measurement data, the head correction posture is obtained by gradually fine-tuning the measurement data. Subsequently, the real-time head posture determined based on the head correction posture is displayed, which can better maintain the stability and continuity of the screen display.

[0016] In some embodiments, the step of correcting the real-time measurement data of the inertial measurement unit based on the correction information to obtain the head correction posture includes: acquiring the angular velocity information of the inertial measurement unit between the first time and the second time; obtaining the zero bias of the angular velocity meter of the inertial measurement unit based on the correction information and the angular velocity information; and correcting the real-time measurement data based on the zero bias of the angular velocity meter to obtain the head correction posture.

[0017] In this embodiment, by optimizing the zero bias of the angular velocity meter using correction information, the reliability of the head correction attitude output by the inertial measurement unit can be improved, and the system stability can be enhanced.

[0018] In some embodiments, determining the real-time head posture based on the head correction posture includes: acquiring model parameters of a 3D preset model, wherein the model parameters of the 3D preset model are used to represent the relative positions of head feature points and the head-mounted near-eye device when the user swings while wearing the head-mounted near-eye device; determining second three-degree-of-freedom posture data based on the model parameters of the 3D preset model; and fusing the first three-degree-of-freedom posture data and the second three-degree-of-freedom posture data to obtain the real-time head posture.

[0019] In this embodiment, the second degree of freedom pose data is determined by obtaining the model parameters of the 3D preset model, and the first three-degree-of-freedom pose data and the second degree-of-freedom pose data are fused to obtain the real-time head pose. The real-time head pose has six degrees of freedom, and the user experience can be improved when the display is based on the real-time head pose.

[0020] In some embodiments, the estimation method further includes: converting the real-time head posture to the virtual camera coordinate system of the virtual camera to obtain posture transformation information, wherein the virtual camera is used to render the image on the display screen of the head-mounted near-eye device; and presenting the image on the display screen of the head-mounted near-eye device based on the posture transformation information.

[0021] In this embodiment, the pose transformation information in the virtual camera coordinate system can be calculated through coordinate transformation, so that the head-mounted near-eye device can render and display the image based on the pose transformation information.

[0022] In some embodiments, before acquiring the image data captured by the camera and the measurement data of the inertial measurement unit, the estimation method further includes: calibrating the camera to obtain the camera's intrinsic parameters; calibrating the head-mounted near-eye device to obtain the calibration parameters of the inertial measurement unit and the calibration parameters of the virtual camera, wherein the virtual camera is used to render the image on the display screen of the head-mounted near-eye device; acquiring the coordinate transformation parameters of the head-mounted near-eye device between a virtual environment and a real environment, wherein the virtual environment is a virtual scene generated based on the real environment; and determining that the estimation system has been calibrated when it is determined that the intrinsic parameters, the calibration parameters of the inertial measurement unit, the calibration parameters of the virtual camera, and the coordinate transformation parameters have been acquired.

[0023] In this embodiment, by calibrating each parameter using the above calibration method, the sampling data of the camera and inertial measurement unit can be corrected, and the display screen can be displayed based on the above calibration parameters to ensure that the correct screen is observed.

[0024] In a second aspect, this application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of the first aspects.

[0025] Thirdly, this application provides a head posture estimation system, which includes: a camera, a head-mounted near-eye device, and the electronic device described in the second aspect; the head-mounted near-eye device includes an inertial measurement unit and a display unit, and the inertial measurement unit, the display unit, and the camera are all electrically connected to the electronic device.

[0026] In this embodiment, the electronic device can be used to execute the head posture estimation method provided in the embodiments of this application, and can use image data and measurement data to correct the inertial measurement unit, thereby improving the accuracy of the head posture estimated by the inertial measurement unit.

[0027] In some embodiments, the estimation system further includes a driving simulation cabin; the camera is located inside the driving simulation cabin, which is used to simulate a driving environment.

[0028] In this embodiment, by setting up a driving simulator, the estimation system can be applied to driving simulation to estimate the human head posture in real time, and display corresponding content on a head-mounted near-eye device based on the user's operation information in the driving simulator, thereby realizing simulated driving.

[0029] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method as described in any of the first aspects.

[0030] Fifthly, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in the first aspect above. Attached Figure Description

[0031] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements / modules and steps with the same reference numerals in the drawings are represented as similar elements / modules and steps. Unless otherwise stated, the figures in the drawings do not constitute a limitation on scale.

[0032] Figure 1 This is a structural block diagram of a head pose estimation system provided in an embodiment of this application;

[0033] Figure 2 This is a schematic diagram of the structure of a calibration plate provided in an embodiment of this application;

[0034] Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of this application;

[0035] Figure 4 This is a flowchart of a head pose estimation method provided in an embodiment of this application;

[0036] Figure 5 This is a schematic diagram of a real environment provided in an embodiment of this application;

[0037] Figure 6 This is a schematic diagram illustrating the display effect of a head-mounted near-eye device provided in an embodiment of this application;

[0038] Figure 7 This is a schematic diagram of another real-world environment provided in the embodiments of this application;

[0039] Figure 8 This is a schematic diagram of a real environment coordinate system and a virtual environment coordinate system provided in an embodiment of this application;

[0040] Figure 9 This is a schematic diagram of a virtual environment provided in an embodiment of this application;

[0041] Figure 10This is a schematic diagram illustrating the combination of a real environment coordinate system and a virtual environment coordinate system provided in an embodiment of this application;

[0042] Figure 11 This is a graph showing the first head state, the second head state, and time, provided in an embodiment of this application.

[0043] Figure 12 This is a graph of a first yaw angle, a second yaw angle, and time provided in an embodiment of this application;

[0044] Figure 13 This is a schematic diagram of a 3D preset model provided in an embodiment of this application;

[0045] Figure 14 This is a schematic diagram of another 3D preset model provided in the embodiments of this application. Detailed Implementation

[0046] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0047] To facilitate understanding of this application, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0048] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram, in some cases, they can be divided differently from those in the device. In addition, the terms "first" and "second" used herein do not limit the data or execution order, but only distinguish between identical or similar items with essentially the same function and effect.

[0049] Currently, in the field of near-eye display technology, to obtain the user's head posture, it is either estimated from images captured by a camera or estimated using an inertial measurement unit (IMU). While camera-based head posture estimation offers high consistency and stability at any given time, its low sampling frame rate makes continuous real-time tracking of head posture impossible. Furthermore, IMU-based posture estimation suffers from significant temperature drift (or drift) issues, leading to substantial accumulated errors during prolonged continuous use. In other words, using a camera alone for head posture estimation cannot achieve real-time tracking, while using an IMU alone results in significant errors.

[0050] To address the aforementioned technical problems, this application provides a method and system for estimating head posture, an electronic device, and a storage medium. This method can utilize image data obtained from a camera and measurement data from an inertial measurement unit (IMU) to obtain correction information for head posture estimation by the IMU, and use this correction information to correct errors in the head posture estimation by the IMU, thereby improving the accuracy of the head posture estimation by the IMU and avoiding the shortcomings of estimating head posture using only a camera or an IMU.

[0051] In a first aspect, embodiments of this application provide a head pose estimation system, please refer to... Figure 1 The estimation system includes a camera 10, a head-mounted near-eye device 20, and an electronic device 30. The head-mounted near-eye device 20 includes an inertial measurement unit 21 and a display unit 22, and the inertial measurement unit 21, the display unit 22, and the camera 10 are all electrically connected to the electronic device 30.

[0052] Camera 10 refers to a device used to capture images including a human head. Specifically, camera 10 may employ a red-green-blue color camera, capable of capturing information from the red, green, and blue color channels. The sampling frequency of camera 10 may be greater than or equal to 10Hz. Typically, camera 10 should be positioned slightly to the lower left front of the user wearing the head-mounted near-eye device 20 to track the human head.

[0053] The head-mounted near-eye device 20 is a near-eye display device that can be worn on the human head, including augmented reality (AR) devices, virtual reality (VR) devices, or mixed reality (MR) devices, and other head-mounted display devices. The AR device can be AR glasses; in the following description, AR glasses will be used as an example of the head-mounted near-eye device 20.

[0054] The inertial measurement unit 21 typically includes an accelerometer and an angular velocity meter, and sometimes a magnetometer. The accelerometer is a sensor used to measure the acceleration of an object along three axes, detecting changes in acceleration in various directions to infer the object's motion state. The angular velocity meter is a sensor used to measure the rotational rate or angular velocity of an object along three axes, detecting the object's rotational state and changes. The magnetometer is a sensor used to measure the surrounding magnetic field. It is typically used in conjunction with sensors such as accelerometers and gyroscopes to provide more comprehensive positioning and navigation information. The accelerometer may include a microelectromechanical system (MEMS) sensor, the angular velocity meter may include a gyroscope, and the magnetometer may include a magnetic induction element. The sampling frequency of the inertial measurement unit 21 may be greater than or equal to 500 Hz.

[0055] The display unit 22 can be a display screen or other device that can directly display images. The display unit 22 can display images based on the control of the electronic device 30, so that the user can observe the images in the display unit 22 and realize augmented reality.

[0056] The electronic device 30 can be used to execute the head posture estimation method provided in the embodiments of this application. It can use the image data obtained by the camera 10 and the measurement data of the inertial measurement unit 21 to obtain the correction information of the head posture estimated by the inertial measurement unit 21, and use the correction information to correct the error of the head posture estimated by the inertial measurement unit 21, thereby improving the accuracy of the head posture estimated by the inertial measurement unit 21 and avoiding the defects of using the camera 10 or the inertial measurement unit 21 alone to estimate the head posture.

[0057] In some embodiments, the estimation system also includes a driving simulator; a camera is located inside the driving simulator, which is used to simulate a driving environment.

[0058] A driving simulator is a device used to simulate real driving scenarios. It may include a real vehicle for driving simulation, which typically includes driving operation instruments such as seats, steering wheels, and pedals, and can realistically simulate the actual driving environment.

[0059] In this embodiment, by setting up a driving simulator, the estimation system can be applied to estimate the human head posture in real time within the driving simulator, and display corresponding content on a head-mounted near-eye device based on the user's operation information within the driving simulator, thereby realizing simulated driving.

[0060] In some embodiments, the estimation system may further include a calibration board for calibrating the estimation system.

[0061] Specifically, the calibration board can be a QR code calibration board specifically designed for camera calibration, and its size can be 40*60cm. Refer to [link / reference] on the calibration board. Figure 2It includes a calibration plate coordinate system and multiple QR code patterns, which are accurately recognized by the camera at different angles and distances, thus enabling the calibration of the camera's intrinsic and extrinsic parameters. The calibration plate coordinate system refers to a two-dimensional rectangular coordinate system constructed on the surface of the calibration plate by mutually perpendicular x1 and y1 axes.

[0062] In this embodiment, by setting a calibration plate, the estimation system can be easily calibrated to obtain its calibration parameters.

[0063] Secondly, this application also provides an electronic device, please refer to... Figure 3 It shows the hardware structure of an electronic device 30 capable of performing the head pose estimation method provided in this application.

[0064] The electronic device 30 includes: at least one processor 31; and a memory 32 communicatively connected to the at least one processor 31. Figure 3 Taking a processor 31 as an example, the memory 32 stores instructions executable by the at least one processor 31. These instructions, when executed by the at least one processor 31, enable the at least one processor 31 to perform the head pose estimation method provided in this application. The processor 31 and the memory 32 can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0065] The memory 32, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the head pose estimation method provided in this application. The processor 31 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 32, thereby implementing the head pose estimation method described in the method embodiment.

[0066] The memory 32 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 32 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 32 may optionally include memory remotely located relative to the processor 31, and these remote memories can be connected to the electronic device 30 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0067] The one or more modules are stored in the memory 32 and, when executed by the one or more processors 31, execute the head pose estimation method in any method embodiment.

[0068] The electronic device can execute the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for executing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0069] Thirdly, this application provides a method for estimating head posture, which is applied to a head posture estimation system. The estimation system includes a camera and a head-mounted near-eye device, the head-mounted near-eye device including an inertial measurement unit. The entity executing the estimation method can be... Figure 1 or Figure 3 Please refer to the electronic devices shown. Figure 4 The estimation method includes:

[0070] Step S10: Acquire image data captured by the camera and measurement data from the inertial measurement unit.

[0071] In this step, the camera and inertial measurement unit (IMU) should have undergone calibration. After calibration, the electronic device can acquire image and measurement data in real time through the camera and IMU. The image data acquired by the camera refers to the image information including the human head captured by the camera. The measurement data from the IMU may include the output data from the angular velocity meter and accelerometer.

[0072] Step S20: Perform time synchronization processing on the image data and measurement data.

[0073] Since the sampling frequency of the camera and the sampling frequency of the inertial measurement unit are usually not equal, time synchronization processing can ensure that the image data and measurement data are aligned in time so that image data and measurement data at the same moment can be acquired later.

[0074] Step S30: Obtain correction information based on the time-synchronized image data and measurement data.

[0075] After time synchronization, the error between the yaw angle estimated from the image data and the yaw angle estimated from the measurement data at the same moment can be calculated, and correction information can be obtained using this error. In this application, the yaw angle refers to the angle between the human head posture on the horizontal plane and the lateral direction of the human body in the real world. The lateral direction of the human body refers to the direction from the left to the right side of the body of a person wearing a head-mounted near-eye device.

[0076] Step S40: Based on the correction information, correct the real-time measurement data of the inertial measurement unit to obtain the head correction attitude.

[0077] The real-time measurement data of the inertial measurement unit can be corrected by using the correction information. For example, the calibration parameters of the inertial measurement unit can be corrected, or the real-time measurement data can be directly corrected to obtain the rotational attitude information of the human head on the three axes in the real-world coordinate system, that is, the first three degrees of freedom attitude data, which is also the head correction attitude.

[0078] Step S50: Determine the real-time head posture based on the head correction posture.

[0079] After obtaining the head correction posture, the head correction posture can be directly determined as the real-time head posture.

[0080] In this embodiment, correction information for head posture estimation by the inertial measurement unit (IMU) is obtained by using image data from the camera and measurement data from the IMU. This correction information is then used to correct the error in head posture estimation by the IMU, resulting in a corrected head posture. Based on the corrected head posture, the real-time head posture is determined. This reduces the error caused by temperature drift of the IMU, improves the accuracy of head posture estimation by the IMU, avoids the shortcomings of using either the camera or the IMU alone to estimate head posture, and minimizes the error caused by temperature drift of the IMU.

[0081] In some embodiments, prior to step S10, the estimation method further includes the following steps:

[0082] Step S11: Calibrate the camera to obtain its intrinsic parameters.

[0083] Specifically, conventional calibration methods can be used to calibrate the camera. For example, refer to... Figure 5 The calibration plate 40 is placed in the shooting area of ​​the camera 10, and its position is moved so that it appears in every position in the image captured by the camera 10. Then, the image is processed, and the intrinsic parameters of the camera are calibrated using a pinhole model or a fisheye model to obtain the intrinsic parameter calibration parameter K of the camera 10.

[0084]

[0085] And obtain the distortion parameters D of camera 10, where c x c y f represents the horizontal and vertical coordinates of the camera's optical center in the camera's pixel coordinate system, respectively. x f y These are the camera's focal length in the horizontal direction and the focal length in the vertical direction, respectively.

[0086] Step S12: Calibrate the head-mounted near-eye device to obtain the calibration parameters of the inertial measurement unit and the virtual camera. The virtual camera is used to render the image on the display screen of the head-mounted near-eye device.

[0087] Specifically, the inertial measurement unit (IMU) can be calibrated using a toolkit compatible with the IMU or existing calibration methods to obtain its calibration parameters. The IMU calibration parameters include the accelerometer correction matrix M. acc Angular velocity meter correction matrix M gyr Accelerometer triaxial zero bias acc Angular velocity meter triaxial zero bias gyr :

[0088]

[0089] bias acc =[b xa b ya b za ] T ;

[0090] bias gyr =[b xg b yg b zg ] T ;

[0091] Among them, s 0a s 1a s 2a denoted as , where a1, b1, and c1 are the dimensional errors of the three axes of the accelerometer, respectively; and s is the assembly error of the three axes of the accelerometer. 1g s 1g s 2g d, e, and f represent the dimensional errors of the three axes of the angular velocity meter, respectively; a2, b2, c2, d, e, and f represent the assembly errors of the three axes of the angular velocity meter, respectively. xa b ya b za These represent the zero bias of the three axes of the accelerometer, b xg b yg b zg These are the zero-bias values ​​for the three axes of the angular velocity meter. The raw acceleration measurement data (accelerometer) acquired by the inertial measurement unit can then be processed based on these calibration parameters. meas and raw angular velocity measurement data gyr meas Correction is performed to obtain the corrected acceleration data acc. calibed and the corrected angular velocity data gyr calibrf The specific calculation process can be found in the following formula:

[0092]

[0093] Next, virtual-real calibration is performed on the head-mounted near-eye device to obtain the calibration parameters of the virtual camera. The specific method for obtaining these parameters can refer to existing technologies. The virtual-real calibration process involves modeling the display screen of the head-mounted near-eye device as the imaging plane of the virtual camera used for rendering the image, and calculating the intrinsic and extrinsic parameters of the virtual camera. For details, please refer to [link to relevant documentation]. Figure 6 The head-mounted near-eye device has two display screens, display screen 1V and display screen 2V, corresponding to the left eye 1E and right eye 2E respectively. This implies that there are two virtual cameras, also corresponding to display screen 1V and display screen 2V respectively. After virtual-real calibration, the intrinsic parameters of the virtual cameras can be obtained. And the transformation relationship between the inertial measurement unit coordinate system and the virtual camera coordinate system. Where the virtual camera number i∈(0,1), For a 3x3 rotation matrix And a 3*1 translation vector The matrix formed, that is, i = 0 or 1.

[0094] Subsequently, the intrinsic parameters of the virtual camera and the extrinsic parameters between the inertial measurement unit and the virtual camera can be used to render the image on the display screen. For example, if an object needs to be placed 3 meters in front of the human eye in the virtual environment, the head-mounted near-eye device can use the intrinsic parameters of the virtual camera. And the transformation relationship between the inertial measurement unit coordinate system and the virtual camera coordinate system. The display position of the object on both screens is calculated so that when the human eye observes the object through both screens, the user can subjectively perceive that the object is 3 meters away from their eyes. Figure 6 As shown, calibration of head-mounted near-eye devices can improve the user experience. Specifically, in... Figure 6 In the diagram, d1 represents the actual distance, and d2 represents the virtual image-to-image distance.

[0095] Step S13: Obtain the coordinate transformation parameters of the head-mounted near-eye device between the virtual environment and the real environment, wherein the virtual environment is a virtual scene generated based on the real environment.

[0096] For example, regarding an estimation system equipped with a driving simulator, please refer to [link / reference]. Figures 7 to 10 It can pre-build coordinate systems for both real and virtual environments, where the virtual vehicle in the virtual environment has the same size as the real vehicle in the real environment.

[0097] For example, a real-world coordinate system, a real-vehicle coordinate system, a real-camera coordinate system, and a real-inertial measurement unit coordinate system can be constructed in a real environment, while a virtual-world coordinate system, a virtual-vehicle coordinate system, and a virtual-inertial measurement unit coordinate system can be constructed in a virtual environment.

[0098] The real-world coordinate system is an absolute coordinate system in the real environment, which can be used to describe the absolute position and orientation of an object in the real world. It takes an arbitrarily selected reference point as its origin O. rw A three-dimensional Cartesian coordinate system is established. The real vehicle coordinate system has its origin O at the center of the real vehicle or another specific location. rcar A three-dimensional Cartesian coordinate system was established. The actual camera coordinate system has its origin O at the camera's optical center. rcam The three-dimensional rectangular coordinate system. The real inertial measurement unit coordinate system refers to the coordinate system within the inertial measurement unit of a head-mounted near-eye device in a real environment, with O as the coordinate system. rimu A three-dimensional Cartesian coordinate system with its origin at a fixed point is typically associated with the device's orientation. A virtual world coordinate system is an absolute coordinate system within a virtual environment, used to describe the absolute position and orientation of objects in the virtual world. It uses an arbitrarily chosen reference point as its origin O. vw A three-dimensional Cartesian coordinate system is established. The virtual vehicle coordinate system takes the center of the virtual vehicle or other specific location in the virtual environment as its origin O. vcar A three-dimensional rectangular coordinate system was established. The virtual inertial measurement unit coordinate system refers to the coordinate system of the virtual inertial measurement unit inside the head-mounted near-eye device in the virtual environment, with O... vimu A three-dimensional rectangular coordinate system with the origin at .

[0099] The transformation parameters between the coordinate systems in the real environment and the virtual environment include the transformation relationship between the real camera coordinate system and the virtual vehicle coordinate system. This includes rotation and position information, where the position information can directly use measurements from the real-world assembly. Additionally, it can be derived from... Figure 7 It can be seen that the rotation information of the real vehicle coordinate system in the real world coordinate system is:

[0100]

[0101] To obtain the transformation relationship between the real camera coordinate system and the virtual vehicle coordinate system, we can first align the vehicle's forward direction in the real environment with the vehicle's forward direction in the virtual environment, and then combine this with... Figure 7 and Figure 9 It can be seen that the transformation relationship between the real vehicle coordinate system and the virtual vehicle coordinate system is... Then, as Figure 5As shown, a calibration plate 40 is placed in front of the camera 10, positioned on the driver's seat inside the real vehicle, ensuring it is perpendicular to the vehicle's forward direction. The calibration plate 40 is observed visually by the camera 10, and the rotational relationship R between the calibration plate's coordinate system and the real vehicle's coordinate system is utilized. car,apriltag_board The rotational relationship between the real camera coordinate system and the calibration plate coordinate system R rgb_cam,apriltag_board Calculate the rotational relationship R between the real camera coordinate system and the real vehicle. rgb_cam,car Finally, through R rgb_cam,car and R car,vir_car The transformation relationship between the real camera coordinate system and the virtual vehicle coordinate system is calculated. This can be combined with... Figure 5 The rotational relationship between the calibration plate coordinate system and the actual vehicle coordinate system can be determined. Simultaneously, the rotational relationship R between the real camera coordinate system and the calibration plate coordinate system can be calculated using the homography transformation method. rgb_cam,apriltag_board .

[0102] Step S14: When it is determined that the intrinsic parameters, the calibration parameters of the inertial measurement unit, the calibration parameters of the virtual camera, and the coordinate transformation parameters have been obtained, the calibration of the estimation system is completed.

[0103] After obtaining the camera's intrinsic parameters, the inertial measurement unit's calibration parameters, the virtual camera's calibration parameters, and the coordinate transformation parameters, it can be determined that the calibration process for the estimation system has been completed.

[0104] In this embodiment, by calibrating each parameter through the above calibration process, the obtained intrinsic parameters of the camera and calibration parameters of the inertial measurement unit can be used to correct the sampling data of the camera and the inertial measurement unit respectively. Subsequently, the head-mounted near-eye device can display the screen based on the above calibration parameters, which can ensure that the virtual environment is aligned with the real environment, ensure that the user observes the correct screen, and provide a more realistic and accurate augmented reality experience.

[0105] In some embodiments, step S20 may include the following steps:

[0106] Step S21A: According to the system clock of the estimation system, mark the image data with a first timestamp and the measurement data with a second timestamp.

[0107] The system clock refers to the estimated default clock of the electronic devices in the system. Image data and measurement data can be uploaded to the electronic devices separately, and then the electronic devices timestamp the image data and measurement data respectively based on the system clock.

[0108] Step S22A: Process the image data marked with the first timestamp and the measurement data marked with the second timestamp to obtain the first head state of the image data at each first timestamp and the second head state of the measurement data at each second timestamp.

[0109] For example, for any image data, first obtain the pose information R of the human head in the image. rgbcam_face Then use the rotation relation R rgb_cam,car Rotation information R of the real vehicle coordinate system in the real-world coordinate system world,car For R rgbcam_face Perform a coordinate transformation to obtain the first head pose R in the world coordinate system. world_face Next, based on the first head pose R world_face The first yaw angle is calculated and the calculation is repeated to obtain the first yaw angle corresponding to the image data at each first time stamp. Then, the first yaw angle data corresponding to each first time stamp can be stored in a double-ended queue with a fixed sliding window in chronological order. Then, the mean and variance of the first yaw angle corresponding to each first time stamp in the sliding window are calculated iteratively by sliding the window. The calculated variances are compared with the fixed variance corresponding to the static state to determine the first head state corresponding to each first time stamp. The first head state includes the first head static state and the first head moving state.

[0110] For the measurement data of the inertial measurement unit, the variance of the three axes of the angular velocity meter and accelerometer at each second time stamp can be calculated, and the variance of the three axes of the angular velocity meter and accelerometer can be compared with the fixed variance of the three axes of the angular velocity meter and accelerometer corresponding to the stationary state to determine the second head state corresponding to each second time stamp. The second head state includes the second head stationary state and the second head moving state.

[0111] The first and second head stationary states refer to the head being in a stationary state, while the first and second head motion states refer to the head being in a motion state.

[0112] Step S23A: Synchronize the time of image data and measurement data using the first timestamp corresponding to the first head static state in the first head state and the second timestamp corresponding to the second head static state in the second head state.

[0113] After obtaining the first header state under each first timestamp and the second header state under each second timestamp, as follows: Figure 11As shown, the time period during which the first head state is in the first head static state (including fdt0, fdt1, fdt2) and the time period during which the second head state is in the second head static state (including idt0, idt1, idt2) can be determined. Then, using the first timestamp of the first head state being in the first head static state and the timetamp of the second head state being in the second head static state...

[0114] The second timestamp of the second head at rest state is used to obtain the time deviation between the image data and the measurement data, and the time of the image data and the time of the measurement data are aligned based on the time deviation.

[0115] In this embodiment, time synchronization is achieved by using the timestamp when the head is in a static state, which reduces the synchronization time and difficulty, consumes less computing power, and can save costs and reduce production complexity.

[0116] In some embodiments, step S20 may include the following steps:

[0117] Step S21B: Based on the estimated system clock, mark the image data with a first timestamp and the measurement data with a second timestamp.

[0118] The system clock refers to the estimated default clock of the electronic devices in the system. Image data and measurement data can be uploaded to the electronic devices separately, and then the electronic devices timestamp the image data and measurement data respectively based on the system clock.

[0119] Step S22B: Process the image data marked with the first time stamp and the measurement data marked with the second time stamp to obtain the first yaw angle of the image data at each first time stamp and the second yaw angle of the measurement data at each second time stamp.

[0120] For any given image data, first obtain the pose information R of the human head in the image. rgbcam_face Then use the rotation relation R rgb_cam,car Rotation information R of the real vehicle coordinate system in the real-world coordinate system world,car For R rgbcam_face Perform a coordinate transformation to obtain the first head pose R in the world coordinate system. world_face Next, based on the first head pose R world_face The first yaw angle is calculated, and the above calculation process is repeated to obtain the first yaw angle corresponding to each image data. Then, the first yaw angle corresponding to consecutive first timestamps is obtained through interpolation. The interpolation method can be b-spline interpolation. For each measurement data output by the inertial measurement unit, the angular velocity in the measurement data can be converted to the real-world coordinate system, and the angular velocity value of the z1 axis can be taken to obtain the second yaw angle corresponding to the measurement data at each second timestamp.

[0121] Step S23B: Synchronize the time of image data and measurement data using the first yaw angle corresponding to each first timestamp and the second yaw angle corresponding to each second timestamp.

[0122] After obtaining the first yaw angle corresponding to each first timestamp and the second yaw angle corresponding to each second timestamp, curves between the first yaw angle and the timestamp, and curves between the second yaw angle and the timestamp, can be obtained respectively, such as... Figure 12 As shown, the similarity between the two curves can then be calculated using a fast dynamic time warping method. For example, fluctuating curve data can be extracted, and the curve trajectories of the first yaw angle and the second yaw angle can be subtracted. The time difference shift_fi_dt between the two can be dynamically adjusted so that the curves of the first yaw angle and the second yaw angle basically overlap and the error is minimized. At this time, the corresponding time difference shift_fi_dt is the time deviation between the image data and the measurement data. After obtaining the time deviation, the time of the image data and the time of the measurement data can be aligned.

[0123] In this embodiment, the accuracy of time synchronization can be improved by using the yaw angle to dynamically synchronize the time of image data and the time of measurement data.

[0124] In some embodiments, step S30 may include the following steps:

[0125] Step S31: Process the image data and measurement data after time synchronization to obtain the first yaw angle corresponding to the image data at the first moment, the first yaw angle corresponding to the image data at the second moment, the second yaw angle corresponding to the measurement data at the first moment, and the second yaw angle corresponding to the measurement data at the second moment.

[0126] For the image and measurement data after time synchronization, we can select the image and measurement data at the first and second time points. Then, for any image data, we can estimate the head pose R in the real camera coordinate system. rgb_cam,face The head pose R in the real-world coordinate system can be calculated using the following formula. world,face This also provides the head posture R in the real car coordinate system. car_face :

[0127] R car_face =R world,faCe

[0128] =(R rgb_cam,face T *R rgb_cam,apriltag_board

[0129] *R car,apriltag_board T ) T

[0130] From R world,face Extract the third column from the matrix as the vector z, that is:

[0131] z = R world,face .col(2);

[0132] The vector z is then projected onto a horizontal plane in the real-world coordinate system and normalized to obtain vector z1. The first yaw angle yaw can then be calculated using the following formula:

[0133] cos(yaw)=z1·[1,0,0。

[0134] For the measurement data, the angular velocity in the measurement data can be converted to the real world coordinate system, and the angular velocity value of the z1 axis can be taken to obtain the second yaw angle corresponding to each measurement data.

[0135] Step S32: Determine the difference between the first yaw angle corresponding to the image data at the first time moment and the second yaw angle corresponding to the measurement data at the first time moment, and obtain the first yaw angle difference. Step S33: Determine the difference between the first yaw angle corresponding to the image data at the second time moment and the second yaw angle corresponding to the measurement data at the second time moment, and obtain the second yaw angle difference. Step S34: Based on the first yaw angle difference and the second yaw angle difference, obtain correction information.

[0136] After obtaining the first yaw angle corresponding to the image data at the first time moment, the first yaw angle corresponding to the image data at the second time moment, the second yaw angle corresponding to the measurement data at the first time moment, and the second yaw angle corresponding to the image data at the second time moment, the difference between the first yaw angle and the second yaw angle corresponding to the image data at the first time moment is taken to obtain the first yaw angle difference value. Similarly, the difference between the first yaw angle difference value and the second yaw angle difference value is taken to obtain the second yaw angle difference value. Then, after obtaining the first yaw angle difference value and the second yaw angle difference value, the difference between the two is taken to obtain the correction information.

[0137] Understandably, if the head posture estimated from the measurement data is accurate, the correction information should be 0. However, in practice, the correction information is not 0, which indicates that there is an error in the head posture estimated from the measurement data. The inertial measurement unit can then be corrected based on this correction information.

[0138] In this embodiment, the error between the estimated yaw angles of the image data and measurement data at two different times can be calculated to determine the correction information. Subsequently, the inertial measurement unit can be calibrated based on this correction information, which can improve the accuracy of the head attitude estimated by the inertial measurement unit.

[0139] In some embodiments, step S40 may include the following steps: Step S41A: Obtain a preset number of correction steps. Step S42A: Obtain step-by-step correction information based on the correction information and the number of correction steps. Step S43A: Correct the real-time measurement data based on the step-by-step correction information to obtain the head correction posture.

[0140] The preset number of calibration steps, n, refers to the preset number of calibrations. Specifically, after obtaining the calibration information yaw... correct Then, the calibration information can be divided by n to obtain the step-by-step calibration information, i.e., the step-by-step calibration information. This step-by-step correction information is used to correct the measurement data once. By using this step-by-step correction information to correct the current measurement data, the current head correction posture can be obtained.

[0141] It is understandable that if the head posture obtained by directly correcting the measurement data with the correction information is used to determine the real-time head posture, the subsequent display using this real-time head posture will lead to problems such as screen distortion and poor display continuity. However, in this embodiment, by performing step-by-step correction on the real-time measurement data and using the step-by-step correction information to perform a correction, the measurement data is gradually fine-tuned to obtain the fine-tuned head posture. When the real-time head posture is subsequently displayed, the stability and continuity of the display can be better maintained. Moreover, the measurement data can be corrected when the user is moving, but not when the user is stationary. This allows the error information to be gradually fine-tuned during the change of the image, making the displayed image reasonable and improving the user experience.

[0142] In some embodiments, step S40 includes step S41B: acquiring angular velocity information of the inertial measurement unit between a first time and a second time; step S42B: obtaining the zero bias of the angular velocity meter of the inertial measurement unit based on the correction information and the angular velocity information; and step S43B: obtaining the head correction attitude based on the real-time measurement data of the zero bias correction of the angular velocity meter.

[0143] Angular velocity information refers to the angular velocity values ​​output by the inertial measurement unit at all times between the first and second moments. After obtaining the angular velocity information, the zero bias of the angular velocity meter can be obtained by combining it with the correction information using extended Kalman filtering, error state Kalman filtering, or least squares method. Subsequently, the real-time measurement data can be corrected in real time based on this zero bias of the angular velocity meter to obtain the head correction attitude.

[0144] In this embodiment, by optimizing the zero bias of the angular velocity meter using correction information, the reliability of the head correction attitude output by the inertial measurement unit can be improved, and the system stability can be enhanced.

[0145] In some embodiments, step S50 may include the following steps: Step S51: Obtain model parameters of a 3D preset model, the model parameters of which represent the relative positions of head feature points and the head-mounted near-eye device when the user is swinging while wearing the device. Step S52: Determine the second three-degree-of-freedom pose data based on the model parameters of the 3D preset model. Step S53: Fuse the first three-degree-of-freedom pose data and the second three-degree-of-freedom pose data to obtain the real-time head pose.

[0146] A 3D preset model refers to a pre-built three-dimensional model used to describe the relative position between a head-mounted near-eye device and a feature point on the head. The model parameters of the 3D preset model may include the horizontal distance *w* and the vertical distance *h* between the head feature point P and the center of the head-mounted near-eye device (which can be the center position between two display screens), i.e., the relative position. The units for the horizontal distance *w* and the vertical distance *h* are meters. The head feature point P can be the center of the neck or other suitable point. For example, regarding the center of the neck, see [link to relevant documentation]. Figure 13 and Figure 14 When the core of the body does not move significantly, the head's movement is a type of motion that rotates up, down, left, and right, and sways left and right around the neck as an axis. When acquiring model parameters, a motion capture system can be used to capture the head's motion posture data, computer vision technology can be used to obtain the positions of head feature points P, and finally, the least squares fitting method is used to obtain and determine the model parameters of the preset 3D model.

[0147] The second and third degrees of freedom (DOF) pose data refers to the translational pose information of the human head along three axes in a real-world coordinate system. After obtaining the model parameters, the second and third degrees of freedom pose data can be calculated using the model parameters and the first and third degrees of freedom pose data. Then, the first and second DDF pose data can be fused to obtain the real-time head pose. For example, the real-time head pose can be obtained by fusing the first and second DDF pose data according to the following formula.

[0148]

[0149] Among them, R wi_corrected For the first three degrees of freedom attitude data, t wi_headmodel This refers to the attitude data for the second and third degrees of freedom.

[0150] In this embodiment, the translational posture information of the human head on three axes, i.e., the second three-degree-of-freedom posture data, is obtained through the model parameters of a 3D preset model. The real-time head posture obtained by subsequently fusing the first three-degree-of-freedom posture data and the second three-degree-of-freedom posture data includes six degrees of freedom. Displaying the real-time head posture based on these six degrees of freedom improves the user experience, achieving a truly immersive experience that seamlessly blends the virtual and real worlds. Furthermore, by including six degrees of freedom in the real-time head posture using the above method, the head posture estimation method provided in this application can be applied to head posture estimation scenarios where the human body is in a fixed position, such as in simulated driving training with a fixed seating position, improving the stability and accuracy of head posture estimation in simulated driving training.

[0151] In some embodiments, the estimation method further includes the following steps: Step S61: Transform the real-time head posture to the virtual camera coordinate system of the virtual camera to obtain posture transformation information, wherein the virtual camera is used to render the image on the display screen of the head-mounted near-eye device; Step S62: Based on the posture transformation information, present the image on the display screen of the head-mounted near-eye device.

[0152] Specifically, after obtaining the real-time head posture, a coordinate transformation needs to be performed to convert it to the virtual camera coordinate system. This yields posture transformation information in the virtual camera coordinate system, enabling the head-mounted near-eye device to render and display the image on the screen based on this posture transformation information.

[0153] Specifically, you can first obtain the transformation relationship T between the real-world coordinate system and the virtual-world coordinate system. world_virworld And the transformation relationship T between the display coordinate system of the head-mounted near-eye device in the real environment and the display coordinate system of the head-mounted near-eye device in the virtual environment. arglass_virglass :

[0154]

[0155] In the real-world environment, the display coordinate system of the head-mounted near-eye device refers to a three-dimensional Cartesian coordinate system established with the center of the head-mounted near-eye device in the real environment, the user's eye position, or other suitable position as the origin. In the virtual environment, the display coordinate system of the head-mounted near-eye device refers to a three-dimensional Cartesian coordinate system established with the center of the head-mounted near-eye device in the virtual environment, the user's eye position, or other suitable position as the origin. Then, the transformation relationship between the inertial measurement unit coordinate system and the virtual camera coordinate system is obtained. This transformation relationship can be obtained through step 12 above, and the pose transformation information tracked by the two virtual cameras can be calculated using the following formula:

[0156]

[0157] Next, the head-mounted near-eye device can utilize the pose change information tracked by two virtual cameras. The corresponding image is rendered in real time, and then the head-mounted near-eye device controls the display screen to show that image.

[0158] In this embodiment, the real-time head posture can be transformed to the virtual camera coordinate system through coordinate transformation to obtain posture transformation information, so that the head-mounted near-eye device can render and display the image based on the posture transformation information.

[0159] This application also provides a computer-readable storage medium storing computer-executable instructions for causing an electronic device to perform the head pose estimation method provided in this application.

[0160] In some embodiments, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0161] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0162] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborative files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0163] As an example, executable instructions can be deployed to execute on a single computing device (including devices such as smart terminals and servers), or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0164] This application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the head pose estimation method as described in the foregoing embodiments.

[0165] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for at least one computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method of estimating a head pose, characterized by, The application is applied to a head posture estimation system, the estimation system comprises a camera and a head-mounted near-eye device, the head-mounted near-eye device comprises an inertial measurement unit, the estimation method comprises: acquiring image data collected by the camera and measurement data of the inertial measurement unit; time synchronizing the image data and the measurement data; obtaining correction information based on the time-synchronized image data and measurement data; correcting real-time measurement data of the inertial measurement unit based on the correction information to obtain a head corrected posture; determining a head real-time posture according to the head corrected posture.

2. The estimation method of claim 1, wherein, The time synchronization of the image data and the measurement data comprises: marking first timestamps on the image data and second timestamps on the measurement data according to a system clock of the estimation system; processing the image data marked with the first timestamps and the measurement data marked with the second timestamps to obtain corresponding first head states of the image data under each first timestamp and corresponding second head states of the measurement data under each second timestamp; synchronizing the time of the image data and the measurement data by using the first timestamp corresponding to the first head state in a first head stationary state and the second timestamp corresponding to the second head state in a second head stationary state.

3. The estimation method of claim 1, wherein, The time synchronization of the image data and the measurement data comprises: marking first timestamps on the image data and second timestamps on the measurement data according to a system clock of the estimation system; processing the image data marked with the first timestamps and the measurement data marked with the second timestamps to obtain corresponding first yaw angles of the image data under each first timestamp and corresponding second yaw angles of the measurement data under each second timestamp; synchronizing the time of the image data and the measurement data by using the first yaw angle corresponding to each first timestamp and the second yaw angle corresponding to each second timestamp.

4. The estimation method according to any one of claims 1 to 3, characterized in that, The time synchronization of the image data and the measurement data comprises: processing the time-synchronized image data and measurement data to obtain a first yaw angle corresponding to the image data at a first time, a first yaw angle corresponding to the image data at a second time, a second yaw angle corresponding to the measurement data at the first time, and a second yaw angle corresponding to the measurement data at the second time; determining a difference between the first yaw angle corresponding to the image data at the first time and the second yaw angle corresponding to the measurement data at the first time to obtain a first yaw angle difference; determining a difference between the first yaw angle corresponding to the image data at the second time and the second yaw angle corresponding to the measurement data at the second time to obtain a second yaw angle difference; obtaining correction information based on the first yaw angle difference and the second yaw angle difference.

5. The estimation method of claim 4, wherein, The correction of the real-time measurement data of the inertial measurement unit based on the correction information to obtain a head corrected posture comprises: acquiring a preset correction step number; obtaining step correction information based on the correction information and the correction step number; correcting the real-time measurement data based on the step correction information to obtain the head correction posture.

6. The estimation method of claim 4, wherein, The method further comprises: obtaining angular velocity information of the inertial measurement unit between the first time and the second time; obtaining the zero offset of the gyroscope of the inertial measurement unit based on the correction information and the angular velocity information; correcting the real-time measurement data based on the zero offset of the gyroscope to obtain the head correction posture.

7. The method of estimating according to any one of claims 1-3, characterized by, The method further comprises: obtaining model parameters of a 3D preset model, the model parameters of the 3D preset model being used to represent the relative position between the head feature point of the user wearing the head-mounted near-eye device and the head-mounted near-eye device when the user swings; determining second three-degree-of-freedom posture data according to the model parameters of the 3D preset model; fusing the first three-degree-of-freedom posture data and the second three-degree-of-freedom posture data to obtain the head real-time posture.

8. The estimation method of claim 7, wherein, The method further comprises: converting the head real-time posture to a virtual camera coordinate system of a virtual camera to obtain posture transformation information, wherein the virtual camera is used to render a picture of a display screen of the head-mounted near-eye device; presenting the picture in the display screen of the head-mounted near-eye device based on the posture transformation information.

9. The method of estimating according to any one of claims 1-3, characterized by, Before the method further comprises: calibrating the camera to obtain intrinsic parameters of the camera; calibrating the head-mounted near-eye device to obtain calibration parameters of the inertial measurement unit and calibration parameters of a virtual camera, wherein the virtual camera is used to render a picture of a display screen of the head-mounted near-eye device; obtaining coordinate conversion parameters of the head-mounted near-eye device between a virtual environment and a real environment, wherein the virtual environment is a virtual scene generated based on the real environment; when it is determined that the intrinsic parameters, the calibration parameters of the inertial measurement unit, the calibration parameters of the virtual camera, and the coordinate conversion parameters are obtained, it is determined that the calibration of the estimation system is completed.

10. An electronic device, comprising: comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

11. A head pose estimation system, characterized by, comprises: a camera, a head-mounted near-eye device, and an electronic device as claimed in claim 10; the head-mounted near-eye device comprises an inertial measurement unit and a display unit, and the inertial measurement unit, the display unit, and the camera are electrically connected to the electronic device.

12. The estimation system of claim 11, wherein, The estimation system further comprises a driving simulation cabin. The camera is arranged inside the driving simulation cabin, and the driving simulation cabin is used to simulate a driving environment. The estimation system further comprises a driving simulation cabin. The camera is arranged inside the driving simulation cabin, and the driving simulation cabin is used to simulate a driving environment.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions for causing a computer to perform the method of any one of claims 1 to 9.

Images