Head posture data collection method, device, electronic device and storage medium

By synchronizing and fixing the inertial measurement unit and camera unit, and obtaining head attitude data based on the target attitude transformation matrix, the flexibility, accuracy and cost problems of head attitude estimation in the prior art are solved, and efficient head attitude is achieved Partial posture collection.

CN118436341BActive Publication Date: 2024-12-03HANGZHOU RUIJIAN ZHIXING TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410903616.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-12-03
Estimated Expiration
2044-07-08

AI Technical Summary

Technical Problem

In the prior art, in head posture estimation, the three-dimensional modeling method based on face modeling is limited by the face modeling accuracy and is easily affected by individual differences. The flexibility and accuracy of obtaining the true value of head posture is low, and the acquisition cost is High, low efficiency, and there is a human obstacle between the camera and the user during the acquisition process, resulting in limited angles.

Method used

After time synchronization of the inertial measurement unit and the imaging unit, the inertial measurement unit is fixed on the imaging unit, and the user's head attitude data is obtained based on the target attitude transformation matrix, offline calibration is realized, and the impact of individual differences is reduced and the acquisition cost is reduced. Improve efficiency and solve the problem of limited acquisition angle.

Benefits of technology

It improves the flexibility and accuracy of head posture data, reduces acquisition costs, improves acquisition efficiency, and solves the problem of limited acquisition angle.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118436341B_ABST
    Figure CN118436341B_ABST
Patent Text Reader

Abstract

The present application provides a method, device, electronic device and storage medium for collecting head posture data. The method includes: in response to an inertial measurement unit moving in a visible area of ​​a camera unit and stopping at a target time, time synchronization is performed on the inertial measurement unit and the camera unit; in response to fixing the inertial measurement unit on the camera unit, a target posture transformation matrix of the camera unit relative to the inertial measurement unit is obtained; in response to the camera unit being aimed at the user's head, an image of the user's head is collected, and the user's head posture data is obtained based on the target posture transformation matrix.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of intelligent cockpits, and in particular to a method, device, electronic device and storage medium for collecting head posture data. Background Art

[0002] With the rapid development of cockpit technology, realizing intelligent and personalized user interaction in cockpit systems has become a key requirement, and head posture estimation is an important part of it. Through head posture estimation, the cockpit system can adjust the angle of the display screen according to the passenger's head movement to ensure the best visual effect. In addition, this technology can also be used in driver monitoring systems to improve the driver's alertness and safety, and prevent the driver from being distracted, fatigued, and driving inappropriately.

[0003] At present, with the rapid development of deep learning, the head posture of passengers is mainly estimated through deep learning model training. This involves obtaining the true value data of the head posture. Usually, the depth information of the face can be obtained through equipment (for example, multi-eye cameras, TOF cameras or lidar, etc.), and the three-dimensional model of the face can be modeled at the same time to calculate the true value of the head posture (head posture data).

[0004] However, the method of obtaining the true value of head posture based on three-dimensional face modeling is limited by the accuracy of face modeling and is easily affected by individual differences. In addition, since online calibration is required, that is, real-time calibration is performed while collecting the true value of head posture, the flexibility and accuracy of obtaining the true value of head posture are low. At the same time, since face modeling is involved, there are individual differences, the collection cost is high and the collection efficiency is low. In addition, since there are often human obstacles between the camera and the user during the collection process, the collection angle is limited. Summary of the invention

[0005] In view of this, the purpose of the present application is to provide a head posture data acquisition method, device, electronic device and storage medium, which can achieve offline calibration of the inertial measurement unit and the camera unit, improve the flexibility and accuracy of obtaining head posture data, and do not need to rely on face modeling, reduce the impact of individual differences, reduce acquisition costs, and improve acquisition efficiency. At the same time, since there is no human obstacle between the camera unit and the user during the acquisition process, the problem of limited acquisition angle is solved.

[0006] In a first aspect, an embodiment of the present application provides a head posture data collection method, which is applied to a head posture data collection system; the head posture data collection system includes an inertial measurement unit and a camera unit; the method includes:

[0007] In response to the inertial measurement unit moving in a visible area of ​​the camera unit and stopping at a target time, time-synchronizing the inertial measurement unit and the camera unit;

[0008] In response to fixing the inertial measurement unit on the camera unit, acquiring a target attitude transformation matrix of the camera unit relative to the inertial measurement unit; wherein the target attitude transformation matrix is ​​an attitude transformation matrix of the camera unit relative to the inertial measurement unit;

[0009] In response to the camera unit being aimed at the user's head, an image of the user's head is collected, and head posture data of the user is acquired based on the target posture transformation matrix.

[0010] In a possible implementation, the time synchronization of the inertial measurement unit and the camera unit includes:

[0011] Recording a first timestamp when the value of the inertial measurement unit stops changing;

[0012] A second timestamp when the inertial measurement unit stops moving is obtained based on observation by the camera unit, and time synchronization is performed on the inertial measurement unit and the camera unit based on a time difference between the first timestamp and the second timestamp.

[0013] In a possible implementation, the head posture data acquisition system further includes a calibration plate; and obtaining a target posture transformation matrix of the camera unit relative to the inertial measurement unit includes:

[0014] In response to moving the inertial measurement unit at different positions and postures, capturing calibration images of the calibration plate based on the camera unit, and extracting feature points of the calibration plate from each of the calibration images based on a preset corner point detection method;

[0015] Obtaining a first posture of the camera unit relative to the calibration plate at a current position and posture based on a preset posture solving method and feature points of the calibration plate, and recording a second posture output by the inertial measurement unit at the current position and posture;

[0016] Calculating a target posture transformation matrix according to the first posture and the second posture. In a possible implementation, calculating a target posture transformation matrix according to the first posture and the second posture includes:

[0017] Performing time synchronization on the first posture and the second posture based on a time difference between the first timestamp and the second timestamp to obtain the synchronized first posture and the second posture;

[0018] Calculating a third posture of the inertial measurement unit and a fourth posture of the camera unit according to the synchronized first posture and second posture; wherein the third posture is the posture of the inertial measurement unit from one position to another position; and the fourth posture is the posture of the camera unit from one position to another position;

[0019] The target posture transformation matrix is ​​calculated based on the third posture of the inertial measurement unit and the fourth posture of the camera unit.

[0020] In a possible implementation, the acquiring the head posture data of the user based on the target posture transformation matrix includes:

[0021] Calculating a first posture relationship between a preset head coordinate system and a preset camera coordinate system;

[0022] Determine a first value of the inertial measurement unit, and calculate a second posture relationship between the head coordinate system and a preset inertial coordinate system based on the first posture relationship, the first value and the target posture transformation matrix;

[0023] In the second posture relationship, in response to moving the camera unit around the user's head at different positions and postures, the user's head posture data is acquired.

[0024] In a possible implementation, the acquiring the head posture data of the user includes:

[0025] determining a second value of the inertial measurement unit;

[0026] The user's head posture data is calculated based on the second value, the target posture transformation matrix and the second posture relationship.

[0027] In a possible implementation, the method further includes:

[0028] Acquire a target head image based on the camera unit within a preset time interval, and calculate head posture verification data based on the target head image;

[0029] A target angle is calculated based on the head posture data and the head posture verification data, and in response to the target angle being greater than a preset threshold, the first posture relationship is recalculated, and the head image and the head posture data are recaptured.

[0030] In a second aspect, an embodiment of the present application further provides a head posture data acquisition device, which is applied to a head posture data acquisition system; the head posture data acquisition system includes an inertial measurement unit and a camera unit; the device includes:

[0031] a synchronization module, configured to perform time synchronization between the inertial measurement unit and the camera unit in response to the inertial measurement unit moving in a visible area of ​​the camera unit and stopping at a target time;

[0032] A first acquisition module, configured to acquire a target attitude transformation matrix of the camera unit relative to the inertial measurement unit in response to fixing the inertial measurement unit on the camera unit; wherein the target attitude transformation matrix is ​​an attitude transformation matrix of the camera unit relative to the inertial measurement unit;

[0033] The second acquisition module is used to collect the head image of the user in response to the camera unit being aimed at the user's head, and to acquire the head posture data of the user based on the target posture transformation matrix.

[0034] In a possible implementation manner, the synchronization module is specifically used to:

[0035] Recording a first timestamp when the value of the inertial measurement unit stops changing;

[0036] A second timestamp when the inertial measurement unit stops moving is obtained based on observation by the camera unit, and time synchronization is performed on the inertial measurement unit and the camera unit based on a time difference between the first timestamp and the second timestamp.

[0037] In a possible implementation, the head posture data acquisition system further includes a calibration board; and the first acquisition module is specifically used to:

[0038] In response to moving the inertial measurement unit at different positions and postures, capturing calibration images of the calibration plate based on the camera unit, and extracting feature points of the calibration plate from each of the calibration images based on a preset corner point detection method;

[0039] Obtaining a first posture of the camera unit relative to the calibration plate at a current position and posture based on a preset posture solving method and feature points of the calibration plate, and recording a second posture output by the inertial measurement unit at the current position and posture;

[0040] A target posture transformation matrix is ​​calculated according to the first posture and the second posture.

[0041] In a possible implementation manner, the first acquisition module is specifically configured to:

[0042] Performing time synchronization on the first posture and the second posture based on a time difference between the first timestamp and the second timestamp to obtain the synchronized first posture and the second posture;

[0043] Calculating a third posture of the inertial measurement unit and a fourth posture of the camera unit according to the synchronized first posture and second posture; wherein the third posture is the posture of the inertial measurement unit from one position to another position; and the fourth posture is the posture of the camera unit from one position to another position;

[0044] The target posture transformation matrix is ​​calculated based on the third posture of the inertial measurement unit and the fourth posture of the camera unit.

[0045] In a possible implementation manner, the second acquisition module is specifically configured to:

[0046] Calculating a first posture relationship between a preset head coordinate system and a preset camera coordinate system;

[0047] Determine a first value of the inertial measurement unit, and calculate a second posture relationship between the head coordinate system and a preset inertial coordinate system based on the first posture relationship, the first value and the target posture transformation matrix;

[0048] In the second posture relationship, in response to moving the camera unit around the user's head at different positions and postures, the user's head posture data is acquired.

[0049] In a possible implementation manner, the second acquisition module is specifically configured to:

[0050] determining a second value of the inertial measurement unit;

[0051] The user's head posture data is calculated based on the second value, the target posture transformation matrix and the second posture relationship.

[0052] In a possible implementation, the head posture data acquisition device further includes:

[0053] A third acquisition module, used to acquire a target head image based on the camera unit within a preset time interval, and calculate head posture verification data based on the target head image;

[0054] A verification module is used to calculate a target angle based on the head posture data and the head posture verification data, and in response to the target angle being greater than a preset threshold, recalculate the first posture relationship and re-collect the head image and the head posture data.

[0055] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the head posture data collection method as described in any one of the first aspects.

[0056] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the head posture data collection method described in any one of the first aspects are executed.

[0057] The embodiment of the present application provides a head posture data acquisition method, device, electronic device and storage medium. In response to the inertial measurement unit moving in the visible area of ​​the camera unit and stopping at the target time, the inertial measurement unit and the camera unit are synchronized in time. In response to fixing the inertial measurement unit on the camera unit, the target posture transformation matrix of the camera unit relative to the inertial measurement unit is obtained. In response to the camera unit aligning the user's head, the user's head image is collected, and the user's head posture data is obtained based on the target posture transformation matrix. In the present application, the inertial measurement unit and the camera unit are fixed on the camera unit after time synchronization, and the user's head posture data is obtained based on the target posture transformation matrix after the user's head image is collected, so as to realize the offline calibration of the inertial measurement unit and the camera unit, improve the flexibility and accuracy of obtaining head posture data, and do not need to rely on face modeling, reduce the influence of individual differences, reduce the acquisition cost, and improve the acquisition efficiency. At the same time, since there is no human obstacle between the camera unit and the user during the acquisition process, the problem of limited acquisition angle is solved. In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 is a flowchart of a head posture data collection method provided according to an embodiment of the present application;

[0060] Figure 2 It is a schematic diagram of the head posture data collection process;

[0061] Figure 3 is a schematic diagram of head posture;

[0062] Figure 4 is a flowchart of a head posture data collection method provided according to another embodiment of the present application;

[0063] Figure 5 is a flowchart of a head posture data collection method provided according to another embodiment of the present application;

[0064] Figure 6 is a flowchart of a head posture data collection method provided according to another embodiment of the present application;

[0065] Figure 7 is a flowchart of a head posture data collection method provided according to another embodiment of the present application;

[0066] Figure 8 is a structural schematic diagram of a head posture data acquisition device provided according to an embodiment of the present application;

[0067] Fig. 9 It is a structural schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0068] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0069] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0070] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0071] With the rapid development of cockpit technology, intelligent and personalized user interaction in cockpit systems has become a key requirement, and head posture estimation is an important part of it. Through head posture estimation, the cockpit system can adjust the angle of the display screen according to the passenger's head movement to ensure the best visual effect. In addition, this technology can also be used in driver monitoring systems to improve driver alertness and safety, and prevent drivers from being distracted, fatigued, and driving inappropriately.

[0072] At present, with the rapid development of deep learning, the head posture of passengers is mainly estimated through deep learning model training. This involves obtaining the true value data of the head posture. Usually, the depth information of the face can be obtained through equipment (for example, multi-eye cameras, TOF cameras or lidar, etc.), and the three-dimensional model of the face can be modeled at the same time to calculate the true value of the head posture (head posture data).

[0073] However, the method of obtaining the true value of head posture based on three-dimensional face modeling is limited by the accuracy of face modeling and is easily affected by individual differences. In addition, since online calibration is required, that is, real-time calibration is performed while collecting the true value of head posture, the flexibility and accuracy of obtaining the true value of head posture are low. At the same time, since face modeling is involved, there are individual differences, the collection cost is high and the collection efficiency is low. In addition, since there are often human obstacles between the camera and the user during the collection process, the collection angle is limited.

[0074] In response to this problem, the present application provides a head posture data acquisition method, device, electronic device and storage medium. After the inertial measurement unit and the camera unit are synchronized in time, the inertial measurement unit is fixed on the camera unit, and the head image of the user is acquired based on the target posture transformation matrix to obtain the head posture data of the user. Offline calibration of the inertial measurement unit and the camera unit is achieved, and the flexibility and accuracy of acquiring head posture data are improved. It does not need to rely on face modeling, reduces the impact of individual differences, reduces acquisition costs, and improves acquisition efficiency. At the same time, since there is no human obstacle between the camera unit and the user during the acquisition process, the problem of limited acquisition angle is solved.

[0075] Figure 1 is a flow chart of a head posture data acquisition method provided in an embodiment of the present application, which is applied to a head posture data acquisition system; the head posture data acquisition system includes an inertial measurement unit and a camera unit; Figure 1 As shown, the head posture data collection method of the embodiment of the present application may specifically include:

[0076] S101 . In response to an inertial measurement unit moving in a visible area of ​​a camera unit and stopping at a target time, time synchronization is performed on the inertial measurement unit and the camera unit.

[0077] S102: In response to fixing the inertial measurement unit on the camera unit, obtaining a target attitude transformation matrix of the camera unit relative to the inertial measurement unit.

[0078] S103 . In response to the camera unit being directed toward the user's head, an image of the user's head is collected, and head posture data of the user is acquired based on a target posture transformation matrix.

[0079] In the above head posture data acquisition method, the inertial measurement unit and the camera unit are time-synchronized and then fixed on the camera unit, and the head image of the user is acquired and the head posture data of the user is acquired based on the target posture transformation matrix, thereby realizing offline calibration of the inertial measurement unit and the camera unit, improving the flexibility and accuracy of acquiring head posture data, and not requiring reliance on face modeling, reducing the impact of individual differences, reducing acquisition costs, and improving acquisition efficiency. At the same time, since there is no human obstacle between the camera unit and the user during the acquisition process, the problem of limited acquisition angles is solved.

[0080] under Figure 1-Figure 7 , the above exemplary steps of the embodiment of the present application are described with reference to specific examples:

[0081] S101 , in response to an inertial measurement unit moving in a visible area of ​​a camera unit and stopping at a target time, time synchronization is performed on the inertial measurement unit and the camera unit.

[0082] In the embodiments of the present application, an inertial measurement unit refers to a device that integrates multiple inertial sensors to provide object motion and direction information; a camera unit refers to a device with an image acquisition function, such as a camera, a camcorder, or other visual perception device; a visible area refers to an effective area in which the camera unit can take pictures; a target time refers to a pre-set time point at which the inertial measurement unit stops moving after moving; the inertial measurement unit is moved in the visible area of ​​the camera unit and stops moving at the target time; at this time, the inertial measurement unit and the camera unit are time-synchronized to obtain the time-synchronized inertial measurement unit and the camera unit for subsequent processing.

[0083] It is understood by those skilled in the art that an inertial measurement unit measures and records the motion state of an object, including speed, direction, acceleration, etc., by using multiple sensors.

[0084] It should be noted that due to data drift caused by hardware and system delays, it is necessary to synchronize the device time, that is, to synchronize the time of the inertial measurement unit and the camera unit.

[0085] S102: In response to fixing the inertial measurement unit on the camera unit, a target attitude transformation matrix of the camera unit relative to the inertial measurement unit is acquired.

[0086] In the embodiment of the present application, the target attitude transformation matrix is ​​the attitude transformation matrix of the camera unit relative to the inertial measurement unit. After the inertial measurement unit and the camera unit are time-synchronized in step S101, the inertial measurement unit is fixed on the camera unit, and the target attitude transformation matrix of the camera unit relative to the inertial measurement unit is obtained for subsequent processing. For example, Figure 2 As shown, IMU represents the inertial measurement unit of the present application, CAM represents the camera unit of the present application, IMU is fixed on CAM, and the target posture transformation matrix of CAM relative to IMU is obtained.

[0087] It should be noted that the inertial measurement unit is fixed on the camera unit (the initial posture of the system) so that the position and posture relationship between the camera unit and the inertial measurement unit remains fixed, thereby realizing offline calibration of the inertial measurement unit and the camera unit, so that data can be directly collected without calibration during the collection process. This solves the problem in the prior art of requiring long-distance real-time calibration while collecting data. Obviously, the flexibility and accuracy of obtaining head posture data have been improved.

[0088] S103, in response to the camera unit being aimed at the user's head, collecting the user's head image, and acquiring the user's head posture data based on the target posture transformation matrix.

[0089] In the embodiment of the present application, the head image is the image of the user's head captured by the camera unit, the camera unit is aimed at the user's head, and the head posture data is the data of the user's head posture. At this time, the user's head image is collected, and the user's head posture data is obtained based on the target posture transformation matrix obtained in step S102 for subsequent processing. For example, Figure 2 As shown, the camera unit CAM is aimed at the head of the user shown in the figure to capture the head image of the user.

[0090] It should be noted that head posture refers to the direction and angle of the head relative to a certain reference point. The reference point in this application is the camera unit. The methods of representing head posture usually include rotation matrix, Euler angle, quaternion, etc. This application takes the yaw angle (Yaw), pitch angle (Pitch), and roll angle (Roll) of Euler angle as examples, where the yaw angle represents the rotation of the head along the vertical axis (up and down direction), which can be understood as the action of looking left and right, the pitch angle represents the rotation of the head along the horizontal axis (left and right direction), which is equivalent to nodding or looking up, and the roll angle represents the rotation of the head along the front-to-back axis, which can be understood as the tilting action of the head, like the action of trying to touch the shoulder, as shown in the following example. Figure 3 shown.

[0091] The head posture data acquisition method provided in the embodiment of the present application is responsive to the inertial measurement unit moving in the visible area of ​​the camera unit and stopping at the target time, the inertial measurement unit and the camera unit are time-synchronized, the inertial measurement unit is fixed on the camera unit, the target posture transformation matrix of the camera unit relative to the inertial measurement unit is obtained, and the camera unit is aligned with the user's head, the user's head image is collected, and the user's head posture data is obtained based on the target posture transformation matrix. The head posture data acquisition method of the present application is to fix the inertial measurement unit on the camera unit after time synchronization with the inertial measurement unit and the camera unit, and obtain the user's head posture data based on the target posture transformation matrix after collecting the user's head image, so as to realize the offline calibration of the inertial measurement unit and the camera unit, improve the flexibility and accuracy of obtaining head posture data, and do not need to rely on face modeling, reduce the influence of individual differences, reduce the acquisition cost, and improve the acquisition efficiency. At the same time, since there is no human obstacle between the camera unit and the user during the acquisition process, the problem of limited acquisition angle is solved.

[0092] Further, such as Figure 4 As shown, the “time synchronization of the inertial measurement unit and the camera unit” in step S102 in the above embodiment may specifically include the following steps:

[0093] S401, recording a first timestamp when the value of the inertial measurement unit stops changing.

[0094] It should be noted that the relevant values ​​can be read out from the inertial measurement unit.

[0095] In the embodiment of the present application, when the inertial measurement unit is moved in the visible area of ​​the camera unit and stops moving at the target time, the first timestamp (represented by Timu) when the value of the inertial measurement unit stops changing is recorded for subsequent processing.

[0096] S402, obtaining a second timestamp when the inertial measurement unit stops moving based on observation by the camera unit, and performing time synchronization on the inertial measurement unit and the camera unit based on a time difference between the first timestamp and the second timestamp.

[0097] In the embodiment of the present application, the camera unit observes the inertial measurement unit to obtain a second timestamp (represented by T_cam) when the inertial measurement unit stops moving, and the time difference between the first timestamp and the second timestamp represents the time difference between the inertial measurement unit and the camera unit. The inertial measurement unit and the camera unit are time-synchronized according to the time difference between the first timestamp and the second timestamp when the inertial measurement unit value stops changing obtained in step S401. For example, the time difference between the first timestamp and the second timestamp, i.e., the time difference △T between the inertial measurement unit and the camera unit, is calculated, and the inertial measurement unit and the camera unit are time-synchronized according to the time difference △T.

[0098] Optionally, the inertial measurement unit and the camera unit are time synchronized based on a preset software synchronization method, that is, the inertial measurement unit and the camera unit are time synchronized according to synchronization software.

[0099] Optionally, the inertial measurement unit and the camera unit may be connected to a same clock source to perform time synchronization on the inertial measurement unit and the camera unit.

[0100] Optionally, the inertial measurement unit and the camera unit can detect the same marker for time synchronization. For example, a marker is mounted on the inertial measurement unit, and synchronization is performed based on the comparison between the time when the camera unit captures the stop of the marker movement and the time when the inertial measurement unit identifies the stop of the marker movement.

[0101] Furthermore, the head posture data acquisition system also includes a calibration plate, which is placed at a fixed position, that is, a fixed position visible to the camera unit. Figure 5 As shown, the step S102 in the above embodiment of “obtaining a target attitude transformation matrix of the camera unit relative to the inertial measurement unit” may specifically include the following steps:

[0102] S501, in response to moving the inertial measurement unit at different positions and postures, capturing calibration images of the calibration plate based on a camera unit, and extracting feature points of the calibration plate from each calibration image based on a preset corner point detection method.

[0103] In an embodiment of the present application, the calibration image is an image of the calibration plate captured by the camera unit, the corner point detection method is a pre-set method for detecting corner points, the feature points are feature points on the calibration plate extracted from the calibration image, the inertial measurement unit is moved at different positions and postures, the camera unit captures the calibration image of the calibration plate at a fixed position, and extracts the feature points on the calibration plate from each calibration image according to the corner point detection method for subsequent processing.

[0104] It is understood by those skilled in the art that the calibration plate has known geometric features, such as black and white squares. By capturing images and analyzing the feature points in them, the internal and external parameters of the camera unit, such as the camera, can be determined, including the focal length, the position of the principal point, the position and orientation of the camera in the world coordinate system, etc.

[0105] S502, obtaining a first posture of the camera unit relative to the calibration plate at the current position and posture based on a preset posture solving method and feature points of the calibration plate, and recording a second posture output by the inertial measurement unit at the current position and posture.

[0106] In the embodiment of the present application, the posture solving method is a pre-set method for solving the posture (i.e., position and attitude). The present application describes the solvePnP (solve Perspective-n-Point) posture solving method as an example, but does not constitute a limitation to this; the first attitude is the attitude of the camera unit relative to the calibration plate at the current position and attitude (expressed by R_cam2chessboard), and the second attitude is the attitude output by the inertial measurement unit at the current position and attitude (expressed by R_imu2inertial). The feature points of the calibration plate obtained in step S501 are calculated by the solvePnP posture solving method, and the first attitude R_cam2chessboard of the camera unit relative to the calibration plate at the current moment, i.e., the current position and attitude, can be obtained, and the second attitude R_imu2inertial output by the inertial measurement unit at the current position and attitude is recorded for subsequent processing.

[0107] It can be understood by those skilled in the art that solvePnP is a method commonly used to calculate the camera pose (position and attitude). The rotation and translation matrix of a camera unit, such as a camera, can be calculated using known 3D space points and 2D image points. The optimal camera pose is solved by minimizing the reprojection error.

[0108] S503: Calculate a target posture transformation matrix according to the first posture and the second posture.

[0109] In the embodiment of the present application, the first posture of the camera unit relative to the calibration plate and the second posture output by the inertial measurement unit at the current position and posture obtained in step S502 are calculated to obtain the target posture transformation matrix (expressed by R_imu2cam) of the camera unit relative to the inertial measurement unit.

[0110] It should be noted that the present application does not impose too many restrictions on the specific method of calculating the target posture transformation matrix according to the first posture and the second posture, and it can be set according to actual conditions.

[0111] As a possible implementation, the first posture and the second posture are synchronized based on the time difference between the first timestamp and the second timestamp to obtain the synchronized first posture and the second posture; the third posture of the inertial measurement unit and the fourth posture of the camera unit are calculated based on the synchronized first posture and the second posture; the target posture transformation matrix is ​​calculated based on the third posture of the inertial measurement unit and the fourth posture of the camera unit. The third posture is the posture of the inertial measurement unit from one position to another position, and the fourth posture is the posture of the camera unit from one position to another position.

[0112] For example, according to the time difference △T (T_imu-T_cam) between the first timestamp and the second timestamp obtained in the above embodiment, the first posture R_cam2chessboard of the camera unit relative to the calibration board at the current position and posture and the second posture R_imu2inertial output by the inertial measurement unit are synchronized in time, and according to the synchronized first posture R_cam2chessboard and second posture R_imu2inertial, the third posture (represented by A) from one position to another position of the inertial measurement unit and the fourth posture (represented by B) from one position to another position of the camera unit are calculated. Since the calibration board is fixed, the position and posture relationship between the third posture A and the fourth posture B is unchanged in different measurements, that is, at different positions and postures. It is beneficial to the third posture A and the fourth posture B, and the target posture transformation matrix is ​​calculated according to the following formula:

[0113] AX=XB

[0114] Among them, X represents the target posture transformation matrix, from which the target posture transformation matrix R_imu2cam is calculated.

[0115] It should be noted that when the target posture transformation matrix is ​​calculated based on the third posture of the inertial measurement unit and the fourth posture of the camera unit, the corresponding hand-eye calibration method can be determined for the target posture transformation matrix, and the third posture and the fourth posture are calculated based on the hand-eye calibration method to obtain multiple posture transformation matrices, and the optimal posture transformation matrix is ​​determined as the target posture transformation matrix. For example, when the target posture transformation matrix is ​​calculated according to the above formula AX=XB, multiple posture transformation matrices X can be obtained by solving the hand-eye calibration method, and the optimal X posture transformation matrix is ​​determined as the target posture transformation matrix. It can be understood by those skilled in the art that hand-eye calibration is usually used to calculate the accurate transformation relationship between the robot arm and the camera (camera unit), and the problem of solving the transformation relationship can usually be expressed by constructing AX=XB, where A represents the transformation matrix from the camera coordinate system to the calibration plate coordinate system, X represents the transformation matrix from the robot arm to the calibration plate coordinate system, and B represents the coordinates of the feature points of the calibration plate observed by the camera in the camera coordinate system. One of the solutions to AX=XB is to use the least squares method, which finds an optimal solution to minimize the error.

[0116] Further, such as Figure 6 As shown, the “obtaining the user's head posture data based on the target posture transformation matrix” in step S103 in the above embodiment may specifically include the following steps:

[0117] S601, calculating a first posture relationship between a preset head coordinate system and a preset camera coordinate system.

[0118] It can be understood by those skilled in the art that an inertial coordinate system is a coordinate system defined relative to a fixed reference object (usually the earth), and is characterized in that there is no acceleration within it. For example, in the common north-east coordinate system, the x-axis of the coordinate system is parallel to the true north direction of the earth's surface, the y-axis is parallel to the true east direction of the earth's surface, and the z-axis points to the direction of gravity on the earth's surface.

[0119] In an embodiment of the present application, the first posture relationship is the posture relationship between the head coordinate system and the camera coordinate system (denoted by R_head2cam in this application). When the camera unit is aimed at the user's head and the user's head image is captured, the first posture relationship R_head2cam is calculated for subsequent processing.

[0120] S602, determining a first value of the inertial measurement unit, and calculating a second posture relationship between the head coordinate system and a preset inertial coordinate system based on the first posture relationship, the first value and a target posture transformation matrix.

[0121] In an embodiment of the present application, the first numerical value is the numerical value obtained by reading the inertial measurement unit (expressed as R_imu2inertial in this application), and the second posture relationship is the posture relationship between the head coordinate system and the inertial coordinate system (expressed as R_inertial2head in this application). The first numerical value R_imu2inertial obtained by the inertial measurement unit is read, and the second posture relationship R_inertial2head is calculated based on the first numerical value R_imu2inertial and the first posture relationship R_head2cam calculated in step S601 and the target posture transformation matrix R_imu2cam in the above embodiment for subsequent processing.

[0122] Optionally, the second posture relation R_inertial2head is calculated according to the following coordinate system transformation formula:

[0123] R_inertial2head=R_imu2inertial.T@R_imu2cam@R_head2cam.T

[0124] Among them, @ is the matrix multiplication operation, and T is the matrix transpose operation.

[0125] S603 , in a second posture relationship, in response to moving the camera unit around the user's head at different positions and postures, acquiring the user's head posture data.

[0126] In the embodiment of the present application, the second posture relationship R_inertial2head is kept unchanged under the second posture relationship R_inertial2head calculated in step S602 (for example, the user's head can be kept still during the acquisition process to ensure that R_inertial2head remains unchanged), and the camera unit is moved around the user's head at different positions and postures, and the camera unit can be rotated appropriately to obtain the user's head posture data, for example, Figure 2 shown.

[0127] It should be noted that when rotating and moving the user's head posture data, it is necessary to ensure that the user's head is always within the field of view of the camera unit, that is, within the visible area.

[0128] It should be noted that the present application does not impose too many restrictions on the specific method of obtaining the user's head posture data, and it can be set according to actual conditions.

[0129] As a possible implementation, determine the second value R_imu2cam@R_imu2intertial of the inertial measurement unit; calculate the user's head posture data based on the second value, the target posture transformation matrix and the second posture relationship. The second value is the value obtained by reading the inertial measurement unit (represented by R_imu2inertial2 in this application to distinguish it from the first value R_imu2inertial), and the head posture data is represented by R_head, which is the true value of the head posture data. For example, the user's head posture data R_head is obtained according to the following formula:

[0130] R_head = R_imu2cam@R_imu2intertial2.T@R_inertial2head.T

[0131] Among them, R_imu2cam represents the target posture transformation matrix, R_imu2intertial2 represents the second value, and R_inertial2head represents the second posture relationship.

[0132] Further, such as Figure 7 As shown, the embodiment of the present application may also include the following steps:

[0133] S701, acquiring a target head image based on a camera unit within a preset time interval, and calculating head posture verification data based on the target head image.

[0134] In the embodiment of the present application, the time interval is a pre-set time interval for acquiring a head image, the target head image is a user head image captured by the camera unit within the time interval, and the head posture verification data is verification data for verifying the head posture data, that is, ideal head posture data. A certain time interval is set, and the target head image is acquired by the camera unit during this time period, and solvePnP is used to calculate the target head image to obtain the head posture verification data R_head2 for verifying the above-calculated head posture data R_head for subsequent processing.

[0135] S702, calculating a target angle based on the head posture data and the head posture verification data, and in response to the target angle being greater than a preset threshold, recalculating the first posture relationship, and re-collecting the head image and the head posture data.

[0136] In the embodiment of the present application, the target angle (expressed in angular) is the angle between the directions corresponding to the head posture data and the head posture verification data. The target angle is calculated based on the head posture data in the above embodiment and the head posture verification data calculated in step S701, and it is determined whether the target angle is greater than a preset threshold (angle threshold). If so, it means that the interference error is large and the acquisition needs to be recalibrated, that is, the first posture relationship needs to be recalculated (equivalent to recalibration), and the head image and head posture data need to be re-collected according to the above process. If not, the acquisition continues.

[0137] Optionally, the target clamp angle can be calculated according to the following formula:

[0138] angular=abs(arccos((trace(R_head2 @R_head)-1 / 2))

[0139] Among them, angular is the target angle, R_head2 represents the head posture verification data, and R_head represents the head posture data.

[0140] Therefore, in order to detect possible interference conditions (such as user head movement) during the acquisition process, a certain time interval is set, the head image is obtained through the camera unit, and the head posture verification data is calculated to judge the interference error and perform corresponding processing, thereby further reducing the interference error and improving the accuracy of the collected head posture data.

[0141] Further, the head image and the head posture data are synchronized; the head detection is performed on the synchronized head image to obtain the head area image of interest. Among them, the head area image of interest is the image of the user's head. For example, the head image contains various redundant elements such as the background of the head, and the head area image of interest can be an image that only includes the user's head after removing various redundant elements such as the background. It can be understood as cropping the head image to obtain a regional image of only the user's head, so as to better target the user's head itself and avoid the influence of other elements (such as the background of the head in the image). It should be noted that synchronizing (aligning) the head image and the head posture data can make the head image and the head posture data a whole set of data (one head image corresponds to one head posture data true value). At this time, the head detection is performed on the head image to obtain the head area image of interest.

[0142] Figure 8 is a schematic diagram of the structure of a head posture data acquisition device provided according to an embodiment of the present application, such as Figure 8 As shown, the head posture data collection device 800 of the embodiment of the present application may specifically include:

[0143] The synchronization module 801 is used to perform time synchronization between the inertial measurement unit and the camera unit in response to the inertial measurement unit moving in the visible area of ​​the camera unit and stopping at a target time.

[0144] The first acquisition module 802 is used to acquire a target attitude transformation matrix of the camera unit relative to the inertial measurement unit in response to fixing the inertial measurement unit on the camera unit; wherein the target attitude transformation matrix is ​​an attitude transformation matrix of the camera unit relative to the inertial measurement unit.

[0145] The second acquisition module 803 is used to collect the user's head image in response to the camera unit being aimed at the user's head, and to acquire the user's head posture data based on the target posture transformation matrix.

[0146] In a possible implementation, the synchronization module is specifically configured to:

[0147] Record the first timestamp when the value of the inertial measurement unit stops changing;

[0148] A second timestamp when the inertial measurement unit stops moving is obtained based on observation by the camera unit, and time synchronization is performed on the inertial measurement unit and the camera unit based on a time difference between the first timestamp and the second timestamp.

[0149] In a possible implementation, the head posture data acquisition system further includes a calibration board; a first acquisition module, specifically configured to:

[0150] In response to moving the inertial measurement unit at different positions and attitudes, capturing calibration images of the calibration plate based on the camera unit, and extracting feature points of the calibration plate from each calibration image based on a preset corner point detection method;

[0151] Based on a preset posture solving method and characteristic points of the calibration plate, a first posture of the camera unit relative to the calibration plate at the current position and posture is obtained, and a second posture output by the inertial measurement unit at the current position and posture is recorded;

[0152] A target posture transformation matrix is ​​calculated according to the first posture and the second posture.

[0153] In a possible implementation manner, the first acquisition module is specifically configured to:

[0154] Performing time synchronization on the first posture and the second posture based on a time difference between the first timestamp and the second timestamp to obtain the synchronized first posture and the second posture;

[0155] Calculate a third posture of the inertial measurement unit and a fourth posture of the camera unit according to the synchronized first posture and second posture; wherein the third posture is the posture of the inertial measurement unit from one position to another position; and the fourth posture is the posture of the camera unit from one position to another position;

[0156] A target posture transformation matrix is ​​calculated based on the third posture of the inertial measurement unit and the fourth posture of the camera unit.

[0157] In a possible implementation manner, the second acquisition module is specifically configured to:

[0158] Calculating a first posture relationship between a preset head coordinate system and a preset camera coordinate system;

[0159] Determine a first value of the inertial measurement unit, and calculate a second posture relationship between the head coordinate system and a preset inertial coordinate system based on the first posture relationship, the first value and the target posture transformation matrix;

[0160] In the second posture relationship, in response to moving the camera unit around the user's head at different positions and postures, the user's head posture data is acquired.

[0161] In a possible implementation manner, the second acquisition module is specifically configured to:

[0162] determining a second value of the inertial measurement unit;

[0163] The user's head posture data is calculated based on the second value, the target posture transformation matrix and the second posture relationship.

[0164] In a possible implementation, the head posture data acquisition device further includes:

[0165] A third acquisition module is used to acquire a target head image based on the camera unit within a preset time interval, and calculate the head posture verification data based on the target head image;

[0166] The verification module is used to calculate the target angle based on the head posture data and the head posture verification data, and in response to the target angle being greater than a preset threshold, recalculate the first posture relationship and re-collect the head image and the head posture data.

[0167] The head posture data acquisition device provided in the embodiment of the present application performs time synchronization on the inertial measurement unit and the camera unit in response to the inertial measurement unit moving in the visible area of ​​the camera unit and stopping at the target time, obtains the target posture transformation matrix of the camera unit relative to the inertial measurement unit in response to the inertial measurement unit being fixed on the camera unit, and acquires the user's head image in response to the camera unit being aimed at the user's head, and acquires the user's head posture data based on the target posture transformation matrix. The head posture data acquisition device of the present application implements offline calibration of the inertial measurement unit and the camera unit by fixing the inertial measurement unit on the camera unit after time synchronization with the inertial measurement unit and the camera unit, and acquires the user's head posture data based on the target posture transformation matrix after acquiring the user's head image, thereby improving the flexibility and accuracy of acquiring head posture data, and does not need to rely on face modeling, reducing the impact of individual differences, reducing acquisition costs, and improving acquisition efficiency. At the same time, since there is no human obstacle between the camera unit and the user during the acquisition process, the problem of limited acquisition angle is solved.

[0168] like Fig. 9 As shown, an electronic device 900 provided in an embodiment of the present application includes: a processor 901, a memory 902 and a bus, wherein the memory 902 stores machine-readable instructions executable by the processor 901. When the electronic device is running, the processor 901 communicates with the memory 902 through the bus, and the processor 901 executes the machine-readable instructions to perform the steps of the head posture data collection method as described above.

[0169] Specifically, the above-mentioned memory 902 and processor 901 can be general-purpose memory and processor, which are not specifically limited here. When the processor 901 runs the computer program stored in the memory 902, the above-mentioned head posture data collection method can be executed.

[0170] Corresponding to the above-mentioned head posture data collection method, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned head posture data collection method are executed.

[0171] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0172] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0174] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the deployment method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0175] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A head posture data acquisition method, applied to a head posture data acquisition system; the head posture data acquisition system comprises an inertial measurement unit and a camera unit; characterized in that: The method comprises: In response to the inertial measurement unit moving in the visible area of ​​the camera unit and stopping at a target time, the inertial measurement unit and the camera unit are time synchronized; the head posture data acquisition system also includes a calibration plate; In response to fixing the inertial measurement unit on the camera unit, in response to moving the inertial measurement unit at different positions and postures, capturing the calibration image of the calibration plate based on the camera unit, and extracting the feature points of the calibration plate from each of the calibration images based on a preset corner point detection method; obtaining a first posture of the camera unit relative to the calibration plate at a current position and posture based on a preset posture solution method and the feature points of the calibration plate, and recording a second posture output by the inertial measurement unit at the current position and posture; calculating a target posture transformation matrix according to the first posture and the second posture; wherein the target posture transformation matrix is ​​a posture transformation matrix of the camera unit relative to the inertial measurement unit; In response to the camera unit being aimed at the user's head, the user's head image is collected, and a first posture relationship between a preset head coordinate system and a preset camera coordinate system is calculated; a first value of the inertial measurement unit is determined, and a second posture relationship between the head coordinate system and a preset inertial coordinate system is calculated based on the first posture relationship, the first value and the target posture transformation matrix; in the second posture relationship, in response to moving the camera unit around the user's head at different positions and postures, the user's head posture data is acquired.

2. The method according to claim 1, characterized in that The time synchronization of the inertial measurement unit and the camera unit includes: Recording a first timestamp when the value of the inertial measurement unit stops changing; A second timestamp when the inertial measurement unit stops moving is obtained based on observation by the camera unit, and time synchronization is performed on the inertial measurement unit and the camera unit based on a time difference between the first timestamp and the second timestamp.

3. The method according to claim 2, characterized in that The step of calculating a target posture transformation matrix according to the first posture and the second posture includes: Performing time synchronization on the first posture and the second posture based on a time difference between the first timestamp and the second timestamp to obtain the synchronized first posture and the second posture; Calculating a third posture of the inertial measurement unit and a fourth posture of the camera unit according to the synchronized first posture and second posture; wherein the third posture is the posture of the inertial measurement unit from one position to another position; and the fourth posture is the posture of the camera unit from one position to another position; The target posture transformation matrix is ​​calculated based on the third posture of the inertial measurement unit and the fourth posture of the camera unit.

4. The method according to claim 1, characterized in that The obtaining of the head posture data of the user includes: determining a second value of the inertial measurement unit; The user's head posture data is calculated based on the second value, the target posture transformation matrix and the second posture relationship.

5. The method according to claim 4, characterized in that The method further comprises: Acquire a target head image based on the camera unit within a preset time interval, and calculate head posture verification data based on the target head image; the head posture data acquisition system also includes a calibration board; A target angle is calculated based on the head posture data and the head posture verification data, and in response to the target angle being greater than a preset threshold, the first posture relationship is recalculated, and the head image and the head posture data are recaptured.

6. A head posture data acquisition device, characterized in that: Applied to head posture data acquisition system; The head posture data acquisition system includes an inertial measurement unit and a camera unit; the device includes: a synchronization module, for performing time synchronization between the inertial measurement unit and the camera unit in response to the inertial measurement unit moving in the visible area of ​​the camera unit and stopping at a target time; the head posture data acquisition system also includes a calibration board; A first acquisition module is used for, in response to fixing the inertial measurement unit on the camera unit and in response to moving the inertial measurement unit at different positions and postures, capturing the calibration image of the calibration plate based on the camera unit, and extracting the feature points of the calibration plate from each of the calibration images based on a preset corner point detection method; obtaining the first posture of the camera unit relative to the calibration plate at the current position and posture based on a preset posture solution method and the feature points of the calibration plate, and recording the second posture output by the inertial measurement unit at the current position and posture; calculating the target posture transformation matrix according to the first posture and the second posture; wherein the target posture transformation matrix is ​​the posture transformation matrix of the camera unit relative to the inertial measurement unit; The second acquisition module is used to collect the user's head image in response to the camera unit being aimed at the user's head, and calculate a first posture relationship between a preset head coordinate system and a preset camera coordinate system; determine a first value of the inertial measurement unit, and calculate a second posture relationship between the head coordinate system and a preset inertial coordinate system based on the first posture relationship, the first value and the target posture transformation matrix; in the second posture relationship, in response to moving the camera unit around the user's head at different positions and postures, obtain the user's head posture data.

7. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the head posture data collection method as described in any one of claims 1 to 5 are performed.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the head posture data collection method as described in any one of claims 1 to 5.