Head posture data acquisition system and method

Abstract

The present invention provides a head posture data acquisition system and method, comprising: a first camera unit, an inertial measurement unit, and a second camera unit; the inertial measurement unit is worn on the user's head and is used to acquire head posture data of the user; the first camera unit and the inertial measurement unit are fixed together and are used to acquire image data of the area they face; the first camera unit faces away from the user's head; the second camera unit is positioned directly opposite the user's head and is used to acquire image data of the area it faces; the first camera unit is used to calibrate the target transformation relationship between the coordinate system of the inertial measurement unit and the coordinate system of the second camera unit to facilitate determination of the user's head posture data in the coordinate system of the second camera unit. The present invention can effectively acquire head posture training data and ensure the training accuracy of the head posture estimation model.

Description

Technical Field

[0001] The present invention belongs to the technical field of data acquisition, and more specifically, relates to a head posture data acquisition system and method. Background Art

[0002] With the continuous advancement of technology, vehicle cabins are becoming increasingly intelligent and user-friendly. Supported by various technologies, users can experience better care and safer service. In-cabin vision applications, the head posture of occupants is often captured. This can be used to analyze the driver's head orientation, determine whether the driver is paying attention to the road, and thus remind the driver to focus on driving. Furthermore, head orientation can be used to analyze the occupant's intentions and behavior, providing a safe and intelligent driving and cabin experience.

[0003] In recent years, deep learning-based head pose estimation methods have garnered widespread attention due to their high accuracy, low latency, and ease of deployment. However, deep learning tasks, especially supervised learning methods, rely heavily on training data. It's difficult to manually label ground truth data for head pose, making it difficult to obtain training data for head pose, impacting the accuracy of head pose estimation models. Summary of the Invention

[0004] The purpose of the present invention is to provide a head posture data acquisition system and method to solve the problem in the prior art that head posture training data is difficult to obtain, which affects the training accuracy of the head posture estimation model.

[0005] To achieve the above objectives, an embodiment of the present invention provides a head posture data acquisition system, the head posture data acquisition system comprising:

[0006] A calibration module and a data acquisition module; the calibration module includes a first camera unit, and the data acquisition module includes an inertial measurement unit and a second camera unit;

[0007] The inertial measurement unit is worn on the user's head and is used to collect posture data of the user's head;

[0008] The first camera unit is fixedly mounted together with the inertial measurement unit and is used to collect image data of the area it is facing; the first camera unit is facing away from the user's head;

[0009] The second camera unit is arranged at a position facing the user's head and is used to collect image data of the area the user is facing;

[0010] Among them, the first camera unit is used to calibrate the target transformation relationship between the coordinate system where the inertial measurement unit is located and the coordinate system where the second camera unit is located; the target transformation relationship is used to convert the posture data collected by the inertial measurement unit to the coordinate system where the second camera unit is located, so as to determine the posture data of the user's head in the coordinate system where the second camera unit is located.

[0011] In a possible implementation, the head posture data acquisition system further includes: a data processing module;

[0012] The data processing module is used to determine a first conversion relationship between a coordinate system where the inertial measurement unit is located and a coordinate system where the first camera unit is located according to the posture data collected by the inertial measurement unit and the image data collected by the first camera unit;

[0013] The data processing module is further configured to determine a second transformation relationship between a coordinate system where the first camera unit is located and a coordinate system where the second camera unit is located based on the image data collected by the first camera unit and the image data collected by the second camera unit;

[0014] The data processing module is further configured to determine a target conversion relationship between the coordinate system where the inertial measurement unit is located and the coordinate system where the second camera unit is located according to the first conversion relationship and the second conversion relationship;

[0015] The data processing module is further configured to convert the posture data collected by the inertial measurement unit according to the target conversion relationship to obtain the posture data of the user's head in the coordinate system where the second camera unit is located.

[0016] In one possible implementation, determining a first conversion relationship between a coordinate system where the inertial measurement unit is located and a coordinate system where the first camera unit is located based on the posture data collected by the inertial measurement unit and the image data collected by the first camera unit includes:

[0017] calibrating internal parameters of the inertial measurement unit;

[0018] calibrating internal parameters of the first camera unit;

[0019] The external parameters of the inertial measurement unit and the external parameters of the first camera unit are calibrated according to the posture data collected by the inertial measurement unit and the image data collected by the first camera unit, and the first conversion relationship is determined.

[0020] In a possible implementation, determining a second transformation relationship between a coordinate system where the first camera unit is located and a coordinate system where the second camera unit is located based on image data captured by the first camera unit and image data captured by the second camera unit includes:

[0021] The external parameters of the first camera unit and the external parameters of the second camera unit are calibrated according to the image data collected by the first camera unit and the image data collected by the second camera unit, and the second conversion relationship is determined.

[0022] In a possible implementation, determining a target transformation relationship between a coordinate system where the inertial measurement unit is located and a coordinate system where the second camera unit is located according to the first transformation relationship and the second transformation relationship includes:

[0023] Determine the target conversion relationship by R_imuToRgb2=R_imuToRgb1@R_Rgb1ToRgb2;

[0024] Among them, R_imuToRgb2 is the target conversion relationship, R_imuToRgb1 is the first conversion relationship, R_Rgb1ToRgb2 is the second conversion relationship, and @ represents matrix multiplication.

[0025] In a possible implementation, the posture data is an angle change of the user's head at different moments;

[0026] The step of converting the posture data collected by the inertial measurement unit according to the target conversion relationship to obtain the posture data of the user's head in the coordinate system where the second camera unit is located includes:

[0027] according to

[0028] R Headpose t0Tot=R_imuToRgb2@R_imut0Tot@(R_imuToRgb2) -1

[0029] Determining an angle change of the user's head at different times in the coordinate system of the second camera unit;

[0030] Among them, R Headpose t0Tot is the angular change of the user's head from time t0 to time t in the coordinate system of the second camera unit, R_imut0Tot is the angular change of the user's head from time t0 to time t collected by the inertial measurement unit, R_imuToRgb2 is the target transformation relationship, and @ represents matrix multiplication.

[0031] In a possible implementation, the head posture data acquisition system further includes:

[0032] Communication module;

[0033] The communication module is used to implement communication among the inertial measurement unit, the first camera unit, and the second camera unit.

[0034] In one possible implementation, the communication module includes a topic communication unit and a protocol communication unit;

[0035] The inertial measurement unit communicates with the first camera unit via the topic communication unit;

[0036] The inertial measurement unit communicates with the second camera unit through the topic communication unit and the protocol communication unit;

[0037] The first camera unit communicates with the second camera unit through the protocol communication unit.

[0038] In a possible implementation, the head posture data acquisition system further includes:

[0039] Data cleaning module;

[0040] The data cleaning module is used to remove abnormal data in the converted posture data;

[0041] The converted posture data is the posture data of the user's head in the coordinate system where the second camera unit is located.

[0042] In another aspect of an embodiment of the present invention, a head posture data collection method applied to the head posture data collection system described above is further provided, the head posture data collection method comprising:

[0043] The inertial measurement unit collects the posture data of the user's head when receiving the data collection instruction, the first camera unit collects the image data of the area it is facing when receiving the data collection instruction, and the second camera unit collects the image data of the area it is facing when receiving the data collection instruction;

[0044] Among them, the posture data collected by the inertial measurement unit, the image data collected by the first camera unit, and the image data collected by the second camera unit are used to determine the target transformation relationship between the coordinate system where the inertial measurement unit is located and the coordinate system where the second camera unit is located; the target transformation relationship is used to convert the posture data collected by the inertial measurement unit to the coordinate system where the second camera unit is located, so as to determine the posture data of the user's head in the coordinate system where the second camera unit is located.

[0045] The beneficial effects of the head posture data acquisition system and method provided by the embodiments of the present invention are:

[0046] An embodiment of the present invention provides a head posture data acquisition system, which includes a data acquisition module. The second camera unit included in the data acquisition module can obtain image data in the training data, and the inertial measurement unit included in the data acquisition module can obtain true value data in the training data. On this basis, the embodiment of the present invention also creatively sets a calibration module, that is, a first camera unit fixedly arranged together with the inertial measurement unit is set. Based on this setting, the target conversion relationship between the inertial measurement unit and the second camera unit can be accurately calibrated with the help of the first camera unit, so as to convert the aforementioned true value data to the coordinate system where the second camera unit is located, so as to facilitate the subsequent training of the head posture estimation model. In other words, the system described based on the embodiment of the present invention can effectively obtain training data of head posture, ensure the training accuracy of the head posture estimation model, and thus effectively solve the problems of the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A schematic diagram of a head posture provided by an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of the structure of a head posture data acquisition system provided by one embodiment of the present invention;

[0050] Figure 3 A schematic diagram of the layout of the head posture data acquisition system provided by the present invention;

[0051] Figure 4 A schematic structural diagram of a head posture data acquisition system provided by another embodiment of the present invention;

[0052] Figure 5 A schematic diagram of conversion between various coordinate systems provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] First, the terms used in the embodiments of the present invention are explained.

[0056] 1) Head posture

[0057] Head posture is used to describe the posture of the head relative to the camera in 3D space, and is generally represented by rotation matrices, rotation vectors, Euler angles, and quaternions (Euler angles are more intuitive and are often used to describe the 3D posture angles of the head). On this basis, a head coordinate system can be established, and the posture of the head can be described by the changes in each posture angle in the head coordinate system. Figure 1 As shown, the head coordinate system has the center of the head as its origin, the X-axis parallel to the line connecting the left and right eyes, the Y-axis parallel to the line connecting the nose and chin endpoints, and the Z-axis perpendicular to both the X and Y axes. Based on this, when the head rotates about the X-axis, the head nods, and the pitch angle changes. When the head rotates about the Y-axis, the head shakes, and the yaw angle changes. When the head rotates about the Z-axis, the head swings, and the roll angle changes. Based on this, head posture can be described.

[0058] 2) Inertial Measurement Unit (IMU)

[0059] IMU stands for Inertial Measurement Unit, which is a component used to measure an object's acceleration, angular velocity, magnetic field, altitude, etc. An IMU includes one or more sensors, such as an inclinometer, accelerometer, gyroscope, magnetometer, barometer, etc. It can obtain the object's motion, heading, attitude angle (roll angle, pitch angle, and yaw angle) through sensor fusion algorithms.

[0060] Next, the background involved in the embodiments of the present invention is described in detail to facilitate understanding of the solutions described in the embodiments of the present invention. The background involved in the embodiments of the present invention is described in detail as follows:

[0061] Head pose estimation is a key area of ​​research in the field of vision. Its primary task is to predict the 3D pose angles of the head from a single image. Due to its widespread application, related research has gradually increased. In recent years, deep learning methods have also garnered widespread attention due to their high precision, low latency, and ease of deployment. The mainstream approach to head pose estimation mentioned above is also deep learning. Its specific technical approach involves first detecting and locating faces in the image to obtain facial images. Training data is then used to train a head pose estimation model, which then determines the head pose angles. Training data relies heavily on ground truth, and the accuracy of these ground truth directly impacts the detection performance of the head pose estimation model.

[0062] Therefore, based on the above background, an embodiment of the present invention proposes a head posture data acquisition system based on an inertial measurement unit and a dual camera unit, aiming to obtain high-accuracy and high-precision head posture angles and improve the training accuracy of the head posture estimation model.

[0063] Based on the above description, you can refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of a head posture data acquisition system provided in one embodiment of the present invention. The head posture data acquisition system 100 includes:

[0064] Calibration module 10 and data acquisition module 20. The calibration module 10 includes a first camera unit 11, and the data acquisition module 20 includes an inertial measurement unit 21 and a second camera unit 22.

[0065] The inertial measurement unit 21 is worn on the user's head and is used to collect posture data of the user's head.

[0066] The first camera unit 11 is fixedly mounted with the inertial measurement unit 21 to collect image data of the area it is facing. The first camera unit 11 faces away from the user's head.

[0067] The second camera unit 22 is arranged at a position facing the user's head, and is used to collect image data of the area the user is facing.

[0068] The first camera unit 11 is used to calibrate the target transformation relationship between the coordinate system of the inertial measurement unit 21 and the coordinate system of the second camera unit 22. The target transformation relationship is used to convert the posture data collected by the inertial measurement unit 21 to the coordinate system of the second camera unit 22 to determine the posture data of the user's head in the coordinate system of the second camera unit 22.

[0069] In this embodiment, since the first camera unit 11 rotates with the user's head, it includes multiple orientation zones. Therefore, the orientation zone of the second camera unit 22 must ensure that it overlaps with at least one orientation zone of the first camera unit 11 to facilitate calibration of the conversion relationship between the two.

[0070] In this embodiment, since the inertial measurement unit 21 collects posture data in its own coordinate system, in order to use it in subsequent training data, the posture data collected by the inertial measurement unit 21 needs to be converted to the coordinate system of the human head image acquisition device (that is, the coordinate system of the second camera unit 22 or the head coordinate system). Therefore, it is necessary to calibrate the target transformation relationship between the coordinate system where the inertial measurement unit 21 is located and the coordinate system where the second camera unit 22 is located. Therefore, in this embodiment, the inertial measurement unit 21 is worn on the user's head to read the posture data of the user's head, and the first camera unit 11 and the inertial measurement unit 21 are fixed together and worn on the user's head. Based on this setting, the transformation relationship between the first camera unit 11 and the inertial measurement unit 21 is fixed, and the transformation relationship between the second camera unit 22 and the inertial measurement unit 21 is variable. On this basis, the inertial measurement unit 21 can use the first camera unit 11, to which the transformation relationship is fixed, to convert the posture data collected by it to the coordinate system where the second camera unit 22 is located, thereby obtaining a high-precision head posture estimation truth value and improving the training accuracy of the head posture estimation model. Furthermore, the data acquisition module only uses the inertial measurement unit 21 and the second camera unit 22 . Compared with the existing method of providing multiple data acquisition devices to improve measurement accuracy, the hardware cost of the embodiment of the present invention is also lower.

[0071] In this embodiment, as a specific implementation method, please refer to Figure 3 , Figure 3 In the figure, IMU represents the inertial measurement unit 21, RGB1 represents the first camera unit 11, and RGB2 represents the second camera unit 22. Figure 3 As shown, in order to avoid the image captured by the second camera unit 22 containing the first camera unit 11 and / or the inertial measurement unit 21, which affects the training accuracy of the subsequent head posture estimation model, an embodiment of the present invention can wear the first camera unit 11 and the inertial measurement unit 21 behind the user's head, and set the second camera unit 22 directly in front of the user (towards the user's head) to better collect various data of the user's head.

[0072] As can be seen from the above description, an embodiment of the present invention provides a head posture data acquisition system, which includes a data acquisition module. The second camera unit included in the data acquisition module can obtain image data in the training data, and the inertial measurement unit included in the data acquisition module can obtain true value data in the training data. On this basis, the embodiment of the present invention also creatively sets a calibration module, that is, a first camera unit fixedly set together with the inertial measurement unit is set. Based on this setting, the target conversion relationship between the inertial measurement unit and the second camera unit can be accurately calibrated with the help of the first camera unit, thereby converting the aforementioned true value data to the coordinate system where the second camera unit is located, so as to facilitate the subsequent training of the head posture estimation model.

[0073] That is to say, the system described in the embodiment of the present invention can effectively obtain training data of head posture, ensure the training accuracy of the head posture estimation model, and thus effectively solve the problems of the existing technology.

[0074] In one possible implementation, refer to Figure 4 The head posture data acquisition system 100 also includes: a data processing module 30.

[0075] The data processing module 30 is configured to determine a first conversion relationship between the coordinate system of the inertial measurement unit 21 and the coordinate system of the first camera unit 11 based on the posture data collected by the inertial measurement unit and the image data collected by the first camera unit 11 .

[0076] The data processing module 30 is further configured to determine a second conversion relationship between the coordinate system of the first camera unit 11 and the coordinate system of the second camera unit 22 based on the image data captured by the first camera unit 11 and the image data captured by the second camera unit 22 .

[0077] The data processing module 30 is further configured to determine a target conversion relationship between the coordinate system where the inertial measurement unit 21 is located and the coordinate system where the second camera unit 22 is located according to the first conversion relationship and the second conversion relationship.

[0078] The data processing module 30 is further configured to convert the posture data collected by the inertial measurement unit according to the target conversion relationship to obtain the posture data of the user's head in the coordinate system where the second camera unit is located.

[0079] In this embodiment, reference may be made to Figure 5 , Figure 5 In the example, R_imuToRgb1 represents the first conversion relationship, and R_rgb1ToRgb2 represents the second conversion relationship. Figure 5As shown, determining the target transformation relationship between the coordinate system of the inertial measurement unit 21 and the coordinate system of the second camera unit 22 can include two steps. The first step is to calibrate the first transformation relationship between the coordinate system of the inertial measurement unit 21 and the coordinate system of the first camera unit 11 in an offline manner. The second step is to calibrate the second transformation relationship between the coordinate system of the first camera unit 11 and the coordinate system of the second camera unit 22 in an online manner.

[0080] The offline method refers to a method in which the collection of relevant data and the calibration of the first conversion relationship can be completed without limiting the object to be collected (ie, the user).

[0081] Since the first camera unit 11 is fixed to the user's head, its spatial position changes as the user's head moves. Different users will have different images when their heads are rotated at the same angle. Therefore, the second transformation relationship needs to be calibrated for each captured subject (user). Therefore, this calibration requires an online method. Specifically, the online method requires collecting relevant data and calibrating the second transformation relationship for each captured subject.

[0082] From the above description, it can be seen that, by combining offline calibration and online calibration, the embodiment of the present invention can calibrate the target transformation relationship between the coordinate system of the inertial measurement unit 21 and the coordinate system of the second camera unit 22 .

[0083] In this embodiment, after determining the target transformation relationship between the coordinate system where the inertial measurement unit 21 is located and the coordinate system where the second camera unit 22 is located, the subsequent data processing module 30 can transform the posture data collected by the inertial measurement unit according to the target transformation relationship to obtain the posture data of the user's head in the coordinate system where the second camera unit is located.

[0084] In one possible implementation, determining a first transformation relationship between a coordinate system where the inertial measurement unit is located and a coordinate system where the first camera unit is located based on posture data collected by the inertial measurement unit and image data collected by the first camera unit includes:

[0085] Calibrate the internal parameters of the inertial measurement unit.

[0086] Calibrate the internal parameters of the first camera unit.

[0087] The external parameters of the inertial measurement unit and the external parameters of the first camera unit are calibrated according to the posture data and the image data collected by the first camera unit, and a first conversion relationship is determined.

[0088] In this embodiment, the internal parameters of the inertial measurement unit are calibrated mainly because the inertial measurement unit has a scale factor error, and the accelerometer, gyroscope, etc. therein may have three non-orthogonal axes and other problems, resulting in a certain amount of noise when the inertial measurement unit calculates the attitude angle, so it is necessary to calibrate the noise parameters of the inertial measurement unit. Among them, the calibration of the internal parameters of the inertial measurement unit mainly involves designing an inertial measurement unit noise model, recording inertial measurement unit data, and using an internal parameter calibration tool to calibrate the internal parameters. For example, the tool imu_utils can be used to calibrate the internal parameters of the inertial measurement unit.

[0089] Among them, the internal parameters of the inertial measurement unit are calibrated, which can be detailed as follows:

[0090] Step 1: Record the data of the IMU at rest.

[0091] Step 2: Play the IMU data packet and obtain the IMU measurement data.

[0092] Step 3: Use the imu_utils tool to process the above measurement data and calibrate the noise of the inertial measurement unit.

[0093] In this embodiment, similar to common color camera calibration, the internal parameter calibration of the first camera unit mainly involves obtaining the internal parameters of the first camera unit. The aforementioned internal parameters include but are not limited to focal length, principal point coordinates, and camera distortion parameters. On this basis, the internal parameters of the first camera unit are calibrated, which can be detailed as follows:

[0094] Step 1: Set up a calibration plate in the direction area of ​​the first camera unit (refer to Figure 2 On this basis, images of the orientation area of ​​the first camera unit can be captured from multiple angles (that is, images of the calibration plate are captured), and the corner points of the calibration plate can be extracted from the images of the calibration plate.

[0095] Step 2: Determine the initial internal parameters of the first camera unit by fitting a circle around the corner points of each row of the calibration plate. It's important to note that parallel lines in the real world are no longer parallel after being projected by the camera unit, but instead intersect at two points. Therefore, for each corner point in a row of the calibration plate, a circle can be fitted onto the image. The distance between the two circles' intersections, divided by pi, determines the initial focal length. The initial value for the principal point coordinates can be half the image resolution of the first camera unit.

[0096] Step 3: Determine the pose of each frame for the first camera unit and use this pose as a variable for subsequent optimization. It's important to note that when using the calibration plate as the reference coordinate system, the 3D coordinates of each corner point in this reference coordinate system are known, as the plate's dimensions are known. Through 3D-2D mapping, the pose of each frame can be determined.

[0097] Step 4: Use the pose of each frame to transform the 3D coordinates in the reference coordinate system to the camera coordinate system. Using the camera's projection model, we obtain the predicted projection positions of the calibration plate's 3D corner points on the image plane. By optimizing the pose of each frame and the internal parameters of the first camera unit to minimize all reprojection errors, the internal parameters that minimize all reprojection errors are the internal parameters of the first camera unit for calibration.

[0098] In this embodiment, the external parameters of the inertial measurement unit and the first camera unit can be calibrated by hand-eye calibration. This process can be described in detail as follows:

[0099] Step 1: Estimate the time delay between the first camera unit and the inertial measurement unit.

[0100] Step 2: Obtain the initial conversion parameters between the first camera unit and the inertial measurement unit and some necessary initial values, such as gravity acceleration, gyroscope bias, etc.

[0101] Step 3: Optimize relevant parameters using the image data captured by the first camera unit and the posture data collected by the inertial measurement unit to calibrate the external parameters of the first camera unit and the inertial measurement unit, and determine a second conversion relationship between the two. The aforementioned relevant parameters include, but are not limited to, reprojection errors of all corner points, measurement errors of the accelerometer and gyroscope of the inertial measurement unit, and bias random walk noise.

[0102] In one possible implementation, determining a second transformation relationship between a coordinate system where the first camera unit is located and a coordinate system where the second camera unit is located based on image data captured by the first camera unit and image data captured by the second camera unit includes:

[0103] The external parameters of the first camera unit and the external parameters of the second camera unit are calibrated according to the image data collected by the first camera unit and the image data collected by the second camera unit, and a second conversion relationship is determined.

[0104] In this embodiment, the second conversion relationship can be determined by calibrating the external parameters of the first camera unit and the second camera unit. The second conversion relationship can be determined in detail as follows:

[0105] Step 1: Set up a calibration plate in the direction area of ​​the first camera unit (refer to Figure 2 On this basis, the first camera unit and the second camera unit can simultaneously capture images of the area they are facing (that is, capture images of the calibration plate).

[0106] Step 2: The first camera unit determines the corner points of the calibration plate based on the image of the calibration plate. On this basis, the rotation matrix and translation matrix of the first camera unit and the world coordinate system can be solved by the PNP method.

[0107] Step 3: The second camera unit determines the corner points of the calibration plate based on the image of the calibration plate. On this basis, the rotation matrix and translation matrix of the second camera unit and the world coordinate system can be solved by the PNP method.

[0108] Step 4: The second conversion relationship can be represented by a rotation matrix and a translation matrix. On this basis, the second conversion relationship between the first camera unit and the second camera unit can be determined by the following formula.

[0109]

[0110] Wherein, R is the rotation matrix between the first camera unit and the second camera unit, is the translation matrix between the first camera unit and the second camera unit, R l is the rotation matrix between the first camera unit and the world coordinate system, R r is the rotation matrix between the second camera unit and the world coordinate system, is the translation matrix between the first camera unit and the world coordinate system, is the translation matrix between the second camera unit and the world coordinate system.

[0111] In one possible implementation, determining a target transformation relationship between a coordinate system where the inertial measurement unit is located and a coordinate system where the second camera unit is located according to the first transformation relationship and the second transformation relationship includes:

[0112] The target conversion relationship is determined by R_imuToRgb2=R_imuToRgb1@R_Rgb1ToRgb2.

[0113] Among them, R_imuToRgb2 is the target conversion relationship, R_imuToRgb1 is the first conversion relationship, R_Rgb1ToRgb2 is the second conversion relationship, and @ represents matrix multiplication.

[0114] In this embodiment, in subsequent practical applications, the posture angle of the user's head can be collected through the inertial measurement unit, and then the posture angle under the inertial measurement unit can be converted into the posture angle under the second camera unit through the target conversion relationship determined by the above formula, so as to determine the true value data in the training data.

[0115] In a possible implementation, the posture data is the angle change of the user's head at different times.

[0116] The posture data collected by the inertial measurement unit is converted according to the target conversion relationship to obtain the posture data of the user's head in the coordinate system where the second camera unit is located, including:

[0117] according to

[0118] R Headpose t0Tot=R_imuToRgb2@R_imut0Tot@(R_imuToRgb2) -1

[0119] Determine the angle change of the user's head at different times in the coordinate system where the second camera unit is located.

[0120] Among them, R Headpose t0Tot is the angular change of the user's head from time t0 to time t in the coordinate system of the second camera unit, R_imut0Tot is the angular change of the user's head from time t0 to time t collected by the inertial measurement unit, R_imuToRgb2 is the target transformation relationship, and @ represents matrix multiplication.

[0121] In this embodiment, in subsequent practical applications, the angle change of the user's head posture angle can be collected through the inertial measurement unit, and then the angle change under the inertial measurement unit can be converted into the angle change under the second camera unit through the above formula, so as to determine the true value data in the training data.

[0122] In one possible implementation, refer to Figure 4 , the head posture data acquisition system 100 further includes:

[0123] Communication module 40.

[0124] The communication module 40 is used to implement communication among the inertial measurement unit 21 , the first camera unit 11 and the second camera unit 22 .

[0125] In this embodiment, in order to ensure the accuracy of calibration and subsequent data acquisition, data needs to be collected as simultaneously as possible during the calibration process of the inertial measurement unit 21, the first camera unit 11, and the second camera unit 22, as well as the subsequent training data acquisition process. Therefore, the embodiment of the present invention also provides a communication module 40 to realize mutual communication between the inertial measurement unit 21, the first camera unit 11, and the second camera unit 22.

[0126] In a possible implementation, the communication module includes a topic communication unit and a protocol communication unit.

[0127] The inertial measurement unit communicates with the first camera unit via the topic communication unit.

[0128] The inertial measurement unit communicates with the second camera unit via the topic communication unit and the protocol communication unit.

[0129] The first camera unit communicates with the second camera unit through the protocol communication unit.

[0130] In this embodiment, the inertial measurement unit and the first camera unit may communicate using a topic communication method of the ROS system (ie, through a topic communication unit), that is, one node publishes a message and the other node subscribes to the message.

[0131] In this embodiment, the inertial measurement unit and the second camera unit can communicate via a topic communication unit and a protocol communication unit (eg, a TCP-IP communication unit).

[0132] In this embodiment, the first camera unit and the second camera unit can communicate with each other through a protocol communication unit (eg, a TCP-IP communication unit).

[0133] In one possible implementation, refer to Figure 4 , the head posture data acquisition system 100 further includes:

[0134] Data cleaning module 50.

[0135] The data cleaning module 50 is used to remove abnormal data in the converted posture data, wherein the converted posture data is the posture data of the user's head in the coordinate system where the second camera unit is located.

[0136] In this embodiment, after determining the posture data of the user's head in the coordinate system of the second camera unit, in order to ensure the accuracy and usability of the data, the embodiment of the present invention can also use the data cleaning module 50 to clean up abnormal data to obtain more accurate posture data. The posture data can also be posture angles, which is not limited in this embodiment.

[0137] In summary of the above embodiments, the present invention proposes a head posture data acquisition system based on an inertial measurement unit and a dual camera unit, which aims to obtain high-accuracy and high-precision posture data and improve the training accuracy of the head posture estimation model. Specifically, the present invention fixes the inertial measurement unit and the first camera unit on the user's head, and the inertial measurement unit reads the posture angle, and combines with the first camera unit to convert the posture data in the inertial measurement unit coordinate system to the head coordinate system (that is, the second camera unit coordinate system), thereby obtaining the posture data in the head coordinate system. Through the present invention, a large amount of high-precision true value data can be obtained, thereby improving the training accuracy. In addition, since this system only uses the inertial measurement unit and the camera unit, the hardware cost is also relatively low.

[0138] In another aspect of an embodiment of the present invention, a head posture data collection method applied to the head posture data collection system described above is further provided. The head posture data collection method includes:

[0139] The inertial measurement unit collects the posture data of the user's head when receiving the data collection instruction, the first camera unit collects the image data of the area it is facing when receiving the data collection instruction, and the second camera unit collects the image data of the area it is facing when receiving the data collection instruction.

[0140] The posture data collected by the inertial measurement unit, the image data collected by the first camera unit, and the image data collected by the second camera unit are used to determine a target transformation relationship between the coordinate system of the inertial measurement unit and the coordinate system of the second camera unit. The target transformation relationship is used to convert the posture data collected by the inertial measurement unit to the coordinate system of the second camera unit to determine the posture data of the user's head in the coordinate system of the second camera unit.

[0141] In this embodiment, the inertial measurement unit, the first camera unit and the second camera unit are all data acquisition units. When they receive data acquisition instructions, all three can collect corresponding data and then send the collected data to the internal / external data processing module for data processing.

[0142] In this embodiment, data collection instructions may be sent to different data collection units in different scenarios.

[0143] For example, when calibrating the transformation relationship between the inertial measurement unit, the first camera unit, and the second camera unit, data collection instructions can be sent to the inertial measurement unit, the first camera unit, and the second camera unit simultaneously. In this case, the inertial measurement unit, the first camera unit, and the second camera unit all collect data, and the data collected by these three units is used to determine the target transformation relationship between the coordinate system of the inertial measurement unit and the coordinate system of the second camera unit. Specifically, the inertial measurement unit, the first camera unit, and the second camera unit can send their collected data to an internal / external data processing module, which processes the data to obtain the target transformation relationship between the coordinate system of the inertial measurement unit and the coordinate system of the second camera unit.

[0144] For example, after the aforementioned target conversion relationship calibration is completed, when the head posture estimation model training data is subsequently acquired, a data acquisition instruction can be sent only to the inertial measurement unit and the second camera unit. At this time, the inertial measurement unit collects the posture data of the user's head (recorded as the first posture data), and the second camera unit collects the image data of the area it is facing (that is, the image data containing the user's head, recorded as the first image data). On this basis, the first image data is the image data in the head posture estimation model training data, and the first posture data converted by the target conversion relationship is the true value data of the head posture estimation model training data (wherein the first posture data can be converted according to the target conversion relationship by an internal / external data processing module), that is, the combination of the first image data and the first posture data converted by the target conversion relationship constitutes the training data of the head posture estimation model.

[0145] In summary, the data collection method according to the embodiment of the present invention can effectively obtain training data of head posture, thereby ensuring the training accuracy of the head posture estimation model.

[0146] In a possible implementation, the head posture data collection method may further include various data processing processes of a data processing module. For details, please refer to the aforementioned embodiment and will not be described again here.

[0147] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A head posture data acquisition system, characterized in that: include: A calibration module and a data acquisition module; the calibration module includes a first camera unit, and the data acquisition module includes an inertial measurement unit and a second camera unit; The inertial measurement unit is worn on the user's head and is used to collect posture data of the user's head; The first camera unit is fixedly mounted together with the inertial measurement unit and is used to collect image data of the area it is facing; the first camera unit is facing away from the user's head; The second camera unit is arranged at a position facing the user's head and is used to collect image data of the area the user is facing; Among them, the first camera unit is used to calibrate the target transformation relationship between the coordinate system where the inertial measurement unit is located and the coordinate system where the second camera unit is located; the target transformation relationship is used to convert the posture data collected by the inertial measurement unit to the coordinate system where the second camera unit is located, so as to determine the posture data of the user's head in the coordinate system where the second camera unit is located.

2. The head posture data acquisition system according to claim 1, characterized in that: The head posture data acquisition system further includes: a data processing module; The data processing module is used to determine a first conversion relationship between a coordinate system where the inertial measurement unit is located and a coordinate system where the first camera unit is located according to the posture data collected by the inertial measurement unit and the image data collected by the first camera unit; The data processing module is further configured to determine a second transformation relationship between a coordinate system where the first camera unit is located and a coordinate system where the second camera unit is located based on the image data collected by the first camera unit and the image data collected by the second camera unit; The data processing module is further configured to determine a target conversion relationship between the coordinate system where the inertial measurement unit is located and the coordinate system where the second camera unit is located according to the first conversion relationship and the second conversion relationship; The data processing module is further configured to convert the posture data collected by the inertial measurement unit according to the target conversion relationship to obtain the posture data of the user's head in the coordinate system where the second camera unit is located.

3. The head posture data acquisition system according to claim 2, characterized in that: The determining, based on the posture data collected by the inertial measurement unit and the image data collected by the first camera unit, a first conversion relationship between a coordinate system where the inertial measurement unit is located and a coordinate system where the first camera unit is located, includes: calibrating internal parameters of the inertial measurement unit; calibrating internal parameters of the first camera unit; The external parameters of the inertial measurement unit and the external parameters of the first camera unit are calibrated according to the posture data collected by the inertial measurement unit and the image data collected by the first camera unit, and the first conversion relationship is determined.

4. The head posture data acquisition system according to claim 2, wherein: The determining, based on the image data collected by the first camera unit and the image data collected by the second camera unit, a second conversion relationship between a coordinate system where the first camera unit is located and a coordinate system where the second camera unit is located includes: The external parameters of the first camera unit and the external parameters of the second camera unit are calibrated according to the image data collected by the first camera unit and the image data collected by the second camera unit, and the second conversion relationship is determined.

5. The head posture data acquisition system according to any one of claims 2 to 4, characterized in that: The determining, according to the first conversion relationship and the second conversion relationship, a target conversion relationship between a coordinate system where the inertial measurement unit is located and a coordinate system where the second camera unit is located includes: Determine the target conversion relationship by R_imuToRgb2=R_imuTiRgb1@R_Rgb1ToRgb2; Among them, R_imuToRgb2 is the target conversion relationship, R_muToRgb1 is the first conversion relationship, R_Rgb1ToRgb2 is the second conversion relationship, and @ represents matrix multiplication.

6. The head posture data acquisition system according to any one of claims 2 to 4, characterized in that: The posture data is the angle change of the user's head at different times; The step of converting the posture data collected by the inertial measurement unit according to the target conversion relationship to obtain the posture data of the user's head in the coordinate system where the second camera unit is located includes: according to <h2 style=";text-align:left;direction:ltr">R<h2 style=";text-align:left;direction:ltr"> Headpose <h2 style=";text-align:left;direction:ltr"> t0Tot=R_imuToRgb2@R_imut0Tot@(R_imuToRgb2)<h2 style=";text-align:left;direction:ltr"> -1 Determining an angle change of the user's head at different times in the coordinate system of the second camera unit; Among them, R Headpose t0Tot is the angular change of the user's head from time t0 to time t in the coordinate system of the second camera unit, R_imut0Tot is the angular change of the user's head from time t0 to time t collected by the inertial measurement unit, R_imuToRgb2 is the target transformation relationship, and @ represents matrix multiplication.

7. The head posture data acquisition system according to any one of claims 1 to 4, characterized in that: The head posture data acquisition system also includes: Communication module; The communication module is used to implement communication among the inertial measurement unit, the first camera unit, and the second camera unit.

8. The head posture data acquisition system according to claim 7, characterized in that: The communication module includes a topic communication unit and a protocol communication unit; The inertial measurement unit communicates with the first camera unit via the topic communication unit; The inertial measurement unit communicates with the second camera unit through the topic communication unit and the protocol communication unit; The first camera unit communicates with the second camera unit through the protocol communication unit.

9. The head posture data acquisition system according to any one of claims 2 to 4, characterized in that: The head posture data acquisition system also includes: Data cleaning module; The data cleaning module is used to remove abnormal data in the converted posture data; The converted posture data is the posture data of the user's head in the coordinate system where the second camera unit is located.

10. A head posture data acquisition method based on the head posture data acquisition system according to any one of claims 1 to 9, characterized in that: include: The inertial measurement unit collects the posture data of the user's head when receiving the data collection instruction, the first camera unit collects the image data of the area it is facing when receiving the data collection instruction, and the second camera unit collects the image data of the area it is facing when receiving the data collection instruction; Among them, the posture data collected by the inertial measurement unit, the image data collected by the first camera unit, and the image data collected by the second camera unit are used to determine the target transformation relationship between the coordinate system where the inertial measurement unit is located and the coordinate system where the second camera unit is located; the target transformation relationship is used to convert the posture data collected by the inertial measurement unit to the coordinate system where the second camera unit is located, so as to determine the posture data of the user's head in the coordinate system where the second camera unit is located.

Images