Human factor intelligent attitude estimation method based on multi-modal data, edge computing equipment and medium

Abstract

The invention discloses a human factor intelligent attitude estimation method based on multi-modal data, an edge computing device and a medium. The human factor intelligent attitude estimation method based on the multi-modal data comprises the following steps: acquiring current IMU head attitude estimation data and current visual head attitude estimation data; fusing the current IMU head attitude estimation data and the current visual head attitude estimation data to obtain fused head attitude estimation data; and replacing the current IMU head attitude estimation data with the fused head attitude data, and continuing estimation based on the fused head attitude data to obtain subsequent IMU head attitude estimation data.

Description

Technical Field

[0001] The present invention relates to technical fields such as inertial navigation and computer vision, and in particular to a human factors intelligent posture estimation method based on multimodal data, an edge computing device and a medium. Background Art

[0002] High-precision head posture estimation plays an important role in improving the human-computer interaction experience, accurately evaluating the effects of health treatments, improving the accuracy of expression recognition, and accurately estimating people's behavioral intentions. Therefore, how to improve the accuracy of head posture estimation has become a focus of attention.

[0003] Related technologies for vision-based head posture estimation rely too much on image quality. The influence of the external environment or the posture of the person may cause information missing in the image, thus affecting the accuracy of posture estimation. In addition, the image processing algorithm is highly complex and cannot run on low-end embedded devices. In addition, individual differences among users may also affect the accuracy of head posture estimation. Summary of the invention

[0004] The embodiments of the present invention aim to solve at least one of the technical problems in the related art to a certain extent. To this end, one object of the present invention is to propose a human factors intelligent posture estimation method, device, edge computing device and storage medium based on multimodal data that can improve the accuracy of posture estimation.

[0005] An embodiment of the present invention provides a human factors intelligent posture estimation method based on multimodal data. The human factors intelligent posture estimation method based on multimodal data includes: obtaining current IMU head posture estimation data and current visual head posture estimation data; fusing the current IMU head posture estimation data and the current visual head posture estimation data to obtain fused head posture estimation data; replacing the current IMU head posture estimation data with the fused head posture data, so as to continue estimating based on the fused head posture data to obtain subsequent IMU head posture estimation data.

[0006] Exemplarily, the current IMU head posture estimation data and the current visual head posture estimation data are fused to obtain fused head posture estimation data, including: determining the error between the current IMU head posture estimation data and the current visual head posture estimation data; updating the current IMU head posture estimation data based on the error, and determining the updated current IMU head posture estimation data as the fused head posture estimation data.

[0007] Exemplarily, updating the current IMU head posture estimation data based on the error includes: determining an error dynamic coefficient based on the error; obtaining an update amount based on the error dynamic coefficient and the error; and subtracting the update amount from the current IMU head posture estimation data to obtain updated current IMU head posture estimation data.

[0008] Exemplarily, obtaining current IMU head posture estimation data includes: obtaining at least one of head posture change data and head posture error data; based on at least one of the head posture change data and the head posture error data, updating the IMU head posture estimation data at the previous moment to obtain current IMU head posture estimation data.

[0009] Exemplarily, based on the head posture change data and the head posture error data, the IMU head posture estimation data at the previous moment is updated to obtain the current IMU head posture estimation data, including: based on the head posture change data, the IMU head posture estimation data at the previous moment is updated to obtain the IMU head posture estimation data after the first update; based on the head posture error data, the IMU head posture estimation data after the first update is updated to obtain the current IMU head posture estimation data.

[0010] Exemplarily, obtaining head posture change data includes: obtaining the angular velocity measurement value at the previous moment, the angular velocity measurement value at the current moment, and the gyro zero bias nominal value at the previous moment; obtaining the posture angle change based on the angular velocity measurement value at the previous moment, the angular velocity measurement value at the current moment, the gyro zero bias nominal value at the previous moment, and the time difference between the previous moment and the current moment; and obtaining the head posture change data based on the posture angle change.

[0011] Exemplarily, obtaining head posture error data includes: obtaining error state data at a current moment according to error state data at a previous moment based on a Kalman update method; and obtaining head posture error data based on the error state data at the current moment.

[0012] Exemplarily, based on the Kalman update method, the error state data of the current moment is obtained according to the error state data of the previous moment, including: constructing an error state space equation based on the error state variable, gyroscope noise and wandering noise; obtaining an error state covariance matrix based on the error state space equation, gyroscope noise and wandering noise; updating the error state data of the previous moment based on the observed value and the error state covariance matrix to obtain the error state data of the current moment.

[0013] Exemplarily, based on the error state space equation, gyro noise and wandering noise, an error state covariance matrix is ​​obtained, including: based on the error state space equation, a state transfer matrix is ​​obtained; based on the gyro noise, wandering noise and the state transfer matrix, an error state covariance matrix is ​​obtained.

[0014] Exemplarily, based on the observation quantity and the error state covariance matrix, the error state data of the previous moment is updated to obtain the error state data of the current moment, including: based on the observation quantity, obtaining the observation matrix, the measurement covariance matrix and the residual matrix of the observation quantity; based on the error state covariance matrix, the observation matrix, the measurement covariance matrix, the residual matrix and the Kalman gain, the error state data of the previous moment is updated to obtain the error state data of the current moment, wherein the Kalman gain is associated with the error state covariance matrix, the observation matrix and the measurement covariance matrix.

[0015] Exemplarily, obtaining current visual head posture estimation data includes: performing face detection and feature extraction on head image data to obtain two-dimensional image feature points; converting the two-dimensional image feature points into three-dimensional space feature points; and obtaining current visual head posture estimation data based on the three-dimensional space feature points.

[0016] An embodiment of the present invention provides a human factors intelligent posture estimation device based on multimodal data. The human factors intelligent posture estimation device based on multimodal data includes: an acquisition module, used to acquire current IMU head posture estimation data and current visual head posture estimation data; a fusion module, used to fuse the current IMU head posture estimation data and the current visual head posture estimation data to obtain fused head posture estimation data; an estimation module, used to replace the current IMU head posture estimation data with the fused head posture data, so as to continue to estimate based on the fused head posture data to obtain subsequent IMU head posture estimation data.

[0017] An embodiment of the present invention provides an edge computing device, including: a processor and a memory, the memory storing programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the above-mentioned human factors intelligent posture estimation method based on multimodal data is implemented.

[0018] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned human factors intelligent posture estimation method based on multimodal data is implemented.

[0019] In the above implementation, the human factors intelligent posture estimation method based on multimodal data includes: obtaining current IMU head posture estimation data and current visual head posture estimation data; fusing the current IMU head posture estimation data and the current visual head posture estimation data to obtain fused head posture estimation data; replacing the current IMU head posture estimation data with the fused head posture data, so as to continue to estimate based on the fused head posture data to obtain subsequent IMU head posture estimation data. Through data fusion, the above method can provide auxiliary information when the computer vision is affected by the IMU data, and the result of the computer vision can also correct the error of the IMU to a certain extent, thereby improving the accuracy and reliability of posture estimation. In addition, when the light is too dark, too bright or there is occlusion, it is difficult for computer vision to accurately extract head feature information, resulting in failure or inaccuracy of posture estimation; the IMU is not affected by light and occlusion, and can provide approximate movement information of the head in these cases, helping the system to maintain the estimation of the head posture, or providing an initial posture estimation value for the computer vision after it resumes normal operation.

[0020] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flow chart of a method for human factors intelligent posture estimation based on multimodal data provided by an embodiment of the present invention;

[0022] Figure 2 A flow chart of a method for obtaining visual head posture estimation data provided by an embodiment of the present invention;

[0023] Figure 3 A block diagram of a human factors intelligent posture estimation device based on multimodal data provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0025] High-precision head posture estimation plays an important role in improving the human-computer interaction experience, accurately evaluating the effects of health treatments, improving the accuracy of expression recognition, and accurately estimating people's behavioral intentions. Therefore, how to improve the accuracy of head posture estimation has become a focus of attention.

[0026] The inventors found in the process of implementing the present invention that the head pose estimation algorithm (Head Pose Perspective-n-Points, HeadPosePnP) based on feature point detection is used. The algorithm relies on accurate facial feature point detection. In actual scenes, if the image quality is poor, such as blur, noise or uneven illumination, the detection accuracy of facial feature points will be affected; in low-light environments, some feature points of the face may be difficult to be accurately identified, or in the case of motion blur, the position of the feature points may be deviated; when a person wears a hat, glasses or the face is blocked by other objects, the feature point information of the blocked area is missing, which will lead to a reduction in the number of feature points of the feature recognition algorithm, thereby affecting the accuracy of head pose estimation; when the head rotation range is too large and the camera cannot capture all facial features, the algorithm will fail. Another related technology uses a deep convolution-based algorithm, which has a high model complexity and cannot run on low-end embedded devices. In addition, due to individual differences, different people have different head shapes, sizes, facial features, etc., and these individual differences may affect the estimation of head pose.

[0027] To this end, an embodiment of the present invention provides a human factors intelligent posture estimation method based on multimodal data, which adopts a combination of IMU (Inertial Measurement Unit) and computer vision, uses computer vision information to correct the posture estimation result of the IMU, and fuses the two data to achieve regular calibration of the IMU zero bias to eliminate the accumulated error caused by the long-term operation of the IMU; alleviates the problem of being unable to perform head posture estimation caused by occlusion and lighting problems in computer vision; can effectively reduce the computational complexity of computer vision and reduce dependence on image quality.

[0028] Figure 1 A flow chart of a method for human factors intelligent posture estimation based on multimodal data provided by an embodiment of the present invention.

[0029] like Figure 1 As shown, the human factors intelligent posture estimation method 100 based on multimodal data includes steps S110 to S130.

[0030] Step S110, obtaining current IMU head posture estimation data and current visual head posture estimation data.

[0031] Exemplarily, IMU, i.e., inertial measurement unit, uses inertial sensors, utilizes integrated three-axis accelerometer, three-axis gyroscope, and three-axis magnetometer, and uses inertial navigation posture estimation algorithm to estimate the head posture to obtain the current IMU head posture estimation data. Visual head posture estimation data, for example, includes the head posture obtained by recognizing the acquired video or image. Head posture estimation data, for example, includes the position, acceleration, angular velocity, etc. of the head.

[0032] Step S120, fusing the current IMU head posture estimation data and the current visual head posture estimation data to obtain fused head posture estimation data.

[0033] Exemplarily, the acquired IMU head posture estimation data and visual head posture estimation data are processed according to certain calculation rules, such as adding or averaging the two, or modifying one based on the other, to obtain fused head posture estimation data.

[0034] Step S130, replacing the current IMU head posture estimation data with the fused head posture data, so as to continue estimating based on the fused head posture data to obtain subsequent IMU head posture estimation data.

[0035] For example, the IMU head posture estimation data at each moment can be obtained by long-term integration calculation based on the IMU unit. The IMU has factors such as gyro bias, sensor drift, temperature noise, and magnetic field noise. As time goes by, the cumulative error will gradually increase. In order to reduce the accumulated error and improve the estimation accuracy of the IMU, the obtained fused head posture data replaces the current IMU head posture estimation data, and the estimation is continued based on the replaced IMU head posture estimation data to improve the estimation accuracy of the IMU.

[0036] In the above embodiment, a combination of IMU head posture estimation data and visual head posture estimation data is adopted, and the posture estimation result of the IMU is corrected using computer vision information. The two are fused to perform data, thereby achieving regular calibration of the IMU zero bias and eliminating the cumulative error caused by the long-term operation of the IMU; the problem of being unable to perform head posture estimation due to occlusion and lighting problems of computer vision when only visual estimation is used is alleviated; and visual data can be collected only when data fusion is required, thereby reducing the frequency of collecting visual data, and effectively reducing the computational complexity of computer vision. Compared with directly using visual head posture estimation data as the final result, the embodiment of the present application is based on visual head posture estimation data and IMU head posture estimation data, which reduces the dependence on image quality and improves the accuracy of head posture estimation.

[0037] In another example, get the current IMU head pose estimation data Including: Get head posture change data Δq k+1 and head posture error data δq k+1 / k At least one of; based on the head posture change data Δq k+1 and head posture error data δq k+1 / k At least one of the above updates the IMU head posture estimation data at the previous moment. Get the current IMU head posture estimation data

[0038] Exemplarily, the following will describe obtaining the head posture change data Δq k+1 process.

[0039] For example, including the three-axis acceleration a m 、Three-axis angular velocity ω m , three-axis magnetic data m m , for a m and m m Do normalization, that is, a m The modulus length is 1, m m The modulus of is 1.

[0040] According to the gyroscope measurement model, the gyroscope state is expressed as shown in formula (1):

[0041] ω m =ω+b+n (1)

[0042] ω=ω m -bn(1")

[0043] In formula (1), ω m represents the angular velocity measurement value, ω represents the true value of the angular velocity, b represents the gyro bias, and n represents the gyro noise. Assuming that the gyro noise is Gaussian white noise, the above parameters satisfy the following formula:

[0044]

[0045]

[0046] E[n(t+Δt)n(t) T ]=N n δ(Δt) (4)

[0047]

[0048]

[0049]

[0050] E[n b (t+Δt)nb (t) T ]=N b δ(Δt) (8)

[0051]

[0052] In the above formula, represents the expected value of the angular velocity measurement value, that is, the nominal value, and formula (2) represents whether the expectation of the measurement value is obtained or the measurement value itself; represents the expected gyro noise, and formula (3) represents the expected gyro noise is the zero vector; δ(Δt) is the impulse function, N n is the gyro noise covariance matrix, and formula (4) represents the correlation between the elements of the gyro noise vector at different times; is the noise variance, and formula (5) represents the definition of the noise covariance matrix; Represents the first-order derivative of the gyro bias, n b represents the wandering noise, and formula (6) indicates that the gyro bias is non-static and is simulated as a random walk process; represents the expectation of the wandering noise, n b It conforms to the Gaussian distribution and satisfies formula (7) and formula (8). Formula (9) is the definition of the zero-biased walk covariance matrix, where is the variance of the wandering noise; E[*] represents the expectation of *.

[0053] By finding the expectation on both sides of formula (1), we can get formula (10):

[0054]

[0055] Combining formula (2) we can get formula (11) and (11'):

[0056]

[0057]

[0058] In formulas (11) and (11'), Represents the expected value of the true value of the angular velocity (i.e. the nominal value).

[0059] From the above content, we can see that the state of IMU can be defined by two parts, one is the expected value of angular velocity measurement, and the other is the expected value of gyro bias. For the convenience of calculation, angular velocity can be represented by attitude quaternion. Therefore, the state of IMU is defined by two parts: nominal attitude quaternion and gyro bias nominal value

[0060] According to formula (7) and formula (11), we can further obtain formula (12) and formula (13):

[0061]

[0062]

[0063] Formula (12) represents the expected gyro bias at the current moment: The expectation of the gyro bias at the last moment extrapolated Equal; Formula (13) represents the expected gyro angular velocity at the current moment Equal to the current measurement minus the expected gyro bias at the current moment

[0064] About obtaining head posture change data Δq k+1 , first, we can obtain the angular velocity measurement value ω at the previous moment k 、The current angular velocity measurement value ω k+1 And the nominal value of the gyro zero bias at the previous moment Based on the angular velocity measurement value ω at the previous moment k 、The angular velocity measurement value ω at the current moment k+1 , the nominal value of the gyro zero bias at the last moment And the time difference Δt between the previous moment and the current moment, the attitude angle change Δθ is obtained k+1 , as shown in formula (14). Then, based on the attitude angle change Δθ k+1 , get the head posture change data Δq k+1 , as shown in formula (15).

[0065] The nominal state extrapolation equation is shown in formula (14):

[0066]

[0067] In formula (14), ω k Represents the angular velocity measured by the IMU at the last moment, ω k+1 Indicates the angular velocity measured by the IMU at the current moment, represents the gyro zero bias at the last moment, Δt represents the time interval between two measurements, and Δθ k+1 It represents the attitude change in the sensor coordinate system within the time interval Δt, that is, the angle change. It can be understood that the attitude angle change Δθ is obtained by the optimized nominal state extrapolation equation k+1 , while ensuring the calculation accuracy, it simplifies the calculation and improves the calculation speed.

[0068] Δθ k+1 The corresponding posture quaternion (head posture change data Δq k+1) is shown in formula (15):

[0069]

[0070] Alternatively, we can use the Δq obtained from formula (15) k+1 Normalize to get the final local change attitude quaternion Δq k+1 , that is, the head posture change data. Get the head posture change data Δq k+1 Afterwards, we can calculate the head posture change data Δq based on the head posture change data Δq. k+1 The nominal attitude quaternion representing the time k is updated for the first time. For details, please refer to the following formula (42).

[0071] In another example, obtaining head posture error data includes:

[0072] (a) Based on the Kalman update method, according to the error state data at the previous moment (such as δθ k / k , Δb k / k ) to obtain the error state data (δθ k+1 / k , Δb k+1 / k );

[0073] (b) Based on the error state data δθ at the current moment k+1 / k , Δb k+1 / k , get the head posture error data δq k+1 / k Among them, δθ k / k Indicates the attitude rotation error at the previous moment, Δb k / k Indicates the gyro bias error at the previous moment, δθ k+1 / k Represents the attitude rotation error at the current moment, Δb k+1 / k Indicates the gyro bias error at the current moment. The specific process is described below.

[0074] Exemplarily, (a) based on the Kalman update method, the error state data at the current moment is obtained according to the error state data at the previous moment, including:

[0075] (a.1) Construct the error state space equation based on the error state variable, gyro noise and wander noise;

[0076] (a.2) Based on the error state space equation, gyro noise and wandering noise, the error state covariance matrix is ​​obtained;

[0077] (a.3) Based on the observed value and the error state covariance matrix, the error state data of the previous moment is updated to obtain the error state data of the current moment.

[0078] For example, during the measurement process, there is an error in the attitude angle. The attitude rotation error δθ is used to represent the attitude angle error, which corresponds to a small change in the rotation angle around each coordinate axis (including roll, pitch, and yaw axes). Substituting the above formula (11) into formula (1) yields the following formula:

[0079]

[0080]

[0081] in, Δb represents the gyro bias error.

[0082] According to formula (6), formula (7) and We can get formula (18):

[0083]

[0084]

[0085] The differential equation of the attitude rotation error δθ is expressed as formula (19):

[0086]

[0087] In formula (19), the subscript “x” represents an antisymmetric matrix.

[0088] Regarding (a.1) based on the error state variable x, gyro noise n and wandering noise n b , construct the error state space equation, including:

[0089] Determine the error state variable x, as shown in formula (20):

[0090]

[0091] Determine the error state space expression, as shown in formula (21):

[0092]

[0093] Formula (21) can be expressed as formula (22), which is the error state space equation expression (differential equation):

[0094]

[0095] in:

[0096] Regarding (a.2) based on the error state space equation, gyro noise n and wandering noise n b , we get the error state covariance matrix (Qc , Q d or P k+1 / k ),include:

[0097] (a.21) Based on the error state space equation, the state transfer matrix Φ(t+Δt,t) is obtained;

[0098] (a.22) Based on gyro noise n and wandering noise n b and the state transfer matrix Φ(t+Δt,t), we get the error state covariance matrix (Q c , Q d or P k+1 / k ).

[0099] For example, for (a.21), based on the error state space equation, the state transfer matrix Φ(t+Δt,t) is obtained, including: According to the error state space equation of the above formula (22), the state transfer matrix Φ(t+Δt,t) can be obtained as shown in formula (23):

[0100]

[0101] About (a.22) based on gyro noise n and wandering noise n b (n、n b Represented as n c ) and the state transfer matrix Φ(t+Δt,t), the error state covariance matrix (Q c , Q d or P k+1 / k )include:

[0102] According to the state space equation expression (differential equation) Calculate its state extrapolation equation, as shown in formula (24):

[0103]

[0104] During the calculation process, while ensuring accuracy, the second-order infinitesimal about time Δt is ignored to improve the calculation efficiency, where Φ(t+Δt,t) is the state transfer matrix from time t to time t+1, and the expression is:

[0105]

[0106] The block matrices C and D are given by formula (25) and formula (26):

[0107]

[0108]

[0109] In formula (25) and formula (26), u is the unit vector corresponding to δθ; C is the rotation matrix corresponding to the attitude rotation error δθ, which changes with the change of δθ, CC T is the identity matrix; for the state extrapolation equation: because The term is small, so in order to simplify the calculation, improve the calculation speed and reduce the calculation complexity while ensuring the calculation accuracy, this term is ignored. Therefore, the error state extrapolation equation can be expressed as formula (27):

[0110]

[0111] Therefore, we can get formula (28) and formula (29):

[0112]

[0113] Formula (28) represents the error state data (δθ k / k , Δb k / k ) to obtain the error state data (δθ k+1 / k , Δb k+1 / k ).

[0114]

[0115] Formula (29) represents the head posture error data obtained based on the error state data at the current moment in (b) above, δq k+1 / k That is the head posture error data.

[0116] At this time, there are still unknown parameters in formulas (28) and (29), and the iterative calculation below needs to be continued.

[0117] based on The continuous-time error state covariance matrix is ​​a 6×6 matrix, which can be expressed as formula (30):

[0118]

[0119] Among them, Q c With gyro noise n, wandering noise n b (n and n b Represented as n c ) related to (combined with a.22), N n represents the gyro noise covariance matrix, N b represents the zero-biased walk covariance matrix.

[0120] The discretized form of the error state covariance matrix is ​​shown in formula (31):

[0121]

[0122] Among them, Q d Related to Φ(t+Δt,t) (combined with a.22), while ensuring the calculation accuracy, in order to simplify the calculation, improve the calculation speed and reduce the calculation complexity, ignore the second-order infinitesimal about time Δt, and get formula (32):

[0123]

[0124] The error state covariance extrapolation equation is shown in formula (33):

[0125] P k+1 / k =ΦP k / k Φ T +Q d (33)

[0126] The error state covariance matrix can be Q c , Q d or P k+1 / k In the following, the difference state covariance matrix is ​​P k+1 / k For example.

[0127] Regarding (a.3), based on the observed value and the error state covariance matrix, the error state data at the previous moment is updated to obtain the error state data at the current moment, including:

[0128] (a.31) Based on the observed quantity, the observation matrix, the measurement covariance matrix of the observed quantity and the residual matrix are obtained;

[0129] (a.32) Based on the error state covariance matrix, observation matrix, measurement covariance matrix, residual matrix and Kalman gain, the error state data of the previous moment is updated to obtain the error state data of the current moment, wherein the Kalman gain is associated with the error state covariance matrix, observation matrix and measurement covariance matrix.

[0130] First, for (a.31), based on the observed quantity, the observation matrix, the measurement covariance matrix of the observed quantity and the residual matrix are obtained, including:

[0131] Update the error state data of the previous moment (i.e., Kalman update), which involves the state equation and observation equation. The state quantity is The input is the angular velocity measurement ω m , the observed quantities are a and m, a represents acceleration and m represents magnetic force.

[0132] The calculation of Kalman gain is shown in formula (34):

[0133] K k+1 =P k+1 / k HT (HP k+1 / k H T +R) -1 (34)

[0134] In the above formula, K k+1 / k is the Kalman gain at the current moment, P k+1 / k is the error state covariance matrix at the current moment, H is the observation matrix (having a mapping relationship with the observations a and m), R is the measurement covariance matrix (indicating the covariance of the measurements of the observations a and m), and R is initialized to a 6×6 diagonal matrix with a diagonal value of 10-3.

[0135] The calculation of the observation matrix is ​​shown in formula (35):

[0136]

[0137] In formula (35),

[0138]

[0139] Where m th Indicates the theoretical magnetic measurement value. According to the theoretical rules, the middle component is set to 0, m th .x means m th The x component of (same for y and z), m m represents the magnetic measurement value, where:

[0140]

[0141]

[0142] and represents the direction cosine matrix, the subscript or superscript b represents the IMU body coordinate system, and the subscript or superscript g represents the earth coordinate system, so formula (36) and formula (37) can be obtained:

[0143]

[0144]

[0145] because

[0146] Formula (37) can be written as formula (38):

[0147]

[0148] in

[0149]

[0150] The calculation of the residual matrix residual is shown in formula (39):

[0151]

[0152] In formula (39), the matrix is ​​a 6x1 matrix.

[0153] Next, for the error state covariance matrix P in (a.32) k+1 / k , the observation matrix H, the measurement covariance matrix R, the residual matrix residual and the Kalman gain K k+1 , for the error state data (δθ k / k , Δb k / k ) is updated to obtain the error state data (δθ k+1 / k , Δb k+1 / k ), including: performing Kalman update on the error state variable and the error state covariance matrix to obtain formula (40) and formula (41):

[0154] x k+1 / k+1 =x k+1 / k +K k+1 residual (40)

[0155]

[0156] Among them, the error state data at the current moment

[0157] In the above embodiment, based on the Kalman update method, the error state data at the current moment is obtained according to the error state data at the previous moment; based on the error state data at the current moment, the head posture error data is obtained, thereby improving the accuracy of head posture estimation.

[0158] The head posture change data Δq is obtained through the above calculation k+1 and head posture error data δq k+1 / k Afterwards, based on the head posture change data Δq k+1 , update the IMU head posture estimation data of the previous moment Get the IMU head posture estimation data after the first update Based on the head posture error data δq k+1 / k , Update the IMU head posture estimation data after the first update Get the current IMU head posture estimation data

[0159] For example, the head posture change data Δq obtained above k+1 and head posture error data δq k+1 / k , the head posture estimation data of the previous moment You can get the current IMU head posture estimation data

[0160] Based on the head posture change data Δq k+1 right Perform the first update and get As shown in formula (42):

[0161]

[0162] In formula (42), represents the nominal attitude quaternion at time k+1, represents the nominal attitude quaternion at time k; (Δb represents the gyro bias error), Also called the nominal angular velocity error, "initialize" means initialization.

[0163] Based on the error state variable at the current moment The nominal state is updated for the second time, as shown in formula (42): where

[0164]

[0165] After the second update That is the current IMU head posture estimation data.

[0166] In the above embodiment, based on the head posture change data, the IMU head posture estimation data of the previous moment is updated to obtain the IMU head posture estimation data after the first update; based on the head posture error data, the IMU head posture estimation data after the first update is updated for the second time to obtain the current IMU head posture estimation data. Through the two updates, the influence of the head posture change over time and the head posture error on the head posture at the next moment is comprehensively considered, thereby improving the estimation accuracy of the IMU.

[0167] Figure 2 A flow chart of a method for obtaining visual head posture estimation data provided in an embodiment of the present invention.

[0168] like Figure 2 As shown, the method 200 for obtaining visual head posture estimation data includes steps S210 to S230.

[0169] Step S210, obtaining current visual head posture estimation data, including: performing face detection and feature extraction on the head image data to obtain two-dimensional image feature points.

[0170] Exemplarily, data collection and preprocessing are first performed: a camera is used to collect head image data in different scenes, including images under different angles, lighting conditions, expression changes, etc. Ensure that the collected images are diverse to improve the generalization ability of the algorithm. The collected images are preprocessed, including denoising, contrast enhancement, brightness adjustment, denoising using median filtering and Gaussian filtering, and enhancing image contrast using histogram equalization. Secondly, face detection and head feature extraction are performed: an efficient face detection algorithm is used to perform face detection on the input image. The face detection algorithm can quickly and accurately detect the face area in the image, providing a basis for subsequent head posture estimation. Finally, key point detection is performed: the key point detection algorithm and feature distribution characteristics are used to locate and detect key parts such as eyes, nose, and mouth to obtain accurate feature point positions in the two-dimensional pixel plane.

[0171] Step S220, converting the two-dimensional image feature points into three-dimensional space feature points.

[0172] Exemplarily, a posture estimation algorithm is used to solve the problem, a linear equation is established based on known two-dimensional projection points and three-dimensional space points (three-dimensional camera coordinate system), and the posture matrix of the camera is obtained by solving the linear equation.

[0173] Step S230, obtaining current visual head posture estimation data based on the three-dimensional space feature points.

[0174] Exemplarily, the camera's posture matrix is ​​used to perform coordinate transformation on two-dimensional projection points to obtain three-dimensional space points, and a visual head posture matrix is ​​obtained based on the three-dimensional space points. The visual head posture matrix can be converted into posture Euler angles.

[0175] Next, the current IMU head posture estimation data and the current visual head posture estimation data are fused to obtain fused head posture estimation data, including: determining the error between the current IMU head posture estimation data and the current visual head posture estimation data; updating the current IMU head posture estimation data based on the error, and determining the updated current IMU head posture estimation data as the fused head posture estimation data.

[0176] Exemplarily, based on the obtained current IMU head posture estimation data and the current visual head posture estimation data, a prediction error matrix is ​​obtained by calculating the error between the two.

[0177] For example, suppose at a certain moment, the Euler angle corresponding to the current IMU head posture measured by the IMU is: Ω = [r i pi y i ] T (The results of IMU measurement The Euler angle corresponding to the current visual head posture predicted by the image is: Φ = [r p p p y p ] T , prediction error Α=[r i -r p p i -p p y i -y p ] T .

[0178] The current IMU head posture estimation data is updated based on the error, including: determining an error dynamic coefficient based on the error; obtaining an update amount based on the error dynamic coefficient and the error; and subtracting the update amount from the current IMU head posture estimation data to obtain updated current IMU head posture estimation data.

[0179] For example, according to the prediction error A, the error dynamic coefficient sigmoid(Α) can be obtained, where sigmoid is a nonlinear activation function, and the calculation method is shown in formula (43):

[0180]

[0181] Where k is a real number and the value of sigmoid(x) is between 0 and 1.

[0182] Based on the error dynamic coefficient and the error, the update amount sigmoid(Α)×Α is obtained; the current IMU head posture estimation data is subtracted from the update amount to obtain the updated current IMU head posture estimation data, as shown in formula (44):

[0183]

[0184] Among them, T is the final prediction result (corrected result of Ω), that is, T is the updated current IMU head posture estimation data.

[0185] In the above embodiment, the IMU head posture estimation data is corrected based on the nonlinear activation function, so that the IMU zero bias is regularly calibrated to eliminate the accumulated error caused by the long-term operation of the IMU, thereby improving the precision and accuracy of the head posture estimation.

[0186] The human factors intelligent attitude estimation method based on multimodal data proposed in this application uses computer vision information to correct the attitude estimation results of IMU. The data of the two are fused to achieve regular calibration of IMU zero bias, thereby eliminating the cumulative error caused by long-term operation of IMU, and improving the accuracy and reliability of attitude estimation: IMU can provide high-frequency real-time motion data, which is not affected by environmental factors such as light and occlusion, but there are problems of error accumulation and zero bias; computer vision can provide high-precision attitude estimation in specific scenarios based on image information, but it is easily affected by the environment; after the two are combined, the data of IMU can provide auxiliary information when computer vision is affected, and the results of computer vision can also correct the errors of IMU to a certain extent, and improve the accuracy and reliability of attitude estimation through data fusion. Enhanced adaptability to complex environments: Computer vision is difficult to accurately extract head feature information when the light is too dark, too bright or there is occlusion, resulting in failure or inaccuracy of attitude estimation; IMU is not affected by light and occlusion, and can provide approximate movement information of the head in these cases to help the system maintain the estimation of the head posture, or provide initial attitude estimation values ​​for computer vision after it resumes normal operation. Reduced computational complexity and resource requirements of the system: The present invention uses IMU and visual sensors, combined with Kalman filter algorithm, so visual detection can be updated at a lower frequency, thereby reducing the requirements for hardware resources. Reduced reliance on image quality: When using computer vision alone for head posture estimation, high image quality requirements are required, and high-resolution and clear images are required to obtain accurate results; after combining with IMU, the reliance on image quality can be reduced to a certain extent.

[0187] Figure 3 A block diagram of a human factors intelligent posture estimation device based on multimodal data provided by another embodiment of the present invention.

[0188] The embodiment of the present invention provides a human factors intelligent posture estimation device 300 based on multimodal data, see Figure 3 The human factors intelligent posture estimation device 300 based on multimodal data includes: an acquisition module 310, a fusion module 320, and an estimation module 330.

[0189] Exemplarily, the acquisition module 310 is used to acquire current IMU head posture estimation data and current visual head posture estimation data.

[0190] Exemplarily, the fusion module 320 is used to fuse the current IMU head posture estimation data and the current visual head posture estimation data to obtain fused head posture estimation data;

[0191] Exemplarily, the estimation module 330 is used to replace the current IMU head posture estimation data with the fused head posture data, so as to continue to estimate based on the fused head posture data to obtain subsequent IMU head posture estimation data.

[0192] It can be understood that for the specific description of the human factors intelligent posture estimation device 300 based on multimodal data, reference can be made to the description of the human factors intelligent posture estimation method based on multimodal data above, which will not be repeated here.

[0193] An embodiment of the present invention provides an edge computing device, which can execute the above-mentioned human factors intelligent posture estimation method based on multimodal data. Through edge computing, there is no need to send data to a remote data center or cloud for processing, which can reduce data transmission time and network delays caused by data transmission, and can also improve data security.

[0194] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method of any one of the above embodiments are implemented.

[0195] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0196] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0197] In the description of the present invention, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0198] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0199] In addition, the terms "first", "second", etc. used in the embodiments of the present invention are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present invention can explicitly or implicitly indicate that the embodiment includes at least one of the features. In the description of the present invention, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0200] In the present invention, unless otherwise clearly specified or limited in the embodiments, the terms "installed", "connected", "connected" and "fixed" etc. in the embodiments should be understood in a broad sense. For example, the connection can be a fixed connection, a detachable connection, or an integrated connection. It can be understood that it can also be a mechanical connection, an electrical connection, etc.; of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements, or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific implementation situation.

[0201] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0202] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A human factors intelligent posture estimation method based on multimodal data, characterized in that: The method comprises: Get the current IMU head pose estimation data and the current visual head pose estimation data; Fusing the current IMU head posture estimation data and the current visual head posture estimation data to obtain fused head posture estimation data; The fused head posture data replaces the current IMU head posture estimation data, so as to continue to estimate based on the fused head posture data to obtain subsequent IMU head posture estimation data.

2. The method according to claim 1, characterized in that The fusing the current IMU head posture estimation data and the current visual head posture estimation data to obtain fused head posture estimation data includes: Determining an error between the current IMU head pose estimation data and the current visual head pose estimation data; The current IMU head posture estimation data is updated based on the error, and the updated current IMU head posture estimation data is determined as the fused head posture estimation data.

3. The method according to claim 2, characterized in that The updating of the current IMU head posture estimation data based on the error includes: Based on the error, determining an error dynamic coefficient; Obtaining an update amount based on the error dynamic coefficient and the error; The current IMU head posture estimation data is subtracted from the update amount to obtain updated current IMU head posture estimation data.

4. The method according to claim 1, characterized in that Obtaining the current IMU head posture estimation data, including: Acquire at least one of head posture change data and head posture error data; Based on at least one of the head posture change data and the head posture error data, the IMU head posture estimation data at the previous moment is updated to obtain the current IMU head posture estimation data.

5. The method according to claim 4, characterized in that Based on the head posture change data and the head posture error data, the IMU head posture estimation data at the previous moment is updated to obtain the current IMU head posture estimation data, including: Based on the head posture change data, updating the IMU head posture estimation data at the previous moment to obtain the IMU head posture estimation data after the first update; Based on the head posture error data, the IMU head posture estimation data after the first update is updated to obtain current IMU head posture estimation data.

6. The method according to claim 4, characterized in that Acquiring the head posture change data, including: Get the angular velocity measurement value at the previous moment, the angular velocity measurement value at the current moment, and the gyro zero bias nominal value at the previous moment; Based on the angular velocity measurement value at the previous moment, the angular velocity measurement value at the current moment, the gyro zero bias nominal value at the previous moment, and the time difference between the previous moment and the current moment, the attitude angle change is obtained; Based on the posture angle change, the head posture change data is obtained.

7. The method according to claim 5, characterized in that Acquiring the head posture error data, including: Based on the Kalman update method, the error state data at the current moment is obtained according to the error state data at the previous moment; Based on the error state data at the current moment, the head posture error data is obtained.

8. The method according to claim 7, characterized in that The Kalman-based update method obtains the error state data at the current moment according to the error state data at the previous moment, including: Based on the error state variables, gyro noise and wandering noise, the error state space equation is constructed; Based on the error state space equation, the gyro noise and the wandering noise, an error state covariance matrix is ​​obtained; Based on the observed value and the error state covariance matrix, the error state data at the previous moment is updated to obtain the error state data at the current moment.

9. The method according to claim 8, characterized in that The error state covariance matrix is ​​obtained based on the error state space equation, the gyro noise and the wandering noise, including: Based on the error state space equation, a state transfer matrix is ​​obtained; An error state covariance matrix is ​​obtained based on the gyro noise, the wandering noise and the state transfer matrix.

10. The method according to claim 8, characterized in that The updating of the error state data at the previous moment based on the observed value and the error state covariance matrix to obtain the error state data at the current moment includes: Based on the observed quantities, the observation matrix, the measurement covariance matrix of the observed quantities and the residual matrix are obtained; Based on the error state covariance matrix, the observation matrix, the measurement covariance matrix, the residual matrix and the Kalman gain, the error state data at the previous moment is updated to obtain the error state data at the current moment, The Kalman gain is associated with the error state covariance matrix, the observation matrix, and the measurement covariance matrix.

11. The method according to any one of claims 1 to 10, characterized in that: Obtaining the current visual head posture estimation data, including: Perform face detection and feature extraction on the head image data to obtain two-dimensional image feature points; Convert two-dimensional image feature points into three-dimensional space feature points; Based on the three-dimensional spatial feature points, the current visual head posture estimation data is obtained.

12. An edge computing device, characterized in that: include: A processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the human factors intelligent posture estimation method based on multimodal data as described in any one of claims 1 to 11 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images