A low-cost mems pose estimation augmentation method based on temporal attention

Abstract

The application provides a low-cost MEMS attitude estimation enhancement method based on timing attention, comprising windowing, continuous sampling and normalization processing of timing sensor data of gyroscope and accelerometer output to construct training samples; the obtained training samples are input into a deep neural network based on timing attention to perform feature coding, feature enhancement and sequence modeling to estimate attitude roll angle and pitch angle; the attitude information output by the IMU is used as the true value, the mean square error loss is adopted and a regularization term is introduced as the overall loss function to train the network parameters. The application proposes a feature enhancement method based on timing attention, designs two attitude estimation network architectures based on sequence and post-processing. In addition, the application proposes a local receptive field mechanism when capturing timing attention. Compared with the capture of global timing attention, the local receptive field mechanism improves the attitude estimation accuracy.

Description

Technical Field

[0001] This invention belongs to the field of posture estimation technology, specifically relating to a low-cost MEMS posture estimation enhancement method based on temporal attention, which can be applied to fields such as robotics, embodied intelligence, gait recognition, and activity recognition. Background Technology

[0002] Attitude estimation has attracted widespread attention in robotics and embodied intelligence. Inertial measurement units (IMUs) based on gyroscopes and accelerometers can be used to achieve attitude estimation. Compared with traditional Kalman filtering and complementary filtering algorithms, deep learning can sense the motion state, thereby reducing the noise introduced into attitude estimation under different motion states.

[0003] In reality, different motion states have varying impacts on pose estimation. For example, during walking, the perturbations on the IMU device platform are relatively small, and the interference is relatively weak, while during running, the interference is more significant. Therefore, learning-based motion state perception is crucial for pose estimation.

[0004] Motion state is typically perceived using temporal sensor data. Traditional methods, such as Kalman filtering and complementary filtering, utilize the temporal outputs of gyroscopes and accelerometers. Deep learning methods tend to perceive features from sensor data within a temporal window, offering more robust performance. Convolutional Neural Networks (CNNs) can treat windowed sensor data as pseudo-images and extract features using computer vision techniques. To capture the contextual relationships between temporal signals, network architectures designed for temporal data, such as LSTM and GRU, are widely used and have achieved good results. Transformer-based network architectures can capture spatial and temporal attention through parallel computation. Effectively constructing training data samples is crucial to better drive deep learning models to fit the mechanism from sensor data to pose estimation.

[0005] Pose estimation is not merely a simple pattern recognition problem; estimation accuracy is equally important. Therefore, constructing a closed set of samples is simple and effective. However, noise can significantly impact the platform when subjected to external forces. Traditional filtering algorithms can combine multiple weak pose estimators to generate relatively accurate pose estimates. Accumulated errors and a lack of awareness of motion patterns reduce the robustness of filtering methods. Deep learning methods, due to their powerful fitting capabilities, are often used for certain parts of pose estimation. Furthermore, accurate pose estimation can be used for advanced tasks such as gait and activity recognition. Therefore, improving the performance of MEMS-based pose estimation is crucial.

[0006] Traditional pose estimation algorithms, such as Kalman filtering and complementary filtering, typically operate on time-series data generated by sensors. For neural network learning methods, sequential data can be processed using network layers such as LSTM and GRU. Sequential data can also be converted into pseudo-images and then used through convolutional neural network architectures to capture local features. Transformer-based attention mechanisms are commonly used to mine spatial context relationships in sequential data. SENet enhances the representational power of convolutional neural networks by constructing channel attention through squeezing and activation operations. Foreground object and structural attention, such as center and corner attention, are widely used in scene perception. Supervised attention mechanisms enable models to capture task-relevant information. In practice, attention mechanisms are often based on feature distances such as Euclidean distance and cosine similarity. The Transformer family utilizes feature multiplication to measure feature relevance.

[0007] The above technologies have the following drawbacks:

[0008] 1. Inexpensive inertial measurement units (IMUs) are not accurate enough in attitude estimation, while more accurate ones are relatively expensive.

[0009] 2. The prediction accuracy of the IMU is affected by the platform's dynamic movements. Drastic movements on the IMU platform, such as the back-and-forth swaying during walking, can produce significant attitude estimation errors. It's worth noting that expensive attitude sensors like the XSens series are more robust under complex motion conditions. However, the increased engineering costs are not negligible. In fact, different motion states have different effects on attitude estimation. For example, during walking, the disturbances on the IMU platform are relatively small, and the interference is relatively weak, while during running, the interference is more significant. Therefore, learning-based motion state perception is crucial for attitude estimation.

[0010] 3. Compared to capturing local and global features, CNNs are not sensitive to temporal relationships. Summary of the Invention

[0011] This invention aims to propose a low-cost MEMS attitude estimation enhancement method based on temporal attention, which establishes a mapping between raw gyroscope and accelerometer data and accurate attitude information, significantly improving the attitude estimation performance of low-cost MEMS IMUs and exhibiting high robustness in various motion modes.

[0012] The specific technical solution is as follows:

[0013] A low-cost MEMS pose estimation enhancement method based on temporal attention includes the following steps:

[0014] S1. Window, continuously sample, and normalize the time-series sensor data output by the gyroscope and accelerometer to construct training samples;

[0015] S2. Input the training samples obtained in step S1 into a deep neural network based on temporal attention to perform feature encoding, feature enhancement and sequence modeling, and estimate the attitude roll angle and pitch angle.

[0016] S3. Using the pose information output by the IMU as the true value, the mean squared error loss is adopted and a regularization term is introduced as the overall loss function to train the network parameters.

[0017] Furthermore, the specific method of step S1 is as follows:

[0018] Within time T, N consecutive discrete sampling sensor data points are sampled to construct a sample input S, as follows:

[0019] ;

[0020] in,( , , ) is the output of the accelerometer, ( , , () is the output of the gyroscope;

[0021] During batch build, N samples are randomly selected. local A series of continuous samples are input and normalized.

[0022] Furthermore, the deep neural network in step S2 includes a state-aware module based on a convolutional neural network, which encodes the sample input S through several linear layers to obtain temporal features; applies a self-attention mechanism to the temporal features to extract temporal attention context features; and combines the temporal attention context features with the temporal features using a sequence-based feature enhancement method or a post-processing feature enhancement method. Then, it uses LSTM or GRU to perform sequence modeling to estimate the attitude roll angle and pitch angle.

[0023] Furthermore, the sequence-based feature enhancement method is as follows:

[0024] Temporal characteristics within the window Through several convolutional layers Processing is performed to extract neighborhood contextual features. ;

[0025] Use and Multiple convolutional layers with non-shared weights deal with Then, it is input into a linear layer to encode Q, K, and V respectively, and attention features are generated based on the Transformer. ;

[0026] Finally, and Connect to enhance features, and obtain the enhanced features. :

[0027] ;

[0028] Original time-series encoded features Compared with enhanced features The input is connected and fed into a sequence network layer. The output of the last timing step is passed through a linear layer to predict attitude roll and pitch, as follows:

[0029] ;

[0030] In this context, MLP and S represent multilayer perceptron and sequence layer, respectively.

[0031] Furthermore, the post-processing feature enhancement method is as follows:

[0032] Neighborhood context features are processed by an LSTM This generates coarse features for pose estimation; simultaneously, attention features are processed using an independent LSTM branch with non-shared parameters. And generate context-adjusted features;

[0033] Feature addition is used to process the two types of features mentioned above, as follows:

[0034] .

[0035] Furthermore, the overall loss function in step S3 is:

[0036] ;

[0037] Among them, L A For mean square error loss, , The estimated attitude is obtained using gyroscopes and accelerometers. λ represents the output pose of the IMU; λ and n represent the regularization weights and the number of parameters, respectively.

[0038] Compared with the prior art, the present invention has at least the following beneficial effects:

[0039] 1. This invention proposes a feature enhancement method based on temporal attention, designs two pose estimation network architectures based on sequence and post-processing, and proposes a local receptive field mechanism when capturing temporal attention, which effectively improves the pose estimation accuracy.

[0040] 2. This invention achieves low mean absolute error (MAE) performance, improving roll and pitch angles by 0.05 and 0.02 respectively compared to the LSTM network architecture; and by 1.50 and 0.59 respectively compared to the complementary filtering algorithm; this invention improves the attitude estimation performance based on low-cost MEMS sensors, making it closer to high-precision attitude estimation products in terms of accuracy.

[0041] 3. This invention can effectively capture the trend of posture changes, exhibits excellent performance during periods of significant amplitude fluctuations, and can adapt quickly during transitions in motion states. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the gyroscope and accelerometer data in this invention;

[0043] Figure 2 This is a block diagram of the overall attitude estimation enhancement in this invention;

[0044] Figure 3 This is a schematic diagram of the feature enhancement module in this invention;

[0045] Figure 4 This is a schematic diagram of the temporal attention receptive field in this invention;

[0046] Figure 5 This is a schematic diagram of the sequence model in this invention;

[0047] Figure 6 This is a schematic diagram of the time-series attention post-processing model in this invention;

[0048] Figure 7 This is a schematic diagram illustrating the model training and validation loss in this invention;

[0049] Figure 8 This is a visualization of the roll attitude estimation in this invention;

[0050] Figure 9 This is a visualization of pitch attitude estimation in this invention;

[0051] Figure 10 This is a visualization of attitude estimation under different motion states in this invention;

[0052] Figure 11 This is a flowchart of the method of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on this invention by other researchers in the art are within the protection scope of this invention.

[0054] Example 1:

[0055] This embodiment proposes a low-cost MEMS attitude estimation enhancement method based on temporal attention, achieving enhanced attitude estimation for inexpensive MEMS devices and providing a low-cost solution for specific IMU application scenarios. Assume the outputs of the gyroscope and accelerometer are respectively ( , , )and( , , The specific steps are as follows:

[0056] Step S1: Windowing time-series sensor data

[0057] Instantaneous data from gyroscopes and accelerometers are insufficient for accurate attitude estimation because they lack robustness to motion. To address this issue, this embodiment employs windowed time-series sensor data, such as... Figure 1 As shown. Specifically, N discrete sampled sensor data points within time T are used to construct a sample input. On one hand, the time-series data input borrows from traditional filtering methods, making full use of sensor data. On the other hand, windowing the time-series data allows the model to better perceive the current motion state. A training sample is defined, and its algebraic expression is shown in the following equation:

[0058] ;

[0059] Multiple training samples are batch-processed and input into the pose estimation network proposed in this invention. During the batch construction process, this invention randomly samples N... local Continuous samples are used to ensure temporal continuity of the samples. Furthermore, the input samples are normalized to facilitate more effective learning of network parameters.

[0060] Step S2: Pose estimation based on temporal attention

[0061] This embodiment proposes a network architecture to improve the attitude estimation performance of traditional MEMS devices, such as... Figure 2As shown, a state-aware module based on a convolutional neural network was designed to perceive the motion state of the hardware platform. Self-attention to temporal data was considered during the motion state perception process. Furthermore, temporal network layers were also applied for pose estimation because they preserve the temporal characteristics of traditional filtering algorithms. Features enhanced by linear encoding and temporal attention were input into temporal network layers such as LSTM and GRU. Figure 2 As shown, sample S is first encoded through multiple linear layers. The 6-dimensional features are transformed into a high-dimensional feature space. The corresponding algebraic representation is shown in the figure, where E represents the feature encoder;

[0062] ;

[0063] When combining temporal attention context features with encoded features, this embodiment proposes a sequence-based feature enhancement and post-processing enhancement method, corresponding to different pose estimation architectures.

[0064] 1) Feature enhancement based on temporal attention: such as Figure 3 As shown, the timing features within the window First, multiple convolutional layers are used. Processing is performed to extract neighborhood contextual features. To capture the interaction between data from different time segments, a module based on the Transformer attention mechanism was employed. Specifically, it uses... Multiple convolutional layers with non-shared weights To handle Then, it is input into a linear layer to encode Q, K, and V respectively, and attention features are generated based on the Transformer. Finally, and Connect them to enhance features. As shown in the following formula, This indicates the enhanced features.

[0065] ;

[0066] Simultaneously, the impact of sensor data from adjacent time points on the current time should be appropriately considered. Therefore, this embodiment uses a local sensing field to calculate temporal attention. For example... Figure 4 As shown, a schematic diagram of the receptive field is depicted, where the current time step only focuses on the previous 2*N steps, the next 2*N steps, and its own attention.

[0067] 2) Sequence module for pose estimation: Augmented features This invention considers local context and temporal attention. It transforms the original temporal coding features... The enhanced features are concatenated and fed into traditional sequence network layers, such as LSTM and GRU, as... Figure 5 As shown. The sequence network layer ensures the temporal iteration of data within the window, consistent with traditional filtering methods. The output of the last temporal step is used to predict attitude roll and pitch through a linear layer. The algebraic representation is given by Equation 4, where MLP and S represent the multilayer perceptron and sequence layer, respectively.

[0068] ;

[0069] Furthermore, this embodiment proposes a post-processing architecture based on temporal attention, aiming to use two independent LSTM or GRU branches for coarse pose estimation and temporal attention-based feature processing, such as... Figure 6 As shown. Specifically, The data is processed by an LSTM to generate coarse features for pose estimation. Simultaneously, a separate LSTM branch with non-shared parameters is used for further processing. This generates context-adjusted features. Feature addition is used to process both types of features, as shown below:

[0070] ;

[0071] In real-world motion, roll and pitch values ​​are often not standardized, which hinders the performance improvement of the network prediction head. To better facilitate the fitting of network parameters, this embodiment uses normalized real labels.

[0072] Step S3: Training Loss Design and Training Method

[0073] This embodiment utilizes the output of inexpensive MEMS devices, such as the gyroscope and accelerometer in the MPU6050, as raw data to estimate attitude. Meanwhile, to supervise the learning of network parameters, the relatively expensive but accurate IMU pose output is used. Defined as the true value. Mean squared error loss is used for measurement. and The distance between them is shown in the following formula:

[0074] ;

[0075] Furthermore, to mitigate overfitting during model training, this embodiment introduces a weight minimization regularization term. The overall loss function is shown in the following equation, where λ and n represent the regularization weights and the number of parameters, respectively;

[0076] .

[0077] Example 2:

[0078] Based on the method provided in Example 1, the following configuration and operation are provided:

[0079] 1) Network parameter configuration: The raw sensor data input to the model consists of 6-dimensional data generated by gyroscopes and accelerometers. The feature encoding layer transforms the raw data into a 128-dimensional feature space; during feature enhancement, the contextual semantics are set to 128 dimensions, and the temporal attention is also set to 128 dimensions. This embodiment uses two Long Short-Term Memory (LSTM) layers as sequence network layers, ultimately generating 128-dimensional features; hyperbolic tangent (Tanh) is used as the activation function of the prediction head. Roll and pitch angles are normalized by dividing by 180 and 90, respectively.

[0080] 2) Data Preprocessing: Sensor data is first normalized, with the maximum value set to [30, 50, 60, 10, 10, 10] and the minimum value set to [-30, -50, -60, -10, -10, -10]; the actual roll and pitch angles are also normalized; training samples are randomly sampled from N... local Samples from consecutive time steps are bound together and extracted. In this embodiment, N local The window size for sequence samples is set to 256 in this invention, with a setting of 4.

[0081] 3) Training and Inference Details: The model was trained using an NVIDIA 4090D graphics card. To minimize computational costs, the raw sensor data was converted to 50Hz. An Adam optimizer with a learning rate of 0.01 was used to drive model optimization. The batch size was set to 512, and the number of training epochs was set to 50.

[0082] This embodiment combines the Adam optimizer and mean squared error loss to achieve optimal model parameters. During training, the proposed method's loss converges, validating the rationality of the network structure design. Figure 7 As shown, the performance on the validation set is slightly lower than that on the training set, mainly because the validation set does not include some training set samples. It is worth noting that the model still shows convergence on the validation set during training.

[0083] This invention achieves low mean absolute error (MAE), demonstrating its superiority. As shown in Table 1, compared to the Long Short-Term Memory (LSTM) network architecture, this invention improves roll and pitch by 0.05 and 0.02, respectively. Notably, compared to the complementary filtering algorithm, the proposed method achieves significant improvements in roll and pitch, reaching 1.50 and 0.59, respectively. Clearly, this invention improves the attitude estimation performance based on low-cost MEMS sensors, bringing its accuracy closer to that of high-precision attitude estimation products.

[0084] Table 1. Average absolute error performance of the present invention

[0085]

[0086] This invention visualizes roll and pitch estimation, such as Figure 8 and Figure 9 As shown. This invention effectively captures the trend of attitude change. Furthermore, it exhibits superior performance in attitude estimation during periods of significant amplitude fluctuations. It is worth noting that, as... Figure 9 As shown, the present invention can adapt quickly during motion state transitions.

[0087] As mentioned above, the dataset proposed in this invention covers various activity states, such as walking, running, climbing stairs, and cycling. Figure 10 As shown, the method proposed in this invention exhibits impressive performance in attitude estimation. The intensity of the platform's motion increases the difficulty of attitude estimation. Clearly, the mean absolute error of attitude estimation is lower when cycling compared to running, walking, and climbing stairs.

[0088] This invention analyzes complexity from two aspects: network parameters and runtime sequence. The hardware platform used is an NVIDIA 4090D GPU card. The model of this invention includes a temporal network layer. Therefore, the size of the perception range has a significant impact on inference time, as shown in Table 2. The decrease in inference efficiency due to an increased perception range is significant and cannot be ignored. The parameters of the model proposed in this invention are 0.83 M.

[0089] Table 2 Inference time performance under different window sizes

[0090]

[0091] The above description is only a preferred embodiment of the present invention. For those skilled in the art, several modifications and substitutions can be made without departing from the principles and concepts of the present invention, and all equivalent modifications and substitutions should fall within the protection scope of the present invention.

Claims

1. A low-cost MEMS pose estimation enhancement method based on temporal attention, characterized in that, Includes the following steps: S1. Window, continuously sample, and normalize the time-series sensor data output by the gyroscope and accelerometer to construct training samples; S2. Input the training samples obtained in step S1 into a deep neural network based on temporal attention to perform feature encoding, feature enhancement and sequence modeling, and estimate the attitude roll angle and pitch angle. S3. Using the pose information output by the IMU as the true value, the mean squared error loss is adopted and a regularization term is introduced as the overall loss function to train the network parameters.

2. The low-cost MEMS pose estimation enhancement method based on temporal attention according to claim 1, characterized in that, The specific method for step S1 is as follows: Within time T, N consecutive discrete sampling sensor data points are sampled to construct a sample input S, as follows: ； in,( , , ) is the output of the accelerometer, ( , , () is the output of the gyroscope; During batch build, N samples are randomly selected. local A series of continuous samples are input and normalized.

3. The low-cost MEMS pose estimation enhancement method based on temporal attention according to claim 1, characterized in that, The deep neural network in step S2 includes a state-aware module based on a convolutional neural network, which encodes the sample input S through several linear layers to obtain temporal features; A self-attention mechanism is applied to the temporal features to extract the temporal attention context features. Then, a sequence-based feature enhancement method or a post-processing feature enhancement method is used to combine the temporal attention context features with the temporal features. LSTM or GRU is used for sequence modeling to estimate the attitude roll and pitch angles.

4. The low-cost MEMS pose estimation enhancement method based on temporal attention according to claim 3, characterized in that, The sequence-based feature enhancement method is as follows: Temporal features within the window Through several convolutional layers Processing is performed to extract neighborhood contextual features. ; Use and Multiple convolutional layers with non-shared weights deal with Then, it is input into a linear layer to encode Q, K, and V respectively, and attention features are generated based on the Transformer. ; Finally, and Connect to enhance features, and obtain the enhanced features. : ； Original time-series encoded features Compared with enhanced features The input is connected and fed into a sequence network layer. The output of the last time series is passed through a linear layer to predict attitude roll and pitch, as follows: ； In this context, MLP and S represent multilayer perceptron and sequence layer, respectively.

5. The low-cost MEMS pose estimation enhancement method based on temporal attention according to claim 4, characterized in that, The post-processing feature enhancement method is as follows: Neighborhood context features are processed by an LSTM This generates coarse features for pose estimation; simultaneously, attention features are processed using an independent LSTM branch with non-shared parameters. And generate context-adjusted features; Feature addition is used to process the two types of features mentioned above, as follows: 。 6. The low-cost MEMS pose estimation enhancement method based on temporal attention according to claim 1, characterized in that, The overall loss function in step S3 is: ； Among them, L A For mean square error loss, , The estimated attitude is obtained using gyroscopes and accelerometers. λ represents the output pose of the IMU; λ and n represent the regularization weights and the number of parameters, respectively.

Images