Humanoid robot real-time speed estimation method and system based on time sequence feature fusion

Abstract

This invention discloses a real-time velocity estimation method and system for humanoid robots based on temporal feature fusion, relating to the field of robotics technology. The method includes: collecting proprioceptive sensory data from the robot's random walks; constructing an LSTM+MLP cascaded hybrid neural network architecture, training and optimizing it to obtain a hybrid neural network model; collecting raw proprioceptive sensory data streams through sensors to obtain standardized feature data; deploying the hybrid neural network model to an embedded main control platform for inference operations, outputting a three-dimensional linear velocity prediction value; performing trapezoidal numerical integration on the three-dimensional linear velocity prediction value to generate displacement odometry information and distributing it to the robot system bus. This invention employs an LSTM+MLP cascaded hybrid neural network architecture, enabling rapid capture of motion physical characteristics. Experimental data shows that the model can achieve a rapid decrease in mean square error and reach stable convergence within a short training period, reducing the migration cost and data dependency of the algorithm across different robot platforms.

Description

Technical Field

[0001] This invention relates to the field of robotics, specifically to a method and system for real-time velocity estimation of humanoid robots based on temporal feature fusion. Background Technology

[0002] In humanoid robots performing indoor navigation, autonomous obstacle avoidance, and environmental mapping tasks, high-frequency and stable odometry feedback is the core foundation for constructing a spatial coordinate system. Currently, mainstream state estimation schemes mainly rely on physically-based extended Kalman filters (EKF). These schemes typically derive the robot's real-time velocity and pose by fusing accelerometer data from an inertial measurement unit (IMU) with positive foot kinematics. However, in practical engineering applications, this technical approach faces the following shortcomings: 1. IMU linear acceleration data suffers from severe noise interference: For example, in the 2012 paper "Stateestimation for legged robots: Consistent fusion of leg kinematics and IMU" published by Bloesch et al., the algorithm uses IMU readings as a predictive model for the filter. It obtains the velocity by performing a first integration on the linear acceleration and the position by performing a second integration. However, during the walking process of humanoid robots, the rotation of the motors and the impact of the feet will generate severe high-frequency vibrations and "glitch" signals. Linear accelerometers are extremely sensitive to this kind of noise. Directly using it as the filter input for integration will cause drastic jumps and severe cumulative drift in the velocity solution, causing the navigation and positioning trajectory to fail rapidly. 2. Over-reliance on foot-end sensing hardware and maintenance challenges: High-precision state estimation usually requires accurate ground contact detection. Existing solutions, such as "Legged Robot State Estimation With Invariant Extended Kalman Filter Using Neural Measurement Network" published by Youm et al. in 2024, attempt to introduce neural measurement networks to predict contact probability. However, such models are mostly designed for quadruped robots with four support points. In bipedal robots, the entire dynamic load of the body must be borne by one foot in the single support phase. This makes the foot pressure sensor very susceptible to damage or severe zero-point drift due to instantaneous high pressure and strong impact. In addition, existing solutions still rely on physical ground contact switches to provide ground value labels during the training phase, which does not completely solve the problem of robust estimation of sensorless feet in bipedal architecture. 3. Over-reliance on expensive sensing hardware: Due to insufficient accuracy of the body odometry, existing technologies are often forced to introduce depth cameras or LiDAR for pose correction. However, such visual or laser algorithms have huge computational overhead and are easily affected by drastic changes in lighting, smoke, dynamic occlusion or feature loss, which increases system complexity and hardware cost. 4. There is a contradiction between the real-time performance of the algorithm and the computing power of the embedded system: In order to pursue estimation accuracy, some solutions adopt complex nonlinear optimization models. However, such models are difficult to maintain a real-time inference frequency of 200Hz on embedded main controllers with limited computing resources (such as RK3588). The resulting computational delay will lead to lag in navigation control feedback, which will seriously affect the robot's motion safety. 5. The model parameter tuning is redundant and the generalization ability is weak: Traditional filters involve a large number of covariance matrix parameter adjustments, which are highly dependent on specific robot models and gaits, and are difficult to adapt to different robot platforms quickly.

[0003] Based on this, we now provide a method and system for real-time velocity estimation of humanoid robots based on temporal feature fusion, which can eliminate the drawbacks of existing technical solutions. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time velocity estimation method and system for humanoid robots based on temporal feature fusion, in order to solve the problems faced by the technical solutions in the background art, such as severe noise interference in IMU linear acceleration data, excessive reliance on foot-end sensing hardware and high-cost sensing hardware, contradiction between algorithm real-time performance and embedded computing power, and redundant model parameter tuning and weak generalization ability.

[0005] To achieve the above objectives, the present invention provides the following technical solution: The real-time velocity estimation method for humanoid robots based on temporal feature fusion includes the following steps: Step S1: Construct a digital twin model of the humanoid robot in a physical simulation environment, collect the body sensory data during the robot's random walk motion, extract the true value of the world system linear velocity, and generate a motion dataset. Step S2: Construct an LSTM+MLP cascaded hybrid neural network architecture. Using the selected 49-dimensional ontology sensory data as input and the world system linear velocity true value as label, train and optimize the LSTM+MLP cascaded hybrid neural network architecture to obtain a hybrid neural network model. Step S3: During the actual operation of the humanoid robot, the raw data stream of the body's senses is collected in real time through several sensors inside the humanoid robot, and the raw data stream of the body's senses is synchronously arranged and time-series aligned according to the characteristic format of the 49-dimensional body sense data to obtain standardized feature data. Step S4: Deploy the optimized hybrid neural network model to the embedded main control platform, input the standardized feature data into the pre-trained model for inference operations, and output the predicted value of the three-dimensional linear velocity of the humanoid robot in the world coordinate system. Step S5: Perform trapezoidal numerical integration on the output three-dimensional linear velocity prediction value to generate continuous robot displacement odometer information and distribute it to the humanoid robot system bus.

[0006] Furthermore, the 49-dimensional body sensory data filtered in step S2 includes the 3-axis angular velocity, 4-axis attitude quaternion output by the IMU, and the angle and angular velocity data of the 21 joint motors of the humanoid robot. The three-axis acceleration data of the IMU is removed. The joint motors include 1 waist joint motor, 8 arm joint motors, and 12 leg joint motors.

[0007] Furthermore, in step S2, the LSTM+MLP cascaded hybrid neural network architecture includes a temporal feature extraction layer and a spatial computation and regression layer. The temporal feature extraction layer is a 2-layer LSTM network used to capture the inertial laws of humanoid robot motion from a 49-dimensional temporal tensor with a sequence length of 20. The spatial computation and regression layer is a 2-layer MLP network, with the last hidden state of the LSTM network as input, used to map high-dimensional temporal features to a 3-dimensional linear velocity space to achieve regression prediction of linear velocity in the world coordinate system.

[0008] Furthermore, in step S2, the LSTM+MLP cascaded hybrid neural network architecture uses the Optuna hyperparameter optimization software framework to determine the optimal hyperparameters, and introduces asymmetric mean squared error (Asymmetric MSE Loss) as the loss function during training. This loss function is the sum of the asymmetric losses of each component, and its calculation formula is as follows: ; in, For loss function, Using the indices of the three-dimensional coordinate axes, traverse... , and Three axes, The penalty coefficient is... For predicting the linear velocity vector, This is the true linear velocity vector.

[0009] Furthermore, in step S5, the trapezoidal integral method is used to perform trapezoidal numerical integration on the output three-dimensional linear velocity prediction value, which is used to calculate the three-axis displacement increment of the humanoid robot in the world coordinate system, for any axis in the world coordinate system. ,exist Displacement at any moment The calculation formula is expressed as: ; in, for The humanoid robot at time in the world coordinate system Cumulative axial displacement For the previous cycle At the time of Historical cumulative displacement in the axial direction for The output of the hybrid neural network model at time step 1 Predicted axial instantaneous linear velocity. For the previous cycle At the time of Historical linear velocity predictions output by a hybrid neural network model along the axial direction. The integration step size is the sampling period. This is the estimated average speed for the current period.

[0010] Furthermore, the random walk motion in step S1 includes forward, backward, lateral movement in any direction, rotation in place, and gait while rotating.

[0011] Furthermore, in step S3, the raw data stream of the body's senses is acquired in real time at a frequency of 500Hz through the LCM lightweight communication protocol, and the time sequence alignment of the multiple asynchronously arriving data streams is performed to ensure that the standardized feature data is compatible with the feature format of the motion dataset. The sensors include an IMU and angle sensors and speed sensors of each joint motor.

[0012] Furthermore, in step S4, the hybrid neural network model is deployed to the RK3588 embedded main control platform, and the standardized feature data is processed through a 20-frame time-series sliding window to output the humanoid robot's position in the world coordinate system in real time at a frequency of 200Hz. , , Three-axis velocity prediction values.

[0013] Furthermore, in step S5, continuous robot displacement odometry information is generated at a frequency of 200Hz, and the continuous robot displacement odometry information is distributed to the humanoid robot system bus through LCM messages or ROS 2 standard topics to achieve the docking effect with the upper-level navigation, obstacle avoidance, localization and mapping algorithms.

[0014] A real-time velocity estimation system for humanoid robots based on temporal feature fusion is used to execute a real-time velocity estimation method for humanoid robots based on temporal feature fusion, including: The offline motion data simulation and annotation module is used to build a digital twin model of a humanoid robot in a physical simulation environment, collect the body sensory data during the robot's random walk motion, extract the true value of the world system linear velocity, and generate a motion dataset. The neural network construction and model training module is used to build, train and optimize the LSTM+MLP cascaded hybrid neural network architecture to obtain a hybrid neural network model. The LSTM layer is used to capture the inertial laws of humanoid robot motion from continuous time series frames, and the MLP layer is used to perform nonlinear spatial mapping on the fused sensory features. The multi-source body data acquisition module is used to collect raw sensory data streams of the humanoid robot in real time through several sensors, and to synchronously arrange and time-series align them to obtain standardized feature data. The embedded model inference and velocity estimation module is used to deploy the hybrid neural network model to the embedded main control platform to complete inference and output the predicted value of the three-dimensional linear velocity of the humanoid robot in the world coordinate system. The real-time calculation and odometry generation module is used to perform trapezoidal numerical integration on the three-dimensional linear velocity prediction values ​​output by the hybrid neural network model and generate continuous robot displacement odometry information.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention sets up a closed-loop calculation system that relies on the internal sensors of the humanoid robot, without relying on external hardware such as vision cameras or lidar. This enables the humanoid robot to obtain stable speed feedback in flat terrain conditions with drastic changes in lighting or limited external sensors, thereby improving the system's environmental adaptability. 2. In feature selection, this invention actively eliminates linear velocity signals that are severely affected by motor vibration and foot impact. By capturing the temporal features of the LSTM layer, it reconstructs the motion inertia law from more stable instantaneous angular velocity and kinematic data, which alleviates the trajectory drift problem caused by the accumulation of acceleration signal noise in traditional solutions and improves the physical consistency of the odometer. 3. This invention adopts the LCM lightweight communication protocol, which reduces the budget, ensures that the speed feedback can be synchronized with the underlying control frequency, and reduces the feedback delay introduced by the state estimation stage. 4. This invention features an LSTM+MLP cascaded hybrid neural network architecture, which enables rapid capture of motion physical characteristics. Experimental data shows that the model can achieve a rapid decrease in mean square error and reach stable convergence within a short training period, reducing the migration cost and data dependence of the algorithm across different robot platforms. 5. By introducing an asymmetric mean square error function, this invention applies a double weight penalty to cases where the predicted value is higher than the true value. This design effectively corrects the "overestimation" phenomenon in velocity estimation, making the integrated odometer trajectory length closer to the actual physical displacement, and improving the reference value of odometer positioning data in navigation tasks. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0017] Figure 2 This is a schematic diagram of the LSTM+MLP cascaded hybrid neural network architecture of the present invention.

[0018] Figure 3 This is a schematic diagram of the system structure of the present invention.

[0019] Figure 4 This is a schematic diagram of the predicted trajectory structure when using the conventional symmetric loss function in existing technologies.

[0020] Figure 5 This is a schematic diagram showing the comparison of the simulation trajectory of the present invention.

[0021] Figure 6 This is a schematic diagram comparing the predicted coordinates and the actual coordinates of the three axes of the world coordinate system according to the present invention.

[0022] Figure 7 This is a trajectory prediction diagram of the actual machine during movement according to the present invention.

[0023] Figure 8 This is a trajectory prediction diagram for the real machine right-angle turn test of the present invention.

[0024] Figure labeling notes: Offline motion data simulation and annotation module 10, neural network construction and model training module 20, multi-source ontology data acquisition module 30, embedded model inference and velocity estimation module 40, real-time calculation and odometer generation module 50. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0026] Example 1 In this embodiment, as Figure 1 and Figure 2 As shown, this invention provides a real-time velocity estimation method for humanoid robots based on temporal feature fusion, specifically including the following steps: Step S1: Construct a digital twin model of the humanoid robot in a physical simulation environment, collect the body sensory data during the robot's random walk motion, extract the true value of the world system linear velocity, and generate a motion dataset. Specifically, this step uses the MuJoCo physics simulation environment to drive the humanoid robot to perform random walks. These random walks include forward movement, backward movement, lateral movement in any direction, stationary rotation, and gait involving rotation and walking. Data is sampled at a frequency of 500Hz, accumulating approximately 20 minutes of motion sequence data. The offline motion data simulation and annotation module 10 retains only the IMU's 3-axis angular velocity, 4-axis attitude quaternions, and the angles and angular velocities of 21 joint motors, totaling 49 features, when exporting data. The IMU's three-axis acceleration data is explicitly removed. The linear velocity of the humanoid robot's center of mass in the world coordinate system is recorded in real time, and the true value of the world coordinate system linear velocity is... As a label, it solves the problem of the difficulty in accurately labeling absolute speed in a real device environment; Step S2: Construct an LSTM+MLP cascaded hybrid neural network architecture. Using the selected 49-dimensional ontology sensory data as input and the world system linear velocity true value as label, train and optimize the LSTM+MLP cascaded hybrid neural network architecture to obtain a hybrid neural network model. Specifically, the filtered 49-dimensional proprioceptive data includes the 3-axis angular velocity output by the IMU ( ), 4-axis attitude quaternion ( ) and the angle data of the motors of the 21 joints of the humanoid robot ( ) and angular velocity data ( The three-axis acceleration data of the IMU were removed. The joint motors include 1 waist joint motor, 8 arm joint motors and 12 leg joint motors. The comparison table of the 21 joint motors of the humanoid robot is shown in Table 1. This invention actively discards the three-axis acceleration data of the IMU and improves the smoothness and anti-drift ability of the overall estimation by sacrificing the high noise feature. Table 1 - Comparison of Motors for 21 Joints of a Humanoid Robot

[0027] Step S3: During the actual operation of the humanoid robot, the raw data stream of the body's senses is collected in real time through several sensors inside the humanoid robot, and the raw data stream of the body's senses is synchronously arranged and time-series aligned according to the characteristic format of the 49-dimensional body sense data to obtain standardized feature data. Specifically, this step involves acquiring raw proprioceptive sensory data streams in real time at a frequency of 500Hz via the LCM lightweight communication protocol, and performing time-series alignment on multiple asynchronously arriving data streams to ensure that the standardized feature data is compatible with the feature format of the motion dataset, and to ensure that the input features are consistent with the input required by the model. The sensors include IMU and angle and speed sensors for each joint motor. Step S4: Deploy the optimized hybrid neural network model to the embedded main control platform, input the standardized feature data into the pre-trained model for inference operations, and output the predicted value of the humanoid robot's three-dimensional linear velocity in the world coordinate system. Specifically, this step deploys the hybrid neural network model (the trained weight file and model) to the RK3588 embedded main control platform. Standardized feature data is processed through a 20-frame (40ms) temporal sliding window. At each time step, after receiving the latest aligned 49-dimensional feature, it is pushed into the temporal sliding window and an inference is triggered. On the RK3588 embedded main control platform, the latency of a single inference is less than 5ms, and the humanoid robot's position in the world coordinate system is output in real time at a frequency of 200Hz. , , Predicted velocity values ​​for three axes; Step S5: Perform trapezoidal numerical integration on the output three-dimensional linear velocity prediction value to generate continuous robot displacement odometry information and distribute it to the humanoid robot system bus. Specifically, this step generates continuous robot displacement odometry information at a frequency of 200Hz, and distributes the continuous robot displacement odometry information to the humanoid robot system bus through LCM messages or ROS 2 standard topics (such as bridging nodes) to achieve the docking effect with the upper-level navigation, obstacle avoidance, localization and mapping algorithms.

[0028] In this embodiment, the LSTM+MLP cascaded hybrid neural network architecture in step S2 includes a temporal feature extraction layer and a spatial computation and regression layer. The temporal feature extraction layer is a two-layer LSTM network used to capture the inertial laws of humanoid robot motion from a 49-dimensional temporal tensor with a sequence length of 20. To prevent overfitting, the present invention sets a weight decay of 3.012e-05 to enhance the model's generalization and inference capabilities. The spatial computation and regression layer is a two-layer MLP network, with the last hidden state of the LSTM network as input. It is used to map high-dimensional temporal features to a 3-dimensional linear velocity space to achieve regression prediction of linear velocity in the world coordinate system. Figure 2 As shown, Figure 2 This is a schematic diagram of an LSTM+MLP cascaded hybrid neural network architecture. , and This represents the hidden state and memory state passed in the LSTM. Specifically, the LSTM+MLP cascaded hybrid neural network architecture has 1 to 3 LSTM layers, preferably 2 layers; LSTM hidden layer size is 64 to 512, preferably 160; MLP layers are 1 to 4, preferably 2 layers; MLP hidden layer size is 64 to 256, preferably 128; sequence length is 5 to 100, preferably 20; learning rate is 0.00001 to 0.01, preferably 0.0005; weight decay is 1e-6 to 1e-3, preferably 3.012e-05; dropout rate is 0.0 to 0.3, preferably 0.0; and batch size is 8 to 128, preferably 16. These parameters can be adjusted according to actual needs. The LSTM+MLP cascaded hybrid neural network architecture uses the Optuna hyperparameter optimization software framework to determine the optimal hyperparameters. To address the problem that predicted velocity values ​​are generally greater than the true values ​​(overestimation), this invention introduces an asymmetric mean squared error (MSE) loss function as the loss function during training. Let the predicted linear velocity vector be... The true linear velocity vector is The loss function is the sum of the asymmetric losses of each component, and its calculation formula is as follows: ; in, For loss function, Using the indices of the three-dimensional coordinate axes, traverse... , and Three axes, The penalty coefficient is... For predicting the linear velocity vector, Given the true linear velocity vector, considering that without a penalty coefficient, using traditional symmetric loss can easily lead to overestimation of velocity predictions and trajectory drift, this invention uses a penalty coefficient... Setting it to 2.0 means that when the model has an overestimation error, the gradient penalty is twice that in the underestimation case, thus forcing the model to output a more robust and conservative velocity estimate.

[0029] In this embodiment, to reduce the discretization error during the single-step integration process, the present invention employs the trapezoidal integration method in step S5 to perform trapezoidal numerical integration on the output three-dimensional linear velocity prediction value. This is used to calculate the three-axis displacement increments of the humanoid robot in the world coordinate system, and to calculate the displacement increments of any axis in the world coordinate system. ,exist Displacement at any moment The calculation formula is expressed as: ; in, for The humanoid robot at time in the world coordinate system Cumulative axial displacement (odometer position). For the previous cycle At the time of Historical cumulative displacement in the axial direction for The output of the hybrid neural network model at time step 1 Predicted axial instantaneous linear velocity. For the previous cycle At the time of Historical linear velocity predictions output by a hybrid neural network model along the axial direction. The integration step size is the sampling period. Since the odometer generates data at a frequency of 200Hz, therefore... , The trapezoidal integral is the average speed estimate for the current period. Compared to the rectangular integral (which directly takes the current value), the trapezoidal integral, by taking the average of two adjacent frames, can effectively compensate for the integral remainder during the linear change of speed, thus improving the accuracy of the odometer.

[0030] Example 2 Among them, such as Figure 3 As shown, the present invention also provides a real-time velocity estimation system for humanoid robots based on temporal feature fusion, used to execute the real-time velocity estimation method for humanoid robots based on temporal feature fusion shown in Example 1, including: The offline motion data simulation and annotation module 10 is used to build a digital twin model of a humanoid robot in a physical simulation environment, collect the body sensory data during the robot's random walking motion, extract the world system linear velocity true value corresponding to the body sensory data, and generate a motion dataset. This module generates a motion dataset with high generalization characteristics by simulating various random walking and rotation of the humanoid robot, providing a benchmark for supervised learning of subsequent neural networks, and solving the problem that the world system linear velocity is difficult to annotate accurately in real time in a real machine environment. This module is used to perform step S1 operation. The neural network construction and model training module 20 is used to construct, train, and optimize the LSTM+MLP cascaded hybrid neural network architecture to obtain a hybrid neural network model. The LSTM layer is used to capture the inertial laws of humanoid robot motion from continuous time series frames, and the MLP layer is used to perform nonlinear spatial mapping on the fused sensory features. The hybrid neural network model only takes the selected 49-dimensional ontological perception features as input to extract angular velocity, quaternion pose, joint angle, and joint velocity, and actively removes IMU linear acceleration signals. This avoids the problem of secondary trajectory drift caused by linear accelerometer deviation from the source of the architecture. In the training phase, the optimal hyperparameters determined by the hyperparameter optimization software framework Optuna are used for iteration, and an asymmetric mean square error (Asymmetric MSE Loss) with a penalty coefficient of 2.0 is introduced to focus on optimizing overestimation errors. Under the above operations, the model has extremely fast convergence characteristics and can be trained on a very small sample set, which is convenient for directly predicting the robot's three-dimensional linear velocity in the world coordinate system. This module is used to execute step S2. The multi-source body data acquisition module 30 is used to acquire the raw sensory data stream of the humanoid robot in real time through several sensors, and to synchronously arrange and time-align it to ensure consistency with the input features of the model and obtain standardized feature data. This module subscribes to angular velocity and quaternion pose data provided by the IMU and angle and angular velocity data provided by each motor joint in real time at a frequency of 500Hz through the LCM (Lightweight Communications and Marshalling) lightweight communication protocol. The LCM communication mechanism can eliminate the serialization overhead of traditional inter-process communication and ensure that the perception data can be read by the subsequent inference module with extremely low latency. This module is used to execute the operation of step S3. The embedded model inference and velocity estimation module 40 is used to deploy the hybrid neural network model to the embedded main control platform to complete inference and output the predicted value of the three-dimensional linear velocity of the humanoid robot in the world coordinate system. This module is responsible for deploying the trained neural network to the embedded main control platform (such as RK3588). After receiving the aligned 49-dimensional time sequence (window length of 20 frames), the hybrid neural network model can complete one inference within 5ms and output the predicted value of the three-dimensional linear velocity of the robot in the world coordinate system in real time. This module is used to execute the step S4 operation. The real-time calculation and odometry generation module 50 is used to perform trapezoidal numerical integration on the three-dimensional linear velocity prediction value output by the hybrid neural network model and generate continuous robot displacement odometry information. This module performs trapezoidal numerical integration on the prediction value output by the hybrid neural network model to generate continuous displacement odometry information at a frequency of 200Hz. This module also has a multi-protocol distribution function, which can send the generated odometry information to the humanoid robot system bus through LCM messages or ROS 2 standard topics (such as bridging nodes), thereby providing a stable spatial position reference for the upper-level navigation, obstacle avoidance, localization and mapping algorithms. This module is used to execute the operation in step S5. In this embodiment, the present invention combines the extraction of the robot's body sensory data, LSTM+MLP cascaded neural network inference, and embedded real-time deployment to construct a closed-loop odometry calculation system that does not rely on linear accelerometer data or external visual sensors, thereby achieving high-frequency and robust estimation of the linear velocity of the humanoid robot's world system in complex noise environments.

[0031] To further demonstrate the practical effect of the real-time velocity estimation method and system for humanoid robots based on temporal feature fusion in this invention, a visual representation of the real-time velocity estimation technology solution for humanoid robots based on temporal feature fusion is provided, such as... Figures 4-8 As shown; Figure 4 This is a schematic diagram of the predicted trajectory structure using the conventional symmetric loss function in existing technologies, with the penalty coefficient shown. =1, Figure 4 Figure a shows a comparison of the X-axis position changing over time. Figure 4 The middle figure (b) shows the Y-axis, displaying the trends of the true values ​​(solid black lines) and model predicted values ​​(dashed red lines) at the X-axis and Y-axis positions over time. This allows for a direct comparison of the fit between the predicted and true curves, and an assessment of the model's prediction accuracy in a single axis direction. According to the figure, the fluctuation trends of the predicted and true values ​​are basically consistent, indicating that the model can capture the positional change pattern well, but there is a certain degree of deviation. The X-axis position fluctuates in the range of -40m to 0m, while the Y-axis position gradually rises from 0m to approximately 80m. Figure 5 This is a schematic diagram comparing the simulated trajectories in the XY plane. The X and Y axis positions are mapped onto a two-dimensional plane to intuitively present the spatial overlap between the humanoid robot's actual movement trajectory and the model's predicted trajectory. The black solid line represents the true trajectory (total length 597.74m), and the red dashed line represents the model's predicted trajectory (total length 593.05m). Key indicators include the final drift of 0.747m and the maximum deviation of 1.683m, used to quantify the model's cumulative error and peak error. As can be seen from the image, the two trajectories are highly similar in shape, indicating that the model performs well in global path tracking, with minor deviations in local details. Figure 6 This is a diagram comparing the predicted coordinates and the actual coordinates on the three axes of the world coordinate system. Figure 6 Figure a shows a comparison of the X-axis position changing over time. Figure 6 Figure b shows a comparison of the Y-axis position changing over time. Figure 6 The middle graph (c) shows the change of the Z-axis position over time. Figure 1 Based on this, a Z-axis dimension is added to fully demonstrate the prediction effect of position in three-dimensional space, and the trends of the X and Y axes are also shown. Figure 1 Consistent with the addition of the Z-axis, the image shows that the true value is basically stable around 0.4m, while the predicted value fluctuates (0~1.1m). This indicates that the predictions of this invention on the X and Y axes are highly consistent with the true values, and the trend is also consistent on the Z-axis. This verifies the effectiveness of the model in multidimensional velocity estimation and can be used to analyze the differences in prediction performance across different dimensions. Figure 7 The image shows the predicted trajectory of the humanoid robot during actual movement. The blue line represents the humanoid robot's walking trajectory predicted by the hybrid neural network model of this invention. The green dots represent the starting point of the movement, and the red dots represent the ending point. According to the image, in the test of the real humanoid robot walking in a straight line for 9m, the predicted cumulative trajectory length is 8.86m, with a relative error of only 1.5%. The trajectory is highly linear, which proves the stability of this method after eliminating accelerometer interference. Figure 8 The image shows the trajectory prediction for a real humanoid robot right-angle turn test. The blue line represents the humanoid robot's walking trajectory predicted by the hybrid neural network model of this invention. According to the image, the turning angle predicted by this invention is approximately 90 degrees in the real humanoid robot right-angle turn test, which is highly similar to the actual rotation angle. according to Figure 7 and Figure 8 It can be seen that in the simulation data test set (not involved in model training and validation), the model predicted trajectory (dashed line) basically coincides with the true trajectory (solid line). In the simulation data test set, the length of the humanoid robot's random walk trajectory is 597.74m, the maximum deviation is only 1.68m, the error is only 0.28%, and the generalization is excellent. Due to the reasonable selection of features and architecture, the model exhibits extremely rapid convergence, achieving a two-order-of-magnitude reduction in loss within the first epoch. Within 10 epochs, the average velocity error across all three dimensions can be reduced to the level of 0.0x m / s. This demonstrates the extremely strong representational ability of the 49-dimensional feature combination and network architecture for the physical laws of humanoid robot motion. One epoch represents one training iteration of the hybrid neural network architecture using the entire training set data. Val Loss represents the loss value on the validation set, and MSE represents the mean squared error on the validation set. represent Mean absolute error of directional velocity represent The mean absolute error of directional velocity and performance data are shown in Table 2.

[0032] Table 2 - Model Performance Diagram

[0033] In summary, this invention constructs a real-time velocity estimation method and system for humanoid robots based on temporal feature fusion, realizing end-to-end closed-loop calculation from proprioceptive sensory data to odometry feedback in the world coordinate system. It can reconstruct motion inertia through a neural network model without relying entirely on linear velocity data, and has the following value: 1. By combining 49-dimensional features, the linear accelerometer signal, which is severely affected by high-frequency vibration interference, is actively eliminated. Utilizing the temporal long short-term memory capability of LSTM, the robot's motion inertia is directly reconstructed from angular velocity and body kinematic features. Experiments show that in a test of walking 9m in a straight line on real flat ground, the velocity integral trajectory length reaches 8.86m, with a cumulative error of only 1.5%. Compared with the traditional inertial state estimation scheme based on acceleration integral, this effectively avoids the problem of large trajectory drift caused by the accumulation of acceleration noise and improves the physical consistency of the odometer. 2. Learning can be completed with only 20 minutes of MuJoCo simulation sampling data. The model has extremely fast convergence characteristics and can reduce the three-axis velocity simulation error to the centimeter level within 10 epochs, shortening the adaptation cycle of the algorithm between different models. 3. Lightweight design of embedded main control platforms such as RK3588, combined with the LCM lightweight communication protocol, achieves excellent performance with a single inference latency of less than 5ms. The 200Hz high-frequency synchronous output ensures that the state estimation can meet the requirements of real-time obstacle avoidance and high dynamic control. By introducing the asymmetric mean square error function Asymmetric MSE Loss and imposing a 2x penalty on overestimation (predicted speed is greater than the true value) during training, the model is forced to output a more robust, accurate and conservative speed estimate. Combined with the trapezoidal integral method, the "virtual length" phenomenon of odometer is effectively suppressed.

[0034] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A real-time velocity estimation method for humanoid robots based on temporal feature fusion, characterized in that, Specifically, the following steps are included: Step S1: Construct a digital twin model of the humanoid robot in a physical simulation environment, collect the body sensory data during the robot's random walk motion, extract the true value of the world system linear velocity, and generate a motion dataset. Step S2: Construct an LSTM+MLP cascaded hybrid neural network architecture. Using the selected 49-dimensional ontology sensory data as input and the world system linear velocity true value as label, train and optimize the LSTM+MLP cascaded hybrid neural network architecture to obtain a hybrid neural network model. Step S3: During the actual operation of the humanoid robot, the raw data stream of the body's senses is collected in real time through several sensors inside the humanoid robot, and the raw data stream of the body's senses is synchronously arranged and time-series aligned according to the characteristic format of the 49-dimensional body sense data to obtain standardized feature data. Step S4: Deploy the optimized hybrid neural network model to the embedded main control platform, input the standardized feature data into the pre-trained model for inference operations, and output the predicted value of the three-dimensional linear velocity of the humanoid robot in the world coordinate system. Step S5: Perform trapezoidal numerical integration on the output three-dimensional linear velocity prediction value to generate continuous robot displacement odometer information and distribute it to the humanoid robot system bus.

2. The real-time velocity estimation method for humanoid robots based on temporal feature fusion according to claim 1, characterized in that, The 49-dimensional body sensory data filtered in step S2 includes the 3-axis angular velocity, 4-axis attitude quaternion output by the IMU, and the angle and angular velocity data of the 21 joint motors of the humanoid robot. The three-axis acceleration data of the IMU is removed. The joint motors include 1 waist joint motor, 8 arm joint motors and 12 leg joint motors.

3. The real-time velocity estimation method for humanoid robots based on temporal feature fusion according to claim 1, characterized in that, In step S2, the LSTM+MLP cascaded hybrid neural network architecture includes a temporal feature extraction layer and a spatial computation and regression layer. The temporal feature extraction layer is a 2-layer LSTM network used to capture the inertial laws of humanoid robot motion from a 49-dimensional temporal tensor with a sequence length of 20. The spatial computation and regression layer is a 2-layer MLP network whose input is the last hidden state of the LSTM network. It is used to map high-dimensional temporal features to a 3-dimensional linear velocity space to achieve regression prediction of linear velocity in the world coordinate system.

4. The real-time velocity estimation method for humanoid robots based on temporal feature fusion according to claim 3, characterized in that, In step S2, the LSTM+MLP cascaded hybrid neural network architecture uses the Optuna hyperparameter optimization software framework to determine the optimal hyperparameters, and introduces asymmetric mean squared error (Asymmetric MSE Loss) as the loss function during training. This loss function is the sum of the asymmetric losses of each component, and its calculation formula is as follows: ； in, For loss function, Using the indices of the three-dimensional coordinate axes, traverse... , and Three axes, The penalty coefficient is... For predicting the linear velocity vector, This is the true linear velocity vector.

5. The real-time velocity estimation method for humanoid robots based on temporal feature fusion according to claim 1, characterized in that, In step S5, the trapezoidal integral method is used to perform trapezoidal numerical integration on the output three-dimensional linear velocity prediction value, which is used to calculate the three-axis displacement increment of the humanoid robot in the world coordinate system. This process is applied to any axis in the world coordinate system. ,exist Displacement at any moment The calculation formula is expressed as: ； in, for The humanoid robot at time in the world coordinate system Cumulative axial displacement For the previous cycle At the time of Historical cumulative displacement in the axial direction for The output of the hybrid neural network model at time step 1 Predicted axial instantaneous linear velocity. For the previous cycle At the time of Historical linear velocity predictions output by a hybrid neural network model along the axial direction. The integration step size is the sampling period. This is the estimated average speed for the current period.

6. The real-time velocity estimation method for humanoid robots based on temporal feature fusion according to claim 1, characterized in that, The random walk motion in step S1 includes forward movement, backward movement, lateral movement in any direction, rotation in place, and gait while rotating.

7. The real-time velocity estimation method for humanoid robots based on temporal feature fusion according to claim 2, characterized in that, In step S3, the raw data stream of the body's sensory organs is acquired in real time at a frequency of 500Hz through the LCM lightweight communication protocol, and the time sequence of the multiple asynchronously arriving data streams is aligned to ensure that the standardized feature data is compatible with the feature format of the motion dataset. The sensors include an IMU and angle sensors and speed sensors for each joint motor.

8. The real-time velocity estimation method for humanoid robots based on temporal feature fusion according to claim 1, characterized in that, In step S4, the hybrid neural network model is deployed to the RK3588 embedded main control platform. The standardized feature data is processed through a 20-frame time-series sliding window, and the humanoid robot's position in the world coordinate system is output in real time at a frequency of 200Hz. , , Three-axis velocity prediction values.

9. The real-time velocity estimation method for humanoid robots based on temporal feature fusion according to claim 1, characterized in that, In step S5, continuous robot displacement odometry information is generated at a frequency of 200Hz. The continuous robot displacement odometry information is distributed to the humanoid robot system bus through LCM messages or ROS 2 standard topics to achieve the docking effect with the upper-level navigation, obstacle avoidance, localization and mapping algorithms.

10. A real-time velocity estimation system for a humanoid robot based on temporal feature fusion, used to execute the real-time velocity estimation method for a humanoid robot based on temporal feature fusion according to any one of claims 1-9, characterized in that, include: The offline motion data simulation and annotation module is used to build a digital twin model of a humanoid robot in a physical simulation environment, collect the body sensory data during the robot's random walk motion, extract the true value of the world system linear velocity, and generate a motion dataset. The neural network construction and model training module is used to build, train and optimize the LSTM+MLP cascaded hybrid neural network architecture to obtain a hybrid neural network model. The LSTM layer is used to capture the inertial laws of humanoid robot motion from continuous time series frames, and the MLP layer is used to perform nonlinear spatial mapping on the fused sensory features. The multi-source body data acquisition module is used to collect raw sensory data streams of the humanoid robot in real time through several sensors, and to synchronously arrange and time-series align them to obtain standardized feature data. The embedded model inference and velocity estimation module is used to deploy the hybrid neural network model to the embedded main control platform to complete inference and output the predicted value of the three-dimensional linear velocity of the humanoid robot in the world coordinate system. The real-time calculation and odometry generation module is used to perform trapezoidal numerical integration on the three-dimensional linear velocity prediction values ​​output by the hybrid neural network model and generate continuous robot displacement odometry information.

Images