A method for predicting falls in humanoid robots by fusing information from multiple sensors

Abstract

This invention designs a fall prediction method for humanoid robots using multi-sensor spatiotemporal information fusion, characterized by the following steps: S1. Multi-sensor data acquisition, S2. Data preprocessing, S3. Construction of a multi-sensor spatiotemporal information fusion network model, S4. Model training, and S5. Model verification and application. This invention first collects multi-sensor data from the humanoid robot, then sequentially performs multi-source information weighting, multi-scale spatial feature extraction, temporal feature extraction, and spatiotemporal feature dimensionality reduction and fusion through a designed multi-sensor spatiotemporal information fusion neural network, finally predicting the fall state of the humanoid robot. This method has high fall recognition accuracy, low false positives, good noise robustness, and a long prediction time, providing sufficient buffer time for subsequent controllers to customize fall solutions.

Description

Technical Field

[0001] This invention relates to the field of humanoid robot technology, and in particular to a method for predicting falls in humanoid robots by fusing spatiotemporal information from multiple sensors. Background Technology

[0002] Breakthroughs in manufacturing technology, iterative upgrades of large-scale artificial intelligence models, and the increasing aging population have driven the rapid development of the humanoid robot industry. As a highly human-like embodied intelligent agent with similar structure and behavior, humanoid robots can perform numerous anthropomorphic and intelligent tasks in human-centered unstructured spaces. They hold broad application prospects in fields such as agricultural production, medical and elderly care, industrial manufacturing, and entertainment performance, and are expected to become a novel and disruptive product. Compared to other multi-legged robots (such as quadruped robots), humanoid robots have a higher center of gravity and a smaller ground support area. They are expected to move autonomously in complex environments such as rugged terrain or confined spaces, inevitably encountering various internal and external disturbances. This poses new challenges to maintaining balance and stability. Due to the rigid structure of the robot, falls can endanger itself and its surrounding environment. Therefore, it is necessary to study humanoid robot fall prevention technology to provide timely warnings before a robot is about to fall due to disturbances, restore stability by switching controllers, or minimize impact forces, thereby ensuring reliable long-term deployment of the robot in various tasks.

[0003] Robot fall prevention strategies generally consist of three consecutive stages: fall prediction, fall control, and fall recovery. Fall prediction directly determines whether the controller can provide sufficient buffer time to customize a fall prevention solution, and is a prerequisite for successful control and protection actions. A significant challenge is balancing the prediction time (the time difference between predicted fall and actual collision time) and the false alarm rate. Existing fall prediction technologies mainly fall into two categories: those based on physical models and those based on data-driven approaches. Physical models include linear inverted pendulum models, point mass models, and rigid body models. These technologies often require joint judgment based on sensor data thresholds and are easy to visualize and understand. However, humanoid robot systems are complex in structure and have high degrees of freedom. The dynamic characteristics of the fall process exhibit complex strong nonlinearity and multiple couplings, making accurate modeling based on numerous assumptions and constraints difficult, thus limiting the application of physical model-based prediction technologies in complex and variable real-world environments. With the advancement of computer science and sensor technology, data-driven methods have become a research hotspot due to their ability to reduce the uncertainty and inaccuracy of dynamic modeling. This type of method aims to extract sensitive fall state features from sensor-perceived data in high-dimensional space. It establishes the relationship between data features and behavioral states through shallow machine learning techniques such as support vector machines and hidden Markov models, or directly constructs an end-to-end prediction model using deep learning techniques like convolutional neural networks. While data-driven methods demonstrate good performance, they still have some limitations in practical engineering applications: ① Most current work treats fall prediction as a binary classification problem of normal and fall states, but the boundary between normal stable states and fall processes is often blurred; ② Single-sensor monitoring is susceptible to interference, fall direction, terrain, and normal behaviors (such as sprinting), and the provided data features may be neither sufficient nor reliable; ③ Due to complex system structures and transmission paths, the signals collected by sensors inevitably contain noise, interfering with feature extraction; ④ Single convolutional networks and recurrent networks can only focus on feature learning from spatial and temporal perspectives, respectively, but humanoid robot perception data has both spatial and temporal characteristics. Summary of the Invention

[0004] The purpose of this invention is to address the limitation of humanoid robots being prone to falling due to poor anti-interference ability in complex unstructured environments, and to provide a fall prediction method for humanoid robots by fusing multi-sensor spatiotemporal information.

[0005] The technical solution of the present invention is as follows:

[0006] A method for predicting falls in humanoid robots by fusing spatiotemporal information from multiple sensors includes the following steps:

[0007] S1. Multi-sensor data acquisition: Three-axis acceleration data and three-axis angular velocity data are acquired through an inertial measurement unit (IMU) sensor located at the center of the robot's torso, and two-axis pressure data are acquired through a pressure sensor array located on the soles of the robot's feet with multiple measurement points. The multi-sensor data (i.e., data from the IMU sensor and the pressure sensor array on the soles of the feet with multiple measurement points) of the humanoid robot under different state disturbances are repeatedly and synchronously recorded with the same time reference, including four types of state time signals: normal, unstable stage I, unstable stage II, and complete fall.

[0008] S2. Data preprocessing: The multidimensional time series obtained in S1 (including time series signals of four states: normal, unstable stage I, unstable stage II, and complete fall) is truncated into multiple sets of short subsequences with fixed time steps using non-overlapping sliding window segmentation technology. This yields multiple sets of trunk acceleration subsequence data, trunk angular velocity subsequence data, and plantar pressure center point subsequence data. Then, each set of trunk acceleration subsequence data, trunk angular velocity subsequence data, and plantar pressure center point subsequence data is used to construct a feature vector in parallel, thereby obtaining a multidimensional channel sample set. This allows the monitoring information from multiple sensors scattered in different locations to be correlated, and the data samples are labeled accordingly. All samples are then randomly sampled and divided into training and test sets, and each sample is standardized.

[0009] S3. Construction of Multi-Sensor Spatiotemporal Information Fusion Network Model: The multi-sensor spatiotemporal information fusion network mainly consists of a multi-source information weighting module, a multi-scale spatial feature extraction module, a temporal feature extraction module, a global information fusion module, and a classification prediction module stacked sequentially. The multi-source information weighting module is used for adaptive weighting of multi-sensor input data. The multi-scale spatial feature extraction module is used to extract rich and complementary spatial features from noisy data. The temporal feature extraction module is used to further mine the temporal relationship information between monitoring data points. The global information fusion module is used for dimensionality reduction and fusion of the extracted spatiotemporal features. The classification prediction module is used to achieve the final prediction of the humanoid robot's fall state.

[0010] S4. Model Training: Input the training set samples into the model built in S3 and train it using a supervised learning strategy. Use cross-entropy loss to measure the error between the predicted probability distribution and the true probability distribution. Use the Adam optimizer to update the network parameters based on the error. After multiple iterations of training, the training is completed when the preset termination condition is met, and the optimal model hyperparameters are saved.

[0011] S5. Model Validation and Application: Input the test set samples into the model trained in S4 for validation, statistically analyze the state prediction results, and calculate the accuracy, precision, recall, F1 score, and early prediction time evaluation metrics. After the test metrics meet the requirements, further deploy the model online in the practical application of early real-time fall prediction for humanoid robots.

[0012] Beneficial effects of the technical solution of this invention

[0013] This invention provides a method for predicting fall in humanoid robots using multi-sensor information fusion. The method first collects multi-sensor data from the humanoid robot, then sequentially performs multi-source information weighting, multi-scale spatial feature extraction, temporal feature extraction, and spatiotemporal feature dimensionality reduction and fusion through a designed multi-sensor spatiotemporal information fusion neural network. Finally, it predicts the fall state of the humanoid robot. This method offers high fall recognition accuracy, low false positives, good noise robustness, and a long prediction time, providing sufficient buffer time for subsequent controllers to customize fall solutions. Attached image description:

[0014] Figure 1 This is a flowchart of the overall process for predicting falls in a humanoid robot using multi-sensor information fusion, as described in this invention.

[0015] Figure 2 This is a state space diagram of a humanoid robot.

[0016] Figure 3 It is a multi-sensor spatiotemporal information fusion network structure;

[0017] Figure 4 It is a multi-source information weighting module;

[0018] Figure 5 It is a multi-scale spatial feature extraction module;

[0019] Figure 6 It is the internal structure of the GRU unit;

[0020] Figure 7 It is a BiGRU structure;

[0021] Figure 8 It is a global information fusion module. Detailed Implementation

[0022] This invention transforms the early fall prediction problem into a time series classification problem, mainly comprising five stages: multi-sensor data acquisition, data preprocessing, construction of a multi-sensor spatiotemporal information fusion network model, model training, model validation, and application. The specific implementation includes the following steps:

[0023] S1. Multi-sensor data acquisition

[0024] To balance adaptability and low cost, inertial measurement unit (IMU) sensors (including accelerometers and gyroscopes) and pressure sensors were selected to monitor time series data as the model's multi-source input data. The IMU sensors were positioned at the center of the torso to measure triaxial acceleration and angular velocity data in the x, y, and z directions; the pressure sensor array was positioned on the soles of both feet to measure pressure data in the x and y directions, with 24 sensor units on each foot. First, sufficient physical protection must be provided to the robot before the experiment to prevent damage to components from falls. Then, when the robot is in a static standing or dynamic walking state, thrusts of different magnitudes, directions, and positions are applied to the robot's torso, or the robot is made to walk on uneven ground. Each operation is performed on the same time base, synchronously recording acceleration, angular velocity, and pressure data during operation. This includes timing signals for four states: normal, unstable stage I, unstable stage II, and complete fall. Unstable stage I refers to the gradual slowing of fluctuations until stability is restored after being disturbed; unstable stage II occurs when the robot is unable to regain balance by adjusting its movements after being disturbed, ultimately falling and potentially colliding with surrounding low-potential objects. A schematic diagram of the humanoid robot's operational state space is shown below. Figure 2 As shown, S1 to S6 represent different states.

[0025] S2. Data Preprocessing

[0026] Data preprocessing includes two main sub-steps: multi-channel feature vector construction and dataset partitioning.

[0027] In the multi-channel feature vector construction step, the different state time series collected by each multi-dimensional sensor are processed using a sliding window segmentation technique to extract multiple short subsequences with fixed time steps. The sliding window time length T can be adjusted to a suitable value according to the sensor sampling frequency and actual experimental conditions. Since the plantar pressure data has multiple measurement points and a large dimension, the feature value of the pressure center point of the force sensing unit is calculated and can be expressed as:

[0028]

[0029] In the formula: x CoP y CoP These are the time series of pressure center points on the x-axis and y-axis, respectively, F i It is the measured force of the i-th force sensing unit, x i y i These are the positions of the force sensing units on the x-axis and y-axis, respectively, and n is the number of pressure sensing units on one foot side.

[0030] Then, the equal-length and equal-time subsequence data of each set of acceleration, angular velocity, and pressure center point are combined in parallel to form feature vectors, constructing a sample set with 10 channels. This allows the monitoring information from multiple sensors scattered in different locations to be correlated, thus improving and supplementing the feature representation of the humanoid robot's fall state. Each sample is attached with a state label, the size of which can be represented as C×T, where C is the number of dimensions of the multi-source sensor data (10), T is the sliding window time length, and the features of each dimension channel are derived from the corresponding sensor information.

[0031] In the dataset partitioning step, all samples are randomly sampled and divided into training and test sets according to a certain proportion. A non-overlapping sliding window partitioning strategy is used to avoid the risk of data leakage in the test set. Furthermore, to reduce the distribution differences between data collected by sensors at different locations, each sample is Z-score standardized.

[0032]

[0033] In the formula: They are samples s τ The mean and variance, This is the standardized result, where T is the sliding window time, or sample length.

[0034] S3. Construction of Multi-Sensor Spatiotemporal Information Fusion Network Model

[0035] The multi-sensor spatiotemporal information fusion network mainly consists of a multi-source information weighting module, a multi-scale spatial feature extraction module, a temporal feature extraction module, a global information fusion module, and a classification and prediction module. First, the multi-source information weighting module adaptively weights the input data from multiple sensors, highlighting the early fall characteristics of sensitive sensors. Next, a multi-scale spatial feature extraction module is designed to fully extract rich spatial features from noisy data. Then, the temporal feature extraction module further mines effective temporal feature information. Next, a global information fusion module is constructed to perform dimensionality reduction and fusion of the extracted spatiotemporal features. Finally, a classification module is used to predict the fall state of the humanoid robot. The structure of the multi-sensor spatiotemporal information fusion network is as follows: Figure 3 As shown:

[0036] (1) Multi-source information weighting module

[0037] Considering that different source sensors have varying importance for sensitive feature information reflecting the robot's fall state, an attention mechanism is introduced to design a multi-source information weighting module that directly acts on the input sample data. This module adaptively recalibrates the effective features of each input channel, assigning greater attention weights to features from important sensor channels, thereby enhancing the distinguishability of different robot behavior states. The structure of the multi-source information weighting module is as follows: Figure 4 As shown, it includes two stages: channel weight learning and channel weighting.

[0038] In the channel weight learning stage, firstly, global max pooling (GMP) and global average pooling (GAP) operations are used to compress the input sample feature maps, respectively. (C is the number of channels, T is the sample length), resulting in two statistical feature vectors. Then, each feature is transformed by a one-dimensional convolutional layer with shared parameters to learn the feature correlation between local channels. The features are then merged using an element-wise addition method, and finally scaled to the (0,1) range by a Sigmoid activation function to obtain the channel weight vector. This process can be represented as:

[0039] ω=σ(Conv(GMP(S))+Conv(GAP(S))) (3)

[0040] In the formula: Conv(·) is a one-dimensional convolution operator, σ(·) is the sigmoid function, and GMP(·) and GAP(·) are the global max pooling operator and the global average pooling operator, respectively, performed along the feature channel dimension.

[0041] In the channel weighting stage, the channel weight vector ω is multiplied by the original input feature map S to allocate the feature importance of each sensor channel dimension. A residual connection structure is used to avoid excessive information loss, resulting in the final weighted feature map.

[0042]

[0043] In the formula: This represents the element-wise multiplication operator.

[0044] (2) Multi-scale spatial feature extraction module

[0045] Under different behavioral states, the data on torso acceleration and angular velocity, as well as plantar pressure data, of humanoid robots often exhibit different characteristic patterns. Specifically, data from normal behavior or after a complete fall shows a global periodic stationarity trend, while data during the fall process shows local transient fluctuations. Furthermore, due to complex working conditions and transmission paths, sensor monitoring data inevitably suffers from noise or redundant information interference, easily blurring the feature boundaries of the early fall process. To fully extract hidden spatial feature information from noisy fall data, three parallel multi-scale feature learning branches are designed using convolutional kernels of different sizes, such as... Figure 5 As shown.

[0046] Specifically, the three convolutional layer branches use kernel sizes of 1×3, 1×7, and 1×16. A zero-padding strategy ensures the consistency of the output feature map size for each branch, and the output results are concatenated and merged along the channel dimension. This process can be represented as:

[0047]

[0048] In the formula: f 1×3 f 1×7 f 1×16 The convolution kernels are 1×3, 1×7, and 1×16, respectively. and S is the output of the three convolution branches. c ′ represents the feature map after concatenation of the three-branch outputs, * represents the convolution operator, and concat(·) represents the concatenation operator along the channel dimension. Smaller kernels help extract local detail information, while larger kernels are geared towards mining features over longer time periods and enhancing noise filtering capabilities.

[0049] (3) Temporal Feature Extraction Module

[0050] To further explore the temporal relationships between data points monitored by robot sensors, a temporal feature extraction module is designed by introducing a gated recurrent unit (GRU). The GRU mainly consists of reset gates and update gates, which are used to capture short-term and long-term temporal dependencies, respectively. Compared with long short-term memory networks, it simplifies the structure and parameters, which helps to improve the training convergence speed and reduce the risk of overfitting. Figure 6 The internal structure of the GRU cell is calculated as follows:

[0051] R t =σ(W r ·[H t-1 ,X t ]) (9)

[0052] Z t =σ(W z ·[H t-1 ,X t (10)

[0053]

[0054] In the formula: X t The input information at time t; R t Z t These are the reset gate and update gate outputs, respectively; W r W z W and H are the weight matrices for the update gate, reset gate, and candidate hidden layer, respectively; t-1 H t These represent the hidden states at the preceding and following time points; denoted as candidate hidden state; σ(·) and tanh(·) are the sigmoid function and hyperbolic tangent function, respectively; ⊙ is the Hadamard product.

[0055] GRU can only learn forward information, therefore a bidirectional GRU (BiGRU) is used to fully capture the complete forward and backward temporal information of the data. Its hidden state output is a weighted sum of the forward GRU hidden state output and the backward GRU hidden state output. The BiGRU structure is as follows: Figure 7 As shown, Figure 7 ZhongX t-2 X t-1 X t The input information at times t-2, t-1, and t are respectively. These are the hidden state outputs of the forward GRU at times t-2, t-1, and t, respectively. These are the hidden state outputs of the backward GRU at times t-2, t-1, and t, respectively. These are the hidden state outputs of BiGRU at times t-2, t-1, and t, respectively.

[0056] (4) Global Information Fusion Module

[0057] The spatiotemporal features extracted from the previous modules are integrated and reduced in dimensionality through a global average pooling layer, and then transformed and fused through a fully connected layer to obtain the final classification feature representation. The specific process can be represented as follows:

[0058] Y = GAP(M) (13)

[0059] z k =W k Y+b k (14)

[0060] In the formula: M and Y are the input and output feature maps of the global average pooling layer, respectively, and z k W is the output of the k-th class node in the fully connected layer. k b k Here, represents the weight and bias parameters of the fully connected layer, respectively, and GAP(·) is the global average pooling operator performed along the feature channel dimension. The global average pooling layer avoids the excessive number of meaningless parameters introduced by conventional flattening layers, effectively reducing the number of parameters and enhancing noise robustness. The structure of the global information fusion module is as follows: Figure 8 As shown.

[0061] (5) Classification prediction module

[0062] The classification prediction module uses the Softmax function to map the feature vectors output by the fully connected layer to a probability distribution with a sum of 1, which can be represented as:

[0063]

[0064] In the formula: z k z j These are the outputs of the k-th and j-th class nodes in the fully connected layer, respectively, where K is the total number of state classes, and p... k Output the probability of the k-th node category, and output the category state corresponding to the highest probability as the prediction result.

[0065] S4. Model Training

[0066] The network model is trained using supervised learning with training set samples, and the error between the predicted probability distribution and the true probability distribution is measured by cross-entropy loss. The formula for calculating the cross-entropy loss function is as follows:

[0067]

[0068] In the formula: p k q represents the probability that the model predicts a sample to be of class k. k The true labels are in one-hot encoded form, and K is the total number of state categories. After the loss value is obtained, the parameters are updated by the Adam optimizer, and the model is trained through multiple iterations to minimize the loss until a preset termination condition is met. Then, the optimal model hyperparameters are saved, and the training is complete.

[0069] S5. Model Validation and Application

[0070] The trained model is validated on a new test set, outputting predicted states. Performance evaluation metrics, including accuracy, precision, recall, F1 score, and prediction time, are calculated to analyze model effectiveness. Accuracy, precision, recall, and F1 score are common classification metrics. Specifically, the prediction time T is... lead Defined as the moment T begins to fall. Fall_init Then, the algorithm detected the time T when the fall occurred. react With the moment of impact T impact The time interval between them, that is:

[0071]

[0072] In the formula, ΔT represents the time interval between the actual start of the fall and the moment of impact. Generally, higher accuracy, precision, recall, and F1 score, along with a longer lead time, result in better performance of the fall prediction model. In practical applications, the prediction algorithm needs to anticipate the state as far ahead as possible before the fall, especially accurately identifying the salvageable unstable stage I of the early fall, so that the subsequent control module can take timely emergency fall prevention actions.

[0073] Once the test results meet the standards, the model can be deployed online on humanoid robots for real-time fall prediction applications.

[0074] Summary of the technical solution of this invention:

[0075] Humanoid robots are an important carrier of embodied intelligence and are expected to become a novel and disruptive product. However, they inevitably fall in practical applications. Fall prediction technology, which is one of the key technologies of humanoid robots, has been rarely studied. This invention focuses on using multi-sensor big data and advanced deep learning technology to predict falls, which can provide a new approach to ensure the reliable deployment of humanoid robots.

[0076] Since humanoid robots exhibit postural instability when they fall, but postural instability is not the same as falling, it is difficult to use a unified and clear standard to determine the boundary between the normal stable state and the falling state. Therefore, unlike previous work that often regarded the falling problem as a binary problem of normal and falling, this invention has made a more refined four-classification: normal, salvageable unstable stage I, unsalvageable unstable stage II, and complete fall.

[0077] To address the issue that using data from a single sensor for fall prediction may be neither sufficient nor reliable, a method is proposed that integrates multi-source information from multi-axis IMU sensors and multi-axis pressure tactile sensors to expand the spatial domain of information and provide more comprehensive and detailed information characterizing the early fall state. At the same time, the use of multiple sensors also makes it easier for subsequent models to distinguish between falls and normal activities (such as robots climbing stairs), thus reducing the false alarm rate.

[0078] Unlike common tasks that process multi-sensor data separately by constructing multi-branch models, this invention combines multi-sensor data into input samples with multi-dimensional channels in parallel and integrates them into a unified, lightweight network framework for training, which helps reduce the number of model parameters. Furthermore, the designed multi-source information weighting module can adaptively correlate monitoring information from multiple sensors located at different positions at the input end, enhancing the expression of key channel features.

[0079] To address the limitation of single network models in extracting effective and sufficient features from noisy data, a temporal feature extraction module combining multi-scale convolutional kernels and bidirectional gated recurrent units is provided. This module helps learn rich and complementary spatiotemporal features of humanoid robot fall states. Furthermore, the large convolutional kernels used in the multi-scale convolutional kernels exhibit good noise robustness.

[0080] The fall prediction method of the present invention can be further extended to application areas such as elderly patients, quadruped or other legged robots, and has strong technology transferability and versatility.

Claims

1. A method for predicting falls in humanoid robots by fusing multi-sensor spatiotemporal information, characterized in that... Includes the following steps: S1. Multi-sensor data acquisition: The Inertial Measurement Unit (IMU) sensor located at the center of the robot's torso collects three-axis acceleration data and three-axis angular velocity data. The pressure sensor array located on the soles of the robot's feet with multiple measurement points collects two-axis pressure data. The multi-sensor data of the humanoid robot under different state disturbances is repeatedly and synchronously recorded with the same time reference, including four types of state time signals: normal, unstable stage I, unstable stage II, and complete fall. S2. Data preprocessing: The multidimensional time series obtained in S1 is truncated into multiple short subsequences with fixed time steps using a non-overlapping sliding window segmentation technique. This yields multiple sets of trunk acceleration subsequence data, trunk angular velocity subsequence data, and plantar pressure center point subsequence data. Then, each set of trunk acceleration subsequence data, trunk angular velocity subsequence data, and plantar pressure center point subsequence data is used to construct a feature vector in parallel, thereby obtaining a multidimensional channel sample set. This allows the monitoring information from multiple sensors scattered in different locations to be correlated, and the data samples are labeled accordingly. All samples are then randomly sampled and divided into training and test sets, and each sample is standardized. S3. Construction of Multi-Sensor Spatiotemporal Information Fusion Network Model: The multi-sensor spatiotemporal information fusion network mainly consists of a multi-source information weighting module, a multi-scale spatial feature extraction module, a temporal feature extraction module, a global information fusion module, and a classification prediction module stacked sequentially. The multi-source information weighting module is used for adaptive weighting of multi-sensor input data. The multi-scale spatial feature extraction module is used to extract rich and complementary spatial features from noisy data. The temporal feature extraction module is used to further mine the temporal relationship information between monitoring data points. The global information fusion module is used for dimensionality reduction and fusion of the extracted spatiotemporal features. The classification prediction module is used to achieve the final prediction of the humanoid robot's fall state. S4. Model Training: Input the training set samples into the model built in S3 and train it using a supervised learning strategy. Use cross-entropy loss to measure the error between the predicted probability distribution and the true probability distribution. Use the Adam optimizer to update the network parameters based on the error. After multiple iterations of training, the training is completed when the preset termination condition is met, and the optimal model hyperparameters are saved. S5. Model Validation and Application: Input the test set samples into the model trained in S4 for validation, statistically analyze the state prediction results, and calculate the accuracy, precision, recall, F1 score and early prediction time evaluation indicators. After the test indicators meet the requirements, further deploy the model online in the practical application of early real-time fall prediction for humanoid robots. The multi-source information weighting module includes two stages: channel weight learning and channel weighting. In the channel weight learning stage, firstly, global max pooling (GMP) and global average pooling (GAP) operations are used to compress the input sample feature maps, respectively. ，C For the number of channels, T Given the sample length, two statistical feature vectors are obtained. , Then, each feature is transformed by a one-dimensional convolutional layer with shared parameters to learn the feature correlation between local channels. The features are then merged using an element-wise addition method, and finally scaled to the (0,1) range by a Sigmoid activation function to obtain the channel weight vector. This process can be represented as: （3） In the formula: For a one-dimensional convolution operator, For the sigmoid function, These are the global max pooling operator and the global average pooling operator, respectively, performed along the feature channel dimension; During the channel weighting stage, the channel weight vector is... Multiply by the original input feature map S This method assigns feature importance across each sensor channel dimension and uses a residual connection structure to avoid excessive information loss, resulting in the final weighted feature map. , （4） In the formula, This represents the element-wise multiplication operator; The multi-scale spatial feature extraction module, in order to fully extract hidden spatial feature information from noisy fall data, employs three parallel multi-scale feature learning branches with convolutional kernels of different sizes. Specifically, the three convolutional layer branches use kernel sizes of 1×3, 1×7, and 1×16. A zero-padding strategy ensures the consistency of the output feature map size for each branch, and the output results are concatenated and merged along the channel dimension. This process can be represented as: (5) (6) (7) (8) In the formula: f 1×3 , f 1×7、 f 1×17 The convolution kernels are 1×3, 1×7, and 1×16, respectively. The outputs of the three convolution branches, This is the feature map after concatenating the outputs of the three branches. For convolution operators, This indicates that the splicing operator along the channel dimension is more effective at extracting local details, while the larger kernel is more suitable for mining features over longer time periods and enhancing noise filtering capabilities. The aforementioned temporal feature extraction module introduces a gated recurrent unit (GRU). The GRU mainly consists of reset gates and update gates, used to capture short-term and long-term dependencies in the temporal sequence, respectively. Compared to Long Short-Term Memory (LSTM) networks, it simplifies the structure and parameters, helping to improve training convergence speed and reduce the risk of overfitting. The internal structure and calculation process of the GRU unit are as follows: (9) (10) (11) (12) In the formula: for t Input information at any time; These are the outputs for resetting the door and updating the door, respectively. These are the weight matrices for the update gate, reset gate, and candidate hidden layer, respectively. These represent the hidden states at the previous and next time steps, respectively. This is the candidate hidden state; These are the sigmoid function and the hyperbolic tangent function, respectively. For Hadama accumulation.

2. The method for predicting falls in humanoid robots by multi-sensor spatiotemporal information fusion according to claim 1, characterized in that, Pressure sensors are placed on the soles of both feet to measure pressure data in the x and y directions. There are 24 sensing unit measurement points on each foot.

3. The method for predicting fall in a humanoid robot by multi-sensor spatiotemporal information fusion according to claim 1, characterized in that, The global information fusion module integrates and reduces the dimensionality of the spatiotemporal features extracted by the previous modules through a global average pooling layer, and then transforms and fuses them through a fully connected layer to obtain the final classification feature representation. The specific process can be represented as follows: (13) (14) In the formula: M, Y These are the input and output feature maps of the global average pooling layer, respectively. For the fully connected layer k The output of each category node, These are the weight parameters and bias parameters of the fully connected layer, respectively. The global average pooling operator is performed along the feature channel dimension. The global average pooling layer can avoid the excessive number of meaningless parameters brought by the conventional flattening layer, effectively reduce the number of parameters, and enhance noise robustness.

4. The method for predicting falls in humanoid robots by multi-sensor spatiotemporal information fusion according to claim 1, characterized in that, The classification prediction module uses the Softmax function to map the feature vectors output by the fully connected layer to a probability distribution with a sum of 1, which can be expressed as: (15) In the formula: These are the fully connected layers. k The and the first j The output of each category node, K It is the total number of state categories. For the first k Output the probability of each node category, and output the category state corresponding to the highest probability as the prediction result.

Images