Self-adaptive safety planning method based on multi-modal intention perception and physiological signals

Abstract

According to the self-adaptive safety planning method based on the multi-modal intention perception and the physiological signals, through fusion of multi-modal information such as vision, IMU and the physiological signals, accurate intention perception and motion trail prediction are achieved, so that balance is achieved between safety margin and efficiency, and the safety performance of the system is improved. Through fusion of human behavior intention prediction, physiological state feedback and environmental perception data, in combination with advanced planning of intention prediction and immediate adjustment of physiological feedback, sudden actions and environmental changes are coped with. The behavior of the robot or the mechanical arm is dynamically adjusted through feature fusion and a human body intention classification algorithm, the collision probability is effectively reduced, and the man-machine cooperation efficiency and the user comfort degree are optimized while safety is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial automation, and in particular to an adaptive safety planning method based on multimodal intention perception and physiological signals. Background Art

[0002] With the development of artificial intelligence and robotics technology, intelligent systems need to make real-time adaptive adjustments based on user intentions and environmental conditions in practical applications to ensure the safety of human-machine collaboration in complex environments.

[0003] Traditional human-machine collaborative systems rely on a single sensor for motion planning, which has the following defects: poor adaptability to dynamic environments and inability to handle the uncertainty of human motion and environmental changes in real time; insufficient safety margin and lack of quantitative constraints on the human physiological state; trajectory prediction does not combine Gaussian process regression and multimodal feedback, resulting in poor adaptability, and traditional single-modal data cannot fully describe human motion intentions. Summary of the Invention

[0004] In order to solve the current technical problems, the main purpose of the present invention is to provide an adaptive safety planning method based on multimodal intention perception and physiological signals. This method realizes the coordinated optimization of safety and efficiency in human-machine collaboration through multi-dimensional data fusion and physiological constraint optimization. It can accurately perceive user intentions and predict motion trajectories, thereby realizing adaptive safety planning and control; it is suitable for human-machine interaction scenarios in fields such as human-machine collaborative assembly in intelligent manufacturing and exoskeleton control in medical rehabilitation.

[0005] In order to achieve the above technical features, the purpose of the present invention is to achieve the following: an adaptive safety planning method based on multimodal intention perception and physiological signals, comprising the following steps: Step 1, multimodal data acquisition: The system collects visual data, IMU data and physiological signals synchronously in real time; Step 2, preprocessing: Preprocess the multimodal data collected in step 1, including: spatiotemporal alignment, denoising, and normalization; Step 3, feature fusion: Perform feature fusion on the data preprocessed in step 2, map the multimodal data into a unified feature space, and receive feedback updates in real time; Step 4, intent classification: The multimodal data fused in step 3 is classified into multiple categories and the probability distribution of intent is output through the spatiotemporal Transformer network. Step 5, motion prediction: Perform motion prediction based on the probability obtained in step 4, generate future trajectory distribution based on Gaussian process regression, and receive feedback updates in real time; Step 6: Generate security domain: Combine the predicted trajectory in step 5 with physiological constraints to construct a time-varying safe space; Step 7, optimization solution: Use model predictive control algorithm to solve the optimal control sequence and send it to execution, and implement the update after receiving feedback; Step 8, execution control: Send the control instruction obtained in step 7 to the actuator; Step 9, Feedback Update: Real-time data is used to update the online model and transmit to steps 3, 5, and 7.

[0006] Preferably, the multimodal data in step 1 includes: Visual data: 3D depth camera collects the coordinates of skeletal joints at a certain frequency; IMU data: Multi-axis sensors collect acceleration and angular velocity signals at a certain frequency; Physiological data: surface electromyography collects electromyographic signals from multiple locations at a certain frequency and collects heart rate signals at a certain frequency.

[0007] Preferably, the data preprocessing method in step 2 is specifically as follows: The heterogeneous data collected in step 1 are temporally and spatially aligned, with the time synchronization error controlled within ±10ms, and frequency differences are compensated by interpolation. Gaussian filtering is used to denoise the visual data; Kalman filtering is used to smooth the IMU data; Butterworth bandpass filtering and multi-point noise reduction with a moving average window are used for physiological signals; and each modality data is standardized to a unified scale using the following formula: ; Where, For the data after standardization, For the collected data, is the mean of the collected data, is the standard deviation of the collected data.

[0008] Preferably, the feature fusion method in step 3 is specifically: Visual features are extracted from each modality after processing in step 3, where the ResNet-50 fully connected layer is 128-dimensional, the IMU temporal features are encoded in two layers of LSTM to 64 dimensions, and the physiological spectrum features are compressed to 64 dimensions through FFT+wavelet transform. The multi-source features are mapped to a 32-dimensional fusion space through an autoencoder and then linearly mapped to a 256-dimensional unified feature vector.

[0009] Preferably, the intention classification method in step 4 is specifically: The feature information fused in step 3 is classified using an FST-Trans network with an input dimension of 256 and four layers of spatiotemporal attention modules. Each layer can independently learn and adjust important spatial positions and time points in the features, outputting a probability distribution of five types of intentions, including walking, raising hands, stretching, grabbing things, and bending over.

[0010] Preferably, the human body motion prediction method in step 5 specifically includes: The intent probability after classification in step 4 is used as input. Based on Gaussian process regression, with an RBF kernel, a length scale of 0.5, and a variance of 1.0, the trajectory distribution in the next 1-2 seconds is predicted. New motion data is collected in real time, the trajectory prediction is continuously adjusted and optimized, and feedback updates are implemented.

[0011] Preferably, the security domain generation method in step 6 specifically includes: According to the predicted trajectory and physiological parameters in step 5, a dynamic safety range is set; considering the physiological limits of the human body and the environmental safety requirements, a time-varying spatial boundary is formed; and during motion and control, human behavior is ensured to be within a safe range.

[0012] Preferably, the optimization solution method in step 7 specifically includes: The model predictive control algorithm is used to solve the optimal control sequence and implement feedback updates. The model predictive control is used to calculate the optimal control instructions within a period of time based on the current state and target. The safety domain threshold is dynamically adjusted based on the user's electromyographic signal in actual operation. δ ( t ) is used to adjust model parameters and control sequences to ensure the adaptability and effectiveness of control.

[0013] Preferably, the control method executed in step 8 specifically includes: The control signal obtained in step 7 is converted, safety is detected, communication signals are transmitted, real-time feedback monitoring status is provided, anomalies are processed and optimized recalculation is performed to achieve safe, stable and efficient closed-loop control of the robotic arm and execution equipment.

[0014] Preferably, the feedback updating method in step 9 specifically includes: Continuously collect and preprocess the robot and sensor data, perform filtering, normalization and exception handling, and transmit them to steps 3, 4, and 7 for online model update; use the dynamic model to predict the state prior, and then combine it with the current observation data to update the fusion state through Bayesian update to obtain the updated state; then use it as the initial state for the next control cycle, adjust the strategy and monitor uncertainty, and recalibrate parameters when necessary; perform model predictive control optimization and real-time closed-loop control based on the updated state to ensure safe operation of the system; cyclically execute prediction, observation, update and optimization, while recording logs and handling exceptions, and triggering safety strategies when necessary.

[0015] The present invention has the following beneficial effects: This invention achieves accurate intention perception and motion trajectory prediction by fusing multimodal information such as vision, IMU, and physiological signals, thereby striking a balance between safety margin and efficiency. By integrating human behavioral intention prediction, physiological state feedback, and environmental perception data, this system combines advance planning based on intention prediction with immediate adjustment based on physiological feedback to respond to sudden movements and environmental changes. It utilizes feature fusion and human intention classification algorithms to dynamically adjust the behavior of the robot or manipulator, effectively reducing the probability of collisions and optimizing human-machine collaboration efficiency and user comfort while ensuring safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention will be further described below with reference to the accompanying drawings and examples.

[0017] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0019] Example 1: See also Figure 1 The present invention provides a closed-loop optimization method that combines intent prediction with a multimodal data fusion mechanism. First, an adaptive safety planning system is constructed. This system achieves accurate intent perception and motion trajectory prediction through the fusion of multimodal information such as vision, IMU, and physiological signals, and an intent classification model based on spatiotemporal Transformer. Dynamic safety domain generation and closed-loop control combined with MPC optimization achieve a balance between safety margin and efficiency, thereby improving the safety and comfort of human-computer interaction. The system specifically includes: Multimodal data acquisition module, visual sensor: uses a 3D depth camera to collect the coordinates of human skeletal joints at a frequency of 30fps; IMU measurement unit: integrates a 9-axis sensor, samples at a frequency of 100Hz, and measures acceleration and angular velocity; physiological signal acquisition: surface electromyography and heart rate monitoring.

[0020] Intention perception and motion prediction module, multimodal feature fusion: mapping visual, IMU, and physiological data into a unified feature vector X ∈ ; Spatiotemporal Transformer Network: Input dimension: 256; Number of layers: 4, each layer contains a spatiotemporal attention module; Output intent probability distribution , a total of 5 categories, including walking, raising hands, stretching, grabbing things, and bending over.

[0021] Physiologically constrained optimization module, metabolic rate estimation model: ; in: , = , It's the heart rate. =5m / s 2 , The root mean square value of the electromyographic signal, Normalized value of acceleration amplitude, , is the Euclidean norm of the instantaneous acceleration of the human body (unit: m / s²), calculated from the IMU data.

[0022] Safety margin function: ; Where, The minimum distance between man and machine, =5m / s 2 , according to the typical safety distance standard for industrial robots (ISO13855): 0.2-0.5 meters. Experimental data: When =0.3 m hour, S ( t ) decays to ≈36.8, which is within the safety margin. Human body acceleration limit: normal action ≤3m / s², emergency dodge can reach 5-8m / s², when hour, S ( t ) decays to ≈36.8.

[0023] The dynamic safety planning module uses dynamic adjustment strategies based on planning and control data to ensure that the system achieves the best balance between safety margin and efficiency.

[0024] Constraints: Joint speed limits ; Collision probability <0.05; EMG threshold <200 μ V.

[0025] Example 2: The present invention provides a closed-loop optimization method combining intent prediction and multimodal data fusion mechanism, comprising the following steps: 1. multimodal data acquisition → 2. preprocessing → 3. feature fusion → 4. intent classification → 5. motion prediction → 6. dynamic safety domain generation → 7. optimization solution → 8. execution control → 9. feedback update.

[0026] 1. Data collection: The system collects visual data, IMU data, and physiological signals synchronously in real time, connecting the signals wirelessly with Bluetooth to ensure synchronization.

[0027] 1.1 Visual Signals: ,in For the Frame image data. A 3D depth vision sensor equipped with an Intel RealSense D435 depth camera with a wavelength of 850nm, a resolution of 1280×720, and a frame rate of 30fps was used to capture the video data stream. For each frame of the image sequence, the coordinates of 17 key human skeletal joints were obtained: the head, neck, shoulder, elbow, wrist, palm, waist, hip, knee, and ankle.

[0028] 1.2IMU signal: ,in For the j Acceleration and angular velocity data at each moment. Using a Bosch BNO0559 IMU with an axial sensor, acceleration and angular velocity are acquired at a 100 Hz frequency. Measurement range: Acceleration: ±16 g, Angular velocity: ±2000° / s, Magnetic field: ±1300 μT.

[0029] 1.3 Physiological signals: ,in For the Physiological data at each moment. Physiological data is obtained through surface electromyography ( ) collects data at a frequency of 500Hz, and the heart rate monitoring device collects data at a frequency of 50Hz. The data is collected through an 8-channel electrode patch, which monitors the myoelectric activity of major muscle groups such as the front thigh muscles and calf muscles in real time.

[0030] 2. Preprocessing: The collected multimodal signals are subjected to spatiotemporal alignment, denoising and feature normalization.

[0031] 2.1 Spatiotemporal Alignment: Timestamp alignment ensures time synchronization across sensor systems, with an error within ±10ms. Interpolation is used to compensate for differences in sampling frequencies to ensure data continuity.

[0032] 2.2 Denoising: Use signal processing methods to filter and remove noise. 1) Visual data denoising: Use mean, median, or Gaussian filtering with a kernel size of 5×5 and a standard deviation of σ=1.0 to reduce random noise and preserve structural information through edge smoothing. 2) IMU data denoising: Use Kalman filtering and process noise covariance matrix (Unit: m² / s 4 ), the observation noise covariance matrix (Unit: m² / s 4 ), smoothing acceleration and angular velocity data and reducing noise interference. 3) Physiological data denoising: Butterworth bandpass filtering with a frequency of 20-450Hz covers the effective frequency range of muscle electrical signals and suppresses power frequency interference (50 / 60Hz) and high-frequency noise. Heart rate signals use a 5-point moving average window, corresponding to 100ms, to balance smoothness and real-time performance.

[0033] 2.3 Feature Standardization: Standardize the features of each modality to ensure consistent dimensions across different data. Calculate the mean and standard deviation of visual data, IMU data, and physiological data; obtain the mean and standard deviation of image pixels or key points, acceleration and angular velocity, and each physiological channel; and use the standardization formula to convert these data into a standard normal distribution: ; ; ; in, and are the mean and standard deviation of each modal data respectively.

[0034] 3. Feature Fusion: The preprocessed visual, IMU and physiological signals are mapped into a unified feature space through a feature fusion algorithm.

[0035] Visual feature extraction, based on a pre-trained ResNet-50 model trained on the ImageNet dataset, with 25.6M parameters, extracts a 128-dimensional feature vector from the penultimate fully connected layer, preserving key texture and spatial information of the object; IMU motion sequence modeling uses a two-layer LSTM network with 64 hidden units in each layer to process IMU time series data and encode dynamic motion patterns into 64-dimensional features; Physiological signal processing: FFT and wavelet transform are performed on surface electromyographic signals, with 16 frequency bands per channel, to generate 128-dimensional spectral features, which are then compressed into 64-dimensional time series representations using LSTM. Fusion and dimensionality reduction: Multimodal alignment is achieved through an autoencoder structure. The encoder is a two-layer fully connected layer that converts from 128 to 64 and then to 32 dimensions. Multi-source features are fused in the 32-dimensional hidden layer, and the decoder performs reverse reconstruction to assist in training. Training optimization, jointly minimizes the reconstruction error and contrast loss, using the Adam optimizer with a learning rate of , and introduced Dropout (0.5) and L2 regularization with a coefficient of 1e-5 to prevent overfitting; The feature output is finally mapped to a 256-dimensional vector through linear transformation, which takes into account both information completeness and computational efficiency, and provides highly discriminative input for downstream intent classification.

[0036] 4. Intent Classification: The FST-Trans network is used to classify the fused features and output the user’s intent probability distribution.

[0037] 4.1 Data Input and Preparation: The network input is a 256-dimensional multimodal fusion feature, which is deeply modeled through a 4-layer spatiotemporal Transformer structure. Each layer of Transformer contains an 8-head self-attention mechanism, where each head has a dimension of 64, which can effectively capture spatiotemporal dependencies; the feedforward network uses a 512-to-256-dimensional fully connected layer, with The activation function and layer normalization enhance the nonlinear expression ability. The network introduces 256-dimensional absolute position encoding to retain the temporal information, and finally The system outputs the probability distribution of five types of intentions: walking, raising hands, reaching, grasping, and bending. During the training phase, the cross-entropy loss function is used, and the Adam optimizer is employed with a learning rate of 5e-5. 50 iterations are performed, with a batch size of 64 and a dropout rate of 0.3 to prevent overfitting. This design significantly improves the recognition accuracy of complex action intentions through the spatiotemporal attention mechanism, while maintaining efficient real-time inference performance.

[0038] 4.2 Constructing FST-Trans network architecture: Through a 4-layer spatiotemporal attention module, each layer contains 8 self-attention heads and the hidden layer dimension is 256, temporal dependencies are extracted; The spatiotemporal attention weights are calculated as: ; in , ensuring gradient stability, Q: the target vector (dimension ), K: the key vector being queried (dimension ), V: actual eigenvalue vector (dimension ).

[0039] The function outputs 5 types of intent probabilities: ; The weight matrix W The 256-dimensional features are mapped to the 5-dimensional intent space. See Table 1 for details. The logit value of each dimension corresponds to the category (walking, raising hands, stretching, grabbing things, bending over). The dimensionality reduction from 256 dimensions to 5 dimensions is achieved through learnable parameters. W Its physical meaning is: Indicates the j The original feature pair i The contribution of class intent; the correspondence between key features and intent can be verified by visualizing Grad-CAM. Covering basic human-machine collaboration actions: Table 1 Five-dimensional intention space table

[0040] Experimental verification of classification dimensions: too few dimensions will lead to confusion of intent, that is, the dimension is less than 3; too many dimensions will increase the computational burden and reduce real-time performance, that is, the dimension is greater than 8.

[0041] 5. Motion Prediction: Based on Gaussian process regression, the future motion trajectory distribution of the classified intention is predicted.

[0042] 5.1 Data Preparation and Preprocessing: 1) Acquire the historical position information of the moving object and align it in time, performing interpolation when necessary; 3) Normalize the position information and action labels to improve the numerical stability and prediction accuracy of the GPR model; 4) Reduce the data dimension by adding features such as timestamp, velocity, and acceleration to enrich the modeling information.

[0043] 5.2 Define Gaussian Process Regression Model: 1) In GPR, the future trajectory 1) The standard regression method is f(.): it takes time, historical location, and action label as input and outputs the future location. 2) The covariance kernel function (RBF) is selected to smooth the trajectory, with a length scale of 0.5 and a variance of 1.0. This kernel encodes temporal, spatial, and action labels.

[0044] 5.3 Model training: 1) Using training data ( , ) to determine the hyperparameters of the kernel function by maximizing the marginal log-likelihood of the Gaussian process; 2) to estimate the hyperparameters using numerical optimization methods such as gradient descent and conjugate gradient; 3) to evaluate model performance through cross-validation to prevent overfitting and make necessary adjustments to the kernel function or input features.

[0045] 5.4 Motion Prediction and Uncertainty Estimation: 1) Use the latest historical data as input and make predictions using the trained GPR model; 2) Use the conditional distribution of GPR to calculate the predicted mean and covariance of the new test input; 3) Output the predicted results It is a Gaussian distribution, with a prediction range of 1-2 seconds in the future. The sampling time of each step is 50ms. Its mean is the most likely trajectory, and the covariance describes the uncertainty of the prediction: ; in and denote the predicted mean and covariance matrix respectively.

[0046] 5.5 Post-processing and result application: 1) Use the covariance matrix to quantitatively assess prediction uncertainty for risk assessment and decision support; 2) Display the prediction trajectory with a mean curve and confidence interval for intuitive understanding; 3) Update historical data and GPR models through a sliding window to improve the accuracy of real-time predictions.

[0047] 5.6 Optionally, model expansion and optimization: 1) Multi-task GPR is used to capture the correlation between multiple motion dimensions; 2) Sparse GPR is used to improve computational efficiency for large-scale data; 3) Different kernel functions (such as RBF and linear kernel) are combined to better characterize the nonlinearity and local smoothness of the data.

[0048] ; in, For historical location information ， is the corresponding action label, For the predicted future trajectory.

[0049] 6. Security domain generation: Combining predicted trajectories with physiological constraints and feedback to construct a time-varying safe space.

[0050] 6.1 Input Data Preparation: 1) Obtaining Predicted Future Trajectory , which can be a continuous function or a discrete point, reflecting the position change of the moving body; 2) Real-time acquisition of user physiological data B(t) and definition of physiological constraint set As the indicator range in the situation; 3) Preset or dynamically calculated safety distance, adjusted according to the motion environment, user status and task requirements, taking into account factors such as obstacle density and speed changes.

[0051] Metabolic rate formula: ; in , = , It's the heart rate. =5m / The root mean square value of the electromyographic signal, Normalized value of acceleration amplitude, , is the Euclidean norm of the instantaneous acceleration of the human body (unit: m / s²), calculated from the IMU data.

[0052] Safety margin function: ; The minimum distance between man and machine, =5m / According to the typical safety distance standard for industrial robots (ISO13855): 0.2-0.5 meters. Experimental data: =0.3 m hour, S ( t ) decays to ≈36.8, which is within the safety margin. Human body acceleration limit: normal action ≤3m / s², emergency dodge can reach 5-8m / s², when hour, S ( t ) decays to ≈36.8.

[0053] Mathematical definition formula of security domain: ; in, Security Domain ， To predict the future trajectory, is the safety distance threshold, is a physiological constraint. Its meaning is as follows: position The predicted future position does not exceed the safety threshold δ(t) = 0.05, ensuring the prediction error and safety margin. The current physiological state B(t) is within the constraint range. Ensure that users have good control and safe movement conditions.

[0054] 6.2 Safe Domain Generation Steps: 1) Obtain and preprocess motion prediction and physiological data B(t) to ensure time synchronization; 2) dynamically adjust to ensure appropriate sensitivity through environmental information and physiological state calculation; 3) verify whether B(t) is within the constraint range triggering the necessary safety mechanisms; 4) Construct a safe area and apply physiological condition filtering; 5) Smooth and update the safe area in real time , used for path planning and safety warning.

[0055] 7. Optimization solution: The model predictive control (MPC) algorithm is used to solve the optimal control sequence to ensure that the motion trajectory is within the safety domain.

[0056] 7.1 Modeling and Problem Formulation: 1) Using Equations Indicates the evolution of the state with the control input, which has predictive properties; 2) Ensure , and may have additional control or state constraints; 3) including Penalize the smoothness of control input and Track the predicted trajectory, using a weighting factor λ to balance the control cost and tracking deviation.

[0057] 7.2 Establish the optimization problem: The MPC parameters are the prediction time domain of 1 second, with 20 prediction points; the control time domain is 0.2 seconds, that is, 4 control steps; the control variables are joint angular velocity and acceleration.

[0058] ; ; in For control input, The current state of the system, is the predicted future trajectory and λ is the trade-off between the coordination control cost and the tracking cost. See Table 2 for the values ​​of λ: Table 2 λ value table

[0059] The constraints are: ; ;

[0060] 7.3 Discretization and prediction time domain determination: 1) Select an appropriate time domain length T based on task requirements and computing resources. The longer the time domain, the greater the computational effort, but may provide longer-term security. When the time domain is shorter, a rolling optimization strategy can be used for continuous adjustment. 2) If the system model is in continuous form, use the Euler method or Runge-Kutta numerical method for discretization to ensure that the discretized model accurately describes the system motion.

[0061] 7.4 Optimization solution steps: 1) Get the current state x(0) and predict the trajectory , security domain , and related thresholds and constraints, determine the initial estimate and prediction time domain T; 2) organize the objective function, dynamic equations and constraints into a standard optimization form and implement it through optimization methods; 3) select the solver QP to solve the discretization problem and handle marginal likelihood and constraint relaxation to maintain numerical stability; 4) extract the optimal input sequence ,implement Update system status to , repeat this process for real-time optimization.

[0062] 7.5 Numerical Debugging and Performance Evaluation: 1) Adjust λ to balance control input smoothness and state tracking error, and modify δ(t) based on feedback if necessary; 2) Test the impact of model errors and measurement noise through simulation, and use robust MPC to enhance fault tolerance; 3) Ensure that the optimization solution time meets real-time requirements. Fast algorithms, parallel computing, or model simplification can be used to improve response speed.

[0063] 8.Execution Control: Control signal processing: Extract the current control input obtained by rolling optimization The signal format is converted to meet the robot's drive requirements, and interpolation and smoothing are used to avoid unstable execution. The communication protocol is Industrial Ethernet (EtherCAT), with a transmission latency of ≤10ms. Anomaly detection utilizes state monitoring, such as rapid changes and abnormal vibrations, triggering an emergency stop mechanism to trigger safety policies. Parameters are continuously adjusted through closed-loop control, and trajectory and control logs are recorded for feedback.

[0064] The steps are as follows: 1) Confirm that the state of the robotic arm is consistent with the MPC model to ensure that it is within the time-varying safety domain and trigger emergency control when an anomaly is detected; 2) Select a suitable communication protocol, assemble and verify the data integrity, send it to the robotic arm controller, and record the sent information; 3) Execute control actions, ensure stability through low-level control, collect sensor feedback in real time, identify motion tracking errors and perform closed-loop adjustments; 4) Record control instructions and feedback status, and set an exception handling mechanism to ensure safety; 5) Recalculate the MPC optimization after each cycle to ensure the real-time performance of the entire closed-loop process.

[0065] 9. Feedback update: 9.1 Online model update steps: 1) Continuously collect actuator and human sensor data, perform data filtering and normalization, check integrity, and correct anomalies; 2) Use the dynamic model and the previous control input to predict the state prior Xp, including state estimation and uncertainty; 3) Combine the prior Xp and the current observation data Xo to construct a likelihood function and perform state fusion to update the model. The likelihood function is: ; in : mean vector (predicted value), R is the observation noise covariance, : precision matrix (inverse of the covariance matrix), : Data dimension. Apply Bayes’ formula: According to Bayes’ theorem, calculate the posterior probability distribution: ; 9.2 Actual implementation steps: 1) Based on prior distribution and likelihood , using the update function BU to calculate the posterior estimate ;2) Calculate the Kalman gain K, , where Pp is the predicted covariance, is the observation matrix; 3) ;4) Update covariance .

[0066] 9.3 Online model update and control process: 1) The updated state Xu is used as the starting state for the next control cycle. The control strategy is adjusted to maintain the dynamic consistency of the model, monitor changes in uncertainty, and recalibrate system parameters or update the model in a timely manner to improve filtering performance. 2) Use Xu to provide a new reference for MPC optimization and control decisions, maintain a real-time feedback loop, and ensure that the system operates within the safety domain and the desired trajectory. 3) Continuously perform state prediction, observation collection, Bayesian update, and control optimization to form a closed-loop regulation. 4) Record the state and covariance information within the update cycle to evaluate performance. When the difference between observation and prediction continues to increase, check for sensor failure or model mismatch, and safety policy intervention is required.

[0067] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. An adaptive safety planning method based on multimodal intention perception and physiological signals, characterized by: The following steps are involved: Step 1, multimodal data acquisition: The system collects visual data, IMU data and physiological signals synchronously in real time; Step 2, preprocessing: Preprocess the multimodal data collected in step 1, including: spatiotemporal alignment, denoising, and normalization; Step 3, feature fusion: Perform feature fusion on the data preprocessed in step 2, map the multimodal data into a unified feature space, and receive feedback updates in real time; Step 4, intent classification: The multimodal data fused in step 3 is classified into multiple categories and the probability distribution of intent is output through the spatiotemporal Transformer network. Step 5, motion prediction: Perform motion prediction based on the probability obtained in step 4, generate future trajectory distribution based on Gaussian process regression, and receive feedback updates in real time; Step 6: Generate security domain: Combine the predicted trajectory in step 5 with physiological constraints to construct a time-varying safe space; Step 7, optimization solution: Use model predictive control algorithm to solve the optimal control sequence and send it to execution, and implement the update after receiving feedback; Step 8, execution control: Send the control instruction obtained in step 7 to the actuator; Step 9, Feedback Update: Real-time data is used to update the online model and transmit to steps 3, 5, and 7.

2. The adaptive safety planning method based on multimodal intention perception and physiological signals according to claim 1 is characterized in that: The multimodal data in step 1 includes: Visual data: 3D depth camera collects the coordinates of skeletal joints at a certain frequency; IMU data: Multi-axis sensors collect acceleration and angular velocity signals at a certain frequency; Physiological data: surface electromyography collects electromyographic signals from multiple locations at a certain frequency and collects heart rate signals at a certain frequency.

3. The adaptive safety planning method based on multimodal intention perception and physiological signals according to claim 1 is characterized in that: The data preprocessing method in step 2 is specifically as follows: The heterogeneous data collected in step 1 are temporally and spatially aligned, with the time synchronization error controlled within ±10ms, and frequency differences are compensated by interpolation. Gaussian filtering is used to denoise the visual data; Kalman filtering is used to smooth the IMU data; Butterworth bandpass filtering and multi-point noise reduction with a moving average window are used for physiological signals; and each modality data is standardized to a unified scale using the following formula: ； Where, For the data after standardization, For the collected data, is the mean of the collected data, is the standard deviation of the collected data.

4. The adaptive safety planning method based on multimodal intention perception and physiological signals according to claim 1 is characterized in that: The feature fusion method in step 3 is specifically as follows: Visual features are extracted from each modality after processing in step 3, where the ResNet-50 fully connected layer is 128-dimensional, the IMU temporal features are encoded in two layers of LSTM to 64 dimensions, and the physiological spectrum features are compressed to 64 dimensions through FFT+wavelet transform. The multi-source features are mapped to a 32-dimensional fusion space through an autoencoder and then linearly mapped to a 256-dimensional unified feature vector.

5. The adaptive safety planning method based on multimodal intention perception and physiological signals according to claim 1 is characterized in that: The intention classification method in step 4 is specifically as follows: The feature information fused in step 3 is classified using an FST-Trans network with an input dimension of 256 and four layers of spatiotemporal attention modules. Each layer can independently learn and adjust important spatial positions and time points in the features, outputting a probability distribution of five types of intentions, including walking, raising hands, stretching, grabbing things, and bending over.

6. The adaptive safety planning method based on multimodal intention perception and physiological signals according to claim 1 is characterized in that: The human body motion prediction method in step 5 specifically includes: The intent probability after classification in step 4 is used as input. Based on Gaussian process regression, with an RBF kernel, a length scale of 0.5, and a variance of 1.0, the trajectory distribution in the next 1-2 seconds is predicted. New motion data is collected in real time, the trajectory prediction is continuously adjusted and optimized, and feedback updates are implemented.

7. The adaptive safety planning method based on multimodal intention perception and physiological signals according to claim 1 is characterized in that: The security domain generation method in step 6 specifically includes: According to the predicted trajectory and physiological parameters in step 5, a dynamic safety range is set; considering the physiological limits of the human body and the environmental safety requirements, a time-varying spatial boundary is formed; and during motion and control, human behavior is ensured to be within a safe range.

8. The adaptive safety planning method based on multimodal intention perception and physiological signals according to claim 1 is characterized in that: The optimization solution method in step 7 specifically includes: The model predictive control algorithm is used to solve the optimal control sequence and implement feedback updates. The model predictive control is used to calculate the optimal control instructions within a period of time based on the current state and target. The safety domain threshold is dynamically adjusted based on the user's electromyographic signal in actual operation. δ ( t ) is used to adjust model parameters and control sequences to ensure the adaptability and effectiveness of control.

9. The adaptive safety planning method based on multimodal intention perception and physiological signals according to claim 1 is characterized in that: The control method executed in step 8 specifically includes: The control signal obtained in step 7 is converted, safety is detected, communication signals are transmitted, real-time feedback monitoring status is provided, anomalies are processed and optimized recalculation is performed to achieve safe, stable and efficient closed-loop control of the robotic arm and execution equipment.

10. The adaptive safety planning method based on multimodal intention perception and physiological signals according to claim 1, characterized in that: The feedback updating method in step 9 specifically includes: Continuously collect and preprocess the robot and sensor data, perform filtering, normalization and exception handling, and transmit them to steps 3, 4, and 7 for online model update; use the dynamic model to predict the state prior, and then combine it with the current observation data to update the fusion state through Bayesian update to obtain the updated state; then use it as the initial state for the next control cycle, adjust the strategy and monitor uncertainty, and recalibrate parameters when necessary; perform model predictive control optimization and real-time closed-loop control based on the updated state to ensure safe operation of the system; cyclically execute prediction, observation, update and optimization, while recording logs and handling exceptions, and triggering safety strategies when necessary.

Images