A method for multimodal data processing of humanoid robots
By employing multimodal data processing methods, synchronous control and circuit safety monitoring of multimodal data for humanoid robots were achieved, solving the problems of control instability and electrical damage caused by inconsistencies in multi-source data, and ensuring the operational reliability and safety of humanoid robots.
Patent Information
- Application Number
- CN202611350600.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-02
- Publication Date
- 2026-11-17
AI Technical Summary
In the existing technology, during the multimodal data processing of humanoid robots, the spatiotemporal reference of multiple source sensors is inconsistent, which leads to semantic conflicts and control jitter. Furthermore, the lack of continuous monitoring of key electrical parameters makes it easy to cause damage and difficult to protect in a timely manner.
A multimodal data processing approach is adopted, including spatiotemporal alignment and feature extraction, deep reinforcement learning fusion decision model, PID motion control strategy and circuit safety monitoring. The spatiotemporal alignment and adaptive calibration module realizes the synchronous control of multimodal data, monitors the circuit safety status in real time, and sets up a multi-level response mechanism to deal with different threat levels.
It achieves efficient processing and synchronous control of multimodal data of humanoid robots, solves the problem of control instability, and ensures the reliability and safety of operation during the execution of complex tasks.
Smart Images

Figure CN123115678A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of humanoid robot technology, and more specifically, to a method for processing multimodal data of humanoid robots. Background Technology
[0002] Humanoid robots are intelligent devices that mimic human form and movement capabilities. They integrate multiple sensors such as vision, force, IMU, and joint encoders. Relying on a perception-decision-control closed loop, they adapt to unstructured human environments in bipedal walking and bi-arm operation, and are an important carrier of embodied intelligence.
[0003] Multimodal data of humanoid robots refers to information from heterogeneous sensors such as vision, force, inertia, and joint position. Due to differences in sampling frequency and coordinate system, these data need to be uniformly calibrated before they can be fused.
[0004] Currently, in the actual processing of multimodal data from humanoid robots:
[0005] ① The spatiotemporal references of multi-source sensors are inconsistent and semantic conflicts are easy to occur, resulting in fusion deviation and control jitter, making it difficult to ensure stable and synchronized humanoid robot movements;
[0006] ② The electrical load of humanoid robots is dynamic and varied, and there is a lack of continuous monitoring of key parameters. Abnormal working conditions can easily cause damage and are difficult to protect in a timely manner.
[0007] Therefore, in view of this, we study and improve the existing structure to provide a multimodal data processing method for humanoid robots, in order to achieve a more practical purpose. Summary of the Invention
[0008] 1. Technical problems to be solved
[0009] To address the problems existing in the prior art, the present invention aims to provide a multimodal data processing method for humanoid robots. This method can achieve efficient processing and synchronous control of multimodal data of humanoid robots, solve the problem of control instability caused by inconsistencies in multi-source data, continuously monitor and comprehensively evaluate key electrical parameters, promptly detect abnormal operating conditions and trigger corresponding protection measures, and comprehensively ensure the operational reliability and safety of humanoid robots during complex task execution.
[0010] 2. Technical Solution
[0011] To solve the above problems, the present invention adopts the following technical solution.
[0012] A method for processing multimodal data of a humanoid robot includes the following steps:
[0013] Step S1: Acquire multimodal sensing data:
[0014] Collect sensor data from multiple modalities of a humanoid robot to construct a multimodal dataset. The sensor data includes visual data, inertial measurement unit data, joint encoder data, and tactile data.
[0015] Step S2, Spatiotemporal Alignment and Feature Extraction:
[0016] Spatiotemporal alignment and feature extraction are performed on the multimodal dataset to obtain a multimodal feature sequence, and the temporal deviation and spatial registration error of each modality are calculated.
[0017] Step S3: Construct a multimodal fusion model:
[0018] A multimodal fusion decision model based on deep reinforcement learning is constructed, which takes multimodal feature sequences as input and generates fusion feature vectors through an adaptive weight allocation strategy.
[0019] Step S4: Determine the data consistency threshold:
[0020] Set a multimodal data consistency threshold. If the time deviation value or spatial registration error value is greater than or equal to the consistency threshold, execute step S5; otherwise, execute step S6.
[0021] Step S5: Calculate the conflict threat value:
[0022] The conflict threat value of multimodal data is calculated by combining the time deviation value and the spatial registration error value;
[0023] Step S6, PID motion control strategy:
[0024] The motion control of a humanoid robot is achieved by using a PID motion controller. The fused feature vector and conflict threat value are used as inputs to the PID motion controller. The fused feature difference and conflict threat value are combined to form an error function, and the control strategy is formulated based on this.
[0025] Step S7: Monitor the circuit safety status:
[0026] Real-time monitoring of circuit safety status, setting circuit safety indicators, and combining the comprehensive values of current, voltage, robot surface temperature, and dangerous vibrations in the circuit to calculate the circuit safety value and determine whether to cut off the power supply to the equipment.
[0027] Furthermore, in step S2, the specific formula for calculating the time deviation value is as follows:
[0028]
[0029] in, This represents the time deviation value of multimodal data. Indicates the total number of modes. Indicates the first The actual timestamp of the acquisition of the modal data. Indicates a reference timestamp. This represents the time deviation weighting coefficient for the i-th mode;
[0030] In step S2, the specific formula for calculating the spatial registration error value is as follows:
[0031]
[0032] in, This represents the spatial registration error value. Indicates the number of spatial feature points. Indicates the first The three-dimensional coordinates of each feature point in the current mode This represents its mapped coordinates in a unified coordinate system. Indicates the first Spatial registration weights for each feature point.
[0033] Furthermore, after spatiotemporal alignment and feature extraction, an online adaptive calibration module is introduced to dynamically compensate for timestamp offsets and spatial calibration parameter drifts caused by temperature drift, mechanical vibration, or long-term operation of the sensor.
[0034] The online adaptive calibration module includes:
[0035] Time calibration unit: based on the calculated time deviation value The Kalman filter is used to predict and compensate the acquisition timestamps of each mode data online, and the calibrated timestamp sequence is output.
[0036] Space calibration unit: based on space registration error value The iterative nearest point algorithm or pose estimation algorithm based on extended Kalman filter is used to update the mapping matrix from each sensor to the unified coordinate system in real time and output the calibrated spatial feature point coordinates.
[0037] Calibration trigger condition: when or When the preset calibration threshold is exceeded continuously, the calibration process is automatically started, and the calibrated parameters are fed back to step S2 to form a closed-loop optimization.
[0038] Furthermore, in step S3, the adaptive weight allocation strategy includes:
[0039] The weights are dynamically adjusted based on the confidence level of each modality data. The formula for generating the fused feature vector is as follows:
[0040]
[0041]
[0042] in, Represents the fused feature vector. Indicates the number of modes. Indicates the first Feature vectors extracted from each modality Indicates the first The weights of each modality Indicates the first Confidence level of each modality This represents the temperature coefficient.
[0043] Furthermore, in step S5, the specific formula for calculating the multimodal data conflict threat value is as follows:
[0044]
[0045] in, This represents the threat value for multimodal data conflicts. These represent the weighting coefficients for time bias, spatial registration error, and feature difference, respectively. Indicates the first The modality and the first Euclidean distance between modal eigenvectors.
[0046] Furthermore, in calculating the conflict threat value of multimodal data... Subsequently, a hierarchical response module was added, which, according to... The numerical range is used to classify conflicts into different levels and match corresponding control strategies.
[0047] The hierarchical response module includes:
[0048] Threat Level Classification Unit:
[0049] The threat levels are set to low, medium, and high, including:
[0050] Low-level: ( (for the first threshold)
[0051] intermediate: (T2 is the second threshold);
[0052] advanced: ;
[0053] The strategy matching unit includes:
[0054] Low level: Maintain normal PID control parameters and only log;
[0055] Intermediate: Automatically increase the conflict threat weight in the PID controller And reduce the weight of differences in fusion features. This makes the controller more focused on consistency;
[0056] Advanced: Triggers safety degradation mode, switches to the most reliable single mode in the multi-modal data for open-loop control, and sends a warning signal to step S7, requesting immediate monitoring of the circuit safety status.
[0057] Furthermore, in step S6, the output error function is:
[0058]
[0059] in, express Output error at time 10:00 and These represent the weights for differences in fused features and the weights for conflict threats, respectively. Represents the reference eigenvector;
[0060] In step S6, the calculation of the control quantity of the PID motion controller includes:
[0061]
[0062] in, express Motion control output at all times Indicates the motion proportionality coefficient. This represents the motion integral time constant, where the integral time is from the start time to the current time.
[0063] Furthermore, in step S7, the calculation of the comprehensive value of dangerous vibration includes: setting a peak vibration velocity threshold. When the actual vibration velocity At that time, this velocity will be included in the dangerous vibration velocity sequence. , No. Comprehensive hazardous vibration value of each joint for:
[0064]
[0065] in, This represents the number of dangerous vibration velocities at the j-th joint. This represents the ϕth velocity value in the dangerous vibration velocity sequence of the j-th joint;
[0066] In step S7, the formula for calculating the circuit safety status value is:
[0067]
[0068] in, Indicates the circuit's safety status value. This represents the maximum value in the entire servo driver current data set. This represents the minimum value in the voltage data set. This represents the average temperature value from all temperature sensors. This represents the maximum value of the combined hazardous vibration value for each joint. These are the weights of the corresponding parameters.
[0069] 3. Beneficial Effects
[0070] Compared with the prior art, the advantages of this invention are:
[0071] ① In this scheme, the multimodal data processing method for humanoid robots acquires information from multiple types of sensors, such as vision, force, inertial measurement, and joint position, through multimodal data acquisition. It performs unified calibration of different modal data on timestamps and spatial coordinate systems through spatiotemporal alignment. It integrates and dynamically optimizes the aligned multimodal features through deep reinforcement learning fusion. It performs real-time detection and quantitative analysis of possible semantic conflicts or command contradictions between various modal information through conflict threat assessment. This achieves efficient processing and synchronous control of multimodal data of humanoid robots and solves the problem of control instability caused by inconsistent multi-source data.
[0072] ② In this scheme, the multimodal data processing method for humanoid robots, on the basis of achieving efficient data processing and control synchronization, further introduces a circuit safety monitoring mechanism to continuously monitor and comprehensively evaluate key electrical parameters, promptly detect abnormal working conditions and trigger corresponding protection measures, thereby comprehensively ensuring the operational reliability and safety of humanoid robots in the process of performing complex tasks. Attached Figure Description
[0073] Figure 1 This is a flowchart illustrating the multimodal data processing method for humanoid robots in this invention.
[0074] Figure 2 This is a system schematic diagram of the online adaptive calibration module in this invention;
[0075] Figure 3 This is a system schematic diagram of the hierarchical response module in this invention. Detailed Implementation
[0076] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0077] Example 1:
[0078] Please see Figure 1 A method for processing multimodal data of a humanoid robot includes the following steps:
[0079] Step S1: Acquire multimodal sensing data:
[0080] Collect sensor data from multiple modalities of a humanoid robot to construct a multimodal dataset. The sensor data includes visual data, inertial measurement unit data, joint encoder data, and tactile data.
[0081] Step S2, Spatiotemporal Alignment and Feature Extraction:
[0082] Spatiotemporal alignment and feature extraction are performed on the multimodal dataset to obtain a multimodal feature sequence, and the temporal deviation and spatial registration error of each modality are calculated.
[0083] Specifically, the formula for calculating the time deviation value is as follows:
[0084]
[0085] in, This represents the time deviation value of multimodal data. Indicates the total number of modes. Indicates the first The actual timestamp of the acquisition of the modal data. Indicates a reference timestamp. This represents the time deviation weighting coefficient for the i-th mode.
[0086] By introducing a time deviation calculation formula, the degree of time asynchrony between different modal data is quantified, providing accurate input for subsequent consistency judgment and conflict threat assessment, and improving the time synchronization accuracy of multimodal data fusion.
[0087] Specifically, the formula for calculating the spatial registration error value is as follows:
[0088]
[0089] in, This represents the spatial registration error value. Indicates the number of spatial feature points. Indicates the first The three-dimensional coordinates of each feature point in the current mode This represents its mapped coordinates in a unified coordinate system. Indicates the first Spatial registration weights for each feature point.
[0090] By using the spatial registration error formula, the spatial alignment of multimodal data was accurately evaluated, effectively reducing spatial misalignment caused by different sensor installation positions or viewing angles, and improving the geometric consistency of multimodal feature fusion.
[0091] Step S3: Construct a multimodal fusion model:
[0092] A multimodal fusion decision model based on deep reinforcement learning is constructed, which takes multimodal feature sequences as input and generates fusion feature vectors through an adaptive weight allocation strategy.
[0093] Specifically, adaptive weight allocation strategies include:
[0094] The weights are dynamically adjusted based on the confidence level of each modality data. The formula for generating the fused feature vector is as follows:
[0095]
[0096]
[0097] in, Represents the fused feature vector. Indicates the number of modes. Indicates the first Feature vectors extracted from each modality Indicates the first The weights of each modality Indicates the first Confidence level of each modality This represents the temperature coefficient.
[0098] By employing confidence-based adaptive weight allocation and Softmax normalization, high-confidence modes occupy a larger proportion in the fusion, enhancing the robustness of the fused features to noise and outlier data, and improving the accuracy of the decision model.
[0099] Step S4: Determine the data consistency threshold:
[0100] Set a multimodal data consistency threshold. If the time deviation value or spatial registration error value is greater than or equal to the consistency threshold, execute step S5; otherwise, execute step S6.
[0101] Step S5: Calculate the conflict threat value:
[0102] The conflict threat value of multimodal data is calculated by combining the time deviation value and the spatial registration error value;
[0103] Specifically, the formula for calculating the multimodal data conflict threat value is as follows:
[0104]
[0105] in, This represents the threat value for multimodal data conflicts. These represent the weighting coefficients for time bias, spatial registration error, and feature difference, respectively. Indicates the first The modality and the first Euclidean distance between modal eigenvectors.
[0106] This conflict threat value formula integrates inconsistencies at the temporal, spatial, and feature levels, and can comprehensively quantify the degree of conflict between multimodal data, providing an effective basis for PID controller adjustment and thus reducing the risk of oscillation.
[0107] Step S6, PID motion control strategy:
[0108] The motion control of a humanoid robot is achieved by using a PID motion controller. The fused feature vector and conflict threat value are used as inputs to the PID motion controller. The fused feature difference and conflict threat value are combined to form an error function, and the control strategy is formulated based on this.
[0109] Specifically, the output error function is:
[0110]
[0111] in, express Output error at time 10:00 and These represent the weights for differences in fused features and the weights for conflict threats, respectively. This represents the reference eigenvector.
[0112] This enables the PID controller to respond simultaneously to data fusion quality and data consistency status, achieving more precise motion control and reducing control deviations caused by multimodal conflicts.
[0113] Specifically, the calculation of control quantities for a PID motion controller includes:
[0114]
[0115] in, express Motion control output at all times Indicates the motion proportionality coefficient. This represents the motion integral time constant, where the integral time is from the start time to the current time.
[0116] By introducing a nonlinear PID control law with a square root term, the ability to suppress large errors is enhanced, while integral saturation is avoided, thus improving the smoothness and response speed of the humanoid robot's motion.
[0117] Step S7: Monitor the circuit safety status:
[0118] Real-time monitoring of circuit safety status, setting circuit safety indicators, and combining the comprehensive values of current, voltage, robot surface temperature, and dangerous vibrations in the circuit to calculate the circuit safety value and determine whether to cut off the power supply to the equipment.
[0119] Specifically, the calculation of the comprehensive value of hazardous vibration includes: setting the peak vibration velocity threshold. When the actual vibration velocity At that time, this velocity will be included in the dangerous vibration velocity sequence. , No. Comprehensive hazardous vibration value of each joint for:
[0120]
[0121] in, This represents the number of dangerous vibration velocities at the j-th joint. This represents the ϕth velocity value in the dangerous vibration velocity sequence of the j-th joint.
[0122] By extracting and accumulating dangerous vibration velocities exceeding peak thresholds, the degree of abnormal vibration of robot joints is effectively quantified, providing key vibration parameters for circuit safety status assessment and further improving the reliability of safety monitoring.
[0123] Specifically, the formula for calculating the circuit safety status value is as follows:
[0124]
[0125] in, Indicates the circuit's safety status value. This represents the maximum value in the entire servo driver current data set. This represents the minimum value in the voltage data set. This represents the average temperature value from all temperature sensors. This represents the maximum value of the combined hazardous vibration value for each joint. These are the weights of the corresponding parameters.
[0126] This circuit safety value formula integrates multiple safety indicators such as current, voltage, temperature, and vibration, enabling real-time and comprehensive assessment of the circuit's safety status, timely disconnection of power supply to prevent equipment damage, and ensuring the safe operation of the humanoid robot.
[0127] Example 2:
[0128] Based on the above embodiment 1, further description is provided.
[0129] See Figure 2 Specifically, after spatiotemporal alignment and feature extraction, an online adaptive calibration module is introduced to dynamically compensate for timestamp offsets and spatial calibration parameter drifts caused by temperature drift, mechanical vibration, or long-term operation of the sensor. The online adaptive calibration module includes:
[0130] Time calibration unit: based on the calculated time deviation value The Kalman filter is used to predict and compensate the acquisition timestamps of each mode data online, and the calibrated timestamp sequence is output.
[0131] Space calibration unit: based on space registration error value The iterative nearest point (ICP) algorithm or pose estimation algorithm based on extended Kalman filter (EKF) is used to update the mapping matrix of each sensor to a unified coordinate system in real time and output the calibrated spatial feature point coordinates.
[0132] Calibration trigger condition: when or When the preset calibration threshold is exceeded continuously, the calibration process is automatically started, and the calibrated parameters are fed back to step S2 to form a closed-loop optimization.
[0133] By introducing an online adaptive calibration module, timestamp offsets and spatial calibration parameter drifts caused by temperature drift, vibration, or long-term operation of the sensors can be dynamically compensated, further improving the accuracy and long-term stability of spatiotemporal alignment of multimodal data. This provides a more reliable input for subsequent deep reinforcement learning fusion models, thereby enhancing the robustness and adaptability of the entire humanoid robot control system.
[0134] See Figure 3 Specifically, in calculating and obtaining the conflict threat value of multimodal data... Subsequently, a hierarchical response module was added, which, according to... The numerical range is used to classify conflicts into different levels and match corresponding control strategies.
[0135] The hierarchical response module includes:
[0136] Threat Level Classification Unit:
[0137] The threat levels are set to low, medium, and high, including:
[0138] Low-level: ( (for the first threshold)
[0139] intermediate: (T2 is the second threshold);
[0140] advanced: ;
[0141] The strategy matching unit includes:
[0142] Low level: Maintain normal PID control parameters and only log;
[0143] Intermediate: Automatically increase the conflict threat weight in the PID controller And reduce the weight of differences in fusion features. This makes the controller more focused on consistency;
[0144] Advanced: Triggers a safety degradation mode, switches to the most reliable single mode (such as vision only or inertial measurement only) in the multimodal data for open-loop control, and sends a warning signal to step S7, requesting immediate monitoring of the circuit safety status.
[0145] By adding a conflict threat classification response module, we have achieved refined management and differentiated response to the degree of conflict in multimodal data, avoiding the problem of response lag or oversensitivity caused by single threshold processing.
[0146] Switching to single-mode open-loop control and linking circuit safety monitoring under advanced threats effectively prevents robot loss of control or hardware damage caused by severe data conflicts, further ensuring the operational safety and reliability of humanoid robots.
[0147] Example 3:
[0148] Further description is provided based on the above embodiments 1 and 2.
[0149] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concepts, should be covered within the scope of protection of the present invention.
Claims
1. A method for processing multimodal data of a humanoid robot, characterized in that: Includes the following steps: Step S1: Acquire multimodal sensing data: Collect sensor data from multiple modalities of a humanoid robot to construct a multimodal dataset. The sensor data includes visual data, inertial measurement unit data, joint encoder data, and tactile data. Step S2, Spatiotemporal Alignment and Feature Extraction: Spatiotemporal alignment and feature extraction are performed on the multimodal dataset to obtain a multimodal feature sequence, and the temporal deviation and spatial registration error of each modality are calculated. Step S3: Construct a multimodal fusion model: A multimodal fusion decision model based on deep reinforcement learning is constructed, which takes multimodal feature sequences as input and generates fusion feature vectors through an adaptive weight allocation strategy. Step S4: Determine the data consistency threshold: Set a multimodal data consistency threshold. If the time deviation value or spatial registration error value is greater than or equal to the consistency threshold, execute step S5; otherwise, execute step S6. Step S5: Calculate the conflict threat value: The conflict threat value of multimodal data is calculated by combining the time deviation value and the spatial registration error value; Step S6, PID motion control strategy: The motion control of a humanoid robot is achieved by using a PID motion controller. The fused feature vector and conflict threat value are used as inputs to the PID motion controller. The fused feature difference and conflict threat value are combined to form an error function, and the control strategy is formulated based on this. Step S7: Monitor the circuit safety status: Real-time monitoring of circuit safety status, setting circuit safety indicators, and combining the comprehensive values of current, voltage, robot surface temperature, and dangerous vibrations in the circuit to calculate the circuit safety value and determine whether to cut off the power supply to the equipment.
2. The method for processing multimodal data of a humanoid robot according to claim 1, characterized in that: In step S2, the specific formula for calculating the time deviation value is as follows: in, This represents the time deviation value of multimodal data. Indicates the total number of modes. Indicates the first The actual timestamp of the acquisition of the modal data. Indicates a reference timestamp. This represents the time deviation weighting coefficient for the i-th mode; In step S2, the specific formula for calculating the spatial registration error value is as follows: in, This represents the spatial registration error value. Indicates the number of spatial feature points. Indicates the first The three-dimensional coordinates of each feature point in the current mode This represents its mapped coordinates in a unified coordinate system. Indicates the first Spatial registration weights for each feature point.
3. The method for processing multimodal data of a humanoid robot according to claim 2, characterized in that: After spatiotemporal alignment and feature extraction, an online adaptive calibration module is introduced to dynamically compensate for timestamp offset and spatial calibration parameter drift caused by temperature drift, mechanical vibration or long-term operation of the sensor. The online adaptive calibration module includes: Time calibration unit: based on the calculated time deviation value The Kalman filter is used to predict and compensate the acquisition timestamps of each mode data online, and the calibrated timestamp sequence is output. Space calibration unit: based on space registration error value The iterative nearest point algorithm or pose estimation algorithm based on extended Kalman filter is used to update the mapping matrix from each sensor to the unified coordinate system in real time and output the calibrated spatial feature point coordinates. Calibration trigger condition: when or When the preset calibration threshold is exceeded continuously, the calibration process is automatically started, and the calibrated parameters are fed back to step S2 to form a closed-loop optimization.
4. The method for processing multimodal data of a humanoid robot according to claim 1, characterized in that: In step S3, the adaptive weight allocation strategy includes: The weights are dynamically adjusted based on the confidence level of each modality data. The formula for generating the fused feature vector is as follows: in, Represents the fused feature vector. Indicates the number of modes. Indicates the first Feature vectors extracted from each modality Indicates the first The weights of each modality Indicates the first Confidence level of each modality This represents the temperature coefficient.
5. The method for processing multimodal data of a humanoid robot according to claim 1, characterized in that: In step S5, the specific formula for calculating the multimodal data conflict threat value is as follows: in, This represents the threat value for multimodal data conflicts. These represent the weighting coefficients for time bias, spatial registration error, and feature difference, respectively. Indicates the first The modality and the first Euclidean distance between modal eigenvectors.
6. The method for processing multimodal data of a humanoid robot according to claim 1, characterized in that: In calculating and obtaining the conflict threat value of multimodal data. Subsequently, a hierarchical response module was added, which, according to... The numerical range is used to classify conflicts into different levels and match corresponding control strategies. The hierarchical response module includes: Threat Level Classification Unit: The threat levels are set to low, medium, and high, including: Low-level: ( (for the first threshold) intermediate: (T2 is the second threshold); advanced: ; The strategy matching unit includes: Low level: Maintain normal PID control parameters and only log; Intermediate: Automatically increase the conflict threat weight in the PID controller And reduce the weight of differences in fusion features. This makes the controller more focused on consistency; Advanced: Triggers safety degradation mode, switches to the most reliable single mode in the multi-modal data for open-loop control, and sends a warning signal to step S7, requesting immediate monitoring of the circuit safety status.
7. The method for processing multimodal data of a humanoid robot according to claim 1, characterized in that: In step S6, the output error function is: in, express Output error at time 10:00 and These represent the weights for differences in fused features and the weights for conflict threats, respectively. Represents the reference eigenvector; In step S6, the calculation of the control quantity of the PID motion controller includes: in, express Motion control output at all times Indicates the motion proportionality coefficient. This represents the motion integral time constant, where the integral time is from the start time to the current time.
8. The method for processing multimodal data of a humanoid robot according to claim 1, characterized in that: In step S7, the calculation of the comprehensive value of dangerous vibration includes: setting a peak vibration velocity threshold. When the actual vibration velocity At that time, this velocity will be included in the dangerous vibration velocity sequence. , No. Comprehensive hazardous vibration value of each joint for: in, This represents the number of dangerous vibration velocities at the j-th joint. This represents the ϕth velocity value in the dangerous vibration velocity sequence of the j-th joint; In step S7, the formula for calculating the circuit safety status value is: in, Indicates the circuit's safety status value. This represents the maximum value in the entire servo driver current data set. This represents the minimum value in the voltage data set. This represents the average temperature value from all temperature sensors. This represents the maximum value of the combined hazardous vibration value for each joint. These are the weights of the corresponding parameters.
