Embodied intelligent robot control method and system based on multi-modal perception and adaptive learning
Patent Information
- Application Number
- CN202610967525.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-01
- Publication Date
- 2026-10-27
Smart Images
Figure CN122975631A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot control technology, specifically relating to an embodied intelligent robot control method and system based on multimodal perception and adaptive learning. Background Technology
[0002] When performing contact-intensive tasks such as precision assembly and flexible object grasping, embodied intelligent robots rely on multimodal perception information to understand the environment and make decisions. Existing technologies generally employ a combination of multimodal fusion and reinforcement learning to construct control strategies. However, such approaches typically treat multimodal fusion as an independent information encoding process and decision control as another independent policy mapping process, lacking deep signal interaction between the two.
[0003] First, the uncertainty information generated by multimodal fusion is not fully utilized. When some sensors produce low-quality data due to occlusion, noise, or malfunction, the fusion network can typically only output a "best guess" environmental state representation vector. However, the reliability of this representation and information about which sensing data stream is unreliable are not passed on to the downstream decision-making and learning modules. This leads to the policy network making incorrect decisions based on unreliable perception, while online learning algorithms may incorrectly attribute perceptual noise to changes in environmental dynamics, thus contaminating the internal world model and causing policy oscillations.
[0004] Secondly, the replanning trigger conditions at the task planning level are too simplistic, typically relying solely on the action probability confidence level output by the policy network. This confidence level reflects the policy's cognitive uncertainty regarding the current state-action space and fails to detect dangers caused by sensor drift or physical contact anomalies (such as slippage or jamming). When contact anomalies occur, a simple confidence threshold may not be sufficient to trigger a timely recovery action, leading to task failure or even hardware damage.
[0005] Therefore, a control architecture is needed to extend multimodal uncertainty from the perception layer to the learning and planning layers, so that the system can use the perceived quality as a global signal to adaptively regulate the learning process and planning logic, thereby achieving stable and safe online adaptation in complex physical interactions. Summary of the Invention
[0006] The purpose of this invention is to provide a control method and system for embodied intelligent robots based on multimodal perception and adaptive learning. This invention mainly aims to achieve the stability of online learning and the security of task planning for embodied intelligent robots in the face of uncertainties in multimodal perception and sudden abnormal physical interactions.
[0007] The specific technical solution adopted by this invention is as follows: A control method and system for embodied intelligent robots based on multimodal perception and adaptive learning defines the uncertainty estimate generated by the multimodal fusion module as a "cross-layer shared reliability metric signal," which acts simultaneously on three layers: Adjusting the confidence weights of empirical data in the adaptive learning control module enables closed-loop regulation of the learning rate by perceived quality; The original action confidence scores from the strategy output are aggregated to generate a comprehensive action confidence score, thereby achieving closed-loop control of perceived quality on planning decisions. When tactile / force sensing detects abnormal physical contact, it not only forces a replanning process but also marks the abnormal data and injects it into the experience playback buffer, establishing direct protection of the learning process from physical safety events.
[0008] Specifically, the method includes: establishing a multimodal perception and state representation module, which is configured to receive visual data streams, tactile data streams, force data streams, and audio data streams, and outputting an environmental state representation vector and uncertainty estimates corresponding to each perception data stream through a multimodal fusion network based on cross-modal attention mechanisms and uncertainty estimation. The uncertainty estimates serve as a reliability metric signal shared across layers. An adaptive learning control module is constructed, which internally maintains a world model and policy network updated online based on meta-gradients, and maintains an inner-loop experience replay buffer. The adaptive learning control module is configured to: receive a complete state input formed by concatenating the environmental state representation vector and the robot's current joint state vector; receive the uncertainty estimates and adjust the confidence weights of corresponding interaction data segments in the inner-loop experience replay buffer based on the uncertainty estimates to suppress the contribution of interaction data segments corresponding to high uncertainty estimates in meta-gradient calculation; predict the hidden state and reward based on the complete state input and the currently executed action through the world model, and output action parameters conditioned on the hidden state through the policy network. A dynamic task planner is constructed and configured to: generate a sub-target sequence based on the task objective description and the environmental state representation vector; receive the original action confidence score output by the policy network and aggregate it with the uncertainty estimate to generate a comprehensive action confidence score; trigger the regeneration of the sub-target sequence when the comprehensive action confidence score is lower than a preset replanning threshold; construct a contact stability evaluation index using the tactile and force data streams, and forcibly trigger the regeneration of the sub-target sequence when the change in the contact stability evaluation index exceeds a preset mutation threshold, while generating an anomaly marker signal; in response to the anomaly marker signal, mark the interaction data segment at the trigger moment as an anomalous segment and inject it into the inner loop experience replay buffer to prevent the anomalous segment from dominating the meta-gradient direction of the world model and the policy network. Action parameters are executed through an actuator, forming a perception closed loop from the execution result to the multimodal perception and state representation module.
[0009] Accordingly, this invention provides a control system comprising: a multimodal sensing module, a multimodal fusion and state representation module, an adaptive learning control module, a dynamic task planner, and an execution interface. Crucially, the "uncertainty estimate" output of the multimodal fusion and state representation module is connected to the confidence weight adjustment input of the adaptive learning control module and the confidence aggregation input of the dynamic task planner, respectively; the anomaly labeling signal output of the dynamic task planner is connected to the anomaly injection input of the adaptive learning control module. Thus, the uncertainty signal and the physical anomaly labeling signal form a closed-loop control network across modules.
[0010] The technical effects achieved by this invention are as follows: This invention defines the uncertainty estimate generated by multimodal perception fusion as a "cross-layer shared reliability metric signal" that runs through the perception, learning, and planning layers, thus constructing a closed-loop collaborative control architecture that is not available in existing technologies, and thereby bringing the following beneficial effects: This invention significantly improves the stability of online learning. By using uncertainty estimates to adjust the confidence weights of interactive data segments in the inner loop's experience replay buffer, the contribution of high-noise, low-quality data to meta-gradient updates is automatically suppressed, avoiding perceptual noise contamination of the world model. Experimental data show that the proposed solution reduces the world model prediction error from 0.320 to 0.140, and the meta-gradient variance in high-noise segments is reduced by more than 70%, fundamentally ensuring smooth convergence of the learning process.
[0011] This invention significantly enhances the success rate and safety of complex contact tasks. By aggregating uncertainty estimates with original action confidence scores to generate a comprehensive action confidence score, the task planner can anticipate perceptual quality deterioration and replan in advance. Simultaneously, the contact stability assessment index based on tactile and force perception not only forces replanning upon detecting physical anomalies but also marks anomalous interaction data fragments and injects them into the experience buffer with extremely low weight, preventing sudden anomalies from disrupting the model's long-term knowledge. Experiments show that this invention increases the task success rate from 80.1% to 95.7%, reduces the average recovery steps after encountering lag from 29.5 steps to 11.8 steps, and shortens the number of interaction fragments required for policy convergence by nearly half. These multi-dimensional leaps in improvement stem from the non-obvious technical effects of the synergistic regulation of the perception-learning-planning closed loop. Attached Figure Description
[0012] Figure 1 This is an overall flowchart of an embodied intelligent robot control method based on multimodal perception and adaptive learning according to the present invention. Figure 2 This is a system block diagram of an embodied intelligent robot control system based on multimodal perception and adaptive learning according to the present invention. Detailed Implementation
[0013] To make the objectives and advantages of this invention clearer, the invention will be specifically described below with reference to embodiments. It should be understood that the following text is merely used to describe one or more specific embodiments of the invention and does not strictly limit the scope of protection specifically claimed by the invention.
[0014] like Figures 1-2 As shown, a control method for an embodied intelligent robot based on multimodal perception and adaptive learning includes: Step 1: Establish a multimodal perception and state representation module and complete multimodal fusion and uncertainty estimation.
[0015] The multimodal perception module includes an RGB-D camera, a tactile sensor array, a six-dimensional force / torque sensor, and a microphone array. It collects data at frequencies of 30Hz, 100Hz, 100Hz, and 16000Hz, respectively. All data streams are then input to the multimodal fusion and state characterization module after being aligned with hardware timestamps.
[0016] The multimodal fusion and state representation module's built-in multimodal fusion network, based on cross-modal attention mechanisms and uncertainty estimation, first uses independent feature encoders to encode the visual, tactile, force, and audio data streams into visual feature maps, tactile feature vectors, force feature vectors, and audio feature vectors, respectively. Subsequently, the cross-modal attention computation layer uses the visual feature map as the query and the feature vectors from other modalities as keys and values to compute cross-modal attention-enhanced features.
[0017] Simultaneously, a parallel uncertainty estimation subnetwork receives the intermediate layer activations from each feature encoder and outputs the variance parameters corresponding to each modality. This uncertainty estimation subnetwork consists of two fully connected layers, with the final output being the log-variance logσ². During training, it learns by applying a negative log-likelihood loss, enabling it to accurately reflect the heteroscedastic uncertainty of each modality in the current scenario.
[0018] The multimodal fusion and state representation module uses the reciprocal of the variance parameter of each modality (i.e., accuracy) as the reliability coefficient to perform a weighted summation of the cross-modal attention enhancement features, resulting in a fused feature map. After global average pooling, the output is an environment state representation vector. The variance parameter of each modality is directly used as the uncertainty estimate vector. This uncertainty estimate vector is explicitly labeled as a cross-layer shared reliability metric signal, and is then processed through... Figure 2 The signal path shown is transmitted to the downstream module.
[0019] Step 2: Construct an adaptive learning control module and perform uncertainty-aware online learning.
[0020] The adaptive learning control module internally comprises a world model updated online based on meta-gradients and a policy network. The world model employs a recurrent state-space model (RSSM), and the policy network is a multilayer perceptron (MLP). Simultaneously, the module maintains an inner-loop experience replay buffer with a capacity of 50, storing interaction data fragments generated from recent interactions. Each interaction data fragment includes: the complete state input, the action performed, the immediate reward, the complete state at the next time step, and a vector of uncertainty estimates for that time step.
[0021] After each round of interaction data is generated, the adaptive learning control module performs the following operations: First, the confidence weight of the interactive data segment is calculated. Specifically, as described in claim 3, the average of the uncertainty estimates of the four sensory data streams (visual, tactile, force, and audio) at that moment is taken as the comprehensive uncertainty U of the segment. A decreasing mapping function w = exp(-U² / σ²), where σ is a temperature coefficient set to 0.5, is used to calculate the confidence weight w between 0 and 1. This means that when the uncertainty estimate spikes due to visual occlusion and tactile noise at a certain moment, the confidence weight of the interactive data segment will automatically decay to near 0.
[0022] Then, the interactive data fragments labeled with confidence weights are stored in the inner loop experience replay buffer.
[0023] Once the buffer accumulates more than a batch of 16 data points, the module triggers a primary gradient online update. During the update, a batch is randomly sampled from the buffer. When calculating the prediction loss of the world model and the loss of the policy network, the loss value of each interaction data segment is multiplied by its corresponding confidence weight. Then, a three-step inner loop gradient descent is performed. The resulting gradient is used as the meta-gradient, which is then used by the Adam optimizer to jointly update the backbone parameters of the world model and the policy network.
[0024] The key effect of this mechanism is that when sensor noise is high, the high uncertainty estimate reduces the contribution of relevant experience to the meta-gradient, which is equivalent to telling the learning module "the observations during this period are unreliable, so do not drastically modify your understanding of the physical world based on them", thereby protecting the stability of the internal model.
[0025] Step 3: Build a dynamic task planner and perform dual-triggered safety replanning and exception protection.
[0026] The dynamic task planner receives environmental state representation vectors and uncertainty estimate vectors from the multimodal fusion and state representation module, as well as raw tactile and force data streams from the multimodal perception module.
[0027] Task decomposition phase: The dynamic task planner has a built-in Transformer-based task decomposition network. Based on the given task objective description (e.g., "insert the metal shaft into the bearing housing") and the current environmental state representation vector, it generates a sequence of sub-objectives through autoregression, such as "align above the bearing housing", "insert vertically downward", "maintain contact and apply axial force", etc.
[0028] In the routine replanning monitoring phase: the policy network outputs action parameters and action probability distributions simultaneously. The dynamic task planner takes the action with the highest probability as the original action confidence. At the same time, it calculates the average of the uncertainty estimates of all perceived data streams at the current moment as the perceived uncertainty. The comprehensive action confidence is generated using the formula: Comprehensive Action Confidence = Original Action Confidence × (1 – Perceived Uncertainty). When this comprehensive action confidence falls below the preset replanning threshold of 0.65 for five consecutive control cycles, the dynamic task planner determines that the current sub-objective is unreachable under the current perception conditions, immediately clears the remaining sub-objective sequence, and regenerates the sub-objective sequence based on the latest environmental state representation vector.
[0029] Physical Anomaly Forced Replanning and Protection Phase: The dynamic task planner uses the contact area signal from the tactile data stream and the torque signal from the force data stream to calculate the contact stability assessment index in real time. This index is defined as the weighted sum of the rate of change of the tactile contact area and the amplitude of the force torque fluctuation. The dynamic task planner maintains a sliding window of length 100 online to record the mean and standard deviation of this index under normal operation. When the instantaneous value of the contact stability assessment index exceeds three standard deviations (i.e., the preset mutation threshold), it is determined that an abnormal contact has occurred, such as the object slipping or the insertion process getting stuck.
[0030] At this point, regardless of the overall action confidence level, the dynamic task planner immediately forces the regeneration of the sub-objective sequence and simultaneously performs two operations: generating an anomaly marker signal and binding it to the current interaction data segment; and injecting the marked anomalous interaction data segment into the inner loop experience replay buffer of the adaptive learning control module. In response to the anomaly marker signal, the adaptive learning control module forcibly sets the confidence weight of this anomalous interaction data segment to 0.001. The core value of this mechanism lies in its prevention of training samples generated by physical accidents (erroneous rewards from slippage, abnormal force feedback from jamming) from dominating the gradient direction in subsequent meta-gradient updates, thus avoiding the world model from incorrectly learning "accidental slippage" as a "normal dynamic property of an object."
[0031] Step 4: Execution and Closure.
[0032] The execution interface converts the motion parameters (target pose and contact force of the end effector) output by the policy network into torque commands for the robot joints, driving the actuators to complete the motion. The execution results are then re-acquired by the multimodal perception module, forming a closed loop.
[0033] Example 2: Hardcore Comparative Experiment and Creative Demonstration This invention designs a unified benchmark test task for "precision insertion of highly complex flexible parts" in a high-fidelity simulation environment and on the Franka Emika Panda real robotic arm platform. This task requires the robot to insert shafts of different materials and surface properties (metal, plastic, and rubber) into corresponding holes under the following extreme conditions: random strong light glare or smoke obscuring vision; the surface friction coefficient of the object randomly switching between 0.1 and 0.6; and random external disturbances causing jamming during insertion.
[0034] This invention trained and tested the following three schemes under the exact same hardware and task distribution: Comparative Example 1; Traditional Meta-Learning Control Scheme: A meta-reinforcement learning controller based on MAML, employing a general cross-modal attention network with uncertainty-free estimation for multimodal fusion. The planner relies solely on the raw action confidence output by the policy network for replanning. This scheme represents a typical existing architecture where "perception-learning-planning" are relatively independent.
[0035] Comparative Example 2; Scheme using uncertainty only in the fusion layer: Building upon Comparative Example 1, the fusion network is replaced with a version featuring an uncertainty estimation subnetwork, using the inverse of variance as a reliability coefficient for weighted fusion. However, this uncertainty estimate is discarded after fusion and not passed to the adaptive learning control module and dynamic task planner. The planner still relies solely on the original action confidence. This scheme represents an intermediate approach that uses uncertainty only locally but does not establish a cross-layer collaborative mechanism.
[0036] The present invention provides a complete cross-layer collaborative closed-loop solution: using the complete system described in Embodiment 1, the uncertainty estimate is used as a reliability metric signal shared across layers, which is fully applied to the adjustment of learning weights and the generation of comprehensive action confidence, and a physical anomaly marking protection mechanism is established.
[0037] Each scheme was run independently for 1000 test rounds, with the task parameters (material, occlusion intensity, and perturbation time) randomly generated for each test. The experimental results are summarized in Table 1. Table 1: Summary of Experimental Results; A significant leap in task success rate and anomaly recovery capability. Comparative Example 1 achieved a success rate of only 73.4%, requiring an average of 38.2 steps to recover from stalling, indicating poor robustness of the independent architecture in the face of complex physical interactions. Comparative Example 2, through the local application of uncertainty-weighted fusion, increased the success rate to 80.1% and reduced the recovery steps to 29.5, demonstrating the value of perceiving uncertainty itself. However, the proposed solution pushes the success rate to 95.7% and drastically reduces stall recovery steps to 11.8. This far exceeds the improvement of Comparative Example 2, and the key lies in the establishment of the "physical anomaly-learning protection" collaborative loop: at the moment of stalling, the proposed solution not only immediately triggers replanning (achieving rapid recovery) by contacting stability indicators, but also marks the anomalous interaction data fragment and injects it into the buffer with extremely low weights to prevent the world model from learning incorrect stall dynamics. After stalling, Comparative Example 2's meta-learning mechanism indiscriminately uses high-noise interaction data fragments from the stalling process for world model updates, leading to short-term model disorder and delaying the recovery opportunity. The "isolation" effect of the present invention on abnormal data produces a non-obvious synergistic effect.
[0038] A fundamental improvement in the stability of online learning. This invention reduces the world model prediction error from 0.320 to 0.140, a reduction of 56.3%. More importantly, in the test segment encountering random visual occlusion and tactile noise (accounting for 30% of the entire task time), the meta-gradient variance is only 0.022, far lower than the 0.075 in Comparative Example 2. This strongly demonstrates the effectiveness of the "perception-learning" collaborative loop: the uncertainty estimate, as a global signal, is only used for fusion weighting in Comparative Example 2, offering no protection for the learning process. High-noise data is still indiscriminately used to calculate the meta-gradient, leading to drastic fluctuations in the update direction. In this invention, the high uncertainty estimate automatically attenuates the confidence weight of the corresponding interaction data segment during learning, essentially installing a perception quality gating filter for the world model's learning process. This shift from "post-correction" to "source control" makes the meta-gradient update direction purer and more stable, directly resulting in faster policy convergence speed, reducing the required interaction segments from 68 to 38. This system-level stability gain is something that cannot be achieved by any scheme that only uses uncertainties locally.
[0039] The predictive advantage of integrated action confidence. Experimental log analysis shows that Comparative Example 2, relying solely on the original action confidence to trigger replanning, exhibits a significant "delay" phenomenon. That is, when sensor noise has already distorted the fused state representation vector, the policy network may still not realize the problem and continue to output erroneous actions with high confidence until it receives a negative reward after interacting with the environment, at which point the confidence drops sharply. By then, replanning is too late. The integrated action confidence used in this invention incorporates real-time perception uncertainty into the calculation, essentially equipping the planner with an "instrument," enabling it to anticipate decision-making errors caused by deteriorating sensor quality, thereby triggering replanning in advance. This is another key mechanism by which the present invention maintains a high success rate in non-stationary noise environments.
[0040] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained in this invention are implemented according to conventional methods in the art unless otherwise specified or limited.
Claims
1. A control method for an embodied intelligent robot based on multimodal perception and adaptive learning, characterized in that, include: A multimodal perception and state representation module is established. The multimodal perception and state representation module is configured to receive visual data streams, tactile data streams, force data streams and audio data streams, and output environmental state representation vectors and uncertainty estimates corresponding to each perception data stream through a multimodal fusion network based on cross-modal attention mechanism and uncertainty estimation. The uncertainty estimates are used as a reliability measurement signal shared across layers. An adaptive learning control module is constructed, which internally maintains a world model and policy network updated online based on meta-gradients, and maintains an inner loop experience replay buffer; wherein, the adaptive learning control module is configured as follows: Receive a complete state input formed by concatenating the environmental state representation vector with the robot's current joint state vector; The uncertainty estimate is received, and the confidence weight of the corresponding interactive data segment in the inner loop experience replay buffer is adjusted according to the uncertainty estimate to suppress the contribution of the interactive data segment corresponding to the high uncertainty estimate in the meta gradient calculation. The world model predicts the hidden state and reward based on the complete state input and the currently executed action, and the policy network outputs action parameters conditioned on the hidden state. Construct a dynamic task planner, which is configured as follows: A sequence of sub-objectives is generated based on the task objective description and the environmental state representation vector; Receive the raw action confidence score output by the policy network, and aggregate the raw action confidence score with the uncertainty estimate to generate a comprehensive action confidence score; When the confidence level of the overall action is lower than a preset replanning threshold, the regeneration of the sub-target sequence is triggered; The tactile data stream and the force data stream are used to construct a contact stability evaluation index. When the change of the contact stability evaluation index exceeds a preset mutation threshold, the regeneration of the sub-target sequence is forcibly triggered, and an anomaly marker signal is generated at the same time. In response to the anomaly marking signal, the interactive data fragment at the triggering time is marked as an anomaly fragment and injected into the inner loop experience replay buffer to prevent the anomaly fragment from dominating the meta gradient direction of the world model and the policy network; The actuator executes the action parameters, forming a perception closed loop from the execution result to the multimodal perception and state representation module.
2. The method according to claim 1, characterized in that, The multimodal fusion network based on cross-modal attention mechanism and uncertainty estimation includes a feature encoder independently set for each modality, a cross-modal attention calculation layer, and an uncertainty estimation sub-network. The uncertainty estimation sub-network outputs the variance parameter corresponding to each modality feature. The multimodal fusion and state representation module uses the reciprocal of the variance parameter as a reliability coefficient to perform weighted fusion of the attention enhancement features of each modality to obtain the environment state representation vector. The variance parameter is directly used as the uncertainty estimate.
3. The method according to claim 1, characterized in that, The step of adjusting the confidence weight of the corresponding interactive data segment in the inner loop experience playback buffer based on the uncertainty estimate includes: for each interactive data segment stored in the inner loop experience playback buffer, using the average or maximum value of the uncertainty estimates of multiple sensing data streams at the time of acquisition, to calculate the confidence weight of the interactive data segment through a decreasing mapping function; when sampling from the inner loop experience playback buffer to perform inner loop gradient descent, using the confidence weight to weight the loss.
4. The method according to claim 1, characterized in that, The aggregation of the original action confidence and the uncertainty estimate includes: using the maximum action probability output by the policy network as the original action confidence, using the average of the uncertainty estimates of all perceived data streams at the current time as the perceived uncertainty, and calculating the comprehensive action confidence as the product of the original action confidence and a factor minus the perceived uncertainty.
5. The method according to claim 1, characterized in that, The contact stability assessment index is a weighted sum of the rate of change of tactile contact area and the fluctuation of force torque; the preset mutation threshold is the three standard deviations of the contact stability assessment index under historical normal contact conditions; when the contact stability assessment index exceeds the preset mutation threshold, it is determined that abnormal contact has occurred.
6. A control system for an embodied intelligent robot based on multimodal perception and adaptive learning, characterized in that, include: The multimodal perception module is configured to acquire visual data streams, tactile data streams, force data streams, and audio data streams in real time; A multimodal fusion and state representation module is connected to the multimodal perception module and incorporates a multimodal fusion network based on cross-modal attention mechanism and uncertainty estimation; the multimodal fusion and state representation module is configured as follows: By fusing the multimodal sensing data, an environmental state representation vector is output; The output corresponds to the uncertainty estimate for each sensing data stream, which can be used as a reliability metric signal shared across modules; An adaptive learning control module, connected to the multimodal fusion and state representation module, internally includes a world model and policy network updated online based on meta-gradients, as well as an inner loop experience replay buffer; the adaptive learning control module is configured as follows: Receive a complete state input formed by concatenating the environmental state representation vector with the robot's current joint state vector; The uncertainty estimate is received, and the confidence weight of the corresponding interactive data segment in the inner loop experience replay buffer is adjusted according to the uncertainty estimate to suppress the contribution of the interactive data segment corresponding to the high uncertainty estimate in the meta gradient calculation. The world model predicts the hidden state and reward based on the complete state input and action, and the policy network generates action parameters based on the hidden state. A dynamic task planner is connected to the multimodal fusion and state representation module, the adaptive learning control module, and the multimodal perception module, respectively; the dynamic task planner is configured as follows: Generate a sequence of sub-objectives based on the task objective description; Receive the raw action confidence score and the uncertainty estimate output by the policy network, and aggregate the two to generate a comprehensive action confidence score; When the confidence level of the overall action is lower than a preset replanning threshold, the regeneration of the sub-target sequence is triggered; The tactile data stream and the force data stream are used to construct a contact stability evaluation index. When the change of the contact stability evaluation index exceeds a preset mutation threshold, the regeneration of the sub-target sequence is forcibly triggered. At the same time, an abnormality marker signal is generated and output to the adaptive learning control module. The adaptive learning control module is further configured to: in response to the anomaly marking signal, mark the interactive data fragment at the trigger time as an anomaly fragment and inject it into the inner loop experience replay buffer to prevent the anomaly fragment from dominating the meta gradient direction of the world model and the policy network; An execution interface is connected to the adaptive learning control module and configured to convert the motion parameters into drive commands for the robot's actuators.
7. The system according to claim 6, characterized in that, The multimodal fusion network based on cross-modal attention mechanism and uncertainty estimation includes a feature encoder independently set for each modality, a cross-modal attention calculation layer, and an uncertainty estimation sub-network. The uncertainty estimation sub-network outputs the variance parameter corresponding to each modality feature. The multimodal fusion and state representation module uses the reciprocal of the variance parameter as a reliability coefficient for weighted fusion and outputs the variance parameter as the uncertainty estimate.
8. The system according to claim 6, characterized in that, The adaptive learning control module calculates the meta-gradient for updating the world model and the backbone parameters of the policy network based on the meta-gradient online update by sampling from the inner loop experience replay buffer and performing inner loop gradient descent; when calculating the gradient, the loss of the sampled interaction data segments is weighted using the confidence weights.
9. The system according to claim 6, characterized in that, The dynamic task planner calculates the comprehensive action confidence as the product of the original action confidence and a factor minus the perceived uncertainty by using the maximum action probability output by the policy network as the original action confidence and the average of the uncertainty estimates of all perceived data streams at the current time as the perceived uncertainty.