Multi-mode body-equipped intelligent robot control method and device

Abstract

The invention relates to the technical field of body-equipped intelligent robots, in particular to a multi-mode body-equipped intelligent robot control method and device, and the method comprises the steps: synchronously collecting visual, auditory, tactile, force sense and body perception information, and unifying the information to the same time-space reference through a cross-mode time-space stamp alignment mechanism; hierarchical feature extraction and fusion are carried out on the multi-modal information, and unified multi-modal scene state representation is generated; reasoning a decision based on the representation by using a body agent framework, and outputting a control instruction; motion planning and control, visual servo tracking in a non-contact stage and dynamic parameter correction in a contact stage are executed according to instructions; optimizing the multi-modal strategy network through an incremental strategy distillation mechanism based on the interactive data flow; the problem of space-time asynchronization of multi-modal sensing information is solved through a cross-modal space-time stamp alignment mechanism.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligent robot technology, and in particular to a control method and device for a multimodal embodied intelligent robot. Background Technology

[0002] In the practical application of embodied intelligent robots, the spatiotemporal asynchrony of multimodal perception information is the primary challenge. Data such as vision, hearing, touch, force, and proprioception come from different sensors. Due to differences in acquisition frequency, time reference, and spatial coordinate system, there are obvious time delays and spatial offsets, making it difficult to effectively correlate cross-modal information and form a consistent cognition of the environment.

[0003] Traditional methods often employ simple splicing or fixed weighting, which cannot dynamically adjust the weights based on the real-time reliability of different modal data. When a certain modality is disturbed by noise, maintaining a fixed weight will cause deviations in the fusion results. In complex environments, feature conflicts are prone to occur, affecting the accuracy of scene state understanding. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal embodied intelligent robot control method and device, which solves the problem that traditional methods often use simple splicing or fixed weighting, which are prone to feature conflicts in complex environments and affect the accuracy of scene state understanding.

[0005] To achieve the above objectives, the present invention provides a multimodal embodied intelligent robot control method and apparatus, comprising the following steps:

[0006] Simultaneously collect visual, auditory, tactile, force, and proprioceptive information, establish a cross-modal spatiotemporal stamp alignment mechanism, and unify data collected by different sensors to the same spatiotemporal reference through timestamp calibration and spatial coordinate system transformation;

[0007] Hierarchical feature extraction is performed on multimodal information, and a unified multimodal scene state representation is generated through a cross-modal attention fusion module. The unified multimodal scene state representation includes vector encoding of environmental geometric topology, semantic entities, physical attributes, and task intent.

[0008] Based on a unified multimodal scene state representation, the embodied agent framework is used for task reasoning and decision-making. The high-order task objective is decomposed into a sequence of executable atomic actions by a task semantic parser, and the multimodal policy network is called to output control commands in real time.

[0009] Multimodal motion planning and control are executed based on control commands. Visual servo trajectory tracking is used in the non-contact phase, and a tactile-visual fusion closed-loop controller is activated in the contact phase to dynamically correct motion parameters.

[0010] Based on robot interaction data streams, a multimodal policy network is optimized online through an incremental policy distillation mechanism.

[0011] Specifically, in the section "Synchronously acquiring visual, auditory, tactile, force, and proprioceptive information, establishing a cross-modal spatiotemporal stamp alignment mechanism, and unifying data collected by different sensors to the same spatiotemporal reference through timestamp calibration and spatial coordinate system transformation," the cross-modal spatiotemporal stamp alignment mechanism specifically includes:

[0012] The acquisition delay of multimodal information is eliminated by timestamp calibration, and the position of visual target, tactile contact point cloud and sound source orientation are unified to the robot base coordinate system by spatial coordinate system transformation, so as to achieve spatial reference consistency.

[0013] Specifically, in the section "heterogeneous feature extraction of multimodal information, and generation of a unified multimodal scene state representation through a cross-modal attention fusion module, wherein the unified multimodal scene state representation includes vector encodings of environmental geometric topology, semantic entities, physical attributes, and task intent," the method specifically includes:

[0014] Spatiotemporal consistency constraints are constructed, and the pose of visual targets, tactile contact point clouds, and sound source orientations are mapped to the robot's base coordinate system using ontology perception data to ensure the consistency of multimodal features in the spatial dimension.

[0015] A gated cross-modal Transformer is used to calculate adaptive weights between visual and tactile features to generate fused features, where the weights are dynamically adjusted by tactile confidence.

[0016] The output UMSR is a spliced ​​tensor of visual-tactile fusion features, auditory command encoding, and body joint states.

[0017] Specifically, in the section "Output UMSR as a spliced ​​tensor of visual-tactile fusion features, auditory command encoding, and proprioceptive joint states":

[0018] The auditory instruction encoding converts audio instructions into vector features through a semantic embedding model, and the body joint state includes joint position, velocity, and torque information.

[0019] Among them, in the section "Based on a unified multimodal scene state representation, using an embodied agent framework for task reasoning and decision-making, decomposing high-order task objectives into executable atomic action sequences through a task semantic parser, and calling a multimodal policy network to output control commands in real time",

[0020] A dual-stream encoder is employed. The first branch uses a convolutional neural network to process the geometric-semantic features of the unified multimodal scene state representation, thereby extracting environmental topology and entity attributes. The second branch uses a recurrent neural network to process the task objective and historical action sequence to capture task logic and temporal dependencies.

[0021] A deep reinforcement learning network with spatiotemporal convolution is used to generate high-level motion commands and low-level joint torques at the output layer, realizing end-to-end mapping of decision-making and control.

[0022] Knowledge distillation is performed through real-time interactive data streams, and network parameters are updated based on the KL divergence of teacher and student strategies.

[0023] In the section "Multimodal motion planning and control based on control commands, visual servo trajectory tracking in the non-contact phase, and dynamic correction of motion parameters by activating a haptic-visual fusion closed-loop controller in the contact phase":

[0024] Visual servo trajectory tracking acquires the target pose in real time through a visual sensor and calculates motion speed commands based on the image Jacobian matrix to achieve high-precision trajectory tracking.

[0025] On the other hand, the present invention also includes a multimodal embodied intelligent robot control device for executing the multimodal embodied intelligent robot control method, including an information acquisition and alignment module, a feature extraction and fusion module, a task decision module, a motion control module, and a strategy optimization module. The information acquisition and alignment module is connected to the feature extraction and fusion module, the feature extraction and fusion module is connected to the task decision module, the task decision module is connected to the motion control module, and the strategy optimization module is connected to both the task decision module and the motion control module.

[0026] The information acquisition and alignment module is used to synchronously acquire multimodal information and unify it to the same benchmark through spatiotemporal calibration;

[0027] The feature extraction and fusion module is used to extract multimodal features and fuse them to generate a unified scene state representation;

[0028] The task decision module makes decisions based on scenario state reasoning, decomposes tasks, and outputs control commands.

[0029] The motion control module is used to plan motion according to instructions, perform visual tracking when not in contact, and dynamically correct parameters when in contact.

[0030] The strategy optimization module optimizes strategies based on interactive data and improves performance by learning from success and failure cases and human instruction.

[0031] This invention discloses a multimodal embodied intelligent robot control method and device. It synchronously collects visual, auditory, tactile, force, and proprioceptive information, unifying it to the same spatiotemporal reference through a cross-modal spatiotemporal stamp alignment mechanism. Hierarchical feature extraction and fusion of multimodal information generate a unified multimodal scene state representation. Based on this representation, it uses an embodied intelligent agent framework for reasoning and decision-making, outputting control commands. Motion planning and control are executed according to the commands, with visual servo tracking in the non-contact phase and dynamic parameter correction in the contact phase. The multimodal policy network is optimized through an incremental policy distillation mechanism based on interactive data streams. This invention solves the spatiotemporal asynchrony problem of multimodal perception information through the cross-modal spatiotemporal stamp alignment mechanism, avoiding the cross-modal information association problem caused by time delay and spatial offset. During fusion, weights can be dynamically adjusted to reduce fusion deviation when a certain modality is affected by noise, avoiding feature conflicts in complex environments, thereby achieving consistent environmental cognition, improving the accuracy of scene state understanding, and enhancing the robot's applicability in complex environments. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0033] Figure 1 This is a flowchart of the multimodal embodied intelligent robot control method of the present invention.

[0034] Figure 2 This is a schematic diagram of the structure of the multimodal embodied intelligent robot control device of the present invention.

[0035] In the diagram: 101-Information Acquisition and Alignment Module, 102-Feature Extraction and Fusion Module, 103-Task Decision Module, 104-Motion Control Module, 105-Strategy Optimization Module. Detailed Implementation

[0036] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0037] Please see Figure 1 ,in Figure 1 This is a flowchart of a multimodal embodied intelligent robot control method.

[0038] This invention provides a control method for a multimodal embodied intelligent robot, comprising the following steps:

[0039] S100: Simultaneously collects visual, auditory, tactile, force, and proprioceptive information, establishes a cross-modal spatiotemporal stamp alignment mechanism, and unifies data collected by different sensors to the same spatiotemporal reference through timestamp calibration and spatial coordinate system transformation.

[0040] Specifically, timestamp calibration eliminates the acquisition delay of multimodal information, and spatial coordinate system transformation unifies the visual target pose, tactile contact point cloud, and sound source orientation to the robot's base coordinate system, thereby achieving spatial reference consistency.

[0041] In this embodiment, timestamp calibration is required because different sensors have different acquisition frequencies and time bases (e.g., a visual frame rate of 30Hz and a tactile sampling rate of 1kHz). Timestamp calibration is needed to achieve time synchronization of multimodal data. Using the high-precision clock of the robot's main control system (e.g., a UTC timestamp) as the reference, the acquisition time t of each sensor's data is recorded. i (i represents the sensor type, such as vision t) v tactile sensation t (etc.), linear interpolation is used to correct timestamp deviations, mapping all modal data to the same time axis, using the formula t. sync =t i +Δt i , where Δt i The time deviation of sensor i (obtained through pre-calibration experiments, unit: seconds), t sync For a unified timestamp after synchronization;

[0042] Spatial coordinate system transformation: The original spatial coordinate systems of each modality data are different (e.g., vision is based on the image coordinate system, and tactile data is based on the end effector coordinate system). It is necessary to map the data to the robot base coordinate system (a three-dimensional coordinate system with the center of the robot base as the origin) through coordinate transformation.

[0043] Visual target pose transformation: The target's coordinates (u, v, z) in the image coordinate system, acquired by the visual sensor (where z is depth), are transformed using the camera's intrinsic parameter matrix K and extrinsic parameter matrix T. cam→base (Camera-to-base coordinate system transformation) is converted into a 3D pose P in the base coordinate system. base The formula is

[0044] Tactile contact point cloud conversion: Contact point cloud of tactile sensor in end effector coordinate system (P ee The transformation matrix T from the end effector to the base coordinate system. ee→base (Calculated from joint angles perceived by the body) is converted into a point cloud P in the base coordinate system. base The formula is P base =T ee→base ·P ee ;

[0045] Sound source azimuth transformation: The sound source azimuth angle (θ, φ) (horizontal angle and elevation angle) output by the microphone array, combined with the distance estimate d, is converted into three-dimensional coordinates (x, y, z) in the base coordinate system. The formula is as follows: Through the aforementioned timestamp calibration and spatial coordinate system transformation, visual, auditory, tactile, force, and proprioceptive data are unified to the same spatiotemporal reference, laying the foundation for subsequent hierarchical feature fusion and scene state understanding.

[0046] S200: Perform hierarchical feature extraction on multimodal information, and generate a unified multimodal scene state representation through a cross-modal attention fusion module. The unified multimodal scene state representation includes vector encoding of environmental geometric topology, semantic entities, physical attributes, and task intent.

[0047] In this embodiment, hierarchical feature extraction employs differentiated processing based on the characteristics of different modalities: visual information undergoes multi-level feature extraction via a convolutional neural network (CNN). First, shallow convolutional layers extract low-order geometric features such as edges and textures. Then, a deep Transformer encoder captures the spatial relationships and semantic attributes between targets (such as the categories and relative positions of "cup" and "table"), outputting a visual feature tensor. (H and W are feature map dimensions, C) v (Number of channels); tactile information is obtained by collecting the stress distribution matrix through a tactile sensor array, which is then transformed into physical property features of the contact point cloud (such as contact area, average pressure, and pressure gradient) via a fully connected network (MLP), outputting a tactile feature vector. (C t (As a feature dimension); auditory information is converted into an audio feature map through Mel spectrum, and the temporal features of the speech commands are extracted by a CNN-LSTM hybrid network to output an auditory feature sequence. (T is the time step, C) a (Feature dimension); the ontology perception information directly collects joint position, velocity, and torque data, which are then standardized to form ontology feature vectors. (N is the number of joints.)

[0048] S201: Construct spatiotemporal consistency constraints, and use ontology perception data to map the visual target pose, tactile contact point cloud, and sound source orientation to the robot's base coordinate system to ensure the consistency of multimodal features in the spatial dimension.

[0049] In this embodiment, based on real-time joint angle data obtained through body perception, the transformation matrix T from the end effector coordinate system to the base coordinate system is updated in real time using the forward kinematics formula of the robot. ee→base (q) (q is the joint angle vector), i.e., T ee→base (q)=T1(q1)·T2(q2)…TN (q N (T) i (where is the transformation matrix for the i-th joint); the visual target pose P visual tactile contact point cloud P tactile Sound source location P audio The characteristic coordinates P under a unified spatial reference are obtained by mapping the corresponding sensor to the base coordinate system using the transformation matrix from the sensor to the base coordinate system (obtained from the spatial coordinate system transformation in step S100). visual '、P tactile '、P audio ', and L is lost through coordinate deviation. spatial =||P visual '-P tactile '||2+||P visual '-P audio '||2 constrains the spatial consistency of multimodal features.

[0050] S202: A gated cross-modal Transformer is used to calculate adaptive weights between visual and tactile features to generate fused features, where the weights are dynamically adjusted by tactile confidence.

[0051] In this embodiment, the gated cross-modal Transformer introduces a gating mechanism on top of the traditional Transformer's self-attention mechanism to control the contribution ratio of visual and tactile features during the fusion process. First, the visual feature tensor and tactile feature vector are dimensionally matched and encoded to meet the Transformer's input format requirements. Then, the cross-modal attention weights between the two modal features are calculated. These weights are not fixed values ​​but are dynamically determined by the tactile confidence level. The tactile confidence level is usually based on the contact stability of the tactile sensor (e.g., whether the contact area is consistently greater than a threshold, and whether the pressure distribution is uniform). The gating mechanism calculates the tactile confidence based on spatial consistency with visual features (e.g., whether the deviation between the tactile contact point and the visual target pose is within a set range). When tactile confidence is high, the gating mechanism assigns higher weights to tactile features, highlighting their role in physical interaction (e.g., pressure feedback when grasping an object). Conversely, when tactile confidence is low (e.g., unstable contact or conflict with visual information), the tactile weights are reduced, relying more on the environmental geometric and semantic information provided by visual features. Through dynamic weight adjustment, the gating cross-modal Transformer can adapt to the reliability of the two modalities in different scenarios, ultimately generating a fused feature F that takes into account both environmental visual information and physical interaction tactile information. fusion This provides a foundation for the subsequent construction of a unified multimodal scene state representation.

[0052] S203: The output UMSR is a spliced ​​tensor of visual-tactile fusion features, auditory command encoding, and body joint states.

[0053] Specifically, the auditory instruction encoding converts audio instructions into vector features through a semantic embedding model, and the body joint state includes joint position, velocity, and torque information.

[0054] In this embodiment, the generation process of auditory command encoding is as follows: First, the collected audio commands are converted into acoustic feature maps such as Mel spectrum. Then, they are processed by a pre-trained semantic embedding model to map the semantic information in the audio signal into fixed-dimensional vector features, i.e., auditory command encoding. This not only preserves the temporal dynamic features of the voice commands but also forms a unified mapping with the features of visual, tactile, and other modalities in the semantic space, ensuring the semantic consistency of cross-modal information. The body joint state integrates the real-time motion data of each joint of the robot, specifically including the current position, motion speed, and torque of each joint. After standardization, these data form a structured vector that can accurately reflect the robot's own motion state and mechanical feedback. Finally, UMSR uses the visual-tactile fusion features (i.e., the F generated in step S202) to form a structured vector. fusion The above auditory command codes (denoted as Embed(audio)) and proprioceptive joint states (denoted as F) joint It is obtained by concatenating according to the feature dimensions, and its mathematical expression is: Dimension C fusion +C sem +C j (where C) fusion For visual-tactile fusion feature dimensions, C sem For the dimension of auditory instruction encoding, C j (For the dimension of the body joint state), this tensor serves as a unified multimodal scene state representation, encompassing not only the geometric, semantic, and physical attributes of the environment, but also the task intent of the user's instructions and the robot's own motion state. This provides comprehensive and consistent input for the robot's subsequent decision-making and control modules, ensuring that the robot can achieve a precise perception-decision-execution closed loop in complex embodied scenarios.

[0055] S300: Based on a unified multimodal scene state representation, it uses an embodied agent framework for task reasoning and decision-making. It decomposes high-order task objectives into executable atomic action sequences through a task semantic parser and calls a multimodal policy network to output control commands in real time.

[0056] In this embodiment, the embodied agent framework uses UMSR as input. First, a task semantic parser deconstructs the high-order task objective in the user instruction. For example, "put the red cup on the table into the sink" is parsed into a sequence of subtasks such as "identify the cup → plan the grasping path → execute grasping → plan the placement path → execute placement". This parsing process combines a predefined task template library with real-time environmental states (such as the position of the cup and the orientation of the sink) to ensure that the decomposed atomic actions are consistent with the semantics of the current scene. Subsequently, the multimodal policy network dynamically focuses on relevant modal information through an attention mechanism based on the decomposed subtask sequence and the current UMSR, and generates control instructions in real time. The network adopts a hierarchical architecture, with high-level policies responsible for action selection and low-level policies responsible for parameter optimization. The policy parameters are continuously optimized through reinforcement learning. During the decision-making process, the system dynamically evaluates the execution conditions and risks of each atomic action. For example, when tactile feedback indicates unstable grasping, the system increases visual attention weights to reposition the object or adjusts the grasping force. Finally, the generated sequence of control commands is sent to the robot's execution system to achieve closed-loop control from perception to decision-making. Throughout the process, UMSR serves as the core representation, ensuring a unified representation of task semantics and environmental state, enabling the robot to flexibly adjust its strategies in complex and dynamic scenarios and complete embodied interaction tasks.

[0057] S301: Employs a dual-stream encoder. The first branch uses a convolutional neural network to process the geometric-semantic features of the unified multimodal scene state representation, thereby extracting environmental topology and entity attributes. The second branch uses a recurrent neural network to process the task objective and historical action sequence to capture task logic and temporal dependencies.

[0058] In this embodiment, the two branches of the dual-stream encoder specifically process static environmental features and dynamic task features, forming complementary feature representations. The first branch, a convolutional neural network (CNN), takes the environment-related geometric-semantic features from the UMSR as input. These features include information such as object shape, position, and surface hardness obtained from visual-tactile fusion. Through multi-layer convolution and pooling operations, it gradually extracts the topological structure and entity attributes of the environment, ultimately outputting a vector that accurately describes the static features of the current environment, providing basic scene information for task execution. The second branch, a recurrent neural network, focuses on the dynamic process of the task, with inputs including the semantically parsed task objective and the robot's already executed tasks. The historical action sequence captures the logical connections and temporal dependencies of the task through the memory mechanism of the recurrent unit, and outputs feature vectors that reflect the progress and dynamic logic of the task. The output features of the two branches are interacted and fused through an attention mechanism. For example, when the task objective is "placement", the system will pay more attention to the topological features of the "water tank" position in the environment, and adjust the current decision by combining the temporal information of "whether the grasp was successful" in the historical actions. In the end, the comprehensive output of the dual-stream encoder includes both the static attributes of the environment and the dynamic logic of the task, providing more accurate feature support for the action planning of the subsequent multimodal policy network. This ensures that the robot can efficiently perform operations in accordance with the temporal logic of the task while understanding the structure of the environment.

[0059] S302: Employs a deep reinforcement learning network with spatiotemporal convolution, where the output layer simultaneously generates high-level motion commands and low-level joint torques, achieving end-to-end mapping between decision-making and control.

[0060] In this embodiment, the deep reinforcement learning network with spatiotemporal convolution takes the fused features output by the dual-stream encoder as input. The spatiotemporal convolution module captures the spatiotemporal correlation of the data. The reinforcement learning framework optimizes the strategy through interaction with the environment. The reward function takes into account task completion, safety and efficiency. The output layer generates high-level action commands and low-level joint torques at the same time, realizing end-to-end mapping of decision-making and control. It can quickly respond to dynamic changes in the scene (such as object sliding), ensure continuous and stable action, and achieve seamless connection between perception and execution by continuously learning and optimizing the use of spatiotemporal features.

[0061] S303: Knowledge distillation is performed through real-time interactive data streams, and network parameters are updated based on the KL divergence of teacher and student policies.

[0062] In this implementation, the real-time interactive data stream contains multimodal feedback during robot action execution. The knowledge distillation process uses this data to transfer the decision-making experience of a well-trained teacher strategy to the student strategy. Specifically, the teacher strategy generates an optimal decision distribution based on complete UMSR features and historical interaction data. The student strategy then attempts to mimic this distribution, using the KL divergence (a measure of the difference in probability distributions) of the two outputs as a loss function to continuously adjust the student network parameters, ensuring that its real-time output of high-level commands and joint torques is as close as possible to the optimal decision of the teacher strategy. This approach preserves the decision-making accuracy of the teacher strategy while meeting the efficiency requirements of real-time robot control through the lightweight design of the student strategy. Furthermore, the real-time data stream dynamically updates the distillation process, allowing the student strategy to quickly adapt to new scenarios (such as unseen object shapes or complex interactive environments), continuously optimizing control performance and ensuring a balance between accuracy and efficiency in the decision model.

[0063] S400: Based on control commands, it performs multimodal motion planning and control. In the non-contact phase, it uses visual servo trajectory tracking. In the contact phase, it activates a haptic-visual fusion closed-loop controller to dynamically correct motion parameters.

[0064] In this embodiment, during the non-contact phase, visual servo trajectory tracking is primarily relied upon: using visual features in the UMSR as a reference, a reference path is generated through a preset trajectory planning algorithm. The deviation between the current position fed back by the visual sensor and the target position is compared in real time, and the joint motion parameters are adjusted by a PID controller to ensure that the robot moves smoothly along the planned trajectory. At this time, the high-frequency sampling of visual information can provide continuous position correction basis for trajectory tracking. When entering the contact phase, the tactile-visual fusion closed-loop controller is activated: the tactile sensor collects physical interaction data such as contact force and pressure distribution in real time, which is fused with the target relative pose information provided by the visual sensor, and a preset force-position hybrid control model is used. Dynamically correcting motion parameters, for example, if the tactile feedback pressure is insufficient during grasping, the controller will combine the visually confirmed position of the finger-object contact point to fine-tune the attitude of the end effector and increase the gripping force; if the vision detects a slight displacement of the object due to contact force, the controller will judge the object's stability based on the tactile pressure gradient and adjust the motion trajectory in real time to avoid slipping. This phased fusion control strategy utilizes both the accuracy of vision in tracking a wide range of trajectories during non-contact situations and the sensitivity of tactile feedback to physical interaction details during contact situations. Through the unified state reference provided by UMSR, seamless switching and coordination between the two modes are achieved, ensuring that the robot completes its tasks efficiently and safely during complex movements.

[0065] S401: Based on the stress distribution of the tactile sensor array, the differential changes of the tactile signal are converted into real-time trajectory adjustment quantities through proportional-integral control.

[0066] In this embodiment, the stress distribution data collected by the tactile sensor array is analyzed in real time. The differential changes in stress are calculated to reflect the dynamic changes in the interaction state. Then, the proportional-integral control algorithm converts these differential signals into specific trajectory adjustment amounts, enabling the robot to respond quickly to subtle changes in the contact process and correct the motion trajectory in real time to maintain stable interaction, ensuring motion accuracy and safety.

[0067] S402: Construct an impedance model in Cartesian space, update the desired pose based on visual servoing, combine external force information from a six-dimensional force sensor, and achieve compliant contact by adjusting impedance parameters.

[0068] In this embodiment, the impedance model in Cartesian space correlates the pose and force of the robot's end effector, and sets parameters such as target stiffness and damping to simulate the characteristics of "flexible interaction". The visual servo system updates the desired pose in real time based on the visual features in the UMSR to ensure that the motion direction conforms to the task objective. At the same time, the six-dimensional force sensor collects the external force during the contact process. The system dynamically adjusts the impedance parameters to make the end effector produce adaptive deformation or trajectory deviation when subjected to force, so as to achieve compliant contact with the environment, avoid damage to objects or robot failure caused by hard collisions, and balance motion accuracy and interaction safety.

[0069] S403: When the tactile pressure gradient exceeds the threshold, the joint compliance control mode is triggered to reduce joint stiffness and avoid mechanical damage.

[0070] In this embodiment, the pressure gradient of the tactile sensor array is monitored in real time. When the value exceeds a preset threshold, the stiffness is reduced by adjusting the output parameters of the joint drive unit, so that the robot joint exhibits elastic buffering characteristics when subjected to force, reducing the squeezing force on the object or the impact force on its own structure, thereby avoiding damage to the object or mechanical parts caused by hard contact, and maintaining the continuity of interaction while ensuring the safety of the task.

[0071] S500: Based on robot interaction data streams, it optimizes multimodal policy networks online through an incremental policy distillation mechanism.

[0072] In this embodiment, the robot interaction data stream encompasses real-time multimodal perception data, executed actions, and task feedback. The incremental policy distillation mechanism continuously updates the multimodal policy network based on this new data: it compares the current network (student policy) with the better-performing decision patterns (dynamically generated temporary teacher policies) in the newly added data, calculates the difference loss between the two outputs, and only fine-tunes the parameters in the network related to the new scenario, rather than retraining the entire network. This retains the effective policies already learned while quickly absorbing new experiences (such as dealing with new object shapes and interaction patterns in complex environments), achieving continuous online optimization of the policy and enabling the robot to gradually improve its task adaptability and decision-making accuracy over long-term use.

[0073] S501: Store successful and failed trajectories into different playback buffers, and record the environment status, action sequence, and task completion rate.

[0074] In this embodiment, the robot's interaction process is classified and recorded: the complete trajectory of a successful task is stored in the successful replay buffer, and the environmental state during the process is saved in detail; the trajectory of a failed task is stored in the failed replay buffer, and key information and corresponding action instructions at the moment of failure are recorded; effective decision patterns are extracted from successful trajectories, and the causes of errors are analyzed from failed trajectories, providing data support for incremental policy distillation.

[0075] S502: The weight of data sampled from the failure trajectory buffer is higher than that from the success trajectory buffer, and the strategy is improved by focusing on learning from failure cases.

[0076] In this implementation, the weight of the data sampled from the failure trajectory buffer is set higher during policy optimization, so the network will pay more attention to failure cases during training: for example, when calculating the loss function, the decision error corresponding to the failure trajectory will be given a greater weight, forcing the network to learn how to correct the actions that caused the failure first; avoid repeating the same mistakes, and at the same time consolidate the effective decision-making pattern by combining the positive experience of the successful trajectory, so that the multimodal policy network can improve its robustness and task success rate in a shorter time.

[0077] S503: Receives human teaching data and fine-tunes the network through behavioral cloning loss to align the robot's strategy with human intentions.

[0078] In this embodiment, the system receives demonstration action data generated by a human operator through a teaching device (such as a robotic arm teaching device or a VR controller), extracts the joint motion trajectory, end effector pose, and force control parameters as the target output, calculates the difference between the current policy network output and the human-taught action using a behavior cloning loss function, and fine-tunes the network parameters accordingly, enabling the robot to gradually imitate human operation methods. At the same time, it combines multimodal perception data to transform human experience into generalizable decision logic, achieving a deep alignment between the policy and human intent.

[0079] This invention discloses a multimodal embodied intelligent robot control method. Through a cross-modal spatiotemporal stamp alignment mechanism, it utilizes timestamp calibration to eliminate the acquisition delay of multimodal information. Combined with spatial coordinate system transformation, it solves the problems of time delay and spatial offset caused by differences in acquisition frequency, time reference, and spatial coordinate system among different sensors. This enables multimodal data to be effectively correlated under the same spatiotemporal reference, providing a foundation for consistent environmental cognition and avoiding the problem of cross-modal information being unable to be correlated due to spatiotemporal misalignment. Secondly, it employs a hierarchical feature extraction and cross-modal attention fusion module, replacing the traditional simple splicing or fixed-weight fusion method. When a certain modality... When subjected to noise interference, it can automatically reduce its weight to reduce fusion bias, avoid feature conflicts in complex environments, and improve the accuracy of scene state understanding. At the same time, by integrating environmental and task information through the Unified Multimodal Scene State Representation (UMSR), combined with the task reasoning, phased motion control, and incremental policy optimization mechanism of the embodied agent framework, the robot can achieve a precise perception-decision-execution closed loop in complex dynamic scenes. It can not only flexibly cope with unknown scenarios such as new objects and complex environments, but also continuously improve its performance by learning from failure cases and human teaching, significantly enhancing the adaptability and reliability of the robot in practical applications.

[0080] On the other hand, the present invention also includes a multimodal embodied intelligent robot control device, please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram of the control device for a multimodal embodied intelligent robot.

[0081] The multimodal embodied intelligent robot control device of the present invention includes an information acquisition and alignment module 101, a feature extraction and fusion module 102, a task decision module 103, a motion control module 104, and a strategy optimization module 105. The information acquisition and alignment module 101 is connected to the feature extraction and fusion module 102, the feature extraction and fusion module 102 is connected to the task decision module 103, the task decision module 103 is connected to the motion control module 104, and the strategy optimization module 105 is connected to both the task decision module 103 and the motion control module 104.

[0082] The information acquisition and alignment module 101 is used to synchronously acquire multimodal information and unify it to the same benchmark through spatiotemporal calibration;

[0083] The feature extraction and fusion module 102 is used to extract multimodal features and fuse them to generate a unified scene state representation;

[0084] The task decision module 103 makes decisions based on scene state reasoning, decomposes tasks, and outputs control commands.

[0085] The motion control module 104 is used to plan motion according to instructions, perform visual tracking when not in contact, and dynamically correct parameters when in contact.

[0086] The strategy optimization module 105 optimizes the strategy based on interactive data and improves performance by learning from success and failure cases and human instruction.

[0087] In this embodiment, the information acquisition and alignment module 101 ensures the spatiotemporal consistency of sensor data such as vision and touch by using a hardware synchronization clock and coordinate transformation algorithm (as described in step S100); the feature extraction and fusion module 102 uses a gated cross-modal Transformer (as described in step S202) to dynamically fuse multimodal features and output a UMSR containing geometric topology and task intent; the task decision module 103 parses the task timing logic based on a dual-stream encoder (as described in step S301) to generate high-level motion commands and low-level joint torques; the motion control module 104 switches between visual servoing and haptic-visual fusion control according to the contact state (as described in step S400) to achieve trajectory tracking and compliant contact; and the strategy optimization module 105 continuously optimizes the strategy network using an incremental distillation mechanism (as described in step S500) and success / failure cases and human teaching data.

[0088] The above-disclosed embodiments are merely one or more preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments and equivalent changes made in accordance with the claims of this application still fall within the scope of this application.

Claims

1. A control method for a multimodal embodied intelligent robot, characterized in that, Includes the following steps: Simultaneously collect visual, auditory, tactile, force, and proprioceptive information, establish a cross-modal spatiotemporal stamp alignment mechanism, and unify data collected by different sensors to the same spatiotemporal reference through timestamp calibration and spatial coordinate system transformation; Hierarchical feature extraction is performed on multimodal information, and a unified multimodal scene state representation is generated through a cross-modal attention fusion module. The unified multimodal scene state representation includes vector encoding of environmental geometric topology, semantic entities, physical attributes, and task intent. Based on a unified multimodal scene state representation, the embodied agent framework is used for task reasoning and decision-making. The high-order task objective is decomposed into a sequence of executable atomic actions by a task semantic parser, and the multimodal policy network is called to output control commands in real time. Multimodal motion planning and control are executed based on control commands. Visual servo trajectory tracking is used in the non-contact phase, and a tactile-visual fusion closed-loop controller is activated in the contact phase to dynamically correct motion parameters. Based on robot interaction data streams, a multimodal policy network is optimized online through an incremental policy distillation mechanism.

2. The multimodal embodied intelligent robot control method as described in claim 1, characterized in that, In the section "Synchronously acquiring visual, auditory, tactile, force, and proprioceptive information, establishing a cross-modal spatiotemporal stamp alignment mechanism, and unifying data collected by different sensors to the same spatiotemporal reference through timestamp calibration and spatial coordinate system transformation," the cross-modal spatiotemporal stamp alignment mechanism specifically includes: The acquisition delay of multimodal information is eliminated by timestamp calibration, and the position of visual target, tactile contact point cloud and sound source orientation are unified to the robot base coordinate system by spatial coordinate system transformation, so as to achieve spatial reference consistency.

3. The multimodal embodied intelligent robot control method as described in claim 2, characterized in that, In the section "Hierarchical feature extraction of multimodal information, and generation of a unified multimodal scene state representation through a cross-modal attention fusion module, wherein the unified multimodal scene state representation includes vector encodings of environmental geometric topology, semantic entities, physical attributes, and task intent", the method specifically includes: Spatiotemporal consistency constraints are constructed, and the pose of visual targets, tactile contact point clouds, and sound source orientations are mapped to the robot's base coordinate system using ontology perception data to ensure the consistency of multimodal features in the spatial dimension. A gated cross-modal Transformer is used to calculate adaptive weights between visual and tactile features to generate fused features, where the weights are dynamically adjusted by tactile confidence. The output UMSR is a spliced ​​tensor of visual-tactile fusion features, auditory command encoding, and body joint states.

4. The multimodal embodied intelligent robot control method as described in claim 3, characterized in that, In the section "Output UMSR as a spliced ​​tensor of visual-tactile fusion features, auditory command encoding, and proprioceptive joint states": The auditory instruction encoding converts audio instructions into vector features through a semantic embedding model, and the body joint state includes joint position, velocity, and torque information.

5. The multimodal embodied intelligent robot control method as described in claim 4, characterized in that, In the section "Based on a unified multimodal scenario state representation, using an embodied agent framework for task reasoning and decision-making, decomposing high-order task objectives into executable atomic action sequences through a task semantic parser, and calling a multimodal policy network to output control commands in real time," A dual-stream encoder is employed. The first branch uses a convolutional neural network to process the geometric-semantic features of the unified multimodal scene state representation, thereby extracting environmental topology and entity attributes. The second branch uses a recurrent neural network to process the task objective and historical action sequence to capture task logic and temporal dependencies. A deep reinforcement learning network with spatiotemporal convolution is used to generate high-level motion commands and low-level joint torques at the output layer, realizing end-to-end mapping of decision-making and control. Knowledge distillation is performed through real-time interactive data streams, and network parameters are updated based on the KL divergence of teacher and student strategies.

6. The multimodal embodied intelligent robot control method as described in claim 5, characterized in that, In the section "Multimodal motion planning and control based on control commands, visual servo trajectory tracking in the non-contact phase, and dynamic correction of motion parameters by activating a haptic-visual fusion closed-loop controller in the contact phase": Visual servo trajectory tracking acquires the target pose in real time through a visual sensor and calculates motion speed commands based on the image Jacobian matrix to achieve high-precision trajectory tracking.

7. A multimodal embodied intelligent robot control device, used to execute the multimodal embodied intelligent robot control method as described in any one of claims 1-6, characterized in that, It includes an information acquisition and alignment module, a feature extraction and fusion module, a task decision module, a motion control module, and a strategy optimization module. The information acquisition and alignment module is connected to the feature extraction and fusion module, the feature extraction and fusion module is connected to the task decision module, the task decision module is connected to the motion control module, and the strategy optimization module is connected to both the task decision module and the motion control module. The information acquisition and alignment module is used to synchronously acquire multimodal information and unify it to the same benchmark through spatiotemporal calibration; The feature extraction and fusion module is used to extract multimodal features and fuse them to generate a unified scene state representation; The task decision module makes decisions based on scenario state reasoning, decomposes tasks, and outputs control commands. The motion control module is used to plan motion according to instructions, perform visual tracking when not in contact, and dynamically correct parameters when in contact. The strategy optimization module optimizes strategies based on interactive data and improves performance by learning from success and failure cases and human instruction.

Images