A robot control method based on large model driving and multi-modal data fusion

By using multimodal data fusion technology driven by large models, the problem of inaccurate task execution in robot control systems in complex environments has been solved, achieving efficient and accurate task execution and dynamic adaptation.

CN122353579APending Publication Date: 2026-07-10EAST CHINA NORMAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
EAST CHINA NORMAL UNIV
Filing Date
2026-04-15
Publication Date
2026-07-10

AI Technical Summary

Technical Problem

Existing robot control systems struggle to effectively combine visual information with verbal commands in complex and dynamic environments, lacking the ability to deeply fuse multimodal data, resulting in inaccurate task execution and insufficient adaptability.

Method used

A multimodal data fusion method driven by a large model is adopted. Deepseek-V3 is used to enhance the fusion of instructions and scenes. CLIP and T5 are used to extract global and local features, and MMDiT and Mamba L*Blocks are combined to optimize features. Finally, action instructions are generated by bidirectional scanning Mamba and DiT layers.

Benefits of technology

It enables robots to perform tasks accurately and adapt dynamically in complex environments, improves their ability to understand and execute complex instructions, and enhances their autonomous decision-making and execution capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122353579A_ABST
    Figure CN122353579A_ABST
Patent Text Reader

Abstract

This invention discloses a robot control method based on large model-driven and multimodal data fusion. The method specifically includes: a user inputting commands via natural language; the robot acquiring scene information of the current environment through visual sensors; deep enhancement of the user commands using the large model Deepseek-V3, combined with visual scene information for fusion; inputting the fused commands into CLIP and T5 text encoders to extract global and local text features, which are then fused using MMDiT for multimodal processing; optimizing the fused features using Mamba L*Blocks to finally generate robot action commands, and generating specific action plans to execute the task through a DiT layer. This invention, by deeply fusing visual and text data and combining large models for command enhancement and scene understanding, enables more accurate and flexible task execution, overcoming the limitations of traditional methods in complex environments and dynamic tasks, and possessing significant intelligent and adaptive value.
Need to check novelty before this filing date? Find Prior Art