To realize improvement of complicated task execution ability under unpredictable circumstance, understanding and execution of high-level abstract instructions, effective integration of multi-
modal feedback, enhancement of ability to adapt itself to change of environment and uncertainty, efficient use of
knowledge base, and securing expandability.SOLUTION: The present invention is directed to a
robot control system which integrates a large-scale
language model, retrieval-enhanced generation technique, a
visual system, a force sense module, a voice recognition and synthesizing module, a multi-
modal integration module, and a
robot control system. A language
processing component recognizes high-level instructions, and dynamically selects and adapts relevant examples from a
knowledge base by using the retrieval-enhanced generation technique. The
visual system generates a three-dimensional expression of an environment. The force sense module measures force of an end
effector. The multi-
modal integration module integrates information from different modalities to realize environment understanding and adaptive action generation. Thus, it is possible to adapt the
system to execution of complicated tasks and change of the environment under an unpredictable environment.SELECTED DRAWING: None