A robot operation method, system and terminal based on a visual language model

Abstract

The application relates to the field of robot intelligent control, and discloses a robot operation method and system based on a visual language model and a terminal, the method comprising the following steps: acquiring a visual image and a natural language instruction, jointly analyzing the visual image and the natural language instruction through a visual language large model to generate an initial planning strategy; controlling a robot to execute an operation action according to the initial planning strategy and collecting original haptic signals in an operation process; inputting the original haptic signals into a haptic language translation model to perform feature extraction and semantic mapping, and obtaining a haptic semantic description; when the haptic semantic description prompts an operation abnormality or a physical attribute that does not conform to a visual expectation, feeding back the haptic semantic description as an enhanced prompt word to the visual language large model; re-inferencing a current task scene according to the visual image and the enhanced prompt word, generating a corrected planning strategy, and controlling the robot to execute an operation action. The application improves the operation precision of the robot.

Images