Natural language navigation method based on radar and vision multi-mode fusion

Abstract

The invention provides a natural language navigation method based on radar and vision multi-mode fusion and belongs to the fields of robot navigation, natural language processing and computer vision. The method comprises the following steps of: firstly, installing a laser radar on a robot with a binocular camera, and training a multi-modal fusion neural network model by using the robot; selecting any real scene, issuing a natural language navigation instruction to the robot, and converting the natural language navigation instruction into a corresponding semantic vector; respectively converting RGB images, depth images and radar information acquired by the robot at each moment into corresponding features; performing feature fusion on the semantic vector, a RGB image feature and a depth image feature to obtain an action feature at the current moment; and after correcting the action feature by using radar features, enabling the neural network model finally to output the action of the robot at the current moment, and enabling the robot to execute the action until a navigation task is completed. According to the natural language navigation method based on the radar and vision multi-mode fusion of the invention, the robot can have good perception ability to the real environment, and the efficiency of obstacle avoidance navigation is improved.

Description

technical field [0001] The invention relates to the fields of robot navigation, natural language processing and computer vision, and is a embodied language navigation method based on multimodal fusion of radar and vision. Background technique [0002] The use of natural language for human-computer interaction with computers has long been pursued because of its important theoretical implications as well as its obvious practical implications. Human beings can use their natural language to use computers without spending a lot of time learning various complex computer languages. At present, using natural language to control robots for autonomous navigation is gradually becoming a research hotspot. Researchers hope that in the future, robots can be controlled by natural language to complete navigation tasks, and navigation tasks are also the basis for other complex tasks of robots, which is of great significance to the development of artificial intelligence. [0003] Visual lan...

AI Technical Summary

Problems solved by technology

Most of the current researchers are improving the accuracy of robot visual language navigation in the simulation environment, but in the real scene, due to the complexity of the environment, the robot often encounters obstacles during the navigation process, relying only on vision and depth information , the robot is often unable to avoid obstacles

Moreover, since the binocular camera is usually installed above the robot, the robot cannot observe the obstacles under its feet

Images