Robot operation method based on visual and auditory fusion

Abstract

The invention discloses a robot operation method based on visual and auditory fusion. The robot operation method comprises the following steps of S1, generating a visual scene image; S2, generating a natural language operation instruction; S3, constructing an auditory data set; S4, constructing an indication expression model; S5, constructing an auditory sense prediction model; S6, constructing an auditory recognizer; S7, constructing an operation model; and S8, repeating the steps S5-S7, and generating the action sequences of the multiple mechanical arms circularly. According to the robot operation method, the visual positioning model and the auditory detection model are constructed and integrated on the mechanical arm operation model, so that a robot system can obtain the visual information, understand that the advanced semantic information in a natural language is positioned to a target with an indication expression relation, and distinguish an object which is difficult to distinguish visually by means of the auditory information; the corresponding operations are executed, so that the multi-modal data processing capacity of the robot and the autonomous operation and environment interaction capacity of the robot are achieved, and the interactivity and intelligence of the multi-modal robot system are improved.

Description

Technical field [0001] The present invention relates to the field of robotic control, and in particular, to a robot operating method based on visual hearing fusion. Background technique [0002] In recent years, with the continuous development of artificial intelligence technology, various advanced algorithms are integrated into the robot, making robots more intelligent. More and more industries can see the service robot to produce life, such as medical robots, hotel rooms, logistics transport robots, etc., which also led to a higher demand for robots, and began to gradually asking robots. Understand the scene of the real world, even able to complete the task according to human natural language instructions. However, human natural language contains uncertainty. The real scene is also complex and variable. Only single modal data can only make the robot to complete orientation, for different target operation tasks, require specific sensors to collect More information Copies differe...

AI Technical Summary

Problems solved by technology

The current embodied intelligent robot system is mainly limited to the simulation environment. For the robot to carry out various simulation experiments in the simulation environment, the ideality of each equipment greatly avoids the cost of data collection and ensures the stability of the entire robot experiment. , so it helps the whole experimental process to go smoothly, but for the real-world robot learning exploration, there are huge physical limitations, and there are few applications of multi-modal information on robots.

When faced with multiple scene targets, human natural language often contains high-level semantic information. For this, robots can only realize simple language information processing and feedback, and cannot accurately locate the target

Moreover, in the real physical environment, there are complex and changeable task requirements, and relying on single-modal information is not enough to complete all tasks. For multiple modal information, the existing technology lacks the ability to realize the same robot to collect different modal processing system

For real physical systems, existing methods cannot verify their system functions without limit

Images