A robot operation method based on visual and auditory fusion

Abstract

The invention discloses a robot operating method based on visual and auditory fusion. The robot operating method includes the following steps: S1 generates visual scene images; S2 generates natural language operation instructions; S3 constructs an auditory data set; S4 constructs an instruction expression model; S5 constructs Auditory prediction model; S6 builds the auditory recognizer; S7 builds the operation model; S8 repeats S5-S7 to generate action sequences of multiple robotic arms in a loop. The robot operation method integrates the visual positioning model and the auditory detection model into the robot arm operation model, so that the robot system can not only obtain visual information, but also understand the advanced semantic information in natural language to locate the target with an instructional expression relationship. Relying on auditory information to distinguish objects that are difficult to distinguish visually and perform corresponding operations, the robot's multi-modal data processing capabilities, the robot's ability to operate autonomously and interact with the environment are realized, and the interactivity and intelligence of the multi-modal robot system are improved.

Description

technical field [0001] The invention relates to the technical field of robot control, in particular to a robot operation method based on visual and auditory fusion. Background technique [0002] In recent years, with the continuous development of artificial intelligence technology, various advanced algorithms have been integrated into robots, making robots more intelligent. More and more industries can see that service robots are used in production and life, such as medical robots, hotel food delivery robots, logistics transportation robots, etc. Understand real-world scenarios, and even be able to complete tasks based on human natural language instructions. However, human natural language contains uncertainty, and the real scene is also complex and changeable. Only single-modal data can only enable the robot to complete simple tasks of orientation. For different target operation tasks, specific sensor collection is required. Additional information addresses different miss...

AI Technical Summary

Problems solved by technology

The current embodied intelligent robot system is mainly limited to the simulation environment. For the robot to carry out various simulation experiments in the simulation environment, the ideality of each equipment greatly avoids the cost of data collection and ensures the stability of the entire robot experiment. , so it helps the whole experimental process to go smoothly, but for the real-world robot learning exploration, there are huge physical limitations, and there are few applications of multi-modal information on robots.

When faced with multiple scene targets, human natural language often contains high-level semantic information. For this, robots can only realize simple language information processing and feedback, and cannot accurately locate the target

Moreover, in the real physical environment, there are complex and changeable task requirements, and relying on single-modal information is not enough to complete all tasks. For multiple modal information, the existing technology lacks the ability to realize the same robot to collect different modal processing system

For real physical systems, existing methods cannot verify their system functions without limit

Images