Object operation instruction following learning method and device based on attention mechanism

Abstract

The invention discloses an object operation instruction following learning method and device based on an attention mechanism. The method comprises the steps: obtaining a target scene image, and extracting the image features of the target scene image; performing feature extraction on the object operation instruction to obtain a text feature of the object operation instruction; fusing the image features and the text features of the object operation instruction through a bilinear model to obtain attention distribution on the image features; the attention distribution acts on the initial image features to be weighted, probability distribution of the operation parameters in respective spaces is obtained according to the weighted image features, and the operation parameter corresponding to the maximum probability value serves as an action parameter. The method solves the problem that an existing object operation instruction following method is low in performance.

Description

technical field [0001] The invention relates to the technical field of machine learning, in particular to an attention mechanism-based object operation instruction following learning method and device. Background technique [0002] Thanks to the rise of deep learning technology and the improvement of computing power, robotics technology has made rapid progress in both theoretical research and industrial applications. Gradually entered the homes of ordinary people. However, what is worrying is that the vast majority of robot applications are still limited to simple and controllable industrial assembly lines. Once entering a dynamic and open environment, especially when faced with complex task goals, robots are often clumsy and unable to make correct decisions autonomously. action. In this case, it is particularly important for humans to communicate with robots through language, command and guide them. In the face of complex tasks, once the robot has the ability to understa...

AI Technical Summary

Problems solved by technology

[0006] For this reason, an object of the present invention is to propose a kind of object operation instruction follow-up learning method based on attention mechanism, this method solves the problem that the performance of existing object operation instruction follow method is low, through attention mechanism, neural network can make neural network The text features of the instruction and the image space features are associated and fused, and an accurate attention map is captured based on the fused features, and the probability distribution of the operation parameters in their respective action spaces is further obtained.

Images