Natural language visual reasoning method based on attention mechanism

Abstract

The invention provides a natural language visual reasoning method based on an attention mechanism. The method mainly comprises a language analysis processing module and three visual processing modules, and comprises the following steps of: firstly, inputting a language expression, processing by utilizing one-hot coding, BiLSTM coding and the like, and calculating phrase embedding expressions and weights for the three visual processing modules according to the language expression; then, a Mask R-CNN detector is used for carrying out target detection on the input image, detection results are respectively input into a subject module, a position module and a relation module, and each module respectively calculates a matching score of the subject module, the position module and the relation module; and finally, calculating the weighted sum of the matching scores of the three modules as an overall matching score, taking the candidate object with the highest overall matching score as an object described by a language expression, and outputting a position frame of the object to complete visual reasoning of the image. The method has better contextual information understanding capability, and expressions of various structures can be processed.

Description

technical field [0001] The invention belongs to the technical fields of computer vision and natural language processing, and in particular relates to an attention mechanism-based natural language visual reasoning method. Background technique [0002] Referential expression understanding refers to locating object regions described by natural language in images. That is: input a picture (including people or other objects), enter a natural language description (referential expression) that can identify a specific object in the picture, the description is an English word, phrase or sentence, and can include the category of the object , position, color, size, and relationship to surrounding objects. It is required to locate the region of the described object in the picture (frame the object with a bounding box and segment it). Referential expression comprehension is a meaningful task that can be applied to image retrieval, such as finding objects with specific attributes in an ...

AI Technical Summary

Problems solved by technology

[0004] In order to overcome the problem of insufficient or inaccurate understanding of the context information of referring expressions in the prior art, the present invention provides a natural language visual reasoning method based on an attention mechanism

Images