Visual dialogue generation system based on semantic alignment

Abstract

The invention relates to a visual dialogue generation system based on semantic alignment. According to the invention, the image information is extracted from two aspects, i.e., a global image information and a local image information. Global image representation based on semantics is obtained through semantic alignment, meanwhile, local dense image description is obtained through dense caption, and high-level semantics of text representation is beneficial to better information acquisition. The two jointly provide clues of image information for generating replies. Meanwhile, comprehensive constraint is carried out from the aspects of text fluency, text coherence and correctness, and generation of replies is guided. In addition, the embodiment of the invention provides a keyword constraint method to constrain the correctness of the reply, so as to enrich the representation form of the generated reply.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of language processing, and in particular to a visual dialogue generation system based on semantic alignment. Background technique [0002] In recent years, with the rapid development of artificial intelligence and robotics, the multimodal semantic understanding of vision and language has received more and more attention and attention in the fields of computer vision and natural language processing. Human-computer interaction cannot only consider a single mode. In real life, the interaction between people is often not limited to a single text, vision or hearing. The multi-modal natural interaction method can not only realize a more friendly interface between machines and humans, but also is the only way to achieve strong artificial intelligence. [0003] Understanding the real world by analyzing vision and language is the primary task of artificial intelligence to achieve human-lik...

AI Technical Summary

Problems solved by technology

This processing ignores the gap in the representation of different modal information. If the image features and semantic information are not well aligned, can we really obtain sufficient information based on the extracted image features to generate reply, still in doubt

[0008] 2. Too much reliance on conversation history instead of image information to generate responses

However, although many models currently try to obtain more and more targeted information from images, they ignore whether the improvement of the effect is caused by the interference caused by adding too much historical information.

[0009] 3. Not considering the textual quality of generative visual dialogue

[0010] From the above analysis, we can see that the traditional visual dialogue generation system needs to be improved

Images