Multi-modal dialogue system and method guided by user attention

Abstract

The invention discloses a multi-modal dialogue system and method guided by user attention. The multi-modal dialogue system and method can respectively encode multi-modal utterances and generate multi-modal replies by adopting a multi-modal encoder and a multi-modal decoder. The system comprises a data acquisition module, a text feature extraction module, a multi-mode encoder and a multi-mode decoder; the data acquisition module acquires text information and visual image information of the commodity; the text feature extraction module generates attention-weighted text features; the multi-mode encoder extracts visual features of the visual image by adopting a convolutional neural network model, and inputs the visual features into the classification-attribute combination tree for traversal toobtain attribute-level visual features; multi-modal decomposition bilinear pooling processing is performed on the visual features and the text features to generate multi-modal utterance vectors; themulti-modal decoder generates a context vector; and based on the context vector, a visual image and a text attribute of a certain amount of required commodity are selected, the visual image and the text attribute are decoded, and a multi-modal commodity representation is generated.

Description

technical field [0001] The present disclosure relates to the field of language processing, and in particular to a multimodal dialogue system and method guided by user attention. Background technique [0002] Dialogue systems have received increasing attention as an intelligent way to interact with computers. However, most current approaches only focus on text-based dialogue systems, completely ignoring the rich semantics conveyed by vision. In fact, with the rapid development of many fields such as online retail and tourism, the demand for multimodal task-oriented dialogue systems is also growing. Furthermore, few methods explicitly consider the hierarchical structure of item taxonomy and users' attention to items. In fact, as the conversation progresses, users tend to focus on the semantic attributes of items, such as color and style, etc. [0003] During the research and development process, the inventor found that the existing task-oriented dialogue system has the foll...

AI Technical Summary

Problems solved by technology

[0004] (1) Most existing dialogue systems only focus on text, ignoring the fact that people tend to use multimodal information communication;

[0005] (2) In order to get the desired product, the user may pay special attention to some aspects or attributes of the product when interacting with the chat robot, while the existing dialogue system is very limited in the user's attention to the attribute level;

Images