A multi-modal dialogue system and method guided by user attention

Abstract

The invention discloses a user attention-guided multi-modal dialogue system and method. A multi-modal encoder and a decoder can be used to encode multi-modal utterances and generate multi-modal replies, respectively. The system includes a data acquisition module, a text feature extraction module, a multimodal encoder and a multimodal decoder; the data acquisition module acquires text information and visual image information of commodities; the text feature extraction module generates attention-weighted text features ; The multi-modal encoder uses the convolutional neural network model to extract the visual features of the visual image, and inputs the visual features into the classification-attribute combination tree for traversal to obtain attribute-level visual features; multi-modal decomposition of visual features and text features Linear pooling processing to generate multi-modal discourse vectors; multi-modal decoders generate context vectors; based on the context vectors, select a certain amount of desired commodity visual images and their text attributes, and decode them to generate multi-modal state of the commodity representation.

Description

technical field [0001] The present disclosure relates to the field of language processing, and in particular to a multimodal dialogue system and method guided by user attention. Background technique [0002] Dialogue systems have received increasing attention as an intelligent way to interact with computers. However, most current approaches only focus on text-based dialogue systems, completely ignoring the rich semantics conveyed by vision. In fact, with the rapid development of many fields such as online retail and tourism, the demand for multimodal task-oriented dialogue systems is also growing. Furthermore, few methods explicitly consider the hierarchical structure of item taxonomy and users' attention to items. In fact, as the conversation progresses, users tend to focus on the semantic attributes of items, such as color and style, etc. [0003] During the research and development process, the inventor found that the existing task-oriented dialogue system has the foll...

AI Technical Summary

Problems solved by technology

[0004] (1) Most existing dialogue systems only focus on text, ignoring the fact that people tend to use multimodal information communication;

[0005] (2) In order to get the desired product, the user may pay special attention to some aspects or attributes of the product when interacting with the chat robot, while the existing dialogue system is very limited in the user's attention to the attribute level;

Images