A visual dialogue method based on multimodal semantic alignment of optimal transmission

Abstract

This invention discloses a visual dialogue method based on optimal transmission multimodal semantic alignment. Its key feature is the construction of a model comprising multimodal feature extraction, optimal transmission-based textual semantic alignment, and optimal transmission-based cross-modal semantic alignment. Given an image, a description of that image, and the dialogue history of the past t-1 rounds surrounding that image, the model can select the correct answer from the candidate answer set for the current question in the t-th round. Compared with existing technologies, this invention introduces optimal transmission-based semantic alignment, explicitly providing training signals for intra-modal and inter-modal alignment. This improves the model's ability to align different text entities pointing to the same entity and different modal entities, helping the model better understand textual information and answer questions, thus improving the accuracy of predicted answers. It has high practical value and promising development prospects for visual dialogue in various real-world application scenarios.

Images