Method and system for generating multimodal charts into text by integrating key chart data

Abstract

The present invention proposes a multimodal chart-to-text generation method and system that integrates key chart data. The method extracts key data and visual features from the chart, and then uses a multimodal attention network and a gate fusion mechanism to fuse the features of the two modalities to fully learn the key semantic features and global visual features of the chart. Finally, the fused features are decoded and a chart description text with high fluency and factual consistency is generated. The present invention adopts a large language model semantic enhancement and multimodal learning method, so that the model can more fully understand and learn the semantics conveyed by the chart. The model is easy to build, the generated text is highly fluent, contains more and more accurate facts, and the text content is more concise.

Images