Multi-modal chart-to-text generation method and system fusing chart key data

Abstract

The invention provides a multi-modal chart-to-text generation method and system for fusing chart key data, and the method comprises the steps: extracting key data and visual features in a chart, and fusing the features of two modals through a multi-modal attention network and a gate fusion mechanism. Key semantic features and global visual features of the chart are fully learned, and finally the fused features are decoded and a chart description text with high fluency and fact consistency is generated. According to the method, a large language model semantic enhancement and multi-modal learning method is adopted, so that the model can fully understand and learn semantics conveyed by the chart. The model is easy to build, the generated text is high in fluency, the contained facts are more and more accurate, and the text content is simpler.

Description

Technical Field

[0001] The present invention relates to the technical field of converting charts into text, and in particular to a method and system for generating multimodal charts into text by integrating key chart data. Background Art

[0002] Chart-to-text is an interdisciplinary research topic in the fields of natural language processing and computer vision. Its goal is to automatically convert the meaning conveyed by a chart (complex bar chart) into a natural language description that is fluent, concise, faithful to the facts of the chart, and easy to understand.

[0003] Existing chart-to-text technologies still have significant defects when processing complex charts. First, the ability to capture important data in charts is insufficient. When processing complex charts, existing technologies have difficulty accurately capturing important data and the connections between data in complex charts, and there are certain errors in the understanding of charts. Secondly, complex charts contain more visual elements, and existing technologies cannot effectively perceive the semantic information conveyed by chart elements from the perspective of the overall chart, resulting in the generated chart description text being too one-sided or misleading. Overall, although chart-to-text technology has made great progress, there is still much room for improvement in understanding complex charts and generating high-quality text descriptions. Summary of the invention

[0004] In view of the above situation, the main purpose of the present invention is to propose a multimodal chart-to-text generation method and system that integrates key chart data to solve the above technical problems.

[0005] The present invention proposes a multimodal chart-to-text generation method integrating key chart data, the method comprising the following steps: Step 1: Extract the underlying data table of the chart, expand the underlying data table to obtain table serialized text, segment the table serialized text, and then input it into the pre-trained language model BERT to obtain the embedding vector of each subword, obtain the feature representation of each cell according to the embedding vector of each subword, and fuse the cell feature representation with the title information at the corresponding position in the underlying data table to obtain the table data value feature; Step 2: Supplement the semantic information of the title of the underlying data table of the chart to obtain supplementary information of the title, fuse the supplementary information of the title with the characteristics of the table data value, and then pass it through the prediction layer to obtain key data related to the chart; Step 3: Segment the chart image according to fixed pixel length and width to obtain a number of image blocks, flatten the image blocks in the channel direction and input them into the Swin Transformer model to obtain the visual features of the chart image; Step 4: Serialize the key data of the chart according to the template and input it into the BART encoder to obtain text features, and then perform multimodal fusion of the text features and visual features to generate multimodal fusion features; Step 5: Input the multimodal fusion features into the BART decoder for decoding to generate the final text.

[0006] The present invention also proposes a multimodal chart-to-text generation system that integrates chart key data, wherein the system applies the multimodal chart-to-text generation method that integrates chart key data as described above, and the system comprises: Key data extraction module, used for: Extract the underlying data table of the chart, expand the underlying data table to obtain table serialized text, segment the table serialized text, and then input it into the pre-trained language model BERT to obtain the embedding vector of each subword. According to the embedding vector of each subword, obtain the feature representation of each cell, and fuse the cell feature representation with the title information at the corresponding position in the underlying data table to obtain the table data value feature; The semantic information of the title of the underlying data table of the chart is supplemented to obtain the supplementary information of the title, and the supplementary information of the title is integrated with the characteristics of the table data value and then passed through the prediction layer to obtain the key data related to the chart; Image feature extraction module, used for: The chart image is segmented into several image blocks according to fixed pixel length and width, and then the image blocks are flattened in the channel direction and input into the Swin Transformer model to obtain the visual features of the chart image; Multimodal feature fusion module, used for: The key data of the chart is serialized according to the template and input into the BART encoder to obtain text features, and then the text features and visual features are multimodally fused to generate multimodal fusion features; Text generation module for: The multimodal fusion features are input into the BART decoder for decoding to generate the final text.

[0007] Compared with the prior art, the present invention has the following beneficial effects: 1. Combining the local semantic information and global visual information of the chart, the key data is automatically extracted from the complex chart first, and then combined with the complementarity of the chart's visual information, the model fully understands the meaning conveyed by the chart in two modes, thereby generating high-quality chart description text.

[0008] 2. The present invention adds additional supplementary information to the horizontal axis labels and legend labels of the chart through a large language model, which helps the model to more fully understand the meaning conveyed by the chart and can more accurately extract key chart data from complex charts.

[0009] 3. The present invention uses the SwinTransformer model to extract image features. SwinTransformer divides the chart image into non-overlapping blocks, and then processes the image blocks through hierarchical feature representation and a multi-head self-attention mechanism based on sliding windows and a multi-layer perceptron, fully understanding the large number of visual elements in complex charts to generate sufficiently effective visual features.

[0010] 4. The present invention uses multimodal attention and gate fusion mechanism to fuse the information of text modality and visual modality. Through two gate mechanism fusions and one multimodal attention calculation, the purpose of fusing text features and visual features is achieved, providing effective feature representation for the final decoder to generate text.

[0011] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of the multimodal chart-to-text generation method for integrating key chart data proposed by the present invention; Figure 2 The overall framework diagram of the multimodal chart-to-text generation method for integrating chart key data proposed by the present invention; Figure 3 Prompt template for large models; Figure 4 This is the image feature extraction process diagram of the SwinTransformer model; Figure 5 This is the principle diagram of the multimodal fusion mechanism; Figure 6 This is a schematic diagram of the structure of the multimodal chart-to-text generation system that integrates chart key data proposed in the present invention. DETAILED DESCRIPTION

[0013] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0014] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0015] See also Figure 1 and Figure 2 This embodiment provides a method for generating text from a multimodal chart by integrating key chart data, the method comprising the following steps: Step 1: Extract the underlying data table of the chart, expand the underlying data table to obtain table serialized text, segment the table serialized text, and then input it into the pre-trained language model BERT to obtain the embedding vector of each subword, obtain the feature representation of each cell according to the embedding vector of each subword, and fuse the cell feature representation with the title information at the corresponding position in the underlying data table to obtain the table data value feature; The format of the underlying data table extracted from the chart is: the non-column header cell of the first column is the horizontal axis label of the chart (i.e., the row title of each row of the table), and the column header cells of the remaining columns are the legend labels of the chart (i.e., the column titles of each column of the table). The number of columns in the entire chart data table is three or more, and the targeted chart is visually visualized as a complex chart with multiple categories of data. Since complex charts contain a large number of discrete data points, using its underlying data table directly as the input of the model to generate the target text will make it difficult for the model to accurately describe the important data in the chart, and may even cause the generated description text to contain factual inconsistencies. The present invention considers first using a key data extraction model to eliminate the influence of other non-important data in complex charts and improve the accuracy of chart data description.

[0016] Serialize the underlying data table of the chart by row, and express the serialized text as: ; in, Indicates the start position of the table. Represents the column header cell, Indicates the first cell of a row. Represents a data cell, Represents the number of rows and columns in the table respectively.

[0017] Use the BERT tokenizer to tokenize the serialized text to get a token sequence of subwords. The corresponding process has the following relationship: ; Among them, the token sequence of the subword is expressed as , , Indicates subwords.

[0018] Use BERT's pre-trained word embedding matrix to extract word embedding vectors, and input the word embedding into the BERT model to obtain the feature representation of the table serialized text. The process is as follows: ; ; in, Represents the word embedding processing function of the BERT pre-trained model, Represents the word embedding representation of the input token sequence, represents the pre-trained BERT encoder, A feature representation representing the serialized text of a table.

[0019] Since the BERT tokenizer will split a complete word into multiple subwords, it is necessary to fuse the representations of multiple subwords of a word to obtain a complete semantic representation of a word. Here, the subword vectors of each word in the table serialized text are averaged to obtain the feature representation of each word in the text. The corresponding process has the following relationship: ; in, Represents the feature representation of all words in the serialized text, express The k-th word feature in , express The number of subwords to be segmented, express No. Subword features.

[0020] Similarly, the number of words in each cell in the chart data table is not the same. In order to obtain the complete semantics of a cell, it is necessary to fuse the features of all words in the cell. The feature representation of all cells in the table can be obtained by averaging the feature representation of all words contained in each cell in the table serialized text. The corresponding process has the following relationship: ; in, represents the cell feature representation of all cells, express The k Cell features, Indicates the number of words contained in a cell. express The first word feature of the cell, express The number of words in the cell.

[0021] After obtaining the feature representation of all cells in the chart data table, the length of the input feature is equal to the serialization length of the data table. By sorting all cell features according to the position of the cells in the data table and the number of rows and columns of the chart data table, the row title feature, column title feature and table data value feature of the table can be extracted respectively. The corresponding process has the following relationship: ; ; ; in, Represent the row header feature, column header feature, and data value feature respectively. Represents the function for extracting row header features, represents the function for extracting column header features, represents a function that extracts features from data values, and Respectively represent the number of rows and columns in the data table.

[0022] The row header features and column header features corresponding to each data cell in the data table are concatenated with the data value features, and then the obtained high-dimensional features are reduced in dimension through linear transformation to reduce the learning difficulty of the model and increase the training speed. After layer normalization, the final tabular data value features that integrate the row label information and column label information are obtained. The corresponding process has the following relationship: ; ; in, Respectively represent the first The row header characteristics of the row, The column header characteristics, and the Row and The data value characteristics of the column, represents the tensor concatenation function, represents the linear transformation function, represents the normalization function, Indicates the data in the table k The data value feature is the intermediate state after the row header and column header features are concatenated. Indicates the data in the table k The final feature is obtained by transforming the intermediate state of the data value feature.

[0023] Step 2: Design a set of large language model prompt templates for the extracted underlying data table of the chart, embed the data table into the prompt template, and then ask questions to the large language model through the prompt template to obtain the supplementary semantic information of the row and column titles of the chart data table, and obtain the feature representation of the supplementary text information after passing through the text encoder. Fuse the feature representation with the table data value feature, and obtain the key data about the chart through the final prediction layer; The horizontal axis labels and legend labels in complex charts have different semantic information from the chart data labels. Chart data labels and elements provide intuitive data distribution features, while the horizontal axis labels of charts show the logic and internal connection of chart data grouping. Legend labels also show the grouping logic of chart data in another dimension. The semantic order of such labels can indirectly express the meaning that the chart author wants to convey. Since there are many types of horizontal axis labels and legend labels in complex charts, judging whether such labels are ordered requires the model to have strong semantic understanding ability and common sense reserves, and to understand the semantics and relevance of almost all labels appearing in the chart. Large language models can adapt well to such tasks with their large pre-trained corpus, strong semantic understanding and reasoning capabilities. Therefore, special prompt templates can be designed for the horizontal axis labels and legend labels of charts to extract their potential order information from such labels.

[0024] like Figure 3 As shown, firstly, the large language model prompt is automatically constructed according to the row title (horizontal axis label) and column title (legend label) of the chart data table, and the row title and column title are replaced into the prompt template. The prompt template can guide the large language model to complete the extraction of the potential order information of the label. The corresponding process has the following relationship: ; ; in, Indicates the large model prompt constructor, Indicates the first row headers, Indicates the first column headers, Represents the constructed row header prompt and column header prompt respectively.

[0025] The row title prompt and column title prompt are input into the large language model to obtain the supplementary text information of the large model for the row title and column title respectively, and then the obtained supplementary text information is encoded using the text encoder to obtain the supplementary semantic information of the large model about the chart label. The corresponding process has the following relationship: ; ; ; ; in, A function representing large language model processing, and Respectively represent the generation of supplementary text for row and column headers, represents a text encoder, and They represent the encoded row header supplementary information and column header supplementary information respectively.

[0026] After passing through a layer of self-attention network, the table data value features are added to the row title supplementary information and column title supplementary information. The result of the addition is then passed through a layer of self-attention network to fully learn the association between the data value features and the semantic supplementary information of the row title and column title, and the data value feature representation of the fused chart data table is obtained. The corresponding process has the following relationship: ; ; ; in, represents the function of the self-attention network, Represents the data value feature representation of the fused chart data table, and Represent the first implicit state and the second implicit state respectively; the data value feature representation of the fused chart data table is passed through a prediction layer to obtain the key data related to the chart. The corresponding process has the following relationship: ; ; ; in, represents the regularization function, represents the activation function, A function representing the index of positions in the output tensor that are greater than 0.5. Represents a function that retrieves the corresponding data based on the position index. The position index of the graph key representing the prediction, Indicates key data, Represents the third implicit state.

[0027] Step 3: Segment the chart image according to fixed pixel length and width to obtain a number of image blocks, flatten the image blocks in the channel direction and input them into the Swin Transformer model to obtain the visual feature representation of the chart image; like Figure 4 As shown, this embodiment first uses the image processing library PIL to divide the input chart image into image blocks of fixed size. The chart image is as follows The patch size is divided into The corresponding process has the following relationship: ; in, represents a chart image, Indicates image blocks, represents the partition function, Each image block has three channels, so each image block has pixels, and project each image patch onto size, and finally the size is After the feature map passes through multiple Swin Transformer blocks and is merged, it is passed through a multi-layer perceptron to obtain the visual feature representation of the chart. The corresponding process has the following relationship: ; ; in, represents the output features of the SwinTransformer model, Represents the function of the SwinTransformer model for image encoding, represents the multi-layer perceptron function, Represents the visual features of a chart image.

[0028] In this step, by dividing the chart image into blocks of fixed size, the computational complexity can be effectively reduced and the model can be helped to capture local features. Then, by alternating the use of standard sliding window self-attention between different layers of the model, the receptive field of the model can be effectively expanded and the fluidity of information can be enhanced. After each window slides, information can be transferred from one window to its adjacent window, gradually achieving feature fusion in a larger range, and finally generating a global visual feature representation of the chart.

[0029] Step 4: Serialize the key data of the chart according to the template and input it into the BART encoder to obtain text feature representation, then perform multimodal fusion of text features and visual features to generate the final multimodal fusion features; The key data of the chart obtained in step 2 is in a structured form. Using it as the input of the model may make it difficult to accurately capture the relationship and meaning between the data when generating text. Therefore, it is necessary to design a serialization template to convert the obtained structured data into text with a natural language sentence structure, and then input the serialized text into the BART encoder to obtain its text feature representation. The corresponding process has the following relationship: ; ; in, Represents a function serialized according to an artificial template, Represents the serialized text. represents the pre-trained BART encoder processing function, Represents a key data serialized text feature.

[0030] In order to make full use of the visual features and key data of the chart, it is necessary to perform multimodal fusion of the visual features and key data features of the chart, so that the model can perceive the overall trend of the chart data from a visual perspective and the key data information of the chart from a textual perspective. Multimodal fusion of visual features and textual features enables the model to perceive the overall trend of the chart data from a visual perspective and the key data information of the chart from a textual perspective, and obtain multimodal chart features. The corresponding process has the following relationship: ; ; in, A function representing multimodal cross-attention layer processing, represents the gate fusion function, represents the initial fusion feature that combines part of the visual information and part of the text information. Represents the final fused multi-modal graph features.

[0031] like Figure 5 As shown in Figure 1, the calculation process of the multimodal cross-attention layer processing function specifically includes the following steps: First, the initial query, key, and value are calculated using text features. The process has the following relationship: ; in, Represent the weight parameters used to generate the initial query, key, and value, respectively. Represents the initial query, key, and value respectively.

[0032] The initial key and value are fused with the visual features through the gate mechanism to obtain the key and value of the fused part of the visual features. ; in, Respectively represent the key and value of the fused partial visual features, They represent the weight parameters for generating the key and value of the fused partial visual features, Represents the information preservation operation, which is used to control the amount of information retained from the visual modality. The information preservation operation is calculated as follows: ; in, Respectively express The weight parameter for linear transformation, represents the activation function; The initial query and the keys and values ​​of the fused partial visual features are used for attention calculation to obtain the initial fused features: ; in, Represents the initial fusion feature. Finally, the initial information and the visual information of the chart are fused through the gate mechanism. The calculation process of the gate fusion function has the following relationship: ; ; in, Express The weight parameter for linear transformation, Express The weight parameter for linear transformation, represents the bias parameter of the fusion gate, represents the fusion gate, Represents a tensor dot product operation.

[0033] Step 5: Input the fused multimodal features into the BART decoder for decoding to generate the final text.

[0034] The BART decoder is a text generation module with relatively excellent performance in the current text generation field. This module can generate grammatically reasonable and coherent sentences. Its autoregressive decoding characteristics enable it to generate text step by step.

[0035] The decoder first receives an initial input token and receives the multimodal graph features as its internal state. This initial token is the starting point of the decoding process. The decoder then uses an autoregressive method to gradually predict and generate the next word or token based on the generated text content and internal state. Once the next token is selected, the decoder updates its internal state to include the information of the newly generated token. This update process includes updating the hidden state, attention weights, etc., so that text can continue to be generated in the next step. The above steps (from the autoregressive generation mechanism to updating the decoder state) will be repeated until a stopping condition is met (such as a specific end tag is generated). <eos>, reaching the preset generation length limit, etc.). When the generation process stops, the decoder will output all the token sequences from the start token to the end token as the final generated text. Finally, the generated text is post-processed to remove unnecessary spaces and special identifiers, punctuation correction, text formatting, etc., to obtain the description text about the chart image.

[0036] Please refer to Figure 6 This embodiment also proposes a multimodal chart-to-text generation system that integrates chart key data. The system applies the multimodal chart-to-text generation method that integrates chart key data as described above. The system includes: Key data extraction module, used for: Extract the underlying data table of the chart, expand the underlying data table to obtain table serialized text, segment the table serialized text, and then input it into the pre-trained language model BERT to obtain the embedding vector of each subword. According to the embedding vector of each subword, obtain the feature representation of each cell, and fuse the cell feature representation with the title information at the corresponding position in the underlying data table to obtain the table data value feature; The semantic information of the title of the underlying data table of the chart is supplemented to obtain the supplementary information of the title, and the supplementary information of the title is integrated with the characteristics of the table data value and then passed through the prediction layer to obtain the key data related to the chart; Image feature extraction module, used for: The chart image is segmented into several image blocks according to fixed pixel length and width, and then the image blocks are flattened in the channel direction and input into the Swin Transformer model to obtain the visual features of the chart image; Multimodal feature fusion module, used for: The key data of the chart is serialized according to the template and input into the BART encoder to obtain text features, and then the text features and visual features are multimodally fused to generate multimodal fusion features; Text generation module for: The multimodal fusion features are input into the BART decoder for decoding to generate the final text.

[0037] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0038] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0039] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.< / eos>

Claims

1. A multimodal chart-to-text generation method integrating chart key data, characterized in that: The method comprises the following steps: Step 1: Extract the underlying data table of the chart, expand the underlying data table to obtain table serialized text, segment the table serialized text, and then input it into the pre-trained language model BERT to obtain the embedding vector of each subword, obtain the feature representation of each cell according to the embedding vector of each subword, and fuse the cell feature representation with the title information at the corresponding position in the underlying data table to obtain the table data value feature; Step 2: Supplement the semantic information of the title of the underlying data table of the chart to obtain supplementary information of the title, fuse the supplementary information of the title with the characteristics of the table data value, and then pass it through the prediction layer to obtain key data related to the chart; Step 3: Segment the chart image according to fixed pixel length and width to obtain a number of image blocks, flatten the image blocks in the channel direction and input them into the Swin Transformer model to obtain the visual features of the chart image; Step 4: Serialize the key data of the chart according to the template and input it into the BART encoder to obtain text features, and then perform multimodal fusion of the text features and visual features to generate multimodal fusion features; Step 5: Input the multimodal fusion features into the BART decoder for decoding to generate the final text.

2. The method for generating multimodal charts from texts by integrating key chart data according to claim 1, characterized in that: In step 1, the underlying data table is expanded to obtain table serialized text, the table serialized text is segmented, and then input into the pre-trained language model BERT to obtain the embedding vector of each subword and the title feature. The method for obtaining the feature representation of each cell according to the embedding vector of each subword specifically includes the following steps: Expand the underlying data table by row to get the table serialization text, where the table serialization text is represented as ; After the text is segmented, a subword token sequence is obtained. The subword token sequence is expressed as ,in, Indicates subwords; Use BERT's pre-trained word embedding matrix to extract word embedding vectors, and input the word embedding into the BERT model to obtain the feature representation of the table serialized text. The corresponding process has the following relationship: ； ； in, Represents the word embedding processing function of the BERT pre-trained model, Represents the word embedding representation of the input token sequence, represents the pre-trained BERT encoder, A feature representation representing the serialized text of a table; The subword vectors of all words in the table serialized text are averaged to obtain the feature representation of all words in the text. The corresponding process has the following relationship: ； in, Represents the feature representation of all words in the serialized text, express The k-th word feature in , express The number of subwords to be segmented, express No. subword features; The feature representations of all words contained in each cell in the table serialized text are averaged to obtain the feature representations of all cells in the table. The corresponding process has the following relationship: ； in, represents the cell feature representation of all cells, express The k-th cell feature in , Indicates the number of words contained in a cell. express The first word feature of the cell, express The number of words in the cell.

3. The method for generating multimodal charts from texts by integrating key chart data according to claim 2, characterized in that: In step 1, the data value feature represented by each cell feature is obtained, and the cell feature representation is merged with the title information at the corresponding position in the underlying data table. The method for obtaining the table data value feature specifically includes the following steps: For all cell features of the chart data table, according to the corresponding position of the cell in the data table, the row title features, column title features and data value features of the data table are extracted respectively. The corresponding process has the following relationship: ； ； ； in, Represent the row header feature, column header feature, and data value feature respectively. Represents the function for extracting row header features, represents the function for extracting column header features, represents a function that extracts features from data values, and Respectively represent the number of rows and columns of the data table; The row title features and column title features corresponding to each data cell in the data table are concatenated and transformed with the data value features to obtain the final table data value features that integrate the row title information and column title information. The corresponding process has the following relationship: ； ； in, Respectively represent the first The row header characteristics of the row, The column header characteristics, and the Row and The data value characteristics of the column, represents the tensor concatenation function, represents the linear transformation function, represents the normalization function, Indicates the data in the table k The data value feature is the intermediate state after the row header and column header features are concatenated. Indicates the data in the table k The final feature is obtained by transforming the intermediate state of the data value feature.

4. The method for generating multimodal charts from texts by integrating key chart data according to claim 3, characterized in that: In step 2, the semantic information of the title of the underlying data table of the chart is supplemented, and the method for obtaining the supplementary information of the title specifically includes the following steps: Construct large language model prompts for the row and column titles of the chart data table, and replace the row and column titles with the prompt template. The corresponding process has the following relationship: ； ； in, Indicates the large model prompt constructor, Indicates the first row headers, Indicates the first column headers, Respectively represent the constructed row header prompt and column header prompt; The row title prompt and column title prompt are input into the large language model to obtain the supplementary text information of the large model for the row title and column title, and then the obtained supplementary text information is encoded using the text encoder to obtain the supplementary information of the large model about the row title and column title of the chart data table. The corresponding process has the following relationship: ； ； ； ； in, A function representing large language model processing, and Respectively represent the generation of supplementary text for row and column headers, represents a text encoder, and They represent the encoded row header supplementary information and column header supplementary information respectively.

5. The method for generating multimodal charts from texts by integrating key chart data according to claim 4, characterized in that: In step 2, the method of fusing the supplementary information of the title with the table data value features and then passing through the prediction layer to obtain key data related to the chart specifically includes the following steps: After the table data value features pass through a layer of self-attention network, they are added to the row header supplementary information and column header supplementary information, and then the result of the addition passes through a layer of self-attention network to obtain the data value feature representation of the fused chart data table. The corresponding process has the following relationship: ； ； ； in, represents the function of the self-attention network, Represents the data value feature representation of the fused chart data table, and Respectively represent the hidden states of different layers of neural networks; The data value feature representation of the fused chart data table is passed through a prediction layer to obtain the key data related to the chart. The corresponding process has the following relationship: ； ； ； in, represents the regularization function, represents the activation function, A function representing the index of positions in the output tensor that are greater than 0.

5. Represents a function that retrieves the corresponding data based on the position index. The position index of the graph key representing the prediction, Indicates key data, Represents the third implicit state.

6. The method for generating multimodal charts from texts by integrating key chart data according to claim 5, characterized in that: In step 3, the chart image is segmented according to fixed pixel length and width to obtain a number of image blocks, and then the image blocks are flattened in the channel direction and input into the Swin Transformer model to obtain the visual feature representation of the chart image. The corresponding process has the following relationship: ； ； ； in, represents a chart image, Indicates image blocks, represents the partition function, represents the output features of the SwinTransformer model, Represents the function of the SwinTransformer model for image encoding, represents the multi-layer perceptron function, Represents the visual features of a chart image.

7. The method for generating multimodal charts from texts by integrating key chart data according to claim 6, characterized in that: In step 4, the obtained key data of the chart is serialized according to the template and then input into the BART encoder to obtain text features, and then the text features and visual features are multimodally fused. The method for generating multimodal fusion features specifically includes the following steps: The key data is serialized according to the manually constructed template to obtain the serialized text, and the serialized text is input into the BART encoder to obtain the text features. The corresponding process has the following relationship: ； ； in, Represents a function serialized according to an artificial template, Represents the serialized text. represents the pre-trained BART encoder processing function, Text features representing serialization of key data; Multimodal fusion of visual features and text features is performed to obtain multimodal chart features. The corresponding process has the following relationship: ； ； in, A function representing multimodal cross-attention layer processing, represents the gate fusion function, represents the initial fusion feature that combines part of the visual information and part of the text information. Represents the final fused multi-modal graph features.

8. The method for generating multimodal charts from texts by integrating key chart data according to claim 7, characterized in that: The calculation process of the multimodal cross-attention layer processing function specifically includes the following steps: The initial query, key, and value are calculated using text features. The process has the following relationship: ； in, Represent the weight parameters used to generate the initial query, key, and value, respectively. Represents the initial query, key, and value respectively; The initial key and value are fused with the visual features through the gate mechanism to obtain the key and value of the fused part of the visual features. ； in, Respectively represent the key and value of the fused partial visual features, They represent the weight parameters used to generate the key and value of the fused partial visual features, Represents the information preservation operation, which is used to control the amount of information retained from the visual modality. The information preservation operation is calculated as follows: ； in, Respectively express The weight parameter for linear transformation, represents the activation function; The initial query and the keys and values ​​of the fused partial visual features are used for attention calculation to obtain the initial fused features: ； in, Represents the initial fusion features.

9. The method for generating multimodal charts from texts by integrating key chart data according to claim 8, characterized in that: The calculation process of the gate fusion function has the following relationship: ； ； in, Express The weight parameter for linear transformation, Express The weight parameter for linear transformation, represents the bias parameter of the fusion gate, represents the fusion gate, Represents a tensor dot product operation.

10. A multimodal chart-to-text generation system integrating chart key data, characterized in that: The system applies the multimodal chart-to-text generation method for integrating chart key data according to any one of claims 1 to 9, and the system comprises: Key data extraction module, used for: Extract the underlying data table of the chart, expand the underlying data table to obtain table serialized text, segment the table serialized text, and then input it into the pre-trained language model BERT to obtain the embedding vector of each subword. According to the embedding vector of each subword, obtain the feature representation of each cell, and fuse the cell feature representation with the title information at the corresponding position in the underlying data table to obtain the table data value feature; The semantic information of the title of the underlying data table of the chart is supplemented to obtain the supplementary information of the title, and the supplementary information of the title is integrated with the characteristics of the table data value and then passed through the prediction layer to obtain the key data related to the chart; Image feature extraction module, used for: The chart image is segmented into several image blocks according to fixed pixel length and width, and then the image blocks are flattened in the channel direction and input into the Swin Transformer model to obtain the visual features of the chart image; Multimodal feature fusion module, used for: The key data of the chart is serialized according to the template and input into the BART encoder to obtain text features, and then the text features and visual features are multimodally fused to generate multimodal fusion features; Text generation module for: The multimodal fusion features are input into the BART decoder for decoding to generate the final text.

Images