Multimodal chart question and answer large model construction method, electronic device and storage medium

Abstract

The application provides a multimodal graph table question and answer large model construction method, an electronic device and a storage medium, comprising: training a graph-text alignment model based on a first sample data set to obtain a trained graph-text feature alignment model; wherein the first sample data set comprises image samples and corresponding text content; training a multimodal graph table question and answer large model with the trained graph-text feature alignment model based on a second sample data set to obtain a trained multimodal graph table question and answer large model as a final multimodal graph table question and answer large model, and the second sample data set comprises context representation information of a graph table sample, an image and question and answer pair data. The multimodal graph table question and answer large model obtained by the application can further improve the graph table question and answer capability of the existing multimodal graph table question and answer large model, and has strong Chinese understanding capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of application of multi-modal large models in chart question answering tasks, in particular to a multi-modal chart question answering large model construction method, an electronic device and a storage medium. BACKGROUND

[0002] Multi-modal large language models represent a significant leap in artificial intelligence technology in understanding and processing diverse information. They can not only process visual, textual, auditory and other single modal information, but also fuse information from different modalities, thereby achieving a more comprehensive and in-depth understanding of the real world. Multi-modal large language models aim to understand and interpret the complex interactions between visual and linguistic data, enabling a deeper understanding of multimedia content. Their importance lies in their ability to process and analyze a wide range of data types, including images and text, which is crucial for the advancement of artificial intelligence applications such as image recognition, natural language processing and automated reasoning. At the same time, multi-modal large language models have shown surprising capabilities. Existing multi-modal large model technologies have made significant improvements by designing different model architectures to encode more powerful multi-modal features or obtain more accurate alignment strategies. However, these methods use frozen language models or visual models, which have limited parameters and result in insufficient modality alignment. Furthermore, during the encoding process of visual models, image sizes are compressed to a smaller uniform size, resulting in the loss of image information. In addition, current multi-modal large models are trained on very few high-quality multi-modal chart instruction datasets. Due to the lack of large-scale high-quality single-modal and multi-modal chart instruction data, they still cannot unlock the capabilities of various multi-modal chart tasks. Therefore, the construction and training method of multi-modal large language models remains a challenging problem. SUMMARY

[0003] To solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0004] According to the first aspect of the present application, a multi-modal chart question answering large model construction method is provided, which includes a large language model, a visual model and a text-image feature alignment model. The method comprises the following steps:

[0005] S100, training the graph-text feature alignment model based on a first sample data set to obtain a trained graph-text feature alignment model; wherein the first sample data set includes n1 graph-text pair data, wherein each graph-text pair data includes corresponding sample images and text content, wherein the sample images are chart images or non-chart images, and the text content of the chart images includes context representation information and numerical information in the chart, and the context representation information includes a title, a legend, X-axis information and Y-axis information in the chart, and the legend includes label categories and color information.

[0006] S200, training the multi-modal chart question answering large model with the trained graph-text feature alignment model based on a second sample data set to obtain a trained multi-modal chart question answering large model as a final multi-modal chart question answering large model; wherein only the parameters of the large language model and the graph-text feature alignment model are updated in the training process of the intermediate multi-modal chart question answering large model, and wherein the second sample data set includes n2 instruction data, and each instruction data includes context representation information of a corresponding chart sample, a chart image and question-answer pair data.

[0007] S100 specifically includes:

[0008] S110, performing first stage training on the graph-text feature alignment model in the initial multi-modal chart question answering large model based on the first sample data set and the frozen visual model to obtain a graph-text feature alignment model after first stage training as an intermediate graph-text feature alignment model.

[0009] S120, performing second stage training on the intermediate graph-text feature alignment model based on the first sample data set and the frozen large language model to obtain a graph-text feature alignment model after second stage training as a trained graph-text feature alignment model.

[0010] According to the second aspect of the present application, a computer readable storage medium is provided, which stores programs or instructions for making a computer execute the steps of the method according to the first aspect of the present application.

[0011] The present application has at least the following beneficial effects:

[0012] The multi-modal chart question and answer large model construction method provided by the embodiment of the application first utilizes chart-text data (including non-chart images and corresponding text description information and chart images and corresponding context representation information) to train a chart-text feature alignment model by freezing a visual model and a large language model respectively, then utilizes chart image, context representation information and chart question and answer data to continue training the multi-modal chart question and answer large model by not updating parameters of the visual model, so that the trained multi-modal chart question and answer large model can further improve the chart question and answer capability relative to the existing multi-modal chart question and answer large model, has strong Chinese understanding capability, and can be directly applied to multi-modal dialog and intelligent question and answer tasks in multiple fields, and has good generalization.

[0013] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the application, nor is it intended to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0015] Figure 1 The flow chart of the multi-modal chart question and answer large model construction method provided by the embodiment of the application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the application will be described clearly and completely in the following with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0018] It is noted that some example embodiments are described as processes or methods depicted as flowcharts. Although the processes are described in a particular, sequential order, many of the steps can be performed in parallel, concomitantly, or concurrently. Further, the order of the steps can be re-arranged. The processes can be terminated when their operations are completed, but can also have additional steps not included in the figure(s). The processes can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0019] The present application mainly relates to the technical of multimodal large model on chart question answering task, that is, given chart picture and question description, return the answer to the given question. At present, based on the system cooperation and end-to-end model training method, most methods mainly rely on powerful language model, ignore the potential ability of visual model, which will lead to the insufficient understanding of the model to the chart picture data, causing the visual model cannot well understand the detail information in the chart picture. Therefore, the present application tries to introduce novel model architecture and training method in the multimodal large model to obtain the fine-grained information of the chart picture, so as to stimulate the potential ability of the multimodal large model in the chart question answering task.

[0020] Firstly, the present application aims to better utilize the title, X-axis, Y-axis, legend and color information in the chart picture. By integrating the information of the five modules with the specific information in the chart, the numerical value, color, position and other information in the chart can be well understood and aligned, which is a core step to solve the chart question answering task.

[0021] Secondly, the present application aims to build a larger and more comprehensive high-quality chart question answering instruction dataset to improve the generalization ability of the model and avoid the influence of poor quality of the chart instruction dataset on the performance of the model. The present application integrates the existing public chart question answering instruction dataset, and performs artificial labeling and quality checking to build a dataset that can fully reflect the diversity and complexity of the chart in actual application.

[0022] Finally, the present application solves the problem in the chart question answering task by instruction fine-tuning, especially the alignment problem between multimodal chart data. By constructing a higher quality and more comprehensive dataset for instruction fine-tuning, the existing problems in the existing research can be well solved, and the accuracy of the multimodal chart question answering task can be improved, and the potential ability of the multimodal large model in the downstream task can be stimulated.

[0023] Specifically, the embodiment of the present application provides a multimodal chart question answering large model construction method, which comprises a large language model, a visual model and a text feature alignment model. As shown in Figure 1 The embodiment of the present application provides a multimodal chart question answering large model construction method, which can include the following steps:

[0024] S100, training the graph-text feature alignment model based on the first sample data set to obtain a trained graph-text feature alignment model.

[0025] In the first sample data set, n1 pieces of graph-text pair data are included, each piece of graph-text pair data includes corresponding sample image and text content, the sample image is a chart image or a non-chart image, the text content of the chart image is context representation information and numerical information in the chart, the context representation information includes title, legend, X-axis information and Y-axis information in the chart, and the legend includes label category and color information.

[0026] In the embodiment of the application, the text content of the non-chart image is text description information of the image.

[0027] In the embodiment of the application, the chart includes a stacked bar chart, a complex line chart, a scatter chart, a pie chart, a conventional line chart, a grouped bar chart, a conventional bar chart, a three-dimensional bar chart and a mixed chart.

[0028] In the embodiment of the application, the context representation information of the chart can be extracted by a context extraction paradigm. The context extraction paradigm aims to extract structured background information from the chart instead of processing specific data points. Given a chart picture data, the goal is to generate a fine context representation that only retains the most relevant details. When extracting title and legend information, the title provides an overview of the chart content, and the legend contains label categories and corresponding colors. The context extraction paradigm uses a template-based method to standardize the extraction of these elements, ensuring consistency across different types of charts. For example, the legend L can be extracted as follows:

[0029]

[0030] The extraction of color C can be formalized as follows:

[0031]

[0032] The X-axis information and Y-axis information contain labels and corresponding value ranges, and the X-axis information and Y-axis information can be formalized as follows:

[0033] X = "USA | UK | Germany | France";

[0034]

[0035] This structured extraction avoids the redundant information usually associated with OCR-based methods, focuses only on the key background information related to reasoning in the chart question and answer task, can exclude unnecessary data, and can reduce noise and redundant information.

[0036] In an illustrative embodiment of the present application, the contextual representation information of the chart is obtained by a multi-modal large model. The multi-modal large model can directly extract information based on a predefined template without additional training. For example, the prompt can be: "extract key information from the chart, including the title, legend, color corresponding to the legend, X-axis information, and Y-axis information." Through such instructions, the model can generate structured fine-grained information from various charts without additional training.

[0037] In another illustrative embodiment of the present application, the contextual representation information of the chart can be obtained by a trained chart contextual representation information extraction model. The chart contextual representation information extraction model can be a multi-modal large model such as Qwen2-VL, InternLM-XComposer2.5, GLM4Vo, etc. The trained chart contextual representation information extraction model can better adapt to the subtle differences of the chart question and answer task, thereby improving the accuracy of generating contextual information.

[0038] In an embodiment of the present application, the chart data in the five data sets of ChartQA, DVQA, FigureQA, PlotQA, and ScigraphQA can be used to train the chart contextual representation information extraction model. Specifically, based on the chart data in the five data sets, the corresponding contextual information can be extracted by a multi-modal large model, and then the extracted contextual information can be manually inspected to obtain labeled data pairs (chart images and contextual representation information) as training data to train the model. Those skilled in the art know that any method of training a chart contextual representation information extraction model using chart data pairs is within the scope of the present application.

[0039] In an embodiment of the present application, the large language model can be an existing large language model, and preferably, can be a Qwen2.5 72b model with strong scalability in multiple languages and high training and inference efficiency.

[0040] In an embodiment of the present application, the large language model is configured to encode received text data to obtain corresponding text feature vectors, wherein the text data includes text content and question and answer pair data.

[0041] In an embodiment of the present application, for text data, the large language model first performs word segmentation processing on each received text to obtain a word segmentation set corresponding to each text, then encodes each word segmentation to obtain a corresponding feature vector, and further combines the feature vectors of the words corresponding to each text to obtain a feature vector of the text.

[0042] In an embodiment of the present application, the visual model is a CLIP ViTL-14 model.

[0043] In the embodiment of the present application, the visual model is used for feature encoding of the received chart image, to obtain a corresponding image feature vector.

[0044] For each chart image data, the visual model will first perform block processing on the image data, such as cutting according to a preset rectangular frame, to obtain all slices in each image, and then encode each slice to obtain an image feature vector of the image.

[0045] In the embodiment of the present application, the image-text feature alignment model is used for image-text feature alignment of the received text feature vector and image feature vector. It can be a Q-Former module. The Q-Former module is composed of an image converter and a text converter, which share the same self-attention layer, wherein the image converter is used to interact with the frozen visual model to extract the visual features most similar to the text. The text converter can act as both a text encoder and a text decoder. The input of the image converter is a series of learnable query embeddings. The input of the text converter is a word embedding. The query vector can interact with itself, the image features extracted by the image encoder, and the text vector of the text encoder to achieve various learning goals.

[0046] The present application realizes the deep fusion and correlation of chart image and text features by introducing the Q-Former module. The core of this method is to accurately connect the implicit features of the image with the text information under a two-stage training framework. During the training process, first, through the freezing of the image and language model, the initialization of the pre-training stage is carried out, laying the foundation for feature alignment. Specifically, in the first stage, representation learning is adopted, so that the Q-Former module can extract visual features more closely related to the text from the image, ensuring that the image features can better serve the text understanding. In the second stage, generation learning is adopted, and with the help of a large language model, the visual features extracted by the Q-Former module are generated and explained, and the visual information is converted into text descriptions with deep semantic understanding, thereby enhancing the semantic correlation between the image and the text. Through this innovative training framework, more accurate and efficient cross-modal feature alignment can be achieved, promoting the research and development of text and image information fusion.

[0047] Specifically, S100 can include:

[0048] S110, based on the first sample data set and the frozen visual model, the image-text feature alignment model in the initial multi-modal chart question and answer large model is trained in the first stage to obtain the image-text feature alignment model after the first stage training, as an intermediate image-text feature alignment model.

[0049] S120, performing second stage training on the intermediate graph-text feature alignment model based on the first sample data set and the frozen large language model, to obtain a graph-text feature alignment model after second stage training as the trained graph-text feature alignment model. Further, in the embodiment of the present application, the graph-text pair data of the non-chart image in the first sample data set can use the graph-text pair data in the LAION-400M, COYO-700M and Google ConCeptual Captions (CC3M) data sets. The CC3M contains 3.3M image-text pairs, the LAION-400M contains 400M image-text pairs, and the COYO-700M contains 747M image-text pairs. The present application extracts 70% of the data from the three data sets for training the model, 15% for testing the model, and 15% for verifying the model.

[0050] The graph-text pair data of the chart image in the first sample data set can be obtained by constructing the chart data in the public chart data set. Specifically, first, the chart data in the public chart data set is obtained, and the context representation information in the chart data in the public chart data set is generated based on the multi-modal large model, then the obtained context representation information is manually audited, and the context representation information generated incorrectly or inaccurately is modified to obtain the graph-text pair data of the chart image. In the embodiment of the present application, the public chart data set used can include ChartSumm, TinyChart, Chart-to-Text, ChartLLaMA, MMC-Instruction, ChartInsight, ChartX, ChartBench and OneChart.

[0051] Further, S110 specifically includes:

[0052] S111, inputting the first sample data set into the frozen visual model to generate an image feature vector of each sample data, to obtain an image feature vector set corresponding to the first sample data set as first training sample data.

[0053] S112, training the current graph-text feature alignment model using the first training sample data to obtain a current trained graph-text feature alignment model and a predicted text content corresponding to the image feature vector set, and taking the current trained graph-text feature alignment model as the intermediate graph-text feature alignment model.

[0054] S210 specifically includes:

[0055] S211, inputting the first sample data set into the frozen large language model to generate a text feature vector of each sample data, to obtain a text feature vector set corresponding to the first sample data set as second training sample data.

[0056] S212, training the intermediate image-text feature alignment model by using the second training sample data set to obtain a trained image-text feature alignment model.

[0057] Further, S112 specifically includes:

[0058] S1120, inputting the first training sample data of the current batch into the current image-text feature alignment model to obtain a corresponding first prediction result; the first prediction result includes a predicted text content corresponding to the first training sample data. The initial value of the current image-text feature alignment model is the initialized image-text feature alignment model.

[0059] S1121, obtaining a current loss function value of the current image-text feature alignment model by using the first prediction result obtained at present and the corresponding real result, and judging whether the current loss function value meets a preset model training end condition; if yes, obtaining a trained image-text feature alignment model and a predicted text content corresponding to the image feature vector set, and taking the current trained image-text feature alignment model as the intermediate image-text feature alignment model, and ending the first stage training; otherwise, performing S1122.

[0060] In the embodiment of the present application, in order to train the image-text feature alignment model to realize the extraction of the visual representation containing the corresponding text content through the query vector, in the training process of the image-text feature alignment model, the model is trained by optimizing three objective functions, each objective function uses different self-attention mask strategies to control the interaction mode between the query vector and the text, and the three objective functions are: image-text contrast learning, image positioning text generation and image-text matching. The image-text contrast learning learns the similarity of image-text pair data through the contrast learning method, the purpose is to learn the alignment strategy of image features and text features, and to maximize the interaction information as much as possible; the image feature representation text generation is used to train the image-text feature alignment model to generate text information; the image-text matching is used to learn the finer-grained information alignment between the image and the text.

[0061] In the embodiment of the present application, the preset model training end condition can be set based on actual needs. For example, the loss is less than or equal to a set loss threshold and remains unchanged within a set time period.

[0062] S1122, updating the parameters of the current image-text feature alignment model based on the current loss function value, taking the first training sample data of the next batch as the first training sample data of the current batch, and performing S1120.

[0063] Further, S212 specifically includes:

[0064] S2120, input the second training sample data of the current batch into the current intermediate graph-text feature alignment model to obtain a corresponding second prediction result; the second prediction result includes the relationship between the prediction text content corresponding to the second training sample data and the first training sample data, for example, it is identified that a certain training sample data a corresponds to a certain prediction text content b, and the corresponding prediction result is (a, b).

[0065] S2121, obtain the current loss function value of the current intermediate graph-text feature alignment model by taking the first prediction result obtained currently and the corresponding real result as input, and determine whether the current loss function value meets the preset model training end condition, if yes, obtain the trained graph-text feature alignment model, and end the second stage training, otherwise, perform S2122. The initial value of the current intermediate graph-text feature alignment model is the intermediate graph-text feature alignment model obtained in S1121.

[0066] S2122, update the parameters of the current intermediate graph-text feature alignment model based on the current loss function value, and take the second training sample data of the next batch as the second training sample data of the current batch, and perform S2120.

[0067] S200, train the multi-modal graph table question and answer large model with the trained graph-text feature alignment model based on the second sample data set to obtain a trained multi-modal graph table question and answer large model as a final multi-modal graph table question and answer large model; wherein, in the training process of the intermediate multi-modal graph table question and answer large model, only the parameters of the large language model and the graph-text feature alignment model are updated, wherein the second sample data set includes n2 instruction data, and each instruction data includes context representation information of a corresponding graph sample, a graph image and question and answer pair data.

[0068] Further, in the embodiment of the application, the second sample data set can be obtained by the following steps:

[0069] S10, obtaining graph data of a public graph dataset.

[0070] In the embodiment of the application, the public graph dataset can include ChartSum, TinyChart, Chart-to-Text, ChartLLaMA, MMC-Instruction, ChartInsight, ChartX, ChartBench and OneChart.

[0071] S11, obtaining context representation information in the graph data by using a multi-modal large model.

[0072] In the embodiment of the application, the context representation information in the graph can be extracted by the multi-modal large model of GPT4o.

[0073] S12, generating a corresponding first question Q1 based on the chart data by using self-instruction.

[0074] S13, modifying Q1 by using a large language model to obtain a second question Q2 with greater complexity.

[0075] In the embodiments of the present application, the Qwen2-VL 72b large language model Q1 can be used for modification.

[0076] S14, saving the context representation information of the chart data, Q2 and the chart image corresponding to the chart data in a structured form to obtain corresponding structured data, and generating an initial answer of the structured data by using a multi-modal large model.

[0077] S15, auditing the initial answer to obtain an audited initial answer as the final answer of Q2.

[0078] In the embodiments of the present application, the initial answer can be audited by manual auditing, and the answers generated by error or inaccuracy can be checked and modified.

[0079] S16, saving Q2 and the final answer of Q2 in json form to obtain question and answer pair data of the chart data.

[0080] Through S10 to S16, high-quality instruction data can be constructed.

[0081] In the training process of training the intermediate multi-modal chart question and answer large model based on the second sample data set, the output of the image-text feature alignment model is the answer of the chart. In this training process, a cross-entropy loss function is used to train the model. Those skilled in the art know that any method for training the intermediate multi-modal chart question and answer large model based on the second sample data set is within the protection scope of the present application.

[0082] In actual application scenarios, the image of the target chart, the context representation information and the question can be input into the final multi-modal chart question and answer large model to obtain the corresponding answer.

[0083] The multi-modal graph question and answer large model construction method provided by the embodiment of the application can further improve the accuracy of the multi-modal model in understanding a graph and improve the fine-grained recognition effect, because a multi-modal large model with a larger parameter quantity is used to extract fine-grained background information such as a title, a legend, a color, an X axis and a Y axis in a graph picture, and the background information is used as prior information of instruction data. The alignment effect is obviously improved compared with the current linear layer mapping mode alignment, by using a Q-Former module to align text features and graph image features. The model has obvious improvement in multi-modal graph question and answer capability by fine-tuning on a high-quality multi-modal graph instruction data set, and the model has strong Chinese understanding capability. The two-stage training method is used on a large-scale graph-text data set and a multi-modal graph question and answer instruction data set, which can be directly applied to multi-modal dialogue and intelligent question and answer tasks in multiple fields, so that the trained multi-modal graph question and answer large model has good generalization.

[0084] The embodiment of the application further provides an electronic device, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the application.

[0085] The embodiment of the application further provides a computer readable storage medium storing computer executable instructions, and the computer executable instructions are used to execute the method described in the embodiment of the application.

[0086] It should be understood that the steps shown above can be reordered, added or deleted. For example, the steps described in the application can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the application can be achieved, which is not limited herein.

[0087] The above specific embodiments do not constitute a limitation on the protection scope of the application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement and improvement within the spirit and principles of the application should be included in the protection scope of the application.

Claims

1. A method for constructing a large-scale multimodal graph-based question-answering model, characterized in that, The multimodal graph question-answering model includes a large language model, a visual model, and a graph-text feature alignment model. The method includes the following steps: S100, the image-text feature alignment model is trained based on the first sample dataset to obtain the trained image-text feature alignment model; wherein, the first sample dataset includes n1 image-text pairs, wherein each image-text pair includes a corresponding sample image and text content, wherein the sample image is a chart image or a non-chart image, and the text content of the chart image includes contextual representation information and numerical information in the chart, wherein the contextual representation information includes the title, legend, X-axis information and Y-axis information in the chart, and the legend includes label category and color information; S200, based on the second sample dataset, train the multimodal chart question-answering large model with the trained image-text feature alignment model to obtain the trained multimodal chart question-answering large model, which serves as the final multimodal chart question-answering large model; wherein, during the training process of the intermediate multimodal chart question-answering large model, only the parameters of the large language model and the image-text feature alignment model are updated; wherein, the second sample dataset includes n2 instruction data, and each instruction data includes the contextual representation information of the corresponding chart sample, the chart image, and the question-answer pair data; Specifically, S100 includes: S110, Based on the first sample dataset and the frozen visual model, the image-text feature alignment model in the initial multimodal chart question answering large model is trained in the first stage to obtain the image-text feature alignment model after the first stage of training, which serves as the intermediate image-text feature alignment model; S120, the intermediate image-text feature alignment model is trained in the second stage based on the first sample dataset and the frozen large language model to obtain the image-text feature alignment model after the second stage of training, which is used as the trained image-text feature alignment model. The second sample dataset was obtained through the following steps: S10, Obtain chart data from the publicly available chart dataset; S11, Use a multimodal large model to obtain contextual representation information in the chart data; S12, using the self-generated instruction framework to generate the corresponding first question Q1 based on the chart data; S13, using a large language model to modify Q1, resulting in a second problem Q2 with greater complexity; S14, save the contextual representation information of the chart data, Q2 and the chart image corresponding to the chart data into a structured form to obtain the corresponding structured data, and use the multimodal large model to generate the initial answer of the structured data; S15, review the initial answer to obtain the reviewed initial answer, which will be the final answer for Q2; S16, save Q2 and the final answer to Q2 in JSON format to obtain the question-answer pair data of the chart data.

2. The method according to claim 1, characterized in that, The large language model is used to encode the features of the received text data to obtain the corresponding text feature vectors. The text data includes text content and question-answer pair data. The visual model is used to encode the features of the received chart images and non-chart images to obtain the corresponding image feature vectors. The image-text feature alignment model is used to align the received text feature vectors and image feature vectors.

3. The method according to claim 1, characterized in that, The image-text feature alignment model is the Q-Former module.

4. The method according to claim 3, characterized in that, The contextual representation information is obtained through a multimodal large model or a trained graph contextual representation information extraction model.

5. The method according to claim 2, characterized in that, S110 specifically includes: S111, input the first sample dataset into the frozen visual model to generate the image feature vector of each sample data, and obtain the set of image feature vectors corresponding to the first sample dataset as the first training sample data; S112, the current image-text feature alignment model is trained using the first training sample data to obtain the currently trained image-text feature alignment model and the predicted text content corresponding to the image feature vector set, and the currently trained image-text feature alignment model is used as the intermediate image-text feature alignment model. S210 specifically includes: S211, input the first sample dataset into the frozen large language model to generate the text feature vector of each sample data, and obtain the set of text feature vectors corresponding to the first sample dataset as the second training sample data; S212, the intermediate image-text feature alignment model is trained using the second training sample dataset to obtain the trained image-text feature alignment model.

6. The method according to claim 5, characterized in that, S112 specifically includes: S1120, Input the first training sample data of the current batch into the current image-text feature alignment model to obtain the corresponding first prediction result; The first prediction result includes the predicted text content corresponding to the first training sample data; S1121: Obtain the current loss function value of the current image-text feature alignment model by taking the first prediction result and the corresponding real result, and determine whether the current loss function value meets the preset model training termination condition. If it meets the condition, obtain the trained image-text feature alignment model and the predicted text content corresponding to the image feature vector set, and use the currently trained image-text feature alignment model as the intermediate image-text feature alignment model, and end the first stage of training. Otherwise, execute S1122. S1122, Update the parameters of the current image-text feature alignment model based on the current loss function value, and use the first training sample data of the next batch as the first training sample data of the current batch, and execute S1120; S212 specifically includes: S2120, Input the second training sample data of the current batch into the current intermediate image-text feature alignment model to obtain the corresponding second prediction result; The second prediction result includes the relationship between the predicted text content corresponding to the second training sample data and the first training sample data. S2121: Obtain the current loss function value of the current intermediate image-text feature alignment model by taking the first prediction result and the corresponding real result, and determine whether the current loss function value meets the preset model training termination condition. If it does, obtain the trained image-text feature alignment model and end the second stage of training; otherwise, execute S2122. S2122, Update the parameters of the current intermediate image-text feature alignment model based on the current loss function value, and use the second training sample data of the next batch as the second training sample data of the current batch, and execute S2120.

7. The method according to claim 1, characterized in that, The large language model is the Qwen2.5 72b model.

8. An electronic device, characterized in that, Including processor and memory; The processor executes the steps of the method as described in any one of claims 1 to 7 by invoking programs or instructions stored in the memory.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a program or instructions that cause a computer to perform the steps of the method as described in any one of claims 1 to 7.

Images