Multi-modal chart question and answer large model construction method, electronic equipment and storage medium

Abstract

The invention provides a multi-modal chart question and answer large model construction method, electronic equipment and a storage medium, and the method comprises the steps: training an image-text alignment model based on a first sample data set, and obtaining a trained image-text feature alignment model; wherein the first sample data set comprises image samples and corresponding text contents; and based on a second sample data set, the multi-modal chart question and answer large model with the trained image-text feature alignment model is trained, the trained multi-modal chart question and answer large model is obtained and serves as the final multi-modal chart question and answer large model, and the second sample data set comprises context representation information, images and question and answer pair data of chart samples. Compared with an existing multi-modal chart question and answer large model, the chart question and answer ability of the obtained multi-modal chart question and answer large model can be further improved, and the multi-modal chart question and answer large model has high Chinese understanding ability.

Description

Technical Field

[0001] The present invention relates to the technical field of application of a multimodal large model in a diagram question-answering task, and in particular to a method for constructing a multimodal diagram question-answering large model, an electronic device and a storage medium. Background Art

[0002] Multimodal large language models represent a major leap forward in AI technology in understanding and processing diverse information. They can not only process information from a single modality such as vision, text, and hearing, but also fuse information from these different modalities to achieve a more comprehensive and in-depth understanding of the real world. Multimodal large language models are designed to understand and interpret the complex interactions between visual and language data, thereby enabling a deeper understanding of multimedia content. Their importance lies in their ability to process and analyze a large number of data types, including images and text, which is essential for the advancement of AI applications such as image recognition, natural language processing, and automatic reasoning. At the same time, multimodal large language models have also shown surprising capabilities. Although existing multimodal large model technologies have made significant improvements by designing different model architectures to encode more powerful multimodal features or obtain more accurate alignment strategies, these methods use frozen language models or visual models, which have insufficient modal alignment due to the limited number of parameters. Furthermore, since the visual model compresses the image size to a smaller uniform size during the encoding process, it will cause the loss of image information. In addition, current multimodal large models are trained on very few high-quality multimodal graph instruction datasets. Due to the lack of large-scale high-quality unimodal and multimodal graph instruction data, they still cannot unlock the capabilities of various multimodal graph tasks. Therefore, the construction and training methods of multimodal large language models are still a challenging problem. Summary of the invention

[0003] In view of the above technical problems, the technical solution adopted by the present invention is:

[0004] According to a first aspect of the present invention, a method for constructing a multimodal graph question-answering model is provided, wherein the multimodal graph question-answering model includes a large language model, a visual model, and a graph-text feature alignment model, and the method includes the following steps:

[0005] S100, training the image-text feature alignment model based on the first sample data set to obtain a trained image-text feature alignment model; wherein the first sample data set includes n1 image-text pair data, wherein each image-text pair data includes a corresponding sample image and text content, wherein the sample image is a chart image or a non-chart image, and the text content of the chart image includes context representation information and numerical information in the chart, and the context representation information includes a title, legend, X-axis information and Y-axis information in the chart, and the legend includes label category and color information.

[0006] S200, based on the second sample data set, train the multimodal graph question and answer big model with the trained graph-text feature alignment model to obtain the trained multimodal graph question and answer big model as the final multimodal graph question and answer big model; wherein, during the training process of the intermediate multimodal graph question and answer big model, only the parameters of the big language model and the graph-text feature alignment model are updated, wherein the second sample data set includes n2 instruction data, each instruction data includes the context representation information of the corresponding graph sample, the graph image and the question and answer pair data.

[0007] Among them, S100 specifically includes:

[0008] S110, based on the first sample data set and the frozen visual model, the first stage of training is performed on the image-text feature alignment model in the initial multimodal graph question-answering model to obtain the image-text feature alignment model after the first stage of training as the intermediate image-text feature alignment model.

[0009] S120, performing a second stage of training on the intermediate image-text feature alignment model based on the first sample data set and the frozen large language model, obtaining the image-text feature alignment model after the second stage of training as the trained image-text feature alignment model. According to a second aspect of the present invention, an electronic device is provided, comprising a processor and a memory; the processor is used to execute the steps of the method described in the first aspect of the present invention by calling a program or instruction stored in the memory.

[0010] According to a second aspect of the present invention, there is provided a computer-readable storage medium storing a program or instructions, wherein the program or instructions enable a computer to execute the steps of the method according to the first aspect of the present invention.

[0011] The present invention has at least the following beneficial effects:

[0012] The method for constructing a multimodal graph question-answering big model provided by an embodiment of the present invention first uses graph-text pair data (including non-graph images and corresponding text description information as well as graph images and corresponding context representation information) to train a graph-text feature alignment model by freezing the visual model and the big language model respectively. Then, the multimodal graph question-answering big model is continued to be trained using graph images, context representation information and graph question-answering pair data without updating the parameters of the visual model. This allows the trained multimodal graph question-answering big model to further improve its graph question-answering capability relative to existing multimodal graph question-answering big models, and has a stronger Chinese comprehension capability. It can also be directly applied to multimodal dialogues and intelligent question-answering tasks in multiple fields, and has good generalization.

[0013] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0015] Figure 1 A flowchart of a method for constructing a multimodal graphical question and answer model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0018] It should be noted that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0019] The present invention mainly relates to the technology of multimodal large models in chart question answering tasks, that is, given a chart image and a question description, returning the answer to a given question. Currently, based on system collaboration and end-to-end model training methods, most methods mainly rely on powerful language models and ignore the potential capabilities of visual models, which will lead to insufficient understanding of the model's chart image data, causing the visual model to be unable to understand the detailed information in the chart image well. Therefore, the present invention attempts to introduce a novel model architecture and training method into the multimodal large model to obtain fine-grained information of the chart image, thereby stimulating the potential capabilities of the multimodal large model in chart question answering tasks.

[0020] First of all, the present invention aims to make better use of the title, X-axis, Y-axis, legend and color information in the chart image. By integrating the information of these five modules with the specific information in the chart, the numerical value, color, position and other information in the chart can be well understood and aligned, which is a core step in solving the chart question and answer task.

[0021] Secondly, the present invention aims to build a larger and more comprehensive high-quality graph question-answering instruction dataset to improve the generalization ability of the model and avoid the poor quality of the graph instruction dataset affecting the model performance. The present invention integrates the existing public graph question-answering instruction datasets, and performs manual annotation and quality inspection to build a dataset that can fully reflect the diversity and complexity of graphs in practical applications.

[0022] Finally, the present invention solves the problems in the graph question answering task by fine-tuning the instructions, especially the alignment problem between multimodal graph data. By constructing a higher quality and more comprehensive dataset for instruction fine-tuning, the problems existing in existing research can be well solved, thereby improving the accuracy of multimodal graph question answering tasks and stimulating the potential capabilities of multimodal large models in downstream tasks.

[0023] Specifically, an embodiment of the present invention provides a method for constructing a multimodal graph question-answering model, wherein the multimodal graph question-answering model includes a large language model, a visual model, and a graph-text feature alignment model. Figure 1 As shown, the embodiment of the present invention provides a method for constructing a multimodal graph question-answering model, which may include the following steps:

[0024] S100, training a picture-text feature alignment model based on a first sample data set to obtain a trained picture-text feature alignment model.

[0025] Among them, the first sample data set includes n1 image-text pair data, wherein each image-text pair data includes a corresponding sample image and text content, wherein the sample image is a chart image or a non-chart image, and the text content of the chart image is context representation information and numerical information in the chart, and the context representation information includes the title, legend, X-axis information and Y-axis information in the chart, and the legend includes label category and color information.

[0026] In the embodiment of the present invention, the text content of the non-graphic image is text description information of the image.

[0027] In the embodiment of the present invention, the charts include stacked bar charts, complex line charts, scatter charts, pie charts, regular line charts, grouped bar charts, regular bar charts, three-dimensional bar charts, mixed charts, and the like.

[0028] In an embodiment of the present invention, the contextual representation information of a chart can be extracted by a context extraction paradigm. The context extraction paradigm aims to extract structured background information from a chart rather than processing specific data points. Given a chart image data, the goal is to generate a refined contextual representation that retains only the most relevant details. When extracting title and legend information, the title provides an overview of the contents of a chart, and the legend contains label categories and corresponding colors. The context extraction paradigm uses a template-based approach to standardize the extraction of these elements to ensure consistency across different types of charts. For example, the legend L can be extracted as follows:

[0029]

[0030] The extraction of color C can be formally expressed as:

[0031]

[0032] The X-axis information and the Y-axis information contain labels and corresponding value ranges. The X-axis information and the Y-axis information can be formally expressed as follows:

[0033] X = "USA|UK|Germany|France";

[0034]

[0035] This structured extraction avoids redundant information typically associated with OCR-based methods, focusing only on key contextual information relevant to reasoning in graph question answering tasks, excluding non-essential data, and reducing noise and redundant information.

[0036] In an exemplary embodiment of the present invention, the contextual representation information of the chart is obtained through the multimodal large model, and the multimodal large model can be directly used to extract information based on a predefined template without additional training. For example, the prompt may be: "Extract key information from the chart, including the title, legend, color corresponding to the legend, X-axis information, and Y-axis information." This type of instruction guides the model to generate structured fine-grained information from a variety of charts without additional training.

[0037] In another exemplary embodiment of the present invention, the contextual representation information of a chart can be obtained by a trained chart contextual representation information extraction model. The chart contextual representation information extraction model can be a multimodal large model such as Qwen2-VL, InternLM-XComposer2.5, GLM4Vo, etc. The trained chart contextual representation information extraction model can better adapt to the nuances of chart question-answering tasks, thereby improving the accuracy of generating contextual information.

[0038] In an embodiment of the present invention, the chart data in the five data sets of ChartQA, DVQA, FigureQA, PlotQA, and ScigraphQA can be used to train the chart context representation information extraction model. Specifically, based on the chart data in these five data sets, the corresponding context information can be extracted through a multimodal large model, and then the extracted context information is manually quality inspected to obtain labeled data pairs (chart images and context representation information) as training data to train the model. It is known to those skilled in the art that any method of using chart data pairs to train the chart context representation information extraction model falls within the scope of protection of the present invention.

[0039] In the embodiment of the present invention, the large language model may be an existing large language model, preferably, a Qwen2.5 72b model having strong scalability and high training and reasoning efficiency in multiple languages.

[0040] In an embodiment of the present invention, the large language model is used to perform feature encoding on received text data to obtain a corresponding text feature vector, wherein the text data includes text content and question-answer pair data.

[0041] In an embodiment of the present invention, for text data, the large language model first performs word segmentation on each received text to obtain a word segmentation set corresponding to each text, then encodes each word segmentation to obtain a corresponding feature vector, and then combines the feature vectors of the words corresponding to each text to obtain the feature vector of the text.

[0042] In the embodiment of the present invention, the visual model is the CLIP ViTL-14 model.

[0043] In the embodiment of the present invention, the visual model is used to perform feature encoding on the received chart image to obtain a corresponding image feature vector.

[0044] For each chart image data, the visual model will first divide the image data into blocks, for example, dividing it according to a preset rectangular frame to obtain all slices in each image, and then encode each slice to obtain the image feature vector of the image.

[0045] In an embodiment of the present invention, a graphic-text feature alignment model is used to perform graphic-text feature alignment on received text feature vectors and image feature vectors. It may be a Q-Former module. The Q-Former module consists of an image converter and a text converter, which share the same self-attention layer, wherein the image converter is used to interact with a frozen visual model to extract visual features that are most similar to the text. The text converter can be used as both a text encoder and a text decoder. The input of the image converter is a series of learnable query embeddings. The input of the text converter is a word embedding. The query vector can interact with itself, the image features extracted by the image encoder, and the text vector of the text encoder at the same time to achieve various learning objectives.

[0046] The present invention realizes the deep fusion and association of chart image and text features by introducing the Q-Former module. The core of this method is to accurately connect the implicit features of the image with the text information under a two-stage training framework. During the training process, the pre-training stage is first initialized by freezing the image and language model to lay the foundation for feature alignment. Specifically, the first stage adopts the representation learning method to enable the Q-Former module to extract visual features that are more closely related to the text from the image, ensuring that the image features can better serve the text understanding. The second stage adopts the generative learning method to generate and interpret the visual features extracted by the Q-Former module with the help of a large language model, and convert the visual information into a text description with deep semantic understanding, thereby enhancing the semantic association between the image and the text. Through this innovative training framework, more accurate and efficient cross-modal feature alignment can be achieved, promoting the research and development of the fusion of text and image information.

[0047] Specifically, S100 may include:

[0048] S110, based on the first sample data set and the frozen visual model, the first stage of training is performed on the image-text feature alignment model in the initial multimodal graph question-answering model to obtain the image-text feature alignment model after the first stage of training as the intermediate image-text feature alignment model.

[0049] S120, based on the first sample data set and the frozen large language model, the intermediate image-text feature alignment model is trained in the second stage to obtain the image-text feature alignment model after the second stage training, as the trained image-text feature alignment model. Furthermore, in an embodiment of the present invention, the image-text pair data of the non-chart image in the first sample data set can use the image-text pair data in the LAION-400M, COYO-700M and Google ConCeptual Captions (CC3M) data sets. Among them, CC3M contains 3.3M image-text pairs, LAION-400M contains 400M image-text pairs, and COYO-700M contains 747M image-text pairs. The present invention extracts 70% of the data from the three data sets for training the model, 15% for testing the model, and 15% for verifying the model.

[0050] The image-text pair data of the chart image in the first sample data set can be obtained by constructing the chart data in the public chart data set. Specifically, first, the chart data in the public chart data set is obtained and the context representation information in the chart data in the public chart data set is generated based on the multimodal large model. Then, the obtained context representation information is manually reviewed, and the generated erroneous or inaccurate context representation information is modified to obtain the image-text pair data of the chart image. In an embodiment of the present invention, the public chart data sets used may include ChartSumm, TinyChart, Chart-to-Text, ChartLLaMA, MMC-Instruction, ChartInsight, ChartX, ChartBench and OneChart.

[0051] Furthermore, S110 specifically includes:

[0052] S111, inputting the first sample data set into a frozen visual model to generate an image feature vector for each sample data, and obtaining a set of image feature vectors corresponding to the first sample data set as first training sample data.

[0053] S112, using the first training sample data to train the current image-text feature alignment model, obtain the currently trained image-text feature alignment model and the predicted text content corresponding to the image feature vector set, and use the currently trained image-text feature alignment model as the intermediate image-text feature alignment model.

[0054] S210 specifically includes:

[0055] S211: Input the first sample data set into a frozen large language model to generate a text feature vector for each sample data, and obtain a set of text feature vectors corresponding to the first sample data set as second training sample data.

[0056] S212: Use the second training sample data set to train the intermediate image-text feature alignment model to obtain a trained image-text feature alignment model.

[0057] Furthermore, S112 specifically includes:

[0058] S1120, input the first training sample data of the current batch into the current image-text feature alignment model to obtain a corresponding first prediction result; the first prediction result includes the predicted text content corresponding to the first training sample data. The initial value of the current image-text feature alignment model is the initialized image-text feature alignment model.

[0059] S1121, obtain the current loss function value of the current image-text feature alignment model using the currently obtained first prediction result and the corresponding true result, and determine whether the current loss function value meets the preset model training end condition. If so, obtain the trained image-text feature alignment model and the predicted text content corresponding to the image feature vector set, and use the currently trained image-text feature alignment model as the intermediate image-text feature alignment model, and end the first stage training. Otherwise, execute S1122.

[0060] In an embodiment of the present invention, in order to train the image-text feature alignment model to extract the visual representation containing the corresponding text content through the query vector, in the training process of the image-text feature alignment model, the model is trained by optimizing three objective functions, each of which uses a different self-attention mask strategy to control the interaction between the query vector and the text, wherein the three objective functions are: image-text contrast learning, image-localized text generation, and image-text matching. Image-text contrast learning learns the similarity of image-text data through contrast learning methods, with the aim of learning the alignment strategy of image features and text features to maximize the interactive information as much as possible; text generation for image feature representation is used to train the image-text feature alignment model to generate text information; image-text matching is used to learn more fine-grained information alignment between images and text.

[0061] In an embodiment of the present invention, the preset model training end condition can be set based on actual needs. For example, the loss is less than or equal to a set loss threshold and remains unchanged within a set time period.

[0062] S1122, update the parameters of the current image-text feature alignment model based on the current loss function value, and use the first training sample data of the next batch as the first training sample data of the current batch, and execute S1120.

[0063] Furthermore, S212 specifically includes:

[0064] S2120, input the second training sample data of the current batch into the current intermediate image-text feature alignment model to obtain the corresponding second prediction result; the second prediction result includes the relationship between the second training sample data and the predicted text content corresponding to the first training sample data, for example, it is recognized that a certain training sample data a corresponds to a certain predicted text content b, then the corresponding prediction result is (a, b).

[0065] S2121, obtain the current loss function value of the current intermediate image-text feature alignment model with the current first prediction result and the corresponding true result, and determine whether the current loss function value meets the preset model training end condition. If so, obtain the trained image-text feature alignment model and end the second stage training. Otherwise, execute S2122. The initial value of the current intermediate image-text feature alignment model is the intermediate image-text feature alignment model obtained in S1121.

[0066] S2122: Update the parameters of the current intermediate image-text feature alignment model based on the current loss function value, and use the second training sample data of the next batch as the second training sample data of the current batch, and execute S2120.

[0067] S200, based on the second sample data set, train the multimodal graph question and answer big model with the trained graph-text feature alignment model to obtain the trained multimodal graph question and answer big model as the final multimodal graph question and answer big model; wherein, during the training process of the intermediate multimodal graph question and answer big model, only the parameters of the big language model and the graph-text feature alignment model are updated, wherein the second sample data set includes n2 instruction data, each instruction data includes the context representation information of the corresponding graph sample, the graph image and the question and answer pair data.

[0068] Furthermore, in the embodiment of the present invention, the second sample data set can be obtained by the following steps:

[0069] S10, obtaining chart data of a public chart dataset.

[0070] In an embodiment of the present invention, the public chart data sets may include ChartSumm, TinyChart, Chart-to-Text, ChartLLaMA, MMC-Instruction, ChartInsight, ChartX, ChartBench and OneChart.

[0071] S11, adopts a multimodal large model to obtain contextual representation information in graph data.

[0072] In an embodiment of the present invention, the contextual representation information in the graph can be extracted through the multimodal large model of GPT4o.

[0073] S12, using a self-instruct framework to generate a corresponding first question Q1 based on the chart data.

[0074] S13, using the large language model to modify Q1 to obtain a second question Q2 with greater complexity.

[0075] In the embodiment of the present invention, the Qwen2-VL 72b large language model Q1 may be used for modification.

[0076] S14, saving the contextual representation information of the chart data, Q2 and the chart image corresponding to the chart data in a structured form, obtaining corresponding structured data, and generating an initial answer to the structured data using a multimodal large model.

[0077] S15, reviewing the initial answer to obtain a reviewed initial answer as the final answer to Q2.

[0078] In the embodiment of the present invention, the initial answers can be reviewed by manual review, and the generated wrong or inaccurate answers can be quality checked and modified.

[0079] S16, save Q2 and the final answer of Q2 into json format to obtain question-answer pair data of chart data.

[0080] Through S10 to S16, high-quality instruction data can be constructed.

[0081] In an embodiment of the present invention, in the training process of training the intermediate multimodal graph question-answering large model based on the second sample data set, the output of the graph-text feature alignment model is the answer to the graph. In this training process, the model is trained using a cross entropy loss function. It is known to those skilled in the art that any method for training the intermediate multimodal graph question-answering large model based on the second sample data set belongs to the protection scope of the present invention.

[0082] In actual application scenarios, the image, contextual representation information and questions of the target chart can be input into the final multimodal chart question-answering model to obtain the corresponding answers.

[0083] The method for constructing a large multimodal chart question-and-answer model provided by an embodiment of the present invention has the following advantages: a large multimodal model with a larger number of parameters is used to extract fine-grained background information of titles, legends, colors, X-axes and Y-axes in chart images, and the background information is used as prior information of instruction data, which can further improve the accuracy of the multimodal model for chart understanding tasks and improve the fine-grained recognition effect; a Q-Former module is used to align text features and chart image features, which has achieved a significant improvement in alignment effect compared to the current modal alignment through linear layer mapping, which has performed well; fine-tuning is performed on a high-quality multimodal chart instruction dataset, and the model has a significant improvement in multimodal chart question-and-answer capabilities, and the model has a strong Chinese comprehension ability; a two-stage training method is used on a large-scale picture-text pair dataset and a multimodal chart question-and-answer instruction dataset, which can be directly applied to multimodal dialogues and intelligent question-and-answer tasks in multiple fields, so that the trained multimodal chart question-and-answer large model has good generalization.

[0084] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the present invention.

[0085] The embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the embodiment of the present invention.

[0086] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0087] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a multimodal graph question-answering model, characterized in that: The multimodal graph question-answering large model includes a large language model, a visual model, and a graph-text feature alignment model. The method includes the following steps: S100, training the image-text feature alignment model based on the first sample data set to obtain a trained image-text feature alignment model; wherein the first sample data set includes n1 image-text pair data, wherein each image-text pair data includes a corresponding sample image and text content, wherein the sample image is a chart image or a non-chart image, and the text content of the chart image includes context representation information and numerical information in the chart, wherein the context representation information includes a title, a legend, X-axis information, and Y-axis information in the chart, and the legend includes a label category and color information; S200, based on the second sample data set, the multimodal graph question-answering large model with the trained graph-text feature alignment model is trained to obtain the trained multimodal graph question-answering large model as the final multimodal graph question-answering large model; wherein, in the training process of the intermediate multimodal graph question-answering large model, only the parameters of the large language model and the graph-text feature alignment model are updated, wherein the second sample data set includes n2 instruction data, each instruction data includes context representation information of the corresponding graph sample, a graph image, and question-answer pair data; Among them, S100 specifically includes: S110, performing a first-stage training on the image-text feature alignment model in the initial multimodal graph question-answering model based on the first sample data set and the frozen visual model, and obtaining the image-text feature alignment model after the first-stage training as an intermediate image-text feature alignment model; S120, performing a second-stage training on the intermediate image-text feature alignment model based on the first sample data set and the frozen large language model, to obtain an image-text feature alignment model after the second-stage training as a trained image-text feature alignment model.

2. The method according to claim 1, characterized in that The large language model is used to perform feature encoding on the received text data to obtain a corresponding text feature vector, wherein the text data includes text content and question-answer pair data; the visual model is used to perform feature encoding on the received chart images and non-chart images to obtain a corresponding image feature vector; the image-text feature alignment model is used to perform image-text feature alignment on the received text feature vector and image feature vector.

3. The method according to claim 1, characterized in that The image-text feature alignment model is a Q-Former module.

4. The method according to claim 3, characterized in that The context representation information is obtained through a multimodal large model or a trained graph context representation information extraction model.

5. The method according to claim 1, characterized in that The second sample data set is obtained by the following steps: S10, obtaining chart data of a public chart dataset; S11, using a multimodal large model to obtain contextual representation information in graph data; S12, using the self-generated instruction framework to generate a corresponding first question Q1 based on the chart data; S13, using the large language model to modify Q1 to obtain the second question Q2 with greater complexity; S14, saving the context representation information of the chart data, Q2 and the chart image corresponding to the chart data in a structured form, obtaining corresponding structured data, and generating an initial answer to the structured data using a multimodal large model; S15, reviewing the initial answer to obtain a reviewed initial answer as the final answer to Q2; S16, save Q2 and the final answer of Q2 into json format to obtain question-answer pair data of chart data.

6. The method according to claim 2, characterized in that S110 specifically includes: S111, inputting the first sample data set into a frozen visual model to generate an image feature vector for each sample data, and obtaining a set of image feature vectors corresponding to the first sample data set as first training sample data; S112, training the current image-text feature alignment model using the first training sample data to obtain the currently trained image-text feature alignment model and the predicted text content corresponding to the image feature vector set, and using the currently trained image-text feature alignment model as the intermediate image-text feature alignment model; S210 specifically includes: S211, inputting the first sample data set into the frozen large language model to generate a text feature vector for each sample data, and obtaining a set of text feature vectors corresponding to the first sample data set as second training sample data; S212: Use the second training sample data set to train the intermediate image-text feature alignment model to obtain a trained image-text feature alignment model.

7. The method according to claim 6, characterized in that S112 specifically includes: S1120, inputting the first training sample data of the current batch into the current image-text feature alignment model to obtain a corresponding first prediction result; the first prediction result includes predicted text content corresponding to the first training sample data; S1121, obtain the current loss function value of the current image-text feature alignment model using the currently obtained first prediction result and the corresponding true result, and determine whether the current loss function value meets the preset model training end condition. If so, obtain the trained image-text feature alignment model and the predicted text content corresponding to the image feature vector set, and use the currently trained image-text feature alignment model as the intermediate image-text feature alignment model, and end the first stage training. Otherwise, execute S1122; S1122, updating the parameters of the current image-text feature alignment model based on the current loss function value, and using the first training sample data of the next batch as the first training sample data of the current batch, and executing S1120; S212 specifically includes: S2120, inputting the second training sample data of the current batch into the current intermediate image-text feature alignment model to obtain a corresponding second prediction result; the second prediction result includes a relationship between the second training sample data and the predicted text content corresponding to the first training sample data; S2121, obtain the current loss function value of the current intermediate image-text feature alignment model using the currently obtained first prediction result and the corresponding true result, and determine whether the current loss function value meets the preset model training end condition. If so, obtain the trained image-text feature alignment model and end the second stage training. Otherwise, execute S2122; S2122: Update the parameters of the current intermediate image-text feature alignment model based on the current loss function value, and use the second training sample data of the next batch as the second training sample data of the current batch, and execute S2120.

8. The method according to claim 1, characterized in that The large language model is the Qwen2.5 72b model.

9. An electronic device, characterized in that: including a processor and a memory; The processor is used to execute the steps of the method according to any one of claims 1 to 8 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a program or an instruction, wherein the program or the instruction enables a computer to execute the steps of the method according to any one of claims 1 to 8.

Images