Chart analysis method based on multi-modal large model, architecture and terminal

Abstract

The invention discloses a chart analysis method, architecture and terminal based on a multi-modal large model, and the method comprises the steps: carrying out the projection processing of a to-be-analyzed chart through a visual encoder of the multi-modal large model, and obtaining a plurality of visual projection tokens; performing conversion processing on the plurality of visual projection tokens by an expert bridging module of the multi-modal large model to respectively obtain a plurality of visual projection sequences; the big language model of the multi-modal big model preprocesses a user question text to obtain a plurality of text projection sequences, and the plurality of visual projection sequences and the plurality of text projection sequences are spliced to obtain a plurality of spliced sequences; and the large language model performs sequence decoding processing on the plurality of spliced sequences to obtain a chart response result corresponding to the user question text. According to the method, the multi-modal large model is constructed, so that the content of the chart can be accurately analyzed, and the analysis efficiency and the response accuracy of the chart are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a chart analysis method, architecture, terminal and computer-readable storage medium based on a multimodal large model. Background Art

[0002] With the rapid growth of chart data, how to achieve fast, efficient and accurate analysis of chart content has become an increasingly important research topic.

[0003] However, in the prior art, the multimodal large model mostly uses natural images during pre-training. Therefore, for charts without data point annotations, the multimodal large model cannot extract reliable values ​​from the charts, which causes the multimodal large model to rely heavily on its own OCR (optical character recognition) capabilities to process charts. If the OCR capability is low, the values ​​obtained based on OCR will be inaccurate, which will lead to low accuracy of the chart analysis results output by the large model. In the prior art, the analysis ability of the multimodal large model in charts can be improved through supervised fine-tuning of chart data, but it will cause a decrease in its general performance, especially its reasoning ability and continuous dialogue ability, resulting in the inability of the multimodal large model to enhance its chart analysis capabilities while maintaining excellent general performance.

[0004] Therefore, the existing technology is unable to achieve accurate analysis of chart content through multimodal large models, and still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a chart analysis method, architecture, terminal and computer-readable storage medium based on a multimodal large model, aiming to solve the problem in the prior art that it is impossible to accurately analyze the chart content through a multimodal large model.

[0006] To achieve the above object, the present invention provides a chart analysis method based on a multimodal large model, and the chart analysis method based on a multimodal large model comprises the following steps: The visual encoder of the multimodal large model projects the chart to be analyzed to obtain multiple visual projection tokens; The expert bridging module of the multimodal large model performs conversion processing on the plurality of visual projection tokens to obtain a plurality of visual projection sequences respectively; The large language model of the multimodal large model preprocesses the user question text to obtain a plurality of text projection sequences, and splices the plurality of visual projection sequences and the plurality of text projection sequences to obtain a plurality of spliced ​​sequences; The large language model performs sequence decoding processing on the plurality of concatenated sequences to obtain a chart answer result corresponding to the user question text.

[0007] Optionally, the diagram analysis method based on the multimodal large model, wherein the visual encoder of the multimodal large model performs projection processing on the diagram to be analyzed to obtain a plurality of visual projection tokens, specifically includes: Acquire a chart to be analyzed, and input the chart to be analyzed into the multimodal macro model; The diagram to be analyzed is projected by the visual encoder in the multimodal large model to obtain a plurality of visual projection tokens.

[0008] Optionally, in the multimodal large model-based chart analysis method, the expert bridging module comprises a gated network and at least one expert connector; Among them, the expert connectors include a table alignment expert connector, a JSON alignment expert connector and a code alignment expert connector.

[0009] Optionally, in the multimodal large model-based graph analysis method, initialization parameters of the expert connector are obtained by pre-training the alignment task, wherein: Obtaining initialization parameters of the table alignment expert connector through pre-training of the chart table alignment task; Obtaining initialization parameters of the JSON alignment expert connector through graph JSON alignment task pre-training; The initialization parameters of the code alignment expert connector are obtained through pre-training of the graph-code alignment task.

[0010] Optionally, the diagram analysis method based on the multimodal large model, wherein the expert bridging module of the multimodal large model converts the plurality of visual projection tokens to obtain a plurality of visual projection sequences, specifically comprising: Inputting a plurality of the visual projection tokens into the expert bridge module, determining a gated network corresponding to each of the visual projection tokens through the expert bridge module, and activating at least one target expert connector in the expert bridge module through the gated network; Each of the visual projection tokens is input into each of the target expert connectors respectively, and each of the visual projection tokens is converted by each of the target expert connectors to obtain a plurality of visual projection sequences respectively.

[0011] Optionally, the diagram analysis method based on the multimodal large model, wherein the activating at least one target expert connector in the expert bridging module through the gated network specifically includes: The gating network calculates the routing weight corresponding to each expert connector in the expert bridging module according to each of the visual projection tokens; The corresponding expert connectors are sorted in descending order according to the routing weights, and a preset number of target expert connectors at the top of the sorting are activated.

[0012] Optionally, the diagram analysis method based on the multimodal large model, wherein each of the visual projection tokens is input into each of the target expert connectors, and each of the visual projection tokens is converted by each of the target expert connectors to obtain multiple visual projection sequences, specifically including: Input each of the visual projection tokens into each of the target expert connectors respectively, and convert each of the visual projection tokens through each of the target expert connectors, so that each of the visual projection tokens obtains a plurality of target visual projection tokens; The target routing weight corresponding to the target expert connector is obtained, and a weighted average processing is performed on a plurality of the target visual projection tokens according to the target routing weight to obtain a visual projection sequence corresponding to the visual projection token.

[0013] Optionally, in the multimodal large model-based graph analysis method, the large language model comprises an encoder, a concatenation module and a decoder; The large language model of the multimodal large model preprocesses the user question text to obtain multiple text projection sequences, and splices the multiple visual projection sequences and the multiple text projection sequences to obtain multiple spliced ​​sequences, specifically including: Obtaining a user question text, inputting the user question text into the large language model, and performing word segmentation and encoding processing on the user question text through an encoder in the large language model to obtain multiple text projection sequences; The plurality of visual projection sequences and the plurality of text projection sequences are spliced ​​together through the splicing module of the large language model to obtain a plurality of spliced ​​sequences.

[0014] Optionally, the diagram analysis method based on the multimodal large model, wherein the large language model performs sequence decoding processing on the plurality of concatenated sequences to obtain a diagram answer result corresponding to the user question text, specifically includes: Inputting a plurality of the concatenated sequences into a decoder of the large language model; The decoder performs sequence decoding processing on the plurality of concatenated sequences to obtain a chart answer result corresponding to the user question text.

[0015] In addition, to achieve the above-mentioned objectives, the present invention also provides a multimodal large model architecture for a chart analysis method based on a multimodal large model, wherein the multimodal large model architecture includes a visual encoder, an expert bridging module and a large language model, and the visual encoder is connected to the large language model through the expert bridging module.

[0016] Optionally, the multimodal large model architecture is pre-trained according to the following steps: Determine an alignment task, and perform alignment pre-training processing on the expert bridge module in the multimodal large model architecture according to the alignment task to obtain an expert connector; Using a supervised fine-tuning method to perform model training and model fine-tuning on the multimodal large model architecture; The alignment task is one or more of a chart table alignment task, a chart JSON alignment task, and a chart code alignment task.

[0017] Optionally, the multimodal large model architecture, wherein the determining of the alignment task and performing alignment pre-training processing on the expert bridging module in the multimodal large model architecture according to the alignment task to obtain the expert connector, comprises: If the alignment task is a chart-table alignment task, chart alignment training is performed on the expert connector in the expert bridging module according to the chart-table alignment task to obtain a table alignment expert connector.

[0018] Optionally, the multimodal large model architecture, wherein the determining of the alignment task and performing alignment pre-training processing on the expert bridging module in the multimodal large model architecture according to the alignment task to obtain an expert connector, further comprises: If the alignment task is a graph JSON alignment task, the expert connector in the expert bridging module is trained for graph JSON alignment according to the graph JSON alignment task to obtain a JSON alignment expert connector.

[0019] Optionally, the multimodal large model architecture, wherein the determining of the alignment task and performing alignment pre-training processing on the expert bridging module in the multimodal large model architecture according to the alignment task to obtain an expert connector, further comprises: If the alignment task is a chart-code alignment task, constructing a training data set according to the chart-code alignment task, wherein the training data set includes chart data and corresponding codes; According to the graph data and the corresponding code, graph-code alignment training is performed on the expert connector in the expert bridge module to obtain a code-aligned expert connector.

[0020] Optionally, in the multimodal large model architecture, the supervised fine-tuning method includes: high-quality knowledge learning training and graph-specific annealing tuning training.

[0021] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a chart analysis program based on a multimodal large model stored in the memory and executable on the processor, wherein the chart analysis program based on the multimodal large model implements the steps of the chart analysis method based on the multimodal large model as described above when executed by the processor.

[0022] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a chart analysis program based on a multimodal large model, and when the chart analysis program based on the multimodal large model is executed by a processor, the steps of the chart analysis method based on the multimodal large model as described above are implemented.

[0023] In the present invention, the visual encoder of the multimodal large model projects the chart to be analyzed to obtain multiple visual projection tokens; the expert bridge module of the multimodal large model converts the multiple visual projection tokens to obtain multiple visual projection sequences respectively; the large language model of the multimodal large model preprocesses the user's question text to obtain multiple text projection sequences, and splices the multiple visual projection sequences and the multiple text projection sequences to obtain multiple spliced ​​sequences; the large language model performs sequence decoding on the multiple spliced ​​sequences to obtain a chart answer result corresponding to the user's question text. The present invention can accurately parse the input chart content by constructing a multimodal large model, so that the chart answer result required by the user can be quickly and accurately output based on the parsing result, effectively improving the parsing efficiency of the multimodal large model for chart data and the accuracy of the output of the chart answer result. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of a preferred embodiment of the method for analyzing a chart based on a multi-modal large model of the present invention; Figure 2 It is a schematic diagram of a bridge module of a hybrid expert architecture of a preferred embodiment of a diagram analysis method based on a multi-modal large model of the present invention; Figure 3 It is a model training flow chart of a multimodal large model architecture of a chart analysis method based on a multimodal large model of the present invention; Figure 4 It is a schematic diagram of expert parameter initialization of a preferred embodiment of the chart analysis method based on a multi-modal large model of the present invention; Figure 5It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0026] With the rapid growth of chart data, how to achieve fast, efficient and accurate analysis of chart content has become an increasingly important research topic. Transformer (neural network architecture)-based multimodal large models are gaining more and more attention due to their powerful reasoning performance and application generalization.

[0027] However, in the prior art, the multimodal large model relies heavily on its own OCR (optical character recognition) capabilities to process charts, and the output chart analysis results have low accuracy. In the prior art, supervised fine-tuning can improve the analysis capabilities of the multimodal large model in terms of charts, but it will cause a decrease in its general performance, especially its reasoning ability and continuous dialogue ability, resulting in the multimodal large model being unable to enhance its chart analysis capabilities while maintaining excellent general performance.

[0028] Faced with this difficulty, current methods are generally based on the design of process framework structure, using multiple auxiliary tools to help the multimodal large model extract more information, integrate it into the input prompt words, and thus improve the performance of the model. However, the design based on the process framework relies heavily on the performance of auxiliary tools, such as optical character recognition (OCR) technology. If the OCR recognition result is wrong, and the large model analyzes, infers, and communicates based on the wrong recognition result, it will output inaccurate conclusions. In addition, for those charts that cannot be processed by OCR, the performance of the large model will drop significantly. At the same time, because the information in the chart cannot be effectively extracted, there is no effective information input into the large model, then the general ability of the large model will also be seriously affected.

[0029] In order to solve the above problems, a multimodal large model is constructed in the present invention. The projection processing and conversion processing of the visual encoder are realized through the multimodal large model to obtain multiple visual projection sequences, and the user question text is converted into multiple text projection sequences and then spliced ​​with the multiple visual projection sequences, and then the chart answer result corresponding to the user question text is output, which can effectively improve the analysis efficiency of the multimodal large model for chart data and the accuracy of the output of the chart answer result.

[0030] The diagram analysis method based on the multimodal large model described in the preferred embodiment of the present invention is as follows: Figure 1As shown, the chart analysis method based on the multimodal large model includes the following steps: Step S10: The visual encoder of the multimodal large model performs projection processing on the chart to be analyzed to obtain multiple visual projection tokens.

[0031] First, the chart to be analyzed is input into the visual encoder in the multimodal large model to obtain N visual projection tokens (i.e., the visual projection tokens in the present invention).

[0032] Specifically, a chart to be analyzed is obtained, and the chart to be analyzed is input into the multimodal large model; the chart to be analyzed is projected by the visual encoder in the multimodal large model to obtain a plurality of visual projection tokens.

[0033] like Figure 2 As shown, get the chart that needs to be parsed (i.e. the chart to be analyzed in the present invention) and the user's Question text (i.e. the user question text in the present invention), and then the chart and the user's question text based on the chart input Input into the multimodal model (i.e., the multimodal large model in the present invention), so as to use the multimodal large model to analyze the chart and the user's question text based on the chart input to be processed.

[0034] In one embodiment, CLIP ViT-Large (a pre-trained model for image-text classification) is used as a visual encoder, taking advantage of its rich prior knowledge gained from training on millions of image-text pairs. Preferably, considering the impact of graph resolution on performance, the input resolution is set to a hyperparameter of c × c (c can be 490, 224, etc.) to strike a balance between efficiency and performance.

[0035] Formally, the visual encoder Draw the chart Projection N A visual projection token (also called a visual projection token) is (dimension is d), the process expression is: , Represents the visual encoder For charts Perform projection processing. For the first visual projection token, For the second visual projection token, For the N A visual projection token.

[0036] Step S20, the expert bridging module of the multimodal large model converts the multiple visual projection tokens to obtain multiple visual projection sequences respectively, wherein the expert bridging module includes a gating network and at least one expert connector, and the expert connector includes a table alignment expert connector, a JSON alignment expert connector and a code alignment expert connector.

[0037] Among them, the present invention also sets up the initialization parameters of the expert connector obtained through alignment task pre-training, including: obtaining the initialization parameters of the table alignment expert connector through chart table alignment task pre-training; obtaining the initialization parameters of the JSON alignment expert connector through chart JSON alignment task pre-training; obtaining the initialization parameters of the code alignment expert connector through chart code alignment task pre-training.

[0038] The present invention proposes a multimodal large model bridging module based on a multi-expert architecture (i.e., the expert bridging module ChartMoE in the present invention), which replaces the linear layer bridging module in the traditional model and replaces it with a parallel expert connector composed of multiple linear layers. The expert bridging module includes a gating network and multiple expert connectors, and the gating network determines which expert connectors need to be activated for each visual token (i.e., the visual projection token in the present invention).

[0039] Specifically, a plurality of the visual projection tokens are input into the expert bridging module, and the gated network calculates the routing weight corresponding to each expert connector in the expert bridging module according to each of the visual projection tokens; the corresponding expert connectors are sorted in descending order according to the routing weights, and a preset number of target expert connectors at the top of the sorting are activated.

[0040] like Figure 2 As shown, the bridge module of the multi-expert architecture proposed in the present invention (i.e., the expert bridge module in the embodiment of the present invention) is used to connect the visual and text branches in the multimodal large model. Each visual projection token (i.e., the visual projection token in the embodiment of the present invention) passes through the expert bridge module to obtain N visual token sequences (i.e., the visual projection sequence in the embodiment of the present invention).

[0041] Input each of the visual projection tokens into each of the target expert connectors respectively, and convert each of the visual projection tokens through each of the target expert connectors, so that each of the visual projection tokens obtains multiple target visual projection tokens; obtain the target routing weight corresponding to the target expert connector, and perform weighted averaging on the multiple target visual projection tokens according to the target routing weight to obtain the visual projection sequence corresponding to the visual projection token.

[0042] Among them, the gating network of the expert bridging module is used to control whether multiple expert connectors are activated. The activated expert connector processes each visual projection Token and obtains the corresponding visual Token sequence after weighted averaging. N visual projection Tokens (i.e., the visual projection tokens in the embodiment of the present invention) can obtain N visual Token sequences (i.e., the visual projection sequence in the embodiment of the present invention).

[0043] like Figure 2 As shown in the figure, the bridge module of the hybrid expert architecture (i.e., the expert bridge module in the embodiment of the present invention, also referred to as MoE architecture) is set in the present invention. The MoE architecture adopts a parallel multi-expert collaboration method. The architecture consists of a gated network and L expert connectors Composition, each expert connector is designed as a linear layer.

[0044] The process of converting the visual projection token to obtain the corresponding visual projection sequence can be expressed as: ,in, represents the softmax function, and Indicates that the weights of routes that are not ranked in the top K will be reset to 0 (if reset to 0, the expert connector corresponding to the route will not be selected). The visual projection token Token (i.e., the target visual projection token in the embodiment of the present invention) after being processed by the expert connector processor is Each visual projection token provided , Gated Network Each expert connector will be calculated The routing weight , and select the top K expert connectors ranked in descending order of routing weight for activation (K is the preset number of target expert connectors selected in the embodiment of the present invention). The processed visual projection token Token is based on Provided target weight Perform weighted averaging to obtain the visual projection sequence (visual Token sequence) after the visual projection token (visual projection token) is converted, marked as .

[0045] Step S30: The large language model of the multimodal large model preprocesses the user question text to obtain multiple text projection sequences, and splices the multiple visual projection sequences and the multiple text projection sequences to obtain multiple spliced ​​sequences, wherein the large language model includes an encoder, a splicing module and a decoder.

[0046] In one embodiment, the large language model LLM model (M LLM ) is used to convert the question text entered by the user After word segmentation and encoding, M text Token sequences (i.e., text projection sequences in the embodiment of the present invention) are obtained.

[0047] Specifically, a user question text is obtained, and the user question text is input into the large language model. The user question text is segmented and encoded by an encoder in the large language model to obtain a plurality of text projection sequences; and a plurality of the visual projection sequences and a plurality of the text projection sequences are spliced ​​by a splicing module of the large language model to obtain a plurality of spliced ​​sequences.

[0048] In one embodiment, the splicing module splices N visual Token sequences (i.e., multiple visual projection sequences in the embodiment of the present invention) with M text Token sequences (i.e., multiple text projection sequences in the embodiment of the present invention) to obtain a spliced ​​Token sequence (i.e., multiple spliced ​​sequences in the embodiment of the present invention).

[0049] It can be understood that in this embodiment, the expert bridge module is used to generate N visual token sequences from the N visual projection tokens V projected by the chart. (where the dimension of the visual projection token is D), and then combined with the M word tokens in the question text The obtained text token sequence (in, is the first word in the text Token sequence, is the second word in the text Token sequence, is the first in the text token sequence The concatenation sequence is used as the input of the large language model. The concatenation sequence can be [visual, text] or [text, visual]. For example, the concatenation sequence can be expressed as [ ; 】or

;

[0050] Step S40: The large language model performs sequence decoding processing on the plurality of concatenated sequences to obtain a chart answer result corresponding to the user question text.

[0051] In one embodiment, a variant of InternLM2-7B-ChatSFT is used as the decoder of the large language model, which is actually a transformer decoder with causal attention mask.

[0052] Specifically, the plurality of concatenated sequences are input into the decoder of the large language model; the decoder performs sequence decoding processing on the plurality of concatenated sequences to obtain a chart answer result corresponding to the user question text.

[0053] For the construction process of the large language model: the LoRA (Low-Rank Adaptation of Large Language Models, large language model fine-tuning technology) parameters obtained after model training can be used to update the parameters of the large language model LLM (perform a sum operation); then, the concatenated Token sequence is decoded by the decoder of the updated large language model LLM to output the generated answer to the query statement based on the input graph (that is, the graph response result corresponding to the user question text in the embodiment of the present invention).

[0054] According to the input chart and question text , the model output proposed by the present invention is It can be expressed as: ,in, is the model output (i.e., the chart response result in the embodiment of the present invention), The visual projection sequence and the text projection sequence are concatenated to obtain a concatenated sequence.

[0055] In summary, the present invention inputs the chart to be analyzed and the user's question text based on the chart into the multimodal large model, and processes the chart to be analyzed and the user's question text through the visual encoder and expert bridge module in the multimodal large model, so that the expert connectors with different specialties in the expert bridge module can play their respective advantages and obtain the visual projection sequence corresponding to the chart; further, by inputting the obtained visual projection sequence and text projection sequence into the large language model together, the chart answer result required by the user can be obtained quickly and accurately, which effectively improves the multimodal large model's efficiency in parsing chart data and the accuracy of the question and answer result output.

[0056] Furthermore, based on the above-mentioned chart analysis method based on the multimodal big model, the present invention also provides a multimodal big model architecture for the chart analysis method based on the multimodal big model, wherein the multimodal big model architecture includes a visual encoder, an expert bridging module and a large language model, and the visual encoder is connected to the large language model through the expert bridging module.

[0057] The basic architecture of the model used in the present invention: The present invention utilizes the ability of the basic model in text-image understanding to demonstrate performance comparable to that of GPT-4 (all-round AI robot). Among them, the basic model (such as the baseline model InternlmXC-v2, Qwen-VL, LLaVA, etc.) includes a visual encoder and a large language model (LLM). The visual encoder and the large language model are connected through an expert bridge module, and different expert connectors in the expert bridge module can be used to accurately analyze the chart data. Compared with the prior art that connects the visual encoder and the large language model through a double-layer multi-layer perceptron (MLP), the present invention can use multiple expert connectors to obtain diverse prior knowledge.

[0058] The multimodal large model architecture of the multimodal large model-based chart analysis method described in the preferred embodiment of the present invention is as follows: Figure 3 As shown, the training process of the multimodal large model architecture includes the following steps: Step S100, determine the alignment task, and perform alignment pre-training processing on the expert bridging module in the multimodal large model architecture according to the alignment task to obtain an expert connector; wherein the alignment task is one or more of a chart table alignment task, a chart JSON alignment task, and a chart code alignment task.

[0059] At present, the existing technology has too single training method for the connection module of multimodal large models, which mainly focuses on the alignment of images with corresponding table texts. This method ignores the extraction of other key information in the image, such as line shape, color and other visual features, which are essential for a comprehensive understanding of the image content. Therefore, future research needs to explore more diverse training strategies to make full use of the rich information in the image.

[0060] At present, the initialization parameters of the expert connector are generally obtained through alignment task pre-training. Unlike previous work, the present invention performs parallel training during the alignment task pre-training stage to obtain the initialization parameters of each expert connector. Previous expert initialization methods mainly initialize expert parameters in two ways: 1. Random initialization, which may cause convergence difficulties during supervised fine-tuning; 2. Collaborative upgrade initialization, that is, setting the parameters of the baseline model as the learnable parameters of each expert connector, which may cause the learnable parameters initialized between expert connectors to be the same, and different expert connectors are difficult to learn the characteristics of different input data during the training process.

[0061] To address the above issues, this paper introduces a more systematic initialization strategy, which enables the expert connector to effectively focus on different components of the graph, thereby outperforming the baseline model in overall performance. Figure 4As shown in (b), the present invention initializes three expert connectors in parallel through three different alignment tasks. During the training process, only the learnable parameters of the expert connectors are updated through gradient back propagation.

[0062] It is understandable that, in order to solve the problem of chart understanding, the present invention proposes a variety of alignment tasks, including chart table alignment, chart JSON alignment, and chart code alignment. The present invention uses these alignment tasks to initialize the parameters of each expert connector, avoiding the homogenization of expert connectors during training, and showing good interpretability and chart understanding capabilities. Among them, the present invention sets three expert connectors obtained through training of multiple alignment tasks, namely: table alignment expert, JSON alignment expert, and code alignment expert.

[0063] Specifically, if the alignment task is a chart and table alignment task, chart alignment training is performed on the expert connector in the expert bridging module according to the chart and table alignment task to obtain a table alignment expert connector.

[0064] In one embodiment, the chart and table alignment task performs chart alignment before supervised fine-tuning, which effectively enhances the model's ability in numerical extraction and chart understanding; the present invention aligns the chart and table (represented in CSV format, such as Figure 4 The alignment is shown in (a) to improve the model's ability to extract tabular data.

[0065] If the alignment task is a graph JSON alignment task, the expert connector in the expert bridging module is trained for graph JSON alignment according to the graph JSON alignment task to obtain a JSON alignment expert connector.

[0066] This embodiment uses JSON instead of table data for alignment pre-training; JSON contains not only the values ​​of the table, but also the drawing properties of the chart, such as font format, line type, color style, etc. This task is more challenging than the alignment between the chart and the table, and is helpful for attribute-based chart editing and highlighting.

[0067] In another embodiment, a chart-to-code training task is also proposed to implement the training of the expert connector. Specifically, the training data set includes a chart and the code for drawing the corresponding chart. The code contains the precise data and drawing properties in the chart. For example, the data format in the chart is a numerical value represented by a Python list, and the drawing properties include color, font, and font size represented by hexadecimal codes, such as Figure 4 As shown in (a) in the figure, based on the training task proposed by the present invention, compared with only using JSON and table alignment, the alignment task with code contains richer information.

[0068] If the alignment task is a chart-code alignment task, a training data set is constructed according to the chart-code alignment task, wherein the training data set includes chart data and corresponding codes; chart-code alignment training is performed on the expert connector in the expert bridging module according to the chart data and the corresponding codes to obtain a code alignment expert connector.

[0069] The present invention proposes to initialize expert parameters using multiple alignment tasks (such as alignment of charts and tables, alignment of charts and JSON (JSON is a lightweight data exchange format) and alignment of charts and drawing codes). This new expert parameter initialization method makes the model training converge faster and more interpretable, and also avoids the problem of uneven expert performance.

[0070] Step S200: Using a supervised fine-tuning method to perform model training and model fine-tuning on the multimodal large model architecture, wherein the supervised fine-tuning method includes: high-quality knowledge learning training and graph-specific annealing tuning training.

[0071] Current large models face a serious problem of knowledge forgetting when performing supervised fine-tuning. This phenomenon not only affects the model's long-term memory ability, but also makes its performance on specific tasks unsatisfactory, limiting the model's scope of application and actual effect.

[0072] To solve the above problems, the current approach is to create more graph training data and then use supervised fine-tuning to improve model performance. In addition, the mainstream approach is to first use a large amount of paired data to only train the connection module (usually a linear layer MLP) between the visual encoder and the LLM (large language model), which is called alignment pre-training. Based on the pre-trained bridge module, more supervised fine-tuning training is then performed.

[0073] For the supervised fine-tuning method: After aligning the expert connectors of three different tasks respectively, the present invention uses Figure 2 The structure shown is used to initialize the learnable parameters of the expert connector in the model. The present invention retains the untrained bridge module and its parameters of the base model to maintain the excellent dialogue ability of the baseline model, which is consistent with the residual optimization principle. At this stage, the present invention uses LoRA (Low-Rank Adaptation of Large Language Models) or full parameter fine-tuning method to train the ChartMoE connector and LLM large language model.

[0074] like Figure 2As shown, the present invention divides this stage into high-quality knowledge learning and graph-specific annealing training. Among them, for high-quality knowledge learning: the present invention adopts MMC to enhance the general knowledge ability of the model. MMC contains a variety of graph types and tasks, such as graph-related question and answer, translation, extraction, reasoning and analysis. Considering the data quality, the present invention only uses the manually verified MMC-Instruction subset. It should be noted that at this stage, the quality of instruction data is more important than the quantity.

[0075] For graph-specific annealing tuning: The present invention requires annealing tuning before evaluating mainstream benchmarks, in which the learning rate is re-increased and instruction tuning is performed using the training sets of ChartQA (a general chart benchmark) and ChartGemma (a lightweight open source chart large model) to adjust the query style and answer format of these benchmarks.

[0076] Reasoning stage: Procedural Thinking Training (PoT) requires a large multimodal model to generate variables and operation codes instead of directly giving answers. This reasoning process solves the problem of mathematical ability by using Python to process logical calculations, which is the shortcoming of all open source models. Considering that the model trained by this method has better numerical extraction capabilities, PoT can significantly improve the model's question-answering performance.

[0077] In summary, the present invention proposes a novel multimodal large model bridging module--ChartMoE. Different from the traditional linear layer bridging module, ChartMoE adopts a multi-expert connector architecture and initializes the parameters of the expert connector by means of alignment tasks. The gating network is responsible for determining the specific expert connector (i.e., the target expert connector in the present invention) to be activated in one reasoning, so that the expert connectors with different specialties can play their respective advantages. In addition, the present invention proposes a variety of innovative alignment tasks to initialize the parameters of different expert connectors, including tasks such as chart table alignment, chart JSON alignment, and chart code alignment. These alignment tasks provide unique initialization methods for the three different expert connectors. The experts initialized in this way show their respective preferences when processing chart images, which not only improves the processing capabilities of different types of charts, but also maintains the original reasoning performance and dialogue capabilities. In addition, the present invention elaborates on the supervised fine-tuning process in detail to ensure the effective convergence and high performance of the bridging model, and proves the effectiveness of the bridging module and alignment tasks proposed in the present invention on multiple real chart datasets and general knowledge question-answering datasets.

[0078] Furthermore, if Figure 5 As shown, based on the above-mentioned chart analysis method and architecture based on the multimodal large model, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 5 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0079] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a chart analysis program 40 based on a multimodal large model is stored on the memory 20, and the chart analysis program 40 based on a multimodal large model can be executed by the processor 10, thereby realizing the chart analysis method based on a multimodal large model in the present application.

[0080] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the multimodal large model-based chart analysis method.

[0081] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0082] In one embodiment, when the processor 10 executes the multimodal large model-based chart analysis program 40 in the memory 20 , the steps of the multimodal large model-based chart analysis method described above are implemented.

[0083] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a chart analysis program based on a multimodal large model, and when the chart analysis program based on the multimodal large model is executed by a processor, the steps of the chart analysis method based on the multimodal large model as described above are implemented.

[0084] In summary, the present invention provides a chart analysis method and related equipment based on a multimodal large model, the method comprising: the visual encoder of the multimodal large model projects the chart to be analyzed to obtain multiple visual projection tokens; the expert bridge module of the multimodal large model converts the multiple visual projection tokens to obtain multiple visual projection sequences respectively; the large language model of the multimodal large model preprocesses the user's question text to obtain multiple text projection sequences, and splices the multiple visual projection sequences and the multiple text projection sequences to obtain multiple splicing sequences; the large language model performs sequence decoding on the multiple splicing sequences to obtain a chart answer result corresponding to the user's question text. The present invention can accurately parse the input chart content by constructing a multimodal large model, so that the chart answer result required by the user can be quickly and accurately output based on the parsing result, effectively improving the parsing efficiency of the multimodal large model for chart data and the accuracy of the output of the chart answer result.

[0085] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0086] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0087] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A chart analysis method based on a multimodal large model, characterized in that: The chart analysis method based on the multimodal large model includes: The visual encoder of the multimodal large model projects the chart to be analyzed to obtain multiple visual projection tokens; The expert bridging module of the multimodal large model performs conversion processing on the plurality of visual projection tokens to obtain a plurality of visual projection sequences respectively; The large language model of the multimodal large model preprocesses the user question text to obtain a plurality of text projection sequences, and splices the plurality of visual projection sequences and the plurality of text projection sequences to obtain a plurality of spliced ​​sequences; The large language model performs sequence decoding processing on the plurality of concatenated sequences to obtain a chart answer result corresponding to the user question text.

2. The method for analyzing graphs based on a multimodal large model according to claim 1, characterized in that: The visual encoder of the multimodal large model performs projection processing on the chart to be analyzed to obtain multiple visual projection tokens, specifically including: Acquire a chart to be analyzed, and input the chart to be analyzed into the multimodal macro model; The diagram to be analyzed is projected by the visual encoder in the multimodal large model to obtain a plurality of visual projection tokens.

3. The method for analyzing graphs based on a multimodal large model according to claim 1, characterized in that: The expert bridging module includes a gating network and at least one expert connector; Among them, the expert connectors include a table alignment expert connector, a JSON alignment expert connector and a code alignment expert connector.

4. The method for analyzing graphs based on a multimodal large model according to claim 3, characterized in that: The initialization parameters of the expert connector are obtained through alignment task pre-training, where: Obtaining initialization parameters of the table alignment expert connector through pre-training of the chart table alignment task; Obtaining initialization parameters of the JSON alignment expert connector through graph JSON alignment task pre-training; The initialization parameters of the code alignment expert connector are obtained through pre-training of the graph-code alignment task.

5. The method for analyzing graphs based on a multimodal large model according to claim 3, characterized in that: The expert bridging module of the multimodal large model converts the plurality of visual projection tokens to obtain a plurality of visual projection sequences, specifically including: Inputting a plurality of the visual projection tokens into the expert bridge module, determining a gated network corresponding to each of the visual projection tokens through the expert bridge module, and activating at least one target expert connector in the expert bridge module through the gated network; Each of the visual projection tokens is input into each of the target expert connectors respectively, and each of the visual projection tokens is converted by each of the target expert connectors to obtain a plurality of visual projection sequences respectively.

6. The method for analyzing graphs based on a multimodal large model according to claim 5, characterized in that: The activating at least one target expert connector in the expert bridge module through the gated network specifically includes: The gating network calculates the routing weight corresponding to each expert connector in the expert bridging module according to each of the visual projection tokens; The corresponding expert connectors are sorted in descending order according to the routing weights, and a preset number of target expert connectors at the top of the sorting are activated.

7. The method for analyzing graphs based on a multimodal large model according to claim 5, characterized in that: The step of inputting each of the visual projection tokens into each of the target expert connectors, and converting each of the visual projection tokens through each of the target expert connectors to obtain a plurality of visual projection sequences, specifically includes: Input each of the visual projection tokens into each of the target expert connectors respectively, and convert each of the visual projection tokens through each of the target expert connectors, so that each of the visual projection tokens obtains a plurality of target visual projection tokens; The target routing weight corresponding to the target expert connector is obtained, and a weighted average processing is performed on a plurality of the target visual projection tokens according to the target routing weight to obtain a visual projection sequence corresponding to the visual projection token.

8. The method for analyzing graphs based on a multimodal large model according to claim 1, characterized in that: The large language model includes an encoder, a concatenation module and a decoder; The large language model of the multimodal large model preprocesses the user question text to obtain multiple text projection sequences, and splices the multiple visual projection sequences and the multiple text projection sequences to obtain multiple spliced ​​sequences, specifically including: Obtaining a user question text, inputting the user question text into the large language model, and performing word segmentation and encoding processing on the user question text through an encoder in the large language model to obtain multiple text projection sequences; The plurality of visual projection sequences and the plurality of text projection sequences are spliced ​​together through the splicing module of the large language model to obtain a plurality of spliced ​​sequences.

9. The method for analyzing graphs based on a multimodal large model according to claim 8, characterized in that: The large language model performs sequence decoding processing on the plurality of concatenated sequences to obtain a chart answer result corresponding to the user's question text, specifically including: Inputting a plurality of the concatenated sequences into a decoder of the large language model; The decoder performs sequence decoding processing on the plurality of concatenated sequences to obtain a chart answer result corresponding to the user question text.

10. A multimodal big model architecture based on the multimodal big model-based chart analysis method according to any one of claims 1 to 9, characterized in that: The multimodal large model architecture includes a visual encoder, an expert bridging module and a large language model, and the visual encoder is connected to the large language model through the expert bridging module.

11. The multimodal large model architecture according to claim 10, characterized in that: The multimodal large model architecture is pre-trained according to the following steps: Determine an alignment task, and perform alignment pre-training processing on the expert bridge module in the multimodal large model architecture according to the alignment task to obtain an expert connector; Using a supervised fine-tuning method to perform model training and model fine-tuning on the multimodal large model architecture; The alignment task is one or more of a chart table alignment task, a chart JSON alignment task, and a chart code alignment task.

12. The multimodal large model architecture according to claim 11, characterized in that: The determining of the alignment task, and performing alignment pre-training processing on the expert bridging module in the multimodal large model architecture according to the alignment task to obtain an expert connector, includes: If the alignment task is a chart-table alignment task, chart alignment training is performed on the expert connector in the expert bridging module according to the chart-table alignment task to obtain a table alignment expert connector.

13. The multimodal large model architecture according to claim 11, characterized in that: The determining of the alignment task, and performing alignment pre-training processing on the expert bridging module in the multimodal large model architecture according to the alignment task to obtain an expert connector, includes: If the alignment task is a graph JSON alignment task, the expert connector in the expert bridging module is trained for graph JSON alignment according to the graph JSON alignment task to obtain a JSON alignment expert connector.

14. The multimodal large model architecture according to claim 11, characterized in that: The determining of the alignment task, and performing alignment pre-training processing on the expert bridging module in the multimodal large model architecture according to the alignment task to obtain an expert connector, includes: If the alignment task is a chart-code alignment task, constructing a training data set according to the chart-code alignment task, wherein the training data set includes chart data and corresponding codes; According to the graph data and the corresponding code, graph-code alignment training is performed on the expert connector in the expert bridge module to obtain a code-aligned expert connector.

15. The multimodal large model architecture according to claim 11, characterized in that: The supervised fine-tuning method includes: high-quality knowledge learning training and graph-specific annealing tuning training.

16. A terminal, characterized in that: The terminal includes: a memory, a processor, and a chart analysis program based on a multimodal large model stored in the memory and executable on the processor. When the chart analysis program based on the multimodal large model is executed by the processor, the steps of the chart analysis method based on the multimodal large model as described in any one of claims 1 to 9 are implemented.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a chart analysis program based on a multimodal large model, and when the chart analysis program based on a multimodal large model is executed by a processor, the steps of the chart analysis method based on a multimodal large model as described in any one of claims 1-9 are implemented.

Images