Graphic question-answering method, system, medium and device based on multimodal large model

Abstract

The present application relates to the technical field of chart question and answer, and in particular to a chart question and answer method, system, medium and device based on a multimodal large model. The method comprises: extracting target text information in the chart to be processed to obtain a chart text feature vector; extracting target image information in the above-mentioned chart to be processed to obtain a chart image feature vector; aligning the comprehensive text feature vector and the chart image feature vector; and generating a target answer corresponding to the question corresponding to the chart to be processed based on the aligned comprehensive text feature vector and the chart image feature vector. The present application is divided into two parts for extracting information from the chart to be processed. The more targeted and accurate the information provided about the chart to be processed is, the higher the accuracy of the answer corresponding to the final icon question and answer is.

Description

Background Art

[0002] Although a series of progress has been made in the existing multimodal graph question answering technology, there are still some significant defects and challenges. Although there are some works in the current multimodal large models that try to solve the graph question answering problem, such as LLaVAR and mPLUG-DocOwl, they mainly focus on specific types of graphs or documents, and use visual encoders to process the graph as a whole, which does not accurately obtain the graph content information. Summary of the invention

[0003] The technical problem to be solved by this application is: how to improve the accuracy of providing information about the content of a chart, so as to improve the accuracy of chart questions and answers.

[0004] In view of the above technical problems, according to the first aspect of the present application, a diagram question answering method based on a multimodal large model is provided, comprising:

[0005] Extracting target text information from the chart to be processed to obtain a chart text feature vector; wherein the target text information is text information corresponding to the intersection of a first text information set and a second text information set; the first text information set is a text information set obtained based on a first optical character recognition unit; and the second text information set is a text information set obtained based on a second optical character recognition unit;

[0006] Extracting target image information from the above-mentioned chart to be processed to obtain a chart image feature vector;

[0007] Aligning the comprehensive text feature vector and the chart image feature vector; wherein the comprehensive text feature vector is obtained by concatenating the chart text feature vector and the question text feature vector; and the question text feature vector is a feature vector generated according to the question corresponding to the chart to be processed;

[0008] A target answer corresponding to the question corresponding to the chart to be processed is generated based on the aligned comprehensive text feature vector and chart image feature vector.

[0009] According to a second aspect of the present application, a diagram question-answering system based on a multimodal large model is provided. The diagram question-answering system based on a multimodal large model comprises: an optical character recognition module, a feature alignment module, an image encoding module and a language encoding module, wherein:

[0010] An optical character recognition module is used to extract target text information in a chart to be processed and obtain a chart text feature vector; wherein the target text information is text information corresponding to the intersection of a first text information set and a plurality of second text information sets; the first text information set is a text information set obtained based on a first optical character recognition unit; and the second text information set is a text information set obtained based on a second optical character recognition unit;

[0011] An image encoding module is used to extract the target image information in the above-mentioned chart to be processed and obtain the chart image feature vector;

[0012] A feature alignment module is used to align the comprehensive text feature vector and the chart image feature vector; wherein the comprehensive text feature vector is obtained by concatenating the chart text feature vector and the question text feature vector; and the question text feature vector is a feature vector generated according to the question corresponding to the chart to be processed;

[0013] The language encoding module is used to generate a target answer corresponding to the question corresponding to the chart to be processed based on the aligned comprehensive text feature vector and the chart image feature vector.

[0014] According to the third aspect of the present application, a non-transitory computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned diagram question and answer method based on a multimodal large model.

[0015] According to a fourth aspect of the present application, there is provided an electronic device comprising a processor and the above-mentioned non-transitory computer-readable storage medium.

[0016] This application has at least the following beneficial effects:

[0017] The present application provides a chart question-and-answer method based on a multimodal large model. First, the target text information in the chart to be processed is extracted to obtain a chart text feature vector; wherein the above-mentioned target text information is the text information corresponding to the intersection of a first text information set and a plurality of second text information sets; the first text information set is a text information set obtained based on a first optical character recognition unit; the second text information set is a text information set obtained based on a second optical character recognition unit; secondly, the target image information in the above-mentioned chart to be processed is extracted to obtain a chart image feature vector; in the present application, the extraction of the chart information to be processed is divided into two parts, one part is to extract the target text information in the chart to be processed, and the other part is to extract the target image information in the chart to be processed; instead of relying solely on the visual encoder to extract information from the chart to be processed, the text and picture in the chart to be processed are respectively extracted and converted into corresponding feature vectors (chart text feature vector and chart image feature vector), so that for the chart to be processed that contains both text information and image information, the information extraction method provided by the present application is more targeted and more accurate. Then, the comprehensive text feature vector and the chart image feature vector are aligned; wherein the above-mentioned comprehensive text feature vector is obtained by splicing the chart text feature vector and the question text feature vector; the question text feature vector is a feature vector generated according to the question corresponding to the chart to be processed; here, for the chart to be processed, the information extraction method of the present application is adopted to obtain the chart text feature vector and the chart image feature vector respectively, the obtained chart text feature vector and the question text feature vector obtained according to the question are spliced, and the spliced ​​comprehensive text feature vector is aligned with the chart image feature vector to obtain a one-to-one correspondence between the chart image feature vector and the comprehensive text feature vector after conversion into a feature vector. Finally, the target answer corresponding to the question corresponding to the chart to be processed is generated according to the aligned comprehensive text feature vector and the chart image feature vector. According to the comprehensive text feature vector and the chart image feature vector, the answer corresponding to the question corresponding to the chart to be processed is obtained. Here, since the information of the chart to be processed obtained above is more accurate, the answer generated here is also more accurate. In summary, the information extraction of the chart to be processed in the present application is divided into two parts. The more accurate the chart information obtained, the higher the accuracy of the answer corresponding to the icon question and answer obtained in the end. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1A flowchart of a diagram question-answering method based on a multimodal large model provided in one embodiment of the present application;

[0020] Figure 2 A structural block diagram of a diagram question-answering system based on a multimodal large model provided for one embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0022] like Figure 1 As shown, a diagram question answering method based on a multimodal large model is provided according to an embodiment of the present application, including:

[0023] S100, extracting target text information from a chart to be processed to obtain a chart text feature vector; wherein the target text information is text information corresponding to the intersection of a first text information set and a second text information set; the first text information set is a text information set obtained based on a first optical character recognition unit; and the second text information set is a text information set obtained based on a second optical character recognition unit.

[0024] Specifically, the diagram question-and-answer method based on a multimodal large model of the present application is implemented based on a diagram question-and-answer system based on a multimodal large model. First, for the diagram to be processed, the text information therein is extracted as the target text information, and then a diagram text feature vector is generated. The target text information here is the text information corresponding to the intersection of the first text information set and several second text information sets, that is, it is obtained by comprehensive voting based on several text information respectively obtained by the first optical character recognition unit and the second optical character recognition unit. That is, the target text information finally obtained is the text information contained in both the first text information set and the second text information set. In this way, the accuracy of the target text information can be improved.

[0025] S200, extracting target image information from the chart to be processed to obtain a chart image feature vector.

[0026] Specifically, in the present application, for the chart to be processed, the information extraction is divided into two parts, one part is to extract the target text information in the chart to be processed, and the other part is to extract the target image information in the chart to be processed; instead of relying solely on the visual encoder to extract information from the chart to be processed, the text and pictures in the chart to be processed are extracted separately and converted into corresponding feature vectors (chart text feature vector and chart image feature vector), so that for the chart to be processed that contains both text information and image information, the information extraction method provided by the present application is more targeted and more accurate.

[0027] S300, aligning the comprehensive text feature vector and the chart image feature vector; wherein the comprehensive text feature vector is obtained by concatenating the chart text feature vector and the question text feature vector; and the question text feature vector is a feature vector generated according to the question corresponding to the chart to be processed.

[0028] Specifically, the comprehensive text feature vector and the chart image feature vector are aligned; wherein the comprehensive text feature vector is obtained by splicing the chart text feature vector and the question text feature vector; the question text feature vector is a feature vector generated according to the question corresponding to the chart to be processed; here, for the chart to be processed, the information extraction method of the present application is adopted to obtain the chart text feature vector and the chart image feature vector respectively, the obtained chart text feature vector and the question text feature vector obtained according to the question are spliced, and the spliced ​​comprehensive text feature vector is aligned with the chart image feature vector to obtain a one-to-one correspondence between the chart image feature vector and the comprehensive text feature vector after conversion into feature vectors.

[0029] S400, generating a target answer corresponding to the question corresponding to the chart to be processed according to the aligned comprehensive text feature vector and the chart image feature vector.

[0030] Specifically, the target answer corresponding to the question corresponding to the chart to be processed is generated according to the aligned comprehensive text feature vector and chart image feature vector. According to the comprehensive text feature vector and chart image feature vector, the answer corresponding to the question corresponding to the chart to be processed is obtained. Here, since the information of the chart to be processed obtained above is more accurate, the answer generated here is also more accurate.

[0031] An embodiment of the present application also provides a diagram question and answer system 100 based on a multimodal big model. The diagram question and answer system 100 based on a multimodal big model is used to implement the diagram question and answer method based on a multimodal big model. The diagram question and answer system based on a multimodal big model is implemented based on a multimodal big model, and includes: an optical character recognition module 110, a feature alignment module 120, an image encoding module 130 and a language encoding module 140. Here, the optical character recognition module 110 and the image encoding module 130 are connected to the feature alignment module 120, and the feature alignment module 120 is connected to the language encoding module 140.

[0032] Among them, the optical character recognition module 110 is used to extract the target text information in the chart to be processed and obtain the chart text feature vector; wherein the above-mentioned target text information is the text information corresponding to the intersection of the first text information set and several second text information sets; the first text information set is the text information set obtained based on the first optical character recognition unit; the second text information set is the text information set obtained based on the second optical character recognition unit.

[0033] Specifically, the input of the optical character recognition module 110 is the chart to be processed, and the output is the target text information. The module adopts OCR technology, which can effectively extract the text information in the chart, so as to better understand the content and structure of the chart. The multimodal chart question-answering model can better understand and parse the text information in the chart to be processed. The module includes a first optical character recognition unit and a second optical character recognition unit. The first optical character recognition unit adopts open function authentication technology or PaddleOCR V3 technology; the second optical character recognition unit adopts open function authentication technology or PaddleOCR V3 technology; and the first optical character recognition unit and the second optical character recognition unit adopt different technologies. First, for the chart to be processed, the text information therein is extracted as the target text information, and then the chart text feature vector is generated. The target text information here is the text information corresponding to the intersection of the first text information set and several second text information sets, that is, it is obtained by comprehensive voting based on several text information obtained by the first optical character recognition unit and the second optical character recognition unit respectively. That is, the target text information finally obtained is the text information contained in the first text information set and the second text information set. In this way, the accuracy of the target text information can be improved.

[0034] The image encoding module 130 is used to extract the target image information in the chart to be processed and obtain the chart image feature vector.

[0035] Specifically, the input of the image encoding module 130 is the chart to be processed, and the output is the chart image feature vector. This application uses clip-vit-large-patch14-336 as an image encoder to process and encode image features. clip-vit-large-patch14-336 is an OCR model that uses the Vision Transformer (ViT) architecture and has undergone a lot of training and optimization. This model is designed for tasks such as text recognition and image text extraction. It can recognize and parse printed and handwritten text and convert it into a computer-readable format.

[0036] The feature alignment module 120 is used to align the comprehensive text feature vector and the chart image feature vector; wherein the comprehensive text feature vector is obtained by concatenating the chart text feature vector and the question text feature vector; and the question text feature vector is a feature vector generated according to the question corresponding to the chart to be processed.

[0037] Specifically, the input of the feature alignment module 120 is a comprehensive text feature vector and a chart image feature vector, and the output is an aligned comprehensive text feature vector and a chart image feature vector. The feature alignment module uses the Q-Former algorithm to perform alignment through an attention mechanism. The attention mechanism can effectively bridge the differences between text and image features. The core of the Q-Former algorithm is a multi-head self-attention mechanism, which can calculate the relationship between text and image features. Here, for the chart to be processed, the information extraction method of the present application is used to obtain a chart text feature vector and a chart image feature vector, respectively, and the obtained chart text feature vector and the question text feature vector obtained according to the question are spliced, and the spliced ​​comprehensive text feature vector is aligned with the chart image feature vector to obtain a one-to-one correspondence between the chart image feature vector and the comprehensive text feature vector after conversion into a feature vector.

[0038] The language encoding module 140 is used to generate a target answer corresponding to the question corresponding to the chart to be processed according to the aligned comprehensive text feature vector and the chart image feature vector.

[0039] Specifically, the input of the language encoding module 140 is the aligned comprehensive text feature vector and chart image feature vector, and the output is the target answer corresponding to the question corresponding to the chart to be processed. Here, a large language model is used to encode text information and generate a language encoding result. This step will include natural language processing technology (large language model) to understand and process text information, combined with image encoding results to generate more accurate question and answer results. In one embodiment, the language encoding module 140 of the present application uses qwen-14B as a large language model to process and encode text features, allowing it to more accurately understand the chart content of the chart to be processed based on the input chart image feature vector and comprehensive text feature vector, and answer the questions corresponding to the chart to be processed.

[0040] In an exemplary embodiment of the present application, the data set used for training the above-mentioned multimodal large model-based diagram question answering system is a comprehensive data set; the above-mentioned comprehensive training set is a data set obtained by integrating the data set of the LLaVAR model and the data set of the mPLUG-DocOwl model and cleaning the non-diagram data. And the above-mentioned multimodal large model-based diagram question answering system is trained according to the instruction fine-tuning technology.

[0041] Specifically, existing multimodal large models often rely on limited data sets for training and testing. These data sets often have limited coverage and cannot fully reflect the diversity and complexity of the real world. In one embodiment of the present application, the above-mentioned diagram question-answering system based on the multimodal large model can be a multimodal diagram question-answering model. When the multimodal diagram question-answering model is trained, in order to enable the multimodal diagram question-answering model to comprehensively handle various types of diagrams and data sources, the training data set used is a comprehensive data set; the above-mentioned comprehensive training set is a data set obtained by integrating the data set of the LLaVAR model and the data set of the mPLUG-DocOwl model and cleaning the non-diagram data. These two data sets contain rich multimodal data. This application will integrate these two data sets to create a larger and more comprehensive data set to ensure the quality and consistency of the data. During the integration process, data cleaning technology is used to remove noise and inconsistent data (non-diagram data) to ensure the high quality of the data set. Furthermore, based on the comprehensive data set, the instruction fine-tuning technology is used to train the above-mentioned diagram question-answering system based on the multimodal large model.

[0042] In this application, by using a larger and more comprehensive comprehensive dataset (a comprehensive dataset integrated from the dataset of the LLaVAR model and the dataset of the mPLUG-DocOwl model) for instruction fine-tuning, the multimodal graph question answering model can be better trained, thereby improving the performance and accuracy of the multimodal graph question answering model.

[0043] In summary, the multimodal large model-based chart question-answering system provided by this application relies on the LLaVAR dataset and the mPLUG-DocOwl dataset, and performs data cleaning technology to create a larger and more comprehensive comprehensive dataset to ensure the high quality and consistency of the data. And then it also combines open functions. And then it uses advanced OCR technology, including open function authentication technology and paddle OCR v3, to extract text information in the chart. Through the comprehensive voting system, this application can select the most reliable results from multiple OCR systems, thereby greatly improving the accuracy of text extraction. And the clip-vit-large-patch14-336 image encoder is used to process and encode the chart to be processed, which can extract the deep features of the chart to be processed, providing a strong foundation for subsequent language encoding. And then according to the Q-Former algorithm, efficient alignment between text feature vectors and image feature vectors is achieved, ensuring that the model can accurately understand the information and structure in the chart, thereby improving the accuracy of question and answer. Finally, by using the qwen-14B large language model to process and encode text features, a more accurate question-answering function is achieved, ensuring that the model can accurately understand and answer questions related to charts.

[0044] An embodiment of the present application also provides a computer program product, which includes program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method described above in this specification according to various exemplary embodiments of the present application.

[0045] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0046] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0047] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0048] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0049] The electronic device according to this embodiment of the present application is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0050] The electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: the at least one processor mentioned above, the at least one storage device mentioned above, and a bus connecting different system components (including storage devices and processors).

[0051] The storage stores program codes, which can be executed by the processor, so that the processor executes the steps described in the above “Exemplary Method” section of this specification according to various exemplary embodiments of the present application.

[0052] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read only memory (ROM).

[0053] The storage may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0054] The bus may represent one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0055] The electronic device can also communicate with one or more external devices (such as keyboards, pointing devices, Bluetooth devices, etc.), and can also communicate with one or more devices that enable users to interact with the electronic device, and / or communicate with any device (such as routers, modems, etc.) that enables the electronic device to communicate with one or more other computing devices. This communication can be carried out through an input / output (I / O) interface. In addition, the electronic device can also communicate with one or more networks (such as local area networks (LANs), wide area networks (WANs) and / or public networks, such as the Internet) through a network adapter. As shown in the figure, the network adapter communicates with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0056] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0057] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present application described in the above "Exemplary Method" section of the present specification.

[0058] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0059] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0060] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0061] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0062] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0063] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0064] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A diagram question answering method based on a multimodal large model, characterized in that: include: Extract target text information from a chart to be processed and obtain a chart text feature vector; wherein the target text information is text information corresponding to the intersection of a first text information set and a second text information set; the first text information set is a text information set obtained based on a first optical character recognition unit; the second text information set is a text information set obtained based on a second optical character recognition unit; the first optical character recognition unit adopts open function authentication technology or PaddleOCR V3 technology; the second optical character recognition unit adopts open function authentication technology or PaddleOCR V3 technology; and the first optical character recognition unit and the second optical character recognition unit adopt different technologies; Extracting target image information from the chart to be processed to obtain a chart image feature vector; Align the comprehensive text feature vector and the chart image feature vector; wherein the comprehensive text feature vector is obtained by concatenating the chart text feature vector and the question text feature vector; the question text feature vector is a feature vector generated according to the question corresponding to the chart to be processed; the feature alignment module uses the Q-Former algorithm to calculate the relationship between the chart text feature vector and the comprehensive text feature vector through the attention mechanism, and bridges the difference between the chart text feature vector and the comprehensive text feature vector to align the chart text feature vector and the comprehensive text feature vector, and obtain a one-to-one correspondence between the chart image feature vector and the comprehensive text feature vector; Generate a target answer corresponding to the question corresponding to the chart to be processed according to the aligned comprehensive text feature vector and chart image feature vector; The data set used for training the multimodal large model-based chart question answering system is a comprehensive data set; the comprehensive data set is a data set obtained by integrating the data set of the LLaVAR model and the data set of the mPLUG-DocOwl model and cleaning the non-chart data.

2. A diagram question answering system based on a multimodal large model, characterized in that: The diagram question-answering system based on the multimodal large model is used to implement the diagram question-answering method based on the multimodal large model described in claim 1. The diagram question-answering system based on the multimodal large model includes: an optical character recognition module, a feature alignment module, an image encoding module and a language encoding module, wherein: An optical character recognition module is used to extract target text information in a chart to be processed and obtain a chart text feature vector; wherein the target text information is text information corresponding to the intersection of a first text information set and a plurality of second text information sets; the first text information set is a text information set obtained based on a first optical character recognition unit; the second text information set is a text information set obtained based on a second optical character recognition unit; the first optical character recognition unit adopts open function authentication technology or PaddleOCR V3 technology; the second optical character recognition unit adopts open function authentication technology or PaddleOCR V3 technology; and the first optical character recognition unit and the second optical character recognition unit adopt different technologies; An image encoding module, used for extracting target image information from the chart to be processed and obtaining a chart image feature vector; A feature alignment module is used to align the comprehensive text feature vector and the chart image feature vector; wherein the comprehensive text feature vector is obtained by concatenating the chart text feature vector and the question text feature vector; the question text feature vector is a feature vector generated according to the question corresponding to the chart to be processed; the feature alignment module uses the Q-Former algorithm to calculate the relationship between the chart text feature vector and the comprehensive text feature vector through the attention mechanism, and bridges the difference between the chart text feature vector and the comprehensive text feature vector to align the chart text feature vector and the comprehensive text feature vector, and obtain a one-to-one correspondence between the chart image feature vector and the comprehensive text feature vector; A language encoding module, used for generating a target answer corresponding to the question corresponding to the chart to be processed according to the aligned comprehensive text feature vector and the chart image feature vector; The data set used for training the multimodal large model-based chart question answering system is a comprehensive data set; the comprehensive training set is a data set obtained by integrating the data set of the LLaVAR model and the data set of the mPLUG-DocOwl model and cleaning the non-chart data.

3. The multimodal large model-based diagram question answering system according to claim 2, characterized in that: The multimodal large model-based graphical question answering system is trained according to instruction fine-tuning technology.

4. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the method as claimed in claim 1.

5. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 4.

Images