Chart visual question-answering method based on dynamic routing and low-rank mixing and related device

Abstract

The invention discloses a chart visual question-answering method based on dynamic routing and low-rank mixing and a related device, and relates to the technical field of visual question-answering, and the method comprises the steps: obtaining original data containing to-be-processed charts and text questions, and generating corresponding answers through the processing of a pre-trained dual-channel fusion model. In the model, a visual encoder is provided to extract semantic features of complex layout of the chart, and a text encoder accurately converts text problem vectors, so that the feature matching degree of the chart and the problem is improved; the chart mixed connection structure dynamically fuses visual features of the data graph and the data table, and the cross-type chart adaptability is enhanced; according to the chart low-rank mixed structure, related vectors are weighted and fused through a LoRA routing network, low-rank adaptation and dynamic fusion are achieved in combination with output of a large language model layer, training parameters are reduced, and the zero sample question and answer ability is improved; the two-stage training strategy reduces the loss of the video memory. According to the method, the universality, accuracy and zero sample performance of chart visual questions and answers are effectively improved, and the training cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of visual question answering technology, and in particular to a graph visual question answering method and related devices based on dynamic routing and low-rank mixing. Background Art

[0002] With the explosive growth of data, data modalities are becoming increasingly diverse, encompassing a wide range of types, including text, images, audio, and structured data. Among these, data in the form of charts and graphs is the most information-rich and yet the most challenging to extract. For example, in fields such as engineering design and geological exploration, numerous data charts and graphs are generated. These charts and graphs often contain a wealth of valuable information, but they contain numerous technical terms and units, and typically only technical personnel in the field can understand the true meaning of the data. To ensure that non-experts in these fields can understand the meaning of these charts and graphs, simple OCR recognition can only identify the textual information but cannot understand the layout structure. Directly inputting this recognized data into a large model also makes it difficult for the model to understand the actual meaning of the data. Only by utilizing a multimodal large model to recognize and logically reason about data charts and graphs can we correctly understand the visual meaning of the charts and the actual meaning of the data.

[0003] Currently, there are two main types of models used for visual question answering: specialized visual language models and generalized multimodal large models. Specialized visual language models typically have fewer parameters and are generally applicable only to a single visual question answering domain. Generalized multimodal large models, on the other hand, not only contain a large amount of data but also, through reinforcement learning on massive datasets, possess strong image semantic understanding capabilities, making them adaptable to a variety of visual question answering domains.

[0004] Visual question answering tasks cover a wide range of areas, but existing models and methods are insufficient for visual question answering of data charts. Even general-purpose multimodal large models have certain limitations for this task. This is because data charts have complex layout structures and contain a relatively large amount of data information. This requires the model to perform logical reasoning based on a correct understanding of the layout of chart elements and the meaning of the data. Furthermore, when some target domains lack corresponding chart data for training, how to improve the zero-shot question answering capabilities of large multimodal models can enable the model to perform well even in the absence of target domain data. Summary of the Invention

[0005] The purpose of this application is to provide a graph visual question answering method and related devices based on dynamic routing and low-rank mixing, which can effectively improve the versatility, accuracy and zero-sample performance of graph visual question answering and reduce training costs.

[0006] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides a graph visual question answering method based on dynamic routing and low-rank mixing, comprising the following steps: Obtain raw data for chart visual question answering. The raw data includes the chart to be processed and text questions about the chart to be processed.

[0007] The original data of chart visual question answering is processed by a pre-trained dual-channel fusion model to generate answers to text questions; the dual-channel fusion model includes a visual encoder, a text encoder, a chart hybrid connection structure, a chart low-rank hybrid structure and a base large model; the visual encoder is used to convert the chart to be processed into a semantic feature representation; the text encoder is used to convert the text question into a vector representation; the chart hybrid connection structure is used to dynamically fuse the visual features of the data graph and data table, and input them together with the vector representation of the text question into the chart low-rank hybrid structure layer; the base large model includes several large language model layers, and the chart low-rank hybrid structure is used to dynamically weighted fuse the graph LoRA vector and the table LoRA vector through the LoRA routing network, and add them to the original output of each large language model layer to achieve low-rank adaptation and dynamic fusion of chart features; the dual-channel fusion model is trained using a two-stage training strategy.

[0008] Optionally, obtaining the original data of the chart visual question answering process specifically includes the following steps: Obtain a chart to be processed; the chart to be processed includes a data chart to be processed and a data table to be processed, the data chart to be processed is an image containing data information, the data chart to be processed is at least one of a line chart, a bar chart, and a scatter chart, and the data table to be processed is a table image containing data.

[0009] Get text questions for the chart to be processed; the text questions are consulting sentences in natural language form.

[0010] Optionally, the pre-trained dual-channel fusion model is used to process the raw data of the chart visual question answering to generate answers to the text questions, specifically including the following steps: The graph to be processed is encoded by the visual encoder to obtain a visual input vector.

[0011] The text question is encoded through the text encoder to obtain a text feature vector.

[0012] The visual input vector is processed through the graph hybrid connection structure to obtain the graph hybrid visual output.

[0013] The graph mixed visual output and text feature vector are input into the large language model layer together, and the original output of the large language model layer is processed by the graph low-rank mixed structure to generate answers to text questions.

[0014] Optionally, the graph hybrid connection structure includes a graph linear layer, a table linear layer, and a visual language routing network; processing the visual input vector through the graph hybrid connection structure to obtain a graph hybrid visual output specifically includes the following steps: The visual input vector is input into the graph linear layer and the table linear layer respectively to obtain the graph data vector and the table data vector.

[0015] The visual input vector is fed into the vision-language routing network to obtain a graph weight score.

[0016] The graph data vector and table data vector are weightedly fused based on the graph weight score to obtain the graph mixed visual output.

[0017] Optionally, the graph low-rank hybrid structure includes a large language model layer, graph LoRA, table LoRA, and LoRA routing network; the graph hybrid visual output and the text feature vector are input into the large language model layer, and the original output of the large language model layer is processed by the graph low-rank hybrid structure to generate an answer to the text question, specifically including the following steps: The mixed visual output of the chart and the text feature vector are taken as input vectors and input into the graph LoRA and table LoRA respectively to obtain the graph LoRA vector and table LoRA vector.

[0018] Input the input vector into the LoRA routing network and obtain the graph LoRA weight score.

[0019] The graph LoRA vector and table LoRA vector are weightedly fused based on the graph LoRA weight score to obtain the graph LoRA hybrid output.

[0020] The graph LoRA mixed output is added to the original output of the large language model layer to obtain a layer output of the graph low-rank mixed structure, which is used as the input of the next large language model layer.

[0021] Generate answers to text questions based on the output of the last layer of the graph's low-rank mixture structure.

[0022] Optionally, the two-stage training strategy includes a single-channel training stage and a dual-channel fusion stage.

[0023] During the single-channel training stage, the parameters of the visual encoder, text encoder, and large language model layers are frozen, and only the parameters of the graph linear layer and table linear layer in the graph hybrid connection structure, as well as the graph LoRA and table LoRA in the graph low-rank hybrid structure are fine-tuned to obtain the single-channel training parameters.

[0024] In the dual-channel fusion stage, the parameters of the visual encoder, text encoder, large language model layer and single-channel training parameters are frozen, and only the parameters of the visual-language routing network in the graph hybrid connection structure and the LoRA routing network in the graph low-rank hybrid structure are fine-tuned to obtain the dual-channel fusion model.

[0025] In the second aspect, the present application provides a graph visual question answering system based on dynamic routing and low-rank mixing, including the following functional modules: The chart visual question answering raw data acquisition module is used to obtain the chart visual question answering raw data; the chart visual question answering raw data includes the chart to be processed and the text questions for the chart to be processed.

[0026] A dual-channel fusion chart visual question answering module is used to process the chart visual question answering raw data through a pre-trained dual-channel fusion model to generate answers to the text questions; the dual-channel fusion model includes a visual encoder, a text encoder, a chart hybrid connection structure, a chart low-rank hybrid structure and a base large model; the visual encoder is used to convert the chart to be processed into a semantic feature representation; the text encoder is used to convert the text question into a vector representation; the chart hybrid connection structure is used to dynamically fuse the visual features of the data graph and the data table, and input them together with the vector representation of the text question into the chart low-rank hybrid structure layer; the base large model includes several large language model layers, and the chart low-rank hybrid structure is used to dynamically weight the fusion of the graph LoRA vector and the table LoRA vector through the LoRA routing network, and add them to the original output of each large language model layer to achieve low-rank adaptation and dynamic fusion of the chart features; the dual-channel fusion model is trained using a two-stage training strategy.

[0027] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the graph visual question answering method based on dynamic routing and low-rank mixing described above.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the graph visual question answering method based on dynamic routing and low-rank mixing described above.

[0029] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the graph visual question answering method based on dynamic routing and low-rank mixing described above.

[0030] According to the specific embodiments provided in this application, this application discloses the following technical effects: The present application provides a diagram visual question answering method and related devices based on dynamic routing and low-rank mixing, which processes the diagram to be processed and the text questions for the diagram to be processed through a pre-trained dual-channel fusion model. Specifically, the diagram to be processed is converted into a semantic feature representation through a visual encoder, which can effectively extract the key visual information in the complex layout structure of the diagram and solve the problem of insufficient feature extraction caused by the complex layout of the data diagram; the text question is converted into a vector representation through a text encoder, which can accurately capture the semantic information of the text question and improve the semantic matching degree between the question and the diagram features; the visual features of the data diagram and the data table are dynamically integrated through the diagram hybrid connection structure, which can adaptively distinguish the structural differences between the data diagram and the data table, enhance the adaptability of the model to different types of diagrams, and improve the cross-type diagram processing. The versatility of the method; through the low-rank hybrid structure of the graph, the LoRA routing network is used to dynamically weight the fusion of graph LoRA vectors and table LoRA vectors, which can flexibly adjust the attention weights of different types of graph features and improve the pertinence of feature fusion. The low-rank adaptation characteristics of LoRA can significantly reduce training parameters and reduce video memory consumption. At the same time, it is added to the original output of the large language model layer to achieve low-rank adaptation and dynamic fusion of graph features, and obtain answers to the text questions, effectively improving the model's zero-sample question-answering ability when there is a lack of training data in the target field; during pre-training, a two-stage training strategy is adopted to train the dual-channel fusion model, which can optimize the parameters of different modules in stages, while ensuring the accuracy of the model, further reducing the video memory loss during training, so that the model can be efficiently trained on consumer-grade hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 A flowchart of a graph visual question answering method based on dynamic routing and low-rank mixing is provided in accordance with an embodiment of the present application.

[0033] Figure 2 A schematic diagram of a dual-channel fusion model in a graph visual question answering method based on dynamic routing and low-rank mixing provided in one embodiment of the present application.

[0034] Figure 3 A schematic diagram of the single-channel training phase of a dual-channel fusion model in a graph visual question answering method based on dynamic routing and low-rank mixing provided in one embodiment of the present application.

[0035] Figure 4A schematic diagram of the dual-channel fusion stage of a dual-channel fusion model in a graph visual question answering method based on dynamic routing and low-rank mixing provided in one embodiment of the present application.

[0036] Figure 5 A schematic diagram of the functional modules of a graph visual question answering system based on dynamic routing and low-rank mixing provided in one embodiment of the present application.

[0037] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0039] Existing large multimodal models are difficult to directly apply to visual question answering tasks for data charts. To address this problem, this application aims to improve the versatility, accuracy, and zero-sample performance of visual question answering on charts and reduce training costs.

[0040] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0041] The embodiment of the present application provides a graph visual question answering method based on dynamic routing and low-rank mixing. In an exemplary embodiment, Figure 1 As shown, the following steps are included: A1. Obtaining raw data for visual question answering of charts; the raw data for visual question answering of charts includes the chart to be processed and text questions about the chart to be processed. In this embodiment, step A1 specifically includes the following steps: A11. Obtain a chart to be processed; the chart to be processed includes a data chart to be processed and a data table to be processed. The data chart to be processed is an image containing data information, and is at least one of a line chart, a bar chart, and a scatter chart. The data table to be processed is a table image containing data.

[0042] A12. Obtain text questions for the chart to be processed; the text questions are consulting statements in natural language.

[0043] A2. Process the original data of visual question answering with the pre-trained dual-channel fusion model to generate answers to text questions. Specifically, Figure 2As shown, the dual-channel fusion model includes a visual encoder, a text encoder, a graph hybrid connection structure, a graph low-rank hybrid structure and a base large model; the visual encoder is used to convert the graph to be processed into a semantic feature representation; the text encoder is used to convert the text question into a vector representation; the graph hybrid connection structure is used to dynamically fuse the visual features of the data graph and the data table, and input them together with the vector representation of the text question into the graph low-rank hybrid structure layer; the base large model includes several large language model layers, and the graph low-rank hybrid structure is used to dynamically weight the fusion graph LoRA vector and the table LoRA vector through the LoRA routing network, and add them to the original output of each large language model layer to achieve low-rank adaptation and dynamic fusion of graph features; the dual-channel fusion model is trained using a two-stage training strategy. In this embodiment, the number of layers of the large language model layer and the graph low-rank hybrid structure are both L.

[0044] Specifically, the base large model is primarily used for decoding multimodal information. In this example, Qwen-7B was selected as the base large model. Qwen-7B has a large number of parameters and has been pre-trained on multiple large-scale, high-quality datasets. It demonstrates excellent performance in areas such as commonsense reasoning and code generation. Qwen-7B employs supervised learning for fine-tuning, improving the model's performance on downstream tasks. Furthermore, Qwen-7B utilizes Proximal Policy Optimization (PPO) for reinforcement learning. By designing components such as the reward model, the model's generated responses are more aligned with human preferences.

[0045] In terms of model architecture, Qwen-7B uses rotational position encoding to integrate position information into the self-attention mechanism, thereby improving the processing ability of long input sequences. In addition, the model uses RMS Norm to replace the traditional layer normalization technology, reducing computational complexity while improving operational efficiency. In addition, Qwen uses SwiGLU as the activation module, and by combining the Swish activation function with the gated linear unit, it enhances the nonlinear ability of the model. Of course, this does not constitute a limitation of the present application. In other optional embodiments, the base model can also be replaced by a multimodal model with a similar model architecture such as ChatGLM, Vicuna, Alpaca, etc. It is understandable that Figure 2 Even the following Figure 3 and Figure 4 The contents shown in the data tables and data diagrams as inputs to the dual-channel fusion model are not to be construed as limitations on the present application and are for illustrative purposes only and have no practical significance.

[0046] Specifically in this embodiment, step A2 includes the following steps: A21. Encode the graph to be processed through the visual encoder to obtain a visual input vector. The visual encoder is a core component in computer vision tasks, responsible for converting raw image data into high-level semantic feature representations. With the development of deep learning, the architecture of the visual encoder has gradually evolved from traditional convolutional neural networks (CNNs) to Transformer-based models (such as ViT), and has demonstrated strong performance in different tasks (classification, detection, segmentation, etc.). In this embodiment, Vision Transformer (ViT) is used as the visual encoder, which is a visual model based on the Transformer architecture. The core idea of ​​ViT is to divide the image into several small patches (patches), and regard these small patches as tokens similar to those in natural language processing tasks, and then use a standard Transformer encoder for processing.

[0047] A22. Encode the text question using a text encoder to obtain a text feature vector. A text encoder is a model or algorithm component that converts natural language text into a numerical representation (usually in vector form). In this example, the word embedding layer of the Vicuna model is used as the text encoder.

[0048] A23. Process the visual input vector using a graph hybrid connection structure to obtain a graph hybrid visual output. In this embodiment, the graph hybrid connection structure includes a graph linear layer, a surface linear layer, and a visual language routing network; step A23 specifically includes the following steps: A231. Input the visual input vector into the graph linear layer and table linear layer respectively to obtain the graph data vector and table data vector. The data graph or data table passes through the visual encoder to form the visual input vector , input them into the graph linear layer and the surface linear layer respectively, and obtain the graph data vector and table data vector , the process can be expressed by formula (1) and formula (2): (1) (2) in, represents the visual input vector, represents the graph linear layer parameters, represents the linear layer parameters, represents the parameters of the visual language routing network, d vin is the dimension of the visual input vector.

[0049] A232, input the visual input vector into the visual language routing network to obtain the graph weight score. Input the visual language routing network to get the graph weight score , which can be expressed by formula (3): (3) in, Represents the parameters of the vision-language routing network.

[0050] A233, based on the graph weight score, weighted fusion is performed on the graph data vector and the table data vector to obtain the graph mixed visual output. Perform weighted fusion of graph data vectors and table data vectors to obtain mixed visual output of graphs , the specific calculation process is shown in formula (4): (4) in, Is a vector connection function, mainly used to connect vectors and Connect them.

[0051] A24. Input the graph hybrid visual output and the text feature vector into the large language model layer, and process the original output of the large language model layer through the graph low-rank hybrid structure to generate an answer to the text question. In this embodiment, the graph low-rank hybrid structure includes a large language model layer, graph LoRA, table LoRA, and LoRA routing network; step A24 specifically includes the following steps: A241, the graph mixed visual output and the text feature vector are used as input vectors, respectively input to the graph LoRA and table LoRA, to obtain the graph LoRA vector and table LoRA vector. Fusion with text feature vector as input vector , respectively input to the graph LoRA and table LoRA, and obtain the graph LoRA vector and table LoRA vector . As shown in formula (5) and formula (6) respectively: (5) (6) in, is the upper projection layer parameter of LoRA, is the lower projection layer parameter of LoRA, It is the upper projection layer parameter of LoRA. It is the lower projection layer parameter of table LoRA, d in is the input vectorh in Dimensions, d out is the dimension of the output vector, d lora_r is the dimension of each projection layer parameter, the superscript T is the transposition symbol, RMS ( ) is the RMS Norm normalization function.

[0052] A242, input the input vector into the LoRA routing network to obtain the LoRA weight score of the graph. Input into the LoRA routing network to get the LoRA weight score of the graph , as shown in formula (7): (7) in, Represents the parameters of the vision-language routing network.

[0053] A243, based on the graph LoRA weight score, performs weighted fusion on the graph LoRA vector and the table LoRA vector to obtain the graph LoRA mixed output. Perform weighted fusion of the graph LoRA vector and the table LoRA vector to obtain the graph LoRA mixed visual output , the specific calculation process is shown in formula (8): (8) A244. Add the graph LoRA mixed output to the original output of the large language model layer to obtain a layer output of the graph low-rank mixed structure as the input of the next large language model layer.

[0054] Then, in step A244, the original output of the large language model layer is converted to and Adding them together, we can get the output of one layer of the low-rank mixed structure of the graph , as shown in formula (9): (9) In this embodiment, the low-rank hybrid structure of the graph is designed to be multi-layered, which is determined by the multi-layer structural characteristics of the large base model on which it relies and the task requirements. For example, in this embodiment, the Qwen-7B selected by the large base model adopts a multi-layer Transformer architecture, each layer of which undertakes the function of gradual feature abstraction and semantic deepening. As an adaptation module embedded in the large base model, the low-rank hybrid structure of the graph needs to correspond to the multi-layer structure of the original model, and achieve refined adaptation of features of different depths through layer-by-layer processing to avoid feature loss or insufficient fusion due to single-layer processing.

[0055] A245. Generate answers to text questions based on the output of the last layer of the graph low-rank mixture structure. Assume that the output of the last layer of the graph low-rank mixture layer is ,So It also needs to go through the linear layer The calculation of the final answer is the distribution of the predicted The probability is shown in formula (10): (10) in, is the vocabulary length, The first i word.

[0056] In an exemplary embodiment, the two-stage training strategy includes a single-channel training stage and a dual-channel fusion stage.

[0057] In the single-channel training phase, Figure 3 As shown, the parameters of the visual encoder, text encoder, and large language model layers are frozen. Only the parameters of the graph linear layer and table linear layer in the graph hybrid connection structure, as well as the graph LoRA and table LoRA parameters in the graph low-rank hybrid structure, are fine-tuned to obtain single-channel training parameters. During the single-channel training phase, the model input has two cases: i) data graph + text question, which trains parameters related to the data graph; ii) data table + text question, which trains parameters related to the data table. After the single-channel training phase, two parameter modules are obtained: ① Parameters of the graph LoRA and graph linear layers related to the data graph; ② Parameters of the table LoRA and table linear layers related to the data table.

[0058] In the dual-channel fusion stage, such as Figure 4 As shown in the figure, the parameters of the visual encoder, text encoder, large language model layer, and single-channel training parameters are frozen, and only the parameters of the visual language routing network in the graph hybrid connection structure and the LoRA routing network in the graph low-rank hybrid structure are fine-tuned to obtain a dual-channel fusion model. In the dual-channel fusion stage, not only the original parameters of the text encoder, visual encoder, and large language model need to be frozen, but also the parameters of the graph linear layer, table linear layer, graph LoRA, and table LoRA obtained in the previous stage need to be frozen. Then, the LoRA routing network and the visual language routing network are set to a state where the parameters can be trained, that is, only the routing parameters of these two parts are fine-tuned. After the dual-channel fusion stage, the LoRA routing network and the visual language routing network with strong generalization ability can be obtained, which can better fuse the dual channels of the graph.

[0059] The above scheme in this embodiment trains the multimodal large model by combining LoRA and the visual language adapter, which reduces the training parameters while enabling the model to have good data learning capabilities. And through training with a small amount of data, the model can obtain strong cross-domain generalization capabilities. Use the outputs of the visual language routing network to dynamically fuse the graph linear layer and the table linear layer, and design the LoRA routing network to dynamically fuse the outputs of the graph LoRA and the table LoRA, so that the dual-channel fusion model as a whole has stronger cross-domain generalization capabilities. Since the dual-channel fusion model contains a more complex module structure, in order to reduce the memory loss during the training process, this embodiment proposes a two-stage training method, which can distribute the memory required for the entire training process to different stages, thereby ensuring that the model can be smoothly trained on a consumer-grade GPU.

[0060] Based on the same inventive concept, the embodiment of the present application also provides a system for implementing the above-mentioned graph visual question answering method based on dynamic routing and low-rank mixing. The implementation solution provided by the system is similar to the implementation solution described in the above method. In an exemplary embodiment, Figure 5 As shown in the figure, a graph visual question answering system based on dynamic routing and low-rank mixing is provided, which includes the following functional modules: The chart visual question answering raw data acquisition module is used to obtain the chart visual question answering raw data; the chart visual question answering raw data includes the chart to be processed and the text questions for the chart to be processed.

[0061] A dual-channel fusion chart visual question answering module is used to process the chart visual question answering raw data through a pre-trained dual-channel fusion model to generate answers to the text questions; the dual-channel fusion model includes a visual encoder, a text encoder, a chart hybrid connection structure, a chart low-rank hybrid structure and a large language model layer; the visual encoder is used to convert the chart to be processed into a semantic feature representation; the text encoder is used to convert the text question into a vector representation; the chart hybrid connection structure is used to dynamically fuse the visual features of the data graph and the data table, and input them together with the vector representation of the text question into the chart low-rank hybrid structure layer; the chart low-rank hybrid structure is used to dynamically weight the fusion of the graph LoRA vector and the table LoRA vector through the LoRA routing network, and add them to the original output of the large language model layer to achieve low-rank adaptation and dynamic fusion of the chart features; the dual-channel fusion model is trained using a two-stage training strategy.

[0062] certainly, Figure 5 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different functions. Figure 5 One or at least two components of the system shown.

[0063] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a graph visual question answering method based on dynamic routing and low-rank mixing provided in the previous embodiment can be implemented.

[0064] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0065] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0066] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0067] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0068] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0069] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0070] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0071] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0072] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A graph visual question answering method based on dynamic routing and low-rank hybrid, characterized by: include: Obtaining original data of visual question answering of charts; the original data of visual question answering of charts includes a chart to be processed and text questions for the chart to be processed; The graph visual question answering raw data is processed by a pre-trained dual-channel fusion model to generate answers to the text questions; the dual-channel fusion model includes a visual encoder, a text encoder, a graph hybrid connection structure, a graph low-rank hybrid structure and a base large model; the visual encoder is used to convert the graph to be processed into a semantic feature representation; The text encoder is used to convert the text question into a vector representation; the graph hybrid connection structure is used to dynamically fuse the visual features of the data graph and the data table, and input them together with the vector representation of the text question into the graph low-rank hybrid structure layer; the base large model includes several large language model layers, and the graph low-rank hybrid structure is used to dynamically weight the fusion of graph LoRA vectors and table LoRA vectors through the LoRA routing network, and add them to the original output of each large language model layer to achieve low-rank adaptation and dynamic fusion of graph features; the dual-channel fusion model is trained using a two-stage training strategy.

2. The graph visual question answering method based on dynamic routing and low-rank mixing according to claim 1 is characterized in that: Obtain the original data of chart visual question answering, including: Obtaining a chart to be processed; the chart to be processed includes a data chart to be processed and a data table to be processed, the data chart to be processed is an image containing data information, the data chart to be processed is at least one of a line chart, a bar chart, and a scatter chart, and the data table to be processed is a table image containing data; A text question for the chart to be processed is obtained; the text question is a consultation sentence in a natural language form.

3. The graph visual question answering method based on dynamic routing and low-rank mixing according to claim 1 is characterized in that: The pre-trained dual-channel fusion model is used to process the raw data of the visual question answering chart to generate answers to the text questions, specifically including: Encoding the to-be-processed chart by the visual encoder to obtain a visual input vector; Encode the text question by the text encoder to obtain a text feature vector; Processing the visual input vector through the graph hybrid connection structure to obtain a graph hybrid visual output; The graph mixed visual output and the text feature vector are input into the large language model layer together, and the original output of the large language model layer is processed by the graph low-rank mixed structure to generate an answer to the text question.

4. The graph visual question answering method based on dynamic routing and low-rank mixing according to claim 3 is characterized in that: The graph hybrid connection structure includes a graph linear layer, a table linear layer, and a visual language routing network; the visual input vector is processed by the graph hybrid connection structure to obtain a graph hybrid visual output, specifically including: Inputting the visual input vector into the graph linear layer and the table linear layer respectively to obtain a graph data vector and a table data vector; Inputting the visual input vector into the visual language routing network to obtain a graph weight score; The graph data vector and the table data vector are weightedly fused based on the graph weight score to obtain the graph mixed visual output.

5. The graph visual question answering method based on dynamic routing and low-rank mixing according to claim 4, characterized in that: The graph low-rank hybrid structure includes a large language model layer, graph LoRA, table LoRA, and LoRA routing network; the graph hybrid visual output and the text feature vector are input into the large language model layer, and the original output of the large language model layer is processed by the graph low-rank hybrid structure to generate an answer to the text question, specifically including: The graph mixed visual output and the text feature vector are used as input vectors, and are input into the graph LoRA and the table LoRA respectively to obtain a graph LoRA vector and a table LoRA vector; Inputting the input vector into the LoRA routing network to obtain a graph LoRA weight score; performing weighted fusion on the graph LoRA vector and the table LoRA vector based on the graph LoRA weight score to obtain a graph LoRA hybrid output; Adding the graph LoRA mixed output to the original output of the large language model layer to obtain a layer output of the graph low-rank mixed structure as the input of the next large language model layer; An answer to the text question is generated based on the output of the last layer of the graph low-rank mixture structure.

6. The graph visual question answering method based on dynamic routing and low-rank mixing according to claim 5, characterized in that: The two-stage training strategy includes a single-channel training stage and a dual-channel fusion stage; In the single-channel training phase, the parameters of the visual encoder, the text encoder, and the large language model layer are frozen, and only the parameters of the graph linear layer and the surface linear layer in the graph hybrid connection structure, and the graph LoRA and surface LoRA in the graph low-rank hybrid structure are fine-tuned to obtain single-channel training parameters; In the dual-channel fusion stage, the parameters of the visual encoder, the text encoder, the large language model layer and the single-channel training parameters are frozen, and only the parameters of the visual language routing network in the graph hybrid connection structure and the LoRA routing network in the graph low-rank hybrid structure are fine-tuned to obtain the dual-channel fusion model.

7. A graph visual question answering system based on dynamic routing and low-rank hybrid, characterized by: include: A chart visual question answering raw data acquisition module is used to acquire chart visual question answering raw data; the chart visual question answering raw data includes a chart to be processed and text questions for the chart to be processed; A dual-channel fusion chart visual question answering module is used to process the chart visual question answering raw data using a pre-trained dual-channel fusion model to generate answers to the text questions; the dual-channel fusion model includes a visual encoder, a text encoder, a chart hybrid connection structure, a chart low-rank hybrid structure, and a large base model; the visual encoder is used to convert the chart to be processed into a semantic feature representation; The text encoder is used to convert the text question into a vector representation; the graph hybrid connection structure is used to dynamically fuse the visual features of the data graph and the data table, and input them together with the vector representation of the text question into the graph low-rank hybrid structure layer; the base large model includes several large language model layers, and the graph low-rank hybrid structure is used to dynamically weight the fusion of graph LoRA vectors and table LoRA vectors through the LoRA routing network, and add them to the original output of each large language model layer to achieve low-rank adaptation and dynamic fusion of graph features; the dual-channel fusion model is trained using a two-stage training strategy.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the graph visual question answering method based on dynamic routing and low-rank mixing according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the graph visual question answering method based on dynamic routing and low-rank mixing according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the graph visual question answering method based on dynamic routing and low-rank mixing according to any one of claims 1 to 6 is implemented.

Images