Graph visual question answering method based on dynamic routing and low rank mixing and related device

Abstract

The application discloses a chart visual question answering method based on dynamic routing and low rank mixing and a related device, relates to the technical field of visual question answering, and comprises the following steps: acquiring original data containing a chart to be processed and a text question, and generating a corresponding answer by processing through a pre-trained double-channel fusion model. In the model, a visual encoder is provided to extract semantic features of a complex layout of the chart, a text encoder is used to accurately convert a text question vector, and the two improve the feature matching degree of the chart and the question; a chart mixed connection structure dynamically fuses visual features of a data graph and a data table, and enhances the cross-type chart adaptability; a chart low rank mixing structure fuses related vectors by means of a LoRA routing network, realizes low rank adaptation and dynamic fusion in combination with a large language model layer output, reduces training parameters, and improves zero sample question answering capability; and a two-stage training strategy reduces the memory loss. The method effectively improves the generality, accuracy and zero sample performance of chart visual question answering, and reduces the training cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual question answering, in particular to a graph visual question answering method based on dynamic routing and low-rank mixing and a related device. BACKGROUND

[0002] With the explosive growth of data, data modalities become increasingly rich, covering text, images, audio, and structured data, and many other types. Among them, the data in the form of graphs contains the most information and is the most difficult to extract. For example, in the fields of engineering design, geological exploration, etc., many data graphs are generated, which often contain a lot of valuable information. However, these data graphs contain many professional terms and units, and only technical personnel in this field can understand the true meaning of the data. If an outsider who is not in this field wants to understand what these graphs express, simply using OCR to recognize the text information in the graphs can only recognize the text information, but cannot understand the layout structure. If the recognized data is directly input into a large model, the large model also cannot understand the actual meaning of the data. Only by using a multi-modal large model to recognize and logically infer the data graphs can the visual meaning of the graphs and the actual meaning of the data be correctly understood.

[0003] At present, there are two kinds of models used for visual question answering tasks, one is a specialized visual language model, and the other is a general multi-modal large model. The specialized visual language model usually has a small number of parameters and is usually only applicable to a single visual question answering field. The general multi-modal large model not only contains a large amount of data, but also has a strong image semantic understanding ability through reinforcement learning on a large amount of data sets, and can adapt to multiple visual question answering fields.

[0004] Visual question answering tasks cover a wide range, but the existing model methods are not sufficient to support the visual question answering of data graphs. Even the general multi-modal large model has certain limitations in this type of task. This is because data graphs have complex layout structures and contain a large amount of data information, which requires the model to logically infer on the basis of correctly understanding the layout of the graph elements and the meaning of the data. Moreover, when some target fields lack corresponding graph data for training, how to improve the zero-shot question answering ability of the multi-modal large model can make the model perform well even in the absence of data in the target field. SUMMARY

[0005] The purpose of the present application is to provide a graph visual question answering method based on dynamic routing and low-rank mixing and a related device, which can effectively improve the generality, accuracy and zero-shot performance of graph visual question answering, and reduce the training cost.

[0006] To achieve the above object, the application provides the following scheme:

[0007] In a first aspect, the application provides a graph visual question answering method based on dynamic routing and low rank mixing, comprising the following steps:

[0008] Obtaining graph visual question answering original data; the graph visual question answering original data includes a to-be-processed graph and a text question for the to-be-processed graph.

[0009] Processing the graph visual question answering original data through a pre-trained double-channel fusion model to generate an answer to the text question; the double-channel fusion model includes a visual encoder, a text encoder, a graph mixed connection structure, a graph low rank mixing structure and a base large model; the visual encoder is used to convert the to-be-processed graph into a semantic feature representation; the text encoder is used to convert the text question into a vector representation; the graph mixed connection structure is used to dynamically fuse the visual features of the data graph and the data table, and input the vector representation of the text question into the graph low rank mixing structure layer together; the base large model includes a plurality of large language model layers, and the graph low rank mixing structure is used to dynamically weight and fuse a graph LoRA vector and a table LoRA vector through a LoRA routing network, and add the original output of each large language model layer to realize low rank adaptation and dynamic fusion of graph features; the double-channel fusion model is trained by using a two-stage training strategy.

[0010] Optionally, obtaining the graph visual question answering original data specifically includes the following steps:

[0011] Obtaining the to-be-processed graph; the to-be-processed graph includes a to-be-processed data graph and a to-be-processed data table, the to-be-processed data graph is a data information containing image, the to-be-processed data graph is at least one of a line chart, a column chart and a scatter chart, and the to-be-processed data table is a data containing table image.

[0012] Obtaining the text question for the to-be-processed graph; the text question is a natural language form of a consultation sentence.

[0013] Optionally, processing the graph visual question answering original data through the pre-trained double-channel fusion model to generate the answer to the text question specifically includes the following steps:

[0014] Encoding the to-be-processed graph through the visual encoder to obtain a visual input vector.

[0015] Encoding the text question through the text encoder to obtain a text feature vector.

[0016] Processing the visual input vector through the graph mixed connection structure to obtain a graph mixed visual output.

[0017] The chart-mixed visual output and the text feature vector are jointly input into a large language model layer, and an original output of the large language model layer is processed through a chart-low-rank mixed structure to generate an answer to the text question.

[0018] Optionally, the chart-mixed connection structure includes a graph linear layer, a table linear layer and a visual language routing network; the visual input vector is processed through the chart-mixed connection structure to obtain a chart-mixed visual output, and the method specifically includes the following steps:

[0019] The visual input vector is input into the graph linear layer and the table linear layer to obtain a graph data vector and a table data vector.

[0020] The visual input vector is input into the visual language routing network to obtain a chart weight score.

[0021] The graph data vector and the table data vector are weighted and fused based on the chart weight score to obtain the chart-mixed visual output.

[0022] Optionally, the chart-low-rank mixed structure includes a large language model layer, a graph LoRA, a table LoRA and a LoRA routing network; the chart-mixed visual output and the text feature vector are jointly input into the large language model layer, and an original output of the large language model layer is processed through the chart-low-rank mixed structure to generate an answer to the text question, and the method specifically includes the following steps:

[0023] The chart-mixed visual output and the text feature vector are jointly input as input vectors into the graph LoRA and the table LoRA to obtain a graph LoRA vector and a table LoRA vector.

[0024] The input vectors are input into the LoRA routing network to obtain a chart LoRA weight score.

[0025] The graph LoRA vector and the table LoRA vector are weighted and fused based on the chart LoRA weight score to obtain a chart LoRA mixed output.

[0026] The chart LoRA mixed output and the original output of the large language model layer are added to obtain a one-layer output of the chart-low-rank mixed structure, which is used as an input of a next-layer large language model layer.

[0027] Based on an output of a last layer of the chart-low-rank mixed structure, an answer to the text question is generated.

[0028] Optionally, the two-stage training strategy includes a single-channel training stage and a dual-channel fusion stage.

[0029] In the single-channel training stage, the parameters of the visual encoder, the text encoder and the large language model layer are frozen, only the parameters of the graph linear layer, the table linear layer in the graph-table mixed connection structure and the graph LoRA and the table LoRA in the graph-table low-rank mixed structure are fine-tuned, to obtain single-channel training parameters.

[0030] In the dual-channel fusion stage, the parameters of the visual encoder, the text encoder, the large language model layer and the single-channel training parameters are frozen, only the parameters of the visual language routing network in the graph-table mixed connection structure and the LoRA routing network in the graph-table low-rank mixed structure are fine-tuned, to obtain a dual-channel fusion model.

[0031] In a second aspect, the present application provides a graph-table visual question answering system based on dynamic routing and low-rank mixing, comprising the following functional modules:

[0032] A graph-table visual question answering original data acquisition module is configured to acquire graph-table visual question answering original data; the graph-table visual question answering original data comprises a to-be-processed graph table and a text question for the to-be-processed graph table.

[0033] A dual-channel fusion graph-table visual question answering module is configured to process the graph-table visual question answering original data through a pre-trained dual-channel fusion model to generate an answer to the text question; the dual-channel fusion model comprises a visual encoder, a text encoder, a graph-table mixed connection structure, a graph-table low-rank mixed structure and a base large model; the visual encoder is configured to convert the to-be-processed graph table into a semantic feature representation; the text encoder is configured to convert the text question into a vector representation; the graph-table mixed connection structure is configured to dynamically fuse the visual features of the data graph and the data table, and input the vector representation of the text question into the graph-table low-rank mixed structure layer together; the base large model comprises a plurality of large language model layers, the graph-table low-rank mixed structure is configured to dynamically weight and fuse the graph LoRA vector and the table LoRA vector through the LoRA routing network, and add the original output of each large language model layer to realize low-rank adaptation and dynamic fusion of the graph-table features; the dual-channel fusion model is trained by using a two-stage training strategy.

[0034] In a third aspect, the present application provides a computer device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the graph-table visual question answering method based on dynamic routing and low-rank mixing described above.

[0035] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the graph-table visual question answering method based on dynamic routing and low-rank mixing described above.

[0036] In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method for chart visual question answering based on dynamic routing and low-rank mixing described above.

[0037] According to the specific embodiments provided in the present application, the present application discloses the following technical effects:

[0038] The present application provides a method for chart visual question answering based on dynamic routing and low-rank mixing and related devices. The pre-trained double-channel fusion model is used to process the to-be-processed chart and the text question for the to-be-processed chart. Specifically, the visual encoder is used to convert the to-be-processed chart into semantic feature representation, which can effectively extract key visual information in the complex layout structure of the chart, solving the problem of insufficient feature extraction caused by complex data chart layout. The text encoder is used to convert the text question into vector representation, which can accurately capture the semantic information of the text question and improve the semantic matching degree of the question and the chart features. The chart mixed connection structure is used to dynamically fuse the visual features of the data chart and the data table, which can adaptively distinguish the structural differences between the data chart and the data table, enhance the adaptability of the model to different types of charts, and improve the universality of cross-type chart processing. The chart low-rank mixing structure uses the LoRA routing network to dynamically weight and fuse the chart LoRA vector and the table LoRA vector, which can flexibly adjust the attention weight of different types of chart features, improve the pertinence of feature fusion, and the low-rank adaptation feature of LoRA can significantly reduce the training parameters and reduce the memory consumption. At the same time, the original output of the large language model layer is added to realize the low-rank adaptation and dynamic fusion of the chart features, obtain the answer to the text question, and effectively improve the zero-shot question answering ability of the model when there is a lack of training data in the target field. During pre-training, the two-stage training strategy is used to train the double-channel fusion model, which can optimize the parameters of different modules in stages, further reduce the memory loss in the training process while ensuring the accuracy of the model, and enable the model to be trained efficiently on consumer-grade hardware. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0040] Figure 1 A flowchart of a method for chart visual question answering based on dynamic routing and low-rank mixing according to an embodiment of the present application is shown.

[0041] Figure 2A schematic diagram of a double-channel fusion model in a graph visual question answering method based on dynamic routing and low-rank mixing provided by an embodiment of the present application.

[0042] Figure 3 A schematic diagram of a single-channel training stage of a double-channel fusion model in a graph visual question answering method based on dynamic routing and low-rank mixing provided by an embodiment of the present application.

[0043] Figure 4 A schematic diagram of a double-channel fusion stage of a double-channel fusion model in a graph visual question answering method based on dynamic routing and low-rank mixing provided by an embodiment of the present application.

[0044] Figure 5 A functional module schematic diagram of a graph visual question answering system based on dynamic routing and low-rank mixing provided by an embodiment of the present application.

[0045] Figure 6 A structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0047] Existing multi-modal large models are difficult to be directly applied to data graph-oriented visual question answering tasks. To solve this problem, the present application aims to improve the generality, accuracy and zero-shot performance of graph visual question answering, and reduce the training cost.

[0048] To make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0049] The graph visual question answering method based on dynamic routing and low-rank mixing provided by the embodiments of the present application, in an exemplary embodiment, as shown in Figure 1 includes the following steps:

[0050] A1, obtaining graph visual question answering original data; the graph visual question answering original data includes a to-be-processed graph and a text question for the to-be-processed graph. In the present embodiment, step A1 specifically includes the following steps:

[0051] A11, acquire a chart to be processed; the chart to be processed includes a data chart to be processed and a data table to be processed, the data chart to be processed is an image containing data information, the data chart to be processed is at least one of a line chart, a column chart and a scatter chart, and the data table to be processed is a table image containing data.

[0052] A12, acquire a text question for the chart to be processed; the text question is a consultation sentence in natural language form.

[0053] A2, processing the chart visual question and answer original data through the pre-trained double-channel fusion model to generate an answer to the text question. Specifically, as shown in Figure 2 the double-channel fusion model includes a visual encoder, a text encoder, a chart mixed connection structure, a chart low-rank mixed structure and a base large model; the visual encoder is used to convert the chart to be processed into a semantic feature representation; the text encoder is used to convert the text question into a vector representation; the chart mixed connection structure is used to dynamically fuse the visual features of the data chart and the data table, and input the vector representation of the text question into the chart low-rank mixed structure layer; the base large model includes a plurality of large language model layers, and the chart low-rank mixed structure is used to dynamically weight and fuse the chart LoRA vector and the table LoRA vector through the LoRA routing network, and add the original output of each layer of the large language model layer to realize the low-rank adaptation and dynamic fusion of the chart features; the double-channel fusion model is trained by using a two-stage training strategy. In this embodiment, the number of layers of the large language model layer and the chart low-rank mixed structure is L.

[0054] Specifically, the base large model is mainly used for decoding of multi-modal information, and Qwen-7B is selected as the base large model in this embodiment. Qwen-7B has a large number of parameters and is pre-trained on multiple large-scale high-quality data sets, and has excellent performance in common sense reasoning, code generation and other fields. Qwen-7B uses a supervised learning method for fine-tuning to improve the performance of the model in downstream tasks. In addition, Qwen-7B uses the proximal policy optimization (PPO) for reinforcement learning, and through the design of reward model and other components, the answers generated by the model can be more in line with human preferences.

[0055] In the model architecture, Qwen-7B uses rotational position encoding to integrate position information into the self-attention mechanism, thereby improving the processing capability for long input sequences. Moreover, the model uses RMS Norm to replace the traditional layer normalization technique, reducing the computational complexity while improving the running efficiency. In addition, Qwen chooses SwiGLU as the activation module, which combines the Swish activation function and the gated linear unit to enhance the nonlinearity of the model

[0056] Of course, this does not constitute a limitation of the present application, and in other alternative embodiments, the base large model can also be replaced by ChatGLM, Vicuna, Alpaca, etc. model architecture similar multi-modal large model. It can be understood that, Figure 2 or the following Figure 3 and Figure 4 Among them, the contents shown in the data table and data graph as the input of the dual-channel fusion model are not considered as a limitation of the scheme of the present application, and only serve as an illustration and do not have actual significance.

[0057] Specifically in the present embodiment, step A2 includes the following steps:

[0058] A21, encode the to-be-processed graph through a visual encoder to obtain a visual input vector. The visual encoder (Visual Encoder) is a core component in computer vision tasks, responsible for converting raw image data into high-level semantic feature representations. With the development of deep learning, the architecture of visual encoders has evolved from traditional convolutional neural networks (CNN) to models based on Transformers (such as ViT), and has shown strong performance in different tasks (classification, detection, segmentation, etc.). In the present embodiment, Vision Transformer (ViT) is used as the visual encoder, which is a visual model based on the Transformer architecture. The core idea of ViT is to divide the image into small blocks (patches) and treat these small blocks as tokens similar to natural language processing tasks, and then use a standard Transformer encoder for processing.

[0059] A22, encode the text question through a text encoder to obtain a text feature vector. The text encoder (TextEncoder) is a model or algorithm component that converts natural language text into numerical representation (usually in the form of a vector). In the present embodiment, the word embedding layer of the Vicuna large model is used as the text encoder.

[0060] A23, process the visual input vector through a graph-table hybrid connection structure to obtain a graph-table hybrid visual output. In the present embodiment, the graph-table hybrid connection structure includes a graph linear layer, a table linear layer, and a visual language routing network; step A23 specifically includes the following steps:

[0061] A231, input the visual input vector into the graph linear layer and the table linear layer respectively to obtain a graph data vector and a table data vector. The data graph or data table is processed by the visual encoder to form a visual input vector , which is input into the graph linear layer and the table linear layer respectively to obtain a graph data vector and a table data vector The process can be represented by formula (1) and formula (2):

[0062] (1)

[0063] (2)

[0064] wherein, represents a visual input vector, represents a graph linear layer parameter, represents a table linear layer parameter, represents a parameter of a visual language routing network, d vin is the dimension of the visual input vector.

[0065] A232, input the visual input vector into the visual language routing network to obtain a graph table weight score. The visual input vector is input into the visual language routing network to obtain a graph table weight score , which can be represented by formula (3):

[0066] (3)

[0067] wherein, represents a parameter of a visual language routing network.

[0068] A233, based on the graph table weight score, the graph data vector and the table data vector are weighted and fused to obtain a graph table mixed visual output. The graph table weight score is used to weight and fuse the graph data vector and the table data vector to obtain a graph table mixed visual output , and the specific calculation process is shown in formula (4):

[0069] (4)

[0070] wherein, is a vector connection function, mainly used to connect vectors and .

[0071] A24, the graph table mixed visual output and the text feature vector are jointly input into a large language model layer, and the original output of the large language model layer is processed through a graph table low-rank mixed structure to generate an answer to the text question. In this embodiment, the graph table low-rank mixed structure includes a large language model layer, a graph LoRA, a table LoRA and a LoRA routing network; step A24 specifically includes the following steps:

[0072] A241. Using the combined visual output of the chart and the text feature vector as input vectors, input them into the graph LoRA and table LoRA respectively, resulting in the graph LoRA vector and the table LoRA vector. (Chart-based visual output) Fusing with text feature vectors as input vectors The inputs are respectively into the graph LoRA and the table LoRA to obtain the graph LoRA vector. LoRA vectors As shown in formulas (5) and (6) respectively:

[0073] (5)

[0074] (6)

[0075] in, These are the parameters of the upper projection layer of the LoRA graph. These are the parameters of the lower projection layer of the LoRA graph. These are the parameters of the upper projection layer of the LoRA table. These are the parameters of the lower projection layer of the LoRA table. d in For input vectors h in Dimensions d out The dimension of the output vector. d lora_r Let T represent the dimensions of each projection layer parameter, with the superscript T indicating transpose. RMS ( ) is the RMS Norm normalization function.

[0076] A242. Input the input vector into the LoRA routing network to obtain the LoRA weight score graph. Inputting into the LoRA routing network yields a graph of LoRA weight scores. As shown in equation (7):

[0077] (7)

[0078] in, Parameters representing the visual language routing network.

[0079] A243. Based on the chart LoRA weight scores, the graph LoRA vectors and table LoRA vectors are weighted and fused to obtain a mixed chart LoRA output. The graph LoRA vectors and table LoRA vectors are weighted and fused to obtain a graph-table LoRA hybrid visual output. The specific calculation process is shown in formula (8):

[0080] (8)

[0081] A244, add the chart LoRA mixed output to the original output of the large language model layer to obtain the output of one layer of the chart low-rank mixed structure as the input of the next layer of the large language model layer.

[0082] Then, the original output of the large language model layer is added in step A244 , to obtain the output of one layer of the chart low-rank mixed structure , as shown in equation (9):

[0083] (9)

[0084] In this embodiment, the chart low-rank mixed structure is designed as multiple layers, which is determined by the multi-layer structure characteristics of the underlying large model and the task requirements. In this embodiment, the Qwen-7B selected as the underlying large model adopts a multi-layer Transformer architecture, and each layer is responsible for feature abstraction and semantic deepening. As an adaptive module embedded in the underlying large model, the chart low-rank mixed structure needs to correspond to the multi-layer structure of the original model, and through layer-by-layer processing, it realizes fine adaptation to different depth features, avoiding feature loss or insufficient fusion caused by single-layer processing.

[0085] A245, based on the output of the last layer of the chart low-rank mixed structure, generate an answer to the text question. Assuming that the output of the last layer of the chart low-rank mixed layer is , then the final answer distribution needs to be calculated through a linear layer , and the probability of is predicted as shown in equation (10):

[0086] (10)

[0087] wherein, is the length of the vocabulary, is the i th word in the vocabulary.

[0088] In an exemplary embodiment, the two-stage training strategy includes a single-channel training stage and a dual-channel fusion stage.

[0089] In the single-channel training stage, as Figure 3As shown, the parameters of the visual encoder, text encoder, and large language model layers are frozen. Only the parameters of the graph linear layer and table linear layer in the graph-graph hybrid connectivity structure, and the graph LoRA and table LoRA in the graph low-rank hybrid structure, are fine-tuned to obtain single-channel training parameters. During the single-channel training phase, there are two types of model input: i) graph data + text problem, training parameters related to the graph data; ii) table data + text problem, training parameters related to the table data. After the single-channel training phase, two parameter modules are obtained: ① parameters of the graph LoRA and graph linear layer related to the graph data; ② parameters of the table LoRA and table linear layer related to the table data.

[0090] During the dual-channel fusion phase, such as Figure 4 As shown, the parameters of the visual encoder, text encoder, and large language model layer, as well as the single-channel training parameters, are frozen. Only the parameters of the visual-language routing network in the graph-symbolic hybrid connection structure and the LoRA routing network in the graph-low-rank hybrid structure are fine-tuned to obtain a dual-channel fusion model. In the dual-channel fusion stage, not only are the original parameters of the text encoder, visual encoder, and large language model frozen, but the parameters of the graph linear layer, table linear layer, graph LoRA, and table LoRA obtained in the previous stage are also frozen. Then, the LoRA routing network and visual-language routing network are set to a trainable state, meaning only these two routing parameters are fine-tuned. After the dual-channel fusion stage, a LoRA routing network and a visual-language routing network with strong generalization ability are obtained, thus enabling better fusion of the graph-symbolic dual pathways.

[0091] The above-described scheme in this embodiment trains a large multimodal model by combining LoRA and a visual-language adapter, reducing training parameters while enabling the model to possess good data learning capabilities. Furthermore, the model achieves strong cross-domain generalization ability with only a small amount of training data. By dynamically fusing the outputs of the graph linear layer and the table linear layer using a visual-language routing network, and designing a LoRA routing network to dynamically fuse the outputs of the graph LoRA and the table LoRA, the dual-channel fusion model as a whole possesses stronger cross-domain generalization ability. Since the dual-channel fusion model contains a relatively complex modular structure, to reduce memory consumption during training, this embodiment proposes a two-stage training method. This method distributes the memory required for the entire training process across different stages, ensuring that the model can be trained smoothly on consumer-grade GPUs.

[0092] Based on the same inventive concept, this application also provides a system for implementing the above-described graph-based visual question-answering method using a hybrid dynamic routing and low-rank approach. The solution provided by this system is similar to the implementation described in the above method. In an exemplary embodiment, such as... Figure 5As shown, a graph visual question answering system based on dynamic routing and low rank mixing is provided, comprising the following functional modules:

[0093] A graph visual question answering original data acquisition module is configured to acquire graph visual question answering original data; the graph visual question answering original data comprises a to-be-processed graph and a text question for the to-be-processed graph.

[0094] A dual-channel fusion graph visual question answering module is configured to process the graph visual question answering original data through a pre-trained dual-channel fusion model to generate an answer to the text question; the dual-channel fusion model comprises a visual encoder, a text encoder, a graph hybrid connection structure, a graph low rank mixing structure and a large language model layer; the visual encoder is configured to convert the to-be-processed graph into a semantic feature representation; the text encoder is configured to convert the text question into a vector representation; the graph hybrid connection structure is configured to dynamically fuse the visual features of the data graph and the data table, and input the vector representation of the text question into the graph low rank mixing structure layer together; the graph low rank mixing structure is configured to dynamically weight and fuse the graph LoRA vector and the table LoRA vector through a LoRA routing network, and add the original output of the large language model layer to realize low rank adaptation and dynamic fusion of the graph features; the dual-channel fusion model is trained using a two-stage training strategy.

[0095] Of course, Figure 5 The architecture shown is only exemplary, and when implementing different functions, some of the components shown in the system can be omitted Figure 5 according to actual needs.

[0096] In an exemplary embodiment, a computer device is provided, which can be a server or a terminal, and an internal structure diagram thereof can be as shown. Figure 6 The computer device comprises a processor, a memory, an input / output interface (I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. When the computer program is executed by the processor, the computer program can implement the graph visual question answering method based on dynamic routing and low rank mixing provided in the foregoing embodiments.

[0097] Those skilled in the art can understand that Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0098] In an exemplary embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method embodiments when executing the computer program.

[0099] In an exemplary embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implements the steps in the above method embodiments when executed by a processor.

[0100] In an exemplary embodiment, a computer program product is provided, including a computer program, and the computer program implements the steps in the above method embodiments when executed by a processor.

[0101] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0102] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to a memory, a database or other medium used in the embodiments provided in the present application can include at least one of a non-volatile and a volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive random access memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric random access memory (FRAM), a phase change memory (PCM), a graphene memory, etc. The volatile memory can include a random access memory (RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc.

[0103] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0104] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0105] The principles and implementation manners of the present application are described by using specific examples in the present application, and the above-mentioned embodiments are only used to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed. In conclusion, the content of the present application should not be understood as a limitation.

Claims

1. A graph visual question answering method based on dynamic routing and low rank mixing, characterized in that, The method comprises the following steps: obtaining chart visual question and answer original data; the chart visual question and answer original data comprises a to-be-processed chart and a text question for the to-be-processed chart; the to-be-processed chart comprises a to-be-processed data chart and a to-be-processed data table; processing the chart visual question and answer original data through a pre-trained double-channel fusion model to generate an answer to the text question; the double-channel fusion model comprises a visual encoder, a text encoder, a chart mixed connection structure, a chart low-rank mixed structure and a base large model; the visual encoder is used to convert the to-be-processed chart into semantic feature representation; the text encoder is used to convert the text question into vector representation; the chart mixed connection structure is used to dynamically fuse the visual features of the to-be-processed data chart and the to-be-processed data table, and input the vector representation of the text question into the chart low-rank mixed structure layer; the base large model comprises a plurality of large language model layers, and the chart low-rank mixed structure is used to dynamically weight fuse a graph LoRA vector and a table LoRA vector through a LoRA routing network, and add the original output of each layer of the large language model layer to realize low-rank adaptation and dynamic fusion of chart features; the double-channel fusion model is trained by using a two-stage training strategy; the chart low-rank mixed structure comprises a large language model layer, a graph LoRA, a table LoRA and a LoRA routing network; processing the chart visual question and answer original data through a pre-trained double-channel fusion model to generate an answer to the text question, specifically comprising: encoding the to-be-processed chart through the visual encoder to obtain a visual input vector; encoding the text question through the text encoder to obtain a text feature vector; processing the visual input vector through the chart mixed connection structure to obtain a chart mixed visual output; inputting the chart mixed visual output and the text feature vector into the large language model layer, and processing the original output of the large language model layer through the chart low-rank mixed structure to generate an answer to the text question; specifically comprising: inputting the chart mixed visual output and the text feature vector as an input vector into the graph LoRA and the table LoRA respectively to obtain a graph LoRA vector and a table LoRA vector; inputting the input vector into the LoRA routing network to obtain a chart LoRA weight score; weighting and fusing the graph LoRA vector and the table LoRA vector based on the chart LoRA weight score to obtain a chart LoRA mixed output; adding the chart LoRA mixed output and the original output of the large language model layer to obtain one layer of output of the chart low-rank mixed structure, which is used as the input of the next layer of large language model layer; generating an answer to the text question based on the output of the last layer of the chart low-rank mixed structure.

2. The method of claim 1, wherein, obtaining chart visual question and answer original data, specifically comprising: Obtain the chart to be processed; the chart to be processed includes a data chart to be processed and a data table to be processed, the data chart to be processed is an image containing data information, the data chart to be processed is at least one of line chart, bar chart, and scatter plot, and the data table to be processed is a table image containing data; Obtain the text question for the chart to be processed; the text question is a consultation statement in natural language form. 3.The dynamic routing and low rank mixed based graph visual question answering method of claim 1, wherein, The graph-based hybrid connection structure includes a graph linear layer, a table linear layer, and a visual language routing network. The visual input vector is processed through this graph-based hybrid connection structure to obtain a graph-based hybrid visual output, specifically including: The visual input vector is input into the graph linear layer and the table linear layer respectively to obtain graph data vector and table data vector; The visual input vector is input into the visual language routing network to obtain the chart weight score; The chart data vector and the table data vector are weighted and fused based on the chart weight scores to obtain the mixed visual output of the chart.

4. The method of claim 3, wherein, The two-stage training strategy includes a single-channel training stage and a dual-channel fusion stage. During the single-channel training phase, the parameters of the visual encoder, the text encoder, and the large language model layer are frozen, and the parameters of the graph linear layer and the table linear layer in the graph hybrid connection structure, as well as the graph LoRA and table LoRA in the graph low-rank hybrid structure, are fine-tuned to obtain the single-channel training parameters. In the dual-channel fusion stage, the parameters of the visual encoder, the text encoder, the large language model layer, and the single-channel training parameters are frozen, and only the parameters of the visual language routing network in the graph hybrid connection structure and the LoRA routing network in the graph low-rank hybrid structure are fine-tuned to obtain the dual-channel fusion model.

5. A graph visual question answering system based on dynamic routing and low rank mixing, characterized in that, The system is used to implement the graph-based visual question answering method based on dynamic routing and low-rank hybrid as described in any one of claims 1-4, and the system includes: The raw data acquisition module for chart visual question and answer is used to acquire raw data for chart visual question and answer; the raw data for chart visual question and answer includes the chart to be processed and the text question for the chart to be processed; The double-channel fusion chart visual question answering module is used for processing the chart visual question answering original data through a pre-trained double-channel fusion model to generate an answer to the text question; the double-channel fusion model comprises a visual encoder, a text encoder, a chart mixed connection structure, a chart low-rank mixed structure and a base large model; the visual encoder is used for converting a to-be-processed chart into a semantic feature representation; the text encoder is used for converting the text question into a vector representation; the chart mixed connection structure is used for dynamically fusing visual features of a data chart and a data table, and inputting the visual features and the vector representation of the text question into a chart low-rank mixed structure layer; the base large model comprises a plurality of large language model layers, and the chart low-rank mixed structure is used for dynamically weighting and fusing a chart LoRA vector and a table LoRA vector through a LoRA routing network, and adding the chart LoRA vector and the table LoRA vector to original outputs of each large language model layer to realize low-rank adaptation and dynamic fusion of chart features; the double-channel fusion model is trained by using a two-stage training strategy.

6. A computer device comprising: A memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the chart visual question answering method based on dynamic routing and low-rank mixing according to any one of claims 1-4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the chart visual question answering method based on dynamic routing and low-rank mixing according to any one of claims 1-4.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the chart visual question answering method based on dynamic routing and low-rank mixing according to any one of claims 1-4. The computer program is executed by the processor to implement the chart visual question answering method based on dynamic routing and low-rank mixing according to any one of claims 1-4.

Images