Power field chart question-answering method, system and equipment and medium

Abstract

The invention belongs to the technical field of intelligent questioning and answering, and particularly relates to an electric power field chart questioning and answering method, system and device based on a multi-modal large language model and a medium. Aiming at the defects of insufficient cross-modal information fusion and insufficient chart understanding ability in the existing power field chart question-answering process, the invention adopts the following technical scheme: the power field chart question-answering method based on the multi-modal large language model comprises the following steps of: setting initial query for input images and questions; processing an input image by using a visual encoder to obtain a vector representation of the image; converting the image vector representation into image hidden representation through an adapter module; inputting the problem into a large language model embedding layer to obtain hidden representation of the problem; and splicing the image hidden representation and the question hidden representation, and inputting the spliced image hidden representation and question hidden representation into the trunk large language model to generate a final answer. According to the method, details of the chart can be accurately captured, it is guaranteed that the image and problem information can be effectively fused, and the fusion effect is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent question answering, and specifically relates to a method, system, device and medium for answering questions in a chart in the electric power field based on a multimodal large language model. Background Art

[0002] With the rapid growth of data in the power industry, various power equipment and monitoring systems have generated a large amount of chart data. For the analysis of chart data, traditional image recognition technology and rule-based chart processing methods usually have great limitations, and they cannot deeply understand the information in the chart. Especially when dealing with different types of charts, they lack cross-modal understanding capabilities.

[0003] In order to better understand chart data and answer user queries, some image question answering systems based on large language models have emerged in recent years. However, these systems still face the problem of how to effectively combine the visual information of the chart with the text information. In the chart question answering scenario in the power field, the structure of the chart is diverse and the information is complex. How to efficiently extract the key information in the chart and generate accurate answers is a difficulty currently faced. Summary of the invention

[0004] The present invention aims to solve the problems of insufficient cross-modal information fusion and insufficient chart understanding ability in the existing power field chart question and answer process, and provides a power field chart question and answer method based on a multimodal large language model. The present invention also provides a power field chart question and answer system, a computer device and a computer-readable storage medium based on a multimodal large language model.

[0005] To achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a diagram question-answering method in the electric power field based on a multimodal large language model, the diagram question-answering method in the electric power field based on a multimodal large language model comprises:

[0006] Step S1, setting an initial query for the input image and question;

[0007] Step S2: Use the visual encoder to process the input image and obtain the vector representation X of the image. v ;

[0008] Step S3: Transform the image vector representation X through the adapter module v Transformed into an image hidden representation H that can be concatenated with the large language model problem hidden representation v ;

[0009] Step S4: Input the question into the embedding layer of the large language model to obtain the hidden representation H of the question q ;

[0010] Step S5: Hide the image vand the problem hidden representation H q After concatenation, the results are input into the backbone language model to generate the final answer A.

[0011] The electric power field diagram question answering method based on the multimodal large language model of the present invention performs vector representation on the image and converts it into an image hidden representation that can be spliced ​​with the large language model question hidden representation, which can accurately capture the details of the diagram and provide an accurate image representation for subsequent reasoning; adjust the representation of the image features to make it consistent with the question hidden representation H q The dimension and distribution of the image and question information are matched, thereby ensuring that the image and question information can be effectively fused and improving the fusion effect.

[0012] As an improvement, step S2 includes:

[0013] Step S21, dividing the input image V into a plurality of non-overlapping blocks;

[0014] Step S22: Process each block using a multi-head self-attention mechanism based on a shifted window;

[0015] Step S23, using a multi-layer perceptron to further process the features of each block;

[0016] Step S24: Generate image feature vector X using visual coding v , expressed as:

[0017] X v =Encoder v (V)

[0018] Among them, Encoder v (·) indicates UniChart visual encoder.

[0019] As an improvement, in step S3, the adapter module adopts a two-layer MLP structure, the adapter module includes two linear layers, and a nonlinear activation function is inserted between the two linear layers. The two-layer MLP structure is a multi-layer perceptron structure with two linear transformations (fully connected layers).

[0020] As an improvement, step S3 includes:

[0021] Step S31: Make the image vector represent X v After the first linear layer and nonlinear activation function processing;

[0022] Step S32: Obtain the image hidden representation H through the second linear layer v , expressed as:

[0023] H v =W 2 ·α(W 1 X v)

[0024] Among them, W 1 With W 2 are the parameter matrices of the linear layer in the two-layer MLP, and α(·) represents the activation function.

[0025] As an improvement, step S4 includes:

[0026] Step S41, input a natural language question Q;

[0027] Step S42, dividing the input question Q into subwords;

[0028] Step S43: Use the vocabulary to convert text words into tokens;

[0029] Step S44: Use a fixed embedding matrix to map the token ID into a vector, expressed as:

[0030] E q =Embed(Q)

[0031] Among them, Embed() represents the embedding layer of the backbone large language model, Q is the question text, E q is the corresponding embedding vector;

[0032] Step S45: adding position code, expressed as:

[0033] P q =E q +PE q

[0034] Among them, P q is the embedding vector with positional encoding, PE q It is a positional encoding, which indicates the position of each token in the sequence.

[0035] Step S45: embedding vector P with position encoding q Input to the Transformer encoder.

[0036] H q =Embed(P q )

[0037] Among them, H q is the hidden representation of the Transformer encoder output.

[0038] As an improvement, in step S5, the image vector represents X v and the problem hidden representation H q The splicing is calculated using the following mathematical formula:

[0039] H = Concat(Hv ,H q );

[0040] Among them, Concat() means concatenation along the feature dimension, that is, the length dimension of the vector.

[0041] As an improvement, in step S5, the main process of the backbone large language model includes:

[0042] Convert H to hidden state hidden_state through the embedding layer;

[0043] The attention mask attention_mask is calculated according to the dimension of H;

[0044] Calculate position_ids based on H and attention_mask;

[0045] The embedded hidden_state, attention_mask and position_ids are input into the core module G of the backbone large language model, which consists of layer normalization, self-attention mechanism, gated linear unit and residual connection;

[0046] Pass through multiple modules G in sequence, each module accepts the current hidden_state and its corresponding layer identifier layer_id, and outputs the updated hidden_state;

[0047] Based on the current hidden_state, convert it into a vector of dictionary size through a fully connected layer;

[0048] Perform the argmax operation on the converted vector to generate the next output token;

[0049] Add the generated token to input_ids as the input for the next iteration;

[0050] Determine whether the generated token is the termination token or whether it has reached the preset maximum length. If the termination condition is met, terminate the iteration, otherwise continue to the next round of generation;

[0051] After the iteration, the final generated token sequence is converted into natural language form according to the decoder and output as the final answer A.

[0052] A diagram question-answering system for the electric power field based on a multimodal large language model, the diagram question-answering system for the electric power field based on a multimodal large language model comprises:

[0053] An initial query setting module, used to set an initial query for an input image and question;

[0054] The image vector module is used to process the input image using the visual encoder to obtain the vector representation X of the image. v ;

[0055] The image hidden representation module is used to represent the image vector X through the adapter module v Transformed into an image hidden representation H that can be concatenated with the large language model problem hidden representation v ;

[0056] The question hidden representation module is used to input the question into the embedding layer of the large language model to obtain the question hidden representation H q ;

[0057] The answer generation module is used to transform the image into a hidden representation H v and the problem hidden representation H q After concatenation, the results are input into the backbone language model to generate the final answer A.

[0058] A computer device includes a processor and a storage medium, wherein the storage medium stores a computer program, characterized in that when the computer program is executed by the processor, the aforementioned power field chart question and answer method based on a multimodal large language model is implemented.

[0059] A computer-readable storage medium, characterized in that: a computer program is stored thereon, and when the computer program is executed, the aforementioned electric power field diagram question and answer method based on a multimodal large language model is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of a diagram question-answering method in the electric power field based on a multimodal large language model according to an embodiment of the present invention.

[0061] Figure 2 It is a sample problem in the multimodal dataset of an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The technical solutions of the embodiments of the present invention are explained and described below, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.

[0063] See also Figure 1 and Figure 2 , the electric power field diagram question answering method based on a multimodal large language model according to an embodiment of the present invention, the electric power field diagram question answering method based on a multimodal large language model comprises:

[0064] Step S1, setting an initial query for the input image and question;

[0065] Step S2: Use the visual encoder to process the input image and obtain the vector representation X of the image. v ;

[0066] Step S3: Transform the image vector representation X through the adapter module v Transformed into an image hidden representation H that can be concatenated with the large language model problem hidden representation v ;

[0067] Step S4: Input the question into the embedding layer of the large language model to obtain the hidden representation H of the question q ;

[0068] Step S5: Hide the image v and the problem hidden representation H q After concatenation, the results are input into the backbone language model to generate the final answer A.

[0069] The electric power field diagram question answering method based on the multimodal large language model of the present invention performs vector representation on the image and converts it into an image hidden representation that can be spliced ​​with the large language model question hidden representation, which can accurately capture the details of the diagram and provide an accurate image representation for subsequent reasoning; adjust the representation of the image features to make it consistent with the question hidden representation H q The dimension and distribution of the image and question information are matched to ensure that the image and question information can be effectively integrated.

[0070] In this embodiment, step S2 includes:

[0071] Step S21, dividing the input image V into a plurality of non-overlapping blocks;

[0072] Step S22: Process each block using a multi-head self-attention mechanism based on a shifted window;

[0073] Step S23, using a multi-layer perceptron to further process the features of each block;

[0074] Step S24: Generate image feature vector X using visual coding v , expressed as:

[0075] X v =Encoder v (V)

[0076] Among them, Encoder v (·) indicates UniChart visual encoder.

[0077] In this embodiment, in step S3, the adapter module adopts a double-layer MLP structure, and the adapter module includes two linear layers, and a nonlinear activation function is inserted between the two linear layers.

[0078] In this embodiment, step S3 includes:

[0079] Step S31: Make the image vector represent X v After the first linear layer and nonlinear activation function processing;

[0080] Step S32: Obtain the image hidden representation H through the second linear layer v , expressed as:

[0081] H v =W 2 ·α(W 1 X v )

[0082] Among them, W 1 With W 2 are the parameter matrices of the linear layer in the two-layer MLP, and α(·) represents the activation function.

[0083] In this embodiment, step S4 includes:

[0084] Step S41, input a natural language question Q;

[0085] Step S42, dividing the input question Q into subwords;

[0086] Step S43: Use the vocabulary to convert text words into tokens;

[0087] Step S44: Use a fixed embedding matrix to map the token ID into a vector, expressed as:

[0088] E q =Embed(Q)

[0089] Among them, Embed() represents the embedding layer of the backbone large language model, Q is the question text, E q is the corresponding embedding vector;

[0090] Step S45: adding position code, expressed as:

[0091] P q =E q +PE q

[0092] Among them, P q is the embedding vector with positional encoding, PE q It is a positional encoding, which indicates the position of each token in the sequence.

[0093] Step S45: embedding vector P with position encoding q Input to the Transformer encoder.

[0094] H q =Embed(P q )

[0095] Among them, H q is the hidden representation of the Transformer encoder output.

[0096] In this embodiment, in step S5, the image vector represents X v and the problem hidden representation H q The splicing is calculated using the following mathematical formula:

[0097] H = Concat(H v ,H q );

[0098] Among them, Concat() means concatenation along the feature dimension, that is, the length dimension of the vector.

[0099] Figure 2 In the question text input, "When In is 3mW and 7mW, what is the difference between P(tot) and P(L)+P(U)?".

[0100] First, the question text is input into the backbone large language model embedding layer (Embed(Q)).

[0101] The purpose of this step is to convert the question text into a high-dimensional vector representation H q , the vector captures the semantic information of the question, such as “difference”, “3mW”, “7mW”, and keywords such as “P(tot)”, “P(L)”, and “P(U)”.

[0102] H q =Embed(Q),

[0103] Among them, Q is the input question, and Embed() represents the embedding layer that converts text into a vector.

[0104] Then, the power graph image V is processed.

[0105] The image V is processed by the visual encoder to generate the image feature vector X v Specifically, first, the image is divided into multiple non-overlapping blocks, and the features of each block are extracted through a multi-head self-attention mechanism based on a shifted window. Then, a multi-layer perceptron (MLP) is used to further process the features of each block, and finally an image feature vector X is generated. v . Image feature vector X v It is expressed as:

[0106] X v =Encoderv (V)

[0107] Among them, Encoder v (·) represents the UniChart visual encoder, and the processed image feature vector X v Contains the key information in the image.

[0108] The purpose of this step is to convert the visual information in the image (such as the shape and value of the curve) into a numerical feature representation for subsequent fusion with the problem representation.

[0109] Traditional models use the CLIP model when dealing with image and text problems, but the CLIP model is mainly designed for a wide range of image-text matching tasks, and cannot accurately capture subtle differences in complex charts (such as multiple curves with similar trends), resulting in loss of chart detail information. In previous methods, due to differences in the representation dimensions and feature distributions of images and problems, simple splicing operations may lead to information loss or make it difficult for the CLIP model to find effective cross-modal associations. In this case, the CLIP model may not be able to accurately extract differences such as "P(tot)" and "P(L)+P(U)". The embodiment of the present invention uses the UniChart visual encoder for charts to replace the CLIP model used in the native LLaVA, combined with a feature extraction method based on the self-attention mechanism, which can more accurately capture the details of the chart and provide a more accurate image representation for subsequent reasoning.

[0110] Secondly, the image feature vector X v Generate image hidden representation H through adapter module v , expressed as:

[0111] H v =W 2 ·α(W 1 X v ),

[0112] Among them, W 1 With W 2 are the parameter matrices of the linear layer in the two-layer MLP, α(·) represents the activation function, and after the two-layer MLP processing, the image feature vector X v is converted into an image hidden representation H v .

[0113] In previous methods, due to the differences in representation dimensions and feature distribution between images and questions, simple concatenation may lead to information loss or make it difficult for the model to find effective cross-modal associations. The double-layer MLP structure of the adapter module in the embodiment of the present invention can adjust the representation of image features to make them consistent with the hidden representation H of the question. qThe dimension and distribution of the image and question information are matched to ensure that the image and question information can be effectively integrated.

[0114] Hide the image to represent H v and the problem hidden representation H q Concatenate them into a new vector H to be input into the backbone large language model for reasoning. The formula is as follows:

[0115] H = Concat(H v ,H q ).

[0116] The concatenated vector H is input into the backbone large language model. The backbone large language model performs reasoning based on the joint representation of the image and the question, identifies the key content of the question requirement and calculates the difference.

[0117] Previous models used general data for training, but lacked vertical data for power scenarios. Therefore, the semantic understanding of questions and images was not accurate enough, and the model might not be able to accurately extract the numerical relationship in the chart, resulting in calculation errors. The method of the embodiment of the present invention uses a large amount of power scenario data for fine-tuning, allowing the model to fully learn power grid knowledge, build a closer connection between question representation and image representation, and ensure that the model can make correct inferences based on questions and image information in the power field.

[0118] The above is only a specific implementation of the invention, but the protection scope of the invention is not limited thereto. Those skilled in the art should understand that the invention includes but is not limited to the contents described in the above specific implementation. Any modification that does not deviate from the functional and structural principles of the invention will be included in the scope of the claims.

Claims

1. A diagram question-answering method for the power sector based on a multimodal large language model, characterized by: The electric power field diagram question answering method based on the multimodal large language model includes: Step S1, setting an initial query for the input image and question; Step S2: Use the visual encoder to process the input image and obtain the vector representation X of the image. v ; Step S3: Use the adapter module to represent the image vector Z v Transformed into an image hidden representation H that can be concatenated with the large language model problem hidden representation v ; Step S4: Input the question into the embedding layer of the large language model to obtain the hidden representation H of the question q ; Step S5: Hide the image v and the problem hidden representation H q After concatenation, the results are input into the backbone language model to generate the final answer A.

2. The method for answering questions in a diagram in the electric power field based on a multimodal large language model according to claim 1 is characterized in that: Step S2 includes: Step S21, dividing the input image V into a plurality of non-overlapping blocks; Step S22: Process each block using a multi-head self-attention mechanism based on a shifted window; Step S23, using a multi-layer perceptron to further process the features of each block; Step S24: Generate image feature vector X using visual coding v , expressed as: X v =Encoder v (V) Among them, Encoder v (·) indicates UniChart visual encoder.

3. The method for answering questions in electric power diagrams based on a multimodal large language model according to claim 1, characterized in that: In step S3, the adapter module adopts a two-layer MLP structure. The adapter module includes two linear layers, and a nonlinear activation function is inserted between the two linear layers.

4. The method for answering questions in a diagram in the electric power field based on a multimodal large language model according to claim 3 is characterized in that: Step S3 include: Step S31: Make the image vector represent X v After the first linear layer and nonlinear activation function processing; Step S32: Obtain the image hidden representation H through the second linear layer v , expressed as: H v =W2·α(W1X v ) Among them, W1 and W2 are the parameter matrices of the linear layer in the two-layer MLP, and α(·) represents the activation function.

5. The method for answering questions in a diagram in the electric power field based on a multimodal large language model according to claim 4 is characterized in that: Step S4 includes: Step S41, input a natural language question Q; Step S42, dividing the input question Q into subwords; Step S43: Use the vocabulary to convert text words into tokens; Step S44: Use a fixed embedding matrix to map the token ID into a vector, expressed as: E q =Embed(Q) Among them, Embed() represents the embedding layer of the backbone large language model, Q is the question text, E q is the corresponding embedding vector; Step S45: adding position code, expressed as: P q =E q +PE q Among them, P q is the embedding vector with positional encoding, PE q It is a positional encoding, which indicates the position of each token in the sequence. Step S45: embedding vector P with position encoding q Input to the Transformer encoder. H q =Embed(P q ) Among them, H q is the hidden representation of the Transformer encoder output.

6. The method for answering questions in electric power diagrams based on a multimodal large language model according to claim 5 is characterized in that: In step S5, the image vector represents X v and the problem hidden representation H q The splicing is calculated using the following mathematical formula: H=Concat(H v ,H q ); Among them, Concat() means concatenation along the feature dimension, that is, the length dimension of the vector.

7. The method for answering questions in electric power diagrams based on a multimodal large language model according to claim 1, characterized in that: In step S5, the main process of the backbone large language model includes: Convert H to hidden state hidden_state through the embedding layer; The attention mask attention_mask is calculated according to the dimension of H; Calculate position_ids based on H and attention_mask; The embedded hidden_state, attention_mask and position_ids are input into the core module G of the backbone large language model, which consists of layer normalization, self-attention mechanism, gated linear unit and residual connection; Pass through multiple modules G in sequence, each module accepts the current hidden_state and its corresponding layer identifier layer_id, and outputs the updated hidden_state; Based on the current hidden_state, convert it into a vector of dictionary size through a fully connected layer; Perform the argmax operation on the converted vector to generate the next output token; Add the generated token to input_ids as the input for the next iteration; Determine whether the generated token is the termination token or whether it has reached the preset maximum length. If the termination condition is met, terminate the iteration, otherwise continue to the next round of generation; After the iteration, the final generated token sequence is converted into natural language form according to the decoder and output as the final answer A.

8. A diagram question-answering system for the power sector based on a multimodal large language model, characterized by: The electric power field diagram question answering system based on the multimodal large language model includes: An initial query setting module, used to set an initial query for an input image and question; The image vector module is used to process the input image using the visual encoder to obtain the vector representation X of the image. v ; The image hidden representation module is used to represent the image vector X through the adapter module v Transformed into an image hidden representation H that can be concatenated with the large language model problem hidden representation v ; The question hidden representation module is used to input the question into the embedding layer of the large language model to obtain the question hidden representation H q ; The answer generation module is used to transform the image into a hidden representation H v and the problem hidden representation H q After concatenation, the results are input into the backbone language model to generate the final answer A.

9. A computer device comprising a processor and a storage medium, wherein the storage medium stores a computer program, characterized in that: When the computer program is executed by a processor, the method for answering questions in a diagram in the electric power field based on a multimodal large language model as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed, the method for answering questions in a diagram in the electric power field based on a multimodal large language model as described in any one of claims 1 to 7 is implemented.

Images