Image-text question and answer method and device based on multi-modal large model and thinking chain and medium

Abstract

The invention discloses an image-text question and answer method and device based on a multi-modal large model and a thinking chain and a medium, belongs to the technical field of natural language processing, and aims to solve the technical problem of how to improve the reasoning ability, the answer accuracy and the interpretability in a complex image-text question and answer task. According to the technical scheme, the method comprises the steps of collecting an image and a question text related to the image; the method comprises the following steps of: preprocessing an image, and converting the preprocessed image into an image token sequence imgtoken through a pre-trained image encoder ViT; the method comprises the following steps: preprocessing a question text related to an image, encoding the preprocessed question text, and converting the preprocessed question text into a question text token sequence; and splicing the image token sequence imgtoken and the question text token sequence query token into a multi-modal input representation.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method, device and medium for answering text and images based on a multimodal large model and a thought chain. Background Art

[0002] With the rapid development of artificial intelligence technology, multimodal large models have made significant progress in image understanding and natural language processing. Multimodal models can process image and text information simultaneously, providing new possibilities for complex image-text question-answering tasks. However, existing question-answering systems still have limitations when dealing with complex questions that require multi-step reasoning, and often have difficulty providing clear reasoning processes and accurate answers. The specific problems are as follows:

[0003] ① Limited reasoning ability: For complex questions that require multi-step reasoning, existing question-answering systems often have difficulty in providing a reasonable reasoning process;

[0004] ② Insufficient answer reliability: Due to the lack of clear reasoning steps, the generated answers may contain logical errors or be irrelevant to the questions;

[0005] ③ Poor explainability: It is difficult to show users how the question-answering system arrives at the final answer, which affects users’ trust in the system.

[0006] Therefore, how to improve the reasoning ability, accuracy of answers and explainability in complex image-text question-answering tasks is a technical problem that needs to be solved urgently. Summary of the invention

[0007] The technical task of the present invention is to provide a method, device and medium for picture-text question and answer based on a multimodal large model and thought chain to solve the problem of how to improve the reasoning ability, accuracy of answers and interpretability in complex picture-text question and answer tasks.

[0008] The technical task of the present invention is achieved in the following way: a picture-text question-answering method based on a multimodal large model and a thought chain, the method is specifically as follows:

[0009] Collect images and text of questions related to the images;

[0010] Preprocess the image and convert the preprocessed image into an image token sequence img_token through the pre-trained image encoder ViT;

[0011] Preprocess the question text related to the image, encode the preprocessed question text, and convert the preprocessed question text into a question text token sequence question_token according to the vocabulary set by the Qwen2-VL neural network;

[0012] The image token sequence img_token and the question text token sequence question_token are concatenated into a multimodal input representation input_multimodal in the following form:

[0013] input_multimodal=<|vision_start|>img_token<|vision_end|>

[0014] question_token;

[0015] Among them, <|vision_start|> and <|vision_end|> are two predefined tokens used to mark the beginning and end of the image token sequence img_token;

[0016] Calculate the initial probability distribution of the multimodal input representation input_multimodal;

[0017] Filter the token at the beginning of a thought chain and generate candidate thought chains; the token at the beginning of a thought chain refers to the token that generates a token to answer a thought chain after generating candidate thought chains starting with the token;

[0018] Evaluate and screen the candidate thinking chains, and select the top K candidate thinking chains as the final reasoning path;

[0019] The top K candidate thought chains are summarized and the Self-Consistency technique is used to generate the final answer.

[0020] Preferably, the image is preprocessed as follows:

[0021] Use the image processing library OpenCV to resize the image to a standard size of 448x448 pixels and RGB color format;

[0022] Normalize the RGB colors of the adjusted image separately, and scale the pixel values ​​to between 0 and 1 in the form of: pixel_normalized = pixel_original / 255.0;

[0023] The preprocessing of the question text related to the image is as follows:

[0024] Abbreviation expansion: expand common abbreviations to their full names using a predefined abbreviation dictionary;

[0025] Rewriting time pronouns: Get the current time and replace the time pronouns in the question text with specific dates or times;

[0026] Remove special characters.

[0027] Preferably, the initial probability distribution calculation for the multimodal input representation input_multimodal is as follows:

[0028] Use the Qwen2-VL neural network to perform forward propagation calculation on the multimodal input representation input_multimodal to obtain the hidden state H;

[0029] Use a linear layer to map the hidden state H to the vocabulary space and obtain logits, denoted as L;

[0030] Apply the softmax function to L to obtain the initial probability distribution. The formula is as follows:

[0031] P0=softmax(L);

[0032] Among them, P0 represents the initial probability distribution.

[0033] More optimally, when screening thought start tokens, the most relevant start tokens are adaptively selected by setting a dynamic probability threshold, which not only ensures diversity but also avoids completely random selection, laying the foundation for generating high-quality and diverse thought chains; the details are as follows:

[0034] Set the probability ratio threshold minP;

[0035] Find the maximum probability value Pmax in P0;

[0036] Calculate the probability lower bound: bound = Pmax*minP;

[0037] Traverse P0 and filter out tokens whose probability is greater than the probability lower bound to form a candidate set T; the token in the candidate set T is the starting token of the thinking chain; each token in the candidate set T is denoted as t.

[0038] Preferably, the candidate thought chain is generated as follows:

[0039] Constructing thought chain

[0040] Thinking Chain Continue writing until the end symbol is generated or the preset maximum length is reached, and obtain the candidate thought chain sequence S t ; The details are as follows:

[0041] ①Use Qwen2-VL neural network to analyze the thinking chain Perform a forward propagation calculation to obtain the output probability distribution Wherein, i is an integer greater than or equal to 0; when i=0, When i>0, Indicates intermediate results;

[0042] ②Select The token with the highest probability is denoted as t next ;

[0043] ③T next Add to thought chain In the process of obtaining the thought chain The formula is:

[0044] ④Record next Probability and The second highest probability

[0045] ⑤Calculate and store confidence The formula is:

[0046] ⑥ Check t next Is it a terminator or a thought chain? Whether the length reaches the preset maximum length:

[0047] If next For the end or thought chain When the length of reaches the preset maximum length, the loop ends and the candidate thinking chain Confidence

[0048] If next Not a terminator and a chain of thoughts If the length of does not reach the preset maximum length, change the value of i to i+1 and repeat steps ① to ⑥;

[0049] Store candidate thought chain S t and confidence level conf t .

[0050] Preferably, candidate chains of thought are evaluated and screened as follows:

[0051] Calculate each candidate thought chain S t The average confidence score t , the formula is as follows:

[0052]

[0053] Among them, len(conf t ) means conf t Length;

[0054] According to the average confidence score t Sort all candidate thought chains in descending order;

[0055] Select the top K candidate thought chains as the final reasoning path.

[0056] Preferably, the top K candidate thought chains are summarized and the final answer is generated using the Self-Consistency technique as follows:

[0057] Construct the Self-Consistency answer prompt text;

[0058] Generate answers using the filled answer prompt text and the language model;

[0059] When it is necessary to display the details of the reasoning, just show the specific content of the candidate thinking chain to the user.

[0060] Preferably, the Self-Consistency answer hint text is constructed as follows:

[0061] Prepare the text template as follows:

[0062] Question: {original question};

[0063] Depending on the question and the image, possible reasoning steps are as follows:

[0064] Candidate thought chain 1: {the content of the first candidate thought chain};

[0065] Candidate thought chain 2: {the content of the second candidate thought chain};

[0066] Candidate thought chain 3: {the content of the third candidate thought chain};

[0067] …

[0068] Candidate thought chain K: {the content of the Kth candidate thought chain};

[0069] According to the reasoning results, use the principle of majority rule to obtain the most likely answer;

[0070] Fill the original question and K candidate thought chains into the template to form a complete prompt text;

[0071] The answers generated using the filled answer prompt text and the language model are as follows:

[0072] Encode the filled prompt text to form a context representation C;

[0073] Initialize an empty output sequence OUT;

[0074] Repeat steps (1) to (8) until the end character is generated or the maximum length specified by the user (such as 1000 tokens) is reached:

[0075] (1) Concatenate the current context representation C and the generated output sequence OUT to form a complete input I;

[0076] (2) Use the language model to perform a forward propagation on the complete input I to obtain the probability distribution P of the next token next ;

[0077] (3) Using the kernel sampling strategy from P next Select the next token, denoted as t next , as follows:

[0078] (4) Select the top with the highest probability k Tokens (such as top k =50);

[0079] (5) For the top with the highest probability k The probabilities of the tokens are renormalized (scaled to sum to 1);

[0080] (6) Random sampling according to the renormalized probability obtains t next ;

[0081] (7) Set t next Add to the output sequence OUT: OUT = OUT + [t next ];

[0082] (8) Decode the output sequence OUT to generate text O, and remove special characters and redundant spaces in text O;

[0083] The processed text O is presented to the user as the final answer.

[0084] An electronic device comprising: a memory and at least one processor;

[0085] Wherein, the memory stores a computer program;

[0086] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the above-mentioned graphic question-and-text question-and-answer method based on the multimodal big model and thinking chain.

[0087] A computer-readable storage medium having a computer program stored therein, wherein the computer program can be executed by a processor to implement the above-mentioned graphic question-and-text question-and-answer method based on a multimodal large model and a thinking chain.

[0088] The graphic question-and-answer method, device and medium based on the multimodal large model and thought chain of the present invention have the following advantages:

[0089] (1) The present invention introduces a multimodal thinking chain mechanism to enhance the overall reasoning ability of the system by generating multiple candidate reasoning processes, and then integrates the candidate reasoning processes into the answer generation process to improve the accuracy and interpretability of the final answer;

[0090] (ii) The present invention adopts an end-to-end approach and can automatically handle various complex picture-text question-answering tasks without human intervention, thereby improving the reasoning ability, accuracy of answers, and interpretability in complex picture-text question-answering tasks;

[0091] (III) The present invention can quickly generate multiple candidate thought chains through the greedy decoding strategy; at the same time, the confidence of each step is recorded, which provides an important basis for subsequent evaluation;

[0092] (iv) The present invention effectively screens out the candidates that are most likely to be effective thought chains by introducing average confidence as an evaluation indicator, thereby improving the quality of subsequent answer generation;

[0093] (V) Each selected thought chain of the present invention has a quantifiable quality indicator, which enhances the user's trust in the answer. It can also display all candidate thought chains, improving the interpretability of the answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] The present invention is further described below in conjunction with the accompanying drawings.

[0095] Attached Figure 1 This is a flowchart of the graphic question-and-answer method based on the multimodal large model and thinking chain. DETAILED DESCRIPTION

[0096] The graphic question-and-answer method, device and medium based on a multimodal large model and a thought chain of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments of the specification.

[0097] Embodiment 1:

[0098] As attached Figure 1 As shown, this embodiment provides a picture-text question-answering method based on a multimodal large model and a thought chain, and the method is specifically as follows:

[0099] S1. Collect images and question texts related to the images;

[0100] S2, preprocess the image and convert the preprocessed image into an image token sequence img_token through the pre-trained image encoder ViT;

[0101] S3, preprocessing the question text related to the image, encoding the preprocessed question text, and converting the preprocessed question text into a question text token sequence question_token according to the vocabulary set by the Qwen2-VL neural network;

[0102] S4. Concatenate the image token sequence img_token and the question text token sequence question_token into a multimodal input representation input_multimodal in the following form:

[0103] input_multimodal=<|vision_start|>img_token<|vision_end|>

[0104] question_token;

[0105] Among them, <|vision_start|> and <|vision_end|> are two predefined tokens used to mark the beginning and end of the image token sequence img_token;

[0106] S5, calculating the initial probability distribution of the multimodal input representation input_multimodal;

[0107] S6. Filter the token at the beginning of a thinking chain and generate candidate thinking chains; wherein the token at the beginning of a thinking chain refers to a token that generates a thinking chain answer after generating candidate thinking chains starting with the token;

[0108] S7, evaluate and screen the candidate thinking chains, and select the top K candidate thinking chains as the final reasoning paths;

[0109] S8. Summarize the top K candidate thought chains and use Self-Consistency technology to generate the final answer.

[0110] The image preprocessing in step S2 of this embodiment is specifically as follows:

[0111] S201. Use the image processing library OpenCV to resize the image to a standard size of 448x448 pixels and RGB color format;

[0112] S202 , normalize the three colors of RGB of the adjusted image respectively, and scale the pixel values ​​to between 0 and 1 in the form of: pixel_normalized=pixel_original / 255.0.

[0113] The preprocessing of the question text related to the image in step S3 of this embodiment is specifically as follows:

[0114] S301, abbreviation expansion: using a predefined abbreviation dictionary to expand common abbreviations into full names;

[0115] For example: expand "AI" to "Artificial Intelligence";

[0116] S302, rewriting time pronouns: obtaining the current time, and replacing the time pronouns in the question text with a specific date or time;

[0117] S303. Remove special characters.

[0118] In step S5 of this embodiment, the initial probability distribution calculation of the multimodal input representation input_multimodal is specifically performed as follows:

[0119] S501, use the Qwen2-VL neural network to perform forward propagation calculation on the multimodal input representation input_multimodal to obtain the hidden state H;

[0120] S502, use a linear layer to map the hidden state H to the vocabulary space, and obtain logits, which are recorded as L;

[0121] S503, apply the softmax function to L to obtain the initial probability distribution, the formula is as follows:

[0122] P0=softmax(L);

[0123] Among them, P0 represents the initial probability distribution.

[0124] When screening the thought start token in step S6 of this embodiment, by setting a dynamic probability threshold, the most relevant start token is adaptively selected, which not only ensures diversity but also avoids completely random selection, laying a foundation for generating high-quality and diverse thought chains; the details are as follows:

[0125] S601-1, setting the probability ratio threshold minP;

[0126] S601-2, find the maximum probability value Pmax in P0;

[0127] S601-3. Calculate the probability lower bound: bound = Pmax*minP;

[0128] S601-4. Traverse P0 and filter out tokens whose probabilities are greater than the probability lower bound to form a candidate set T; wherein the tokens in the candidate set T are the starting tokens of the thinking chain; each token in the candidate set T is denoted as t.

[0129] The generation of candidate thought chains in step S6 of this embodiment is specifically as follows:

[0130] S602-1. Constructing a thought chain

[0131] S602-2, Thinking Chain Continue writing until the end symbol is generated or the preset maximum length (such as 50 tokens) is reached to obtain the candidate thought chain sequence S t ; The details are as follows:

[0132] ①Use Qwen2-VL neural network to analyze the thinking chain Perform a forward propagation calculation to obtain the output probability distribution Wherein, i is an integer greater than or equal to 0; when i=0, When i>0, Indicates intermediate results;

[0133] ②Select The token with the highest probability is denoted as t next ;

[0134] ③T next Add to thought chain In the process of obtaining the thought chain The formula is:

[0135] ④Record next Probability and The second highest probability

[0136] ⑤Calculate and store confidence The formula is:

[0137] ⑥ Check t next Is it a terminator or a thought chain? Whether the length reaches the preset maximum length:

[0138] If next For the end or thought chain When the length of reaches the preset maximum length, the loop ends and the candidate thinking chain Confidence

[0139] If next Not a terminator and a chain of thoughts If the length of does not reach the preset maximum length, change the value of i to i+1 and repeat steps ① to ⑥;

[0140] S602-3. Store candidate thought chain S t and confidence level conf t .

[0141] The evaluation and screening of candidate thought chains in step S7 of this embodiment are specifically as follows:

[0142] S701, calculate each candidate thought chain S t The average confidence score t , the formula is as follows:

[0143]

[0144] Among them, len(conf t ) means conf t Length;

[0145] S702, according to the average confidence score t Sort all candidate thought chains in descending order;

[0146] S703: Select the top K candidate thinking chains as the final reasoning path.

[0147] In step S8 of this embodiment, the top K candidate thought chains are summarized and the final answer is generated using the Self-Consistency technology as follows:

[0148] S801, constructing a Self-Consistency answer prompt text;

[0149] S802, generating an answer using the filled answer prompt text and the language model;

[0150] S803: When the reasoning details need to be displayed, the specific content of the candidate thought chain is displayed to the user. The specific text of the Self-Consistency answer in step S801 of this embodiment is as follows:

[0151] S80101. Prepare the text template as follows:

[0152] Question: {original question};

[0153] Depending on the question and the image, possible reasoning steps are as follows:

[0154] Candidate thought chain 1: {the content of the first candidate thought chain};

[0155] Candidate thought chain 2: {the content of the second candidate thought chain};

[0156] Candidate thought chain 3: {the content of the third candidate thought chain};

[0157] …

[0158] Candidate thought chain K: {the content of the Kth candidate thought chain};

[0159] According to the reasoning results, use the principle of majority rule to obtain the most likely answer;

[0160] S80102. Fill the original question and K candidate thinking chains into the template to form a complete prompt text.

[0161] In step S802 of this embodiment, the answer generated using the filled answer prompt text and the language model (such as GPT-4, Tongyi Qianwen, Wenxin Yiyan, etc.) is specifically as follows:

[0162] S80201. Encode the filled prompt text to form a context representation C;

[0163] S80202, initialize an empty output sequence OUT;

[0164] S80203. Repeat steps (1) to (8) until the end character is generated or the maximum length specified by the user (e.g., 1000 tokens) is reached:

[0165] (1) Concatenate the current context representation C and the generated output sequence OUT to form a complete input I;

[0166] (2) Use the language model to perform a forward propagation on the complete input I to obtain the probability distribution P of the next token next ;

[0167] (3) Using the kernel sampling strategy from P next Select the next token, denoted as t next , as follows:

[0168] (4) Select the top with the highest probability k Tokens (such as top k =50);

[0169] (5) For the top with the highest probability k The probabilities of the tokens are renormalized (scaled to sum to 1);

[0170] (6) Random sampling according to the renormalized probability obtains t next ;

[0171] (7) Set t next Add to the output sequence OUT: OUT = OUT + [t next ];

[0172] (8) Decode the output sequence OUT to generate text O, and remove special characters and redundant spaces in text O;

[0173] S80204. Present the processed text O to the user as the final answer.

[0174] Embodiment 2:

[0175] This embodiment also provides an electronic device, including: a memory and a processor;

[0176] Wherein, the memory stores computer-executable instructions;

[0177] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the graphic question-and-answer method based on the multimodal large model and the thinking chain in any embodiment of the present invention.

[0178] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0179] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0180] Embodiment 3:

[0181] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the graphic question-and-answer method based on a multimodal large model and a thinking chain in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0182] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0183] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0184] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0185] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A picture-text question-answering method based on a multimodal large model and thought chain, characterized in that: The method is as follows: Collect images and text of questions related to the images; Preprocess the image and convert the preprocessed image into an image token sequence img_token through the pre-trained image encoder ViT; Preprocess the question text related to the image, encode the preprocessed question text, and convert the preprocessed question text into a question text token sequence question_token; The image token sequence img_token and the question text token sequence question_token are concatenated into a multimodal input representation input_multimodal in the following form: input_multimodal=<|vision_start|>img_token<|vision_end|> question_token; Among them, <|vision_start|> and <|vision_end|> are two predefined tokens used to mark the beginning and end of the image token sequence img_token; Calculate the initial probability distribution of the multimodal input representation input_multimodal; Filter the token at the beginning of a thought chain and generate candidate thought chains; the token at the beginning of a thought chain refers to the token that generates a token to answer a thought chain after generating candidate thought chains starting with the token; Evaluate and screen the candidate thinking chains, and select the top K candidate thinking chains as the final reasoning path; The top K candidate thought chains are summarized and the Self-Consistency technique is used to generate the final answer.

2. The method for answering text and pictures based on a multimodal large model and a thought chain according to claim 1 is characterized in that: The image preprocessing is as follows: Use the image processing library OpenCV to resize the image to a standard size of 448x448 pixels and RGB color format; Normalize the RGB colors of the adjusted image separately, and scale the pixel values ​​to between 0 and 1 in the form of: pixel_normalized = pixel_original / 255.0; The preprocessing of the question text related to the image is as follows: Abbreviation expansion: expand common abbreviations to their full names using a predefined abbreviation dictionary; Rewriting time pronouns: Get the current time and replace the time pronouns in the question text with specific dates or times; Remove special characters.

3. The method for picture-text question-answering based on multimodal large model and thought chain according to claim 1 or 2, characterized in that: The initial probability distribution calculation for the multimodal input representation input_multimodal is as follows: Use the Qwen2-VL neural network to perform forward propagation calculation on the multimodal input representation input_multimodal to obtain the hidden state H; Use a linear layer to map the hidden state H to the vocabulary space and obtain logits, denoted as L; Apply the softmax function to L to obtain the initial probability distribution. The formula is as follows: P0=softmax(L); Among them, P0 represents the initial probability distribution.

4. The method for answering text and pictures based on a multimodal large model and thought chain according to claim 3 is characterized in that: When filtering thought start tokens, the most relevant start tokens are adaptively selected by setting a dynamic probability threshold; the details are as follows: Set the probability ratio threshold minP; Find the maximum probability value Pmax in P0; Calculate the probability lower bound: bound = Pmax*minP; Traverse P0 and filter out tokens whose probability is greater than the probability lower bound to form a candidate set T; the token in the candidate set T is the starting token of the thinking chain; each token in the candidate set T is denoted as t.

5. The method for picture-text question-answering based on multimodal large model and thought chain according to claim 4 is characterized in that: The specific steps to generate candidate thought chains are as follows: Constructing thought chain Thinking Chain Continue writing until the end symbol is generated or the preset maximum length is reached, and obtain the candidate thought chain sequence S t ; The details are as follows: ①Use Qwen2-VL neural network to analyze the thinking chain Perform a forward propagation calculation to obtain the output probability distribution Wherein, i is an integer greater than or equal to 0; when i=0, When i>0, Indicates intermediate results; ②Select The token with the highest probability is denoted as t next ; ③T next Add to thought chain In the process of obtaining the thought chain The formula is: ④Record next Probability and The second highest probability ⑤Calculate and store confidence The formula is: ⑥ Check t next Is it a terminator or a thought chain? t i+1 Whether the length reaches the preset maximum length: If next For the end or thought chain When the length of reaches the preset maximum length, the loop ends and the candidate thinking chain Confidence If next Not a terminator and a chain of thoughts If the length of does not reach the preset maximum length, change the value of i to i+1 and repeat steps ① to ⑥; Store candidate thought chain S t and confidence level conf t .

6. The method for picture-text question-answering based on multimodal large model and thought chain according to claim 5 is characterized in that: The evaluation and screening of candidate thinking chains are as follows: Calculate each candidate thought chain S t The average confidence score t , the formula is as follows: Among them, len(conf t ) means conf t Length; According to the average confidence score t Sort all candidate thought chains in descending order; Select the top K candidate thought chains as the final reasoning path.

7. The method for picture-text question-answering based on multimodal large model and thought chain according to claim 6 is characterized in that: The top K candidate thought chains are summarized and the final answer is generated using the Self-Consistency technique as follows: Construct the Self-Consistency answer prompt text; Generate answers using the filled answer prompt text and the language model; When it is necessary to display the details of the reasoning, just show the specific content of the candidate thinking chain to the user.

8. The method for picture-text question-answering based on multimodal large model and thought chain according to claim 7 is characterized in that: The prompt text for building a Self-Consistency answer is as follows: Prepare the text template as follows: Question: {original question}; Depending on the question and the image, possible reasoning steps are as follows: Candidate thought chain 1: {the content of the first candidate thought chain}; Candidate thought chain 2: {the content of the second candidate thought chain}; Candidate thought chain 3: {the content of the third candidate thought chain}; …… Candidate thought chain K: {the content of the Kth candidate thought chain}; According to the reasoning results, use the principle of majority rule to obtain the most likely answer; Fill the original question and K candidate thought chains into the template to form a complete prompt text; The answers generated using the filled answer prompt text and the language model are as follows: Encode the filled prompt text to form a context representation C; Initialize an empty output sequence OUT; Repeat steps (1) to (8) until the end character is generated or the maximum length specified by the user is reached: (1) Concatenate the current context representation C and the generated output sequence OUT to form a complete input I; (2) Use the language model to perform a forward propagation on the complete input I to obtain the probability distribution P of the next token next ; (3) Using the kernel sampling strategy from P next Select the next token, denoted as t next , as follows: (4) Select the top with the highest probability k Tokens; (5) For the top with the highest probability k The probability of each token is renormalized; (6) Random sampling according to the renormalized probability obtains t next ; (7) Set t next Add to the output sequence OUT: OUT = OUT + [t next ]; (8) Decode the output sequence OUT to generate text O, and remove special characters and redundant spaces in text O; The processed text O is presented to the user as the final answer.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the graphic question and answer method based on the multimodal big model and thinking chain as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the graphic question-and-answer method based on a multimodal large model and a thinking chain as described in any one of claims 1 to 8.

Images