Trustworthy multimodal large model reasoning method and device based on self-questioning and answering

Abstract

The present invention provides a trustworthy multimodal large model reasoning method and device based on self-questioning and answering, which relates to the field of artificial intelligence technology. The method includes: obtaining multimodal data, and performing feature extraction to obtain corresponding multimodal token features; obtaining questions for multimodal data, and inputting the questions and multimodal token features into the multimodal large model to perform a first question and answer; and inputting the obtained first answer text and prompt words into the multimodal large model to perform a second question and answer, obtaining a second answer text, determining a similarity score between the first answer text and the second answer text, and when the similarity score is greater than a similarity threshold, using the first answer text as the answer text for the question. Through this application, the defects of the large language model being too dependent on a question and answer knowledge base when performing question and answering, and the generated answer text having low accuracy and credibility are overcome.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a credible multimodal large model reasoning method and device based on self-questioning and answering. Background Art

[0002] With the development of the Internet, the types of content generated by users are becoming increasingly diverse, covering data in different modalities such as text, images, videos, and audio. Traditional single-modal models (processing only text or only images) are difficult to cope with the needs of multiple information fusion. For example, image retrieval or text retrieval in books based on a specified image, and extracting audio information and image information from videos all require comprehensive processing of data in multiple modalities.

[0003] Multimodal large models are an important step towards general artificial intelligence (AGI). General artificial intelligence requires the system to be able to process information in any modality and perform general reasoning and decision-making, such as answering text questions based on input multimodal data. Multimodal understanding is the key to achieving this goal. By integrating multimodal data, large models can have more comprehensive and profound understanding capabilities, and develop in the direction of being more intelligent and more adaptable to diverse tasks. At present, multimodal large models have the following shortcomings in text question answering: (1) They rely on a huge question and answer knowledge base. When question and answer data is insufficient, large models will have the defect of using different questioning methods to answer the same question content, but getting very different answers; (2) Large models have the problem of "answer hallucination". The answer results seem credible, but in fact they are very different from the real situation. The same question intention will give different wrong answers in different questioning methods, making the answer less credible. Summary of the invention

[0004] The present invention provides a credible multimodal large model reasoning method and device based on self-questioning and answering, which is used to overcome the defects of the large language model in the prior art that it is too dependent on the question and answer knowledge base when performing question and answering, and the generated answer text has low accuracy and credibility.

[0005] The present invention provides a credible multimodal large model reasoning method based on self-questioning and answering, and the method comprises the following steps:

[0006] Acquire multimodal data, and call an encoder of a corresponding modality to perform feature extraction processing on the multimodal data to obtain corresponding multimodal token features;

[0007] Obtaining a question for the multimodal data, and inputting the question and the multimodal token feature into a multimodal macro model to perform a first question-answering operation;

[0008] Inputting a first answer text and a prompt word obtained by the multimodal large model for the first question and answer into the multimodal large model for the second question and answer to obtain a second answer text, wherein the prompt word is used to prompt the multimodal large model to ask a question according to the first answer text;

[0009] A similarity score between the first answer text and the second answer text is determined, and when the similarity score is greater than a similarity threshold, the first answer text is used as an answer text to the question.

[0010] In some embodiments, the multimodal data includes at least one of: text; image; video; audio; the multimodal token features include at least one of: text token features; image token features; video token features; audio token features.

[0011] In some embodiments, determining the similarity score between the first answer text and the second answer text includes:

[0012] Calling a text encoder to respectively extract a first text feature of the first answer text and a second text feature of the second answer text;

[0013] The cosine similarity between the first text feature and the second text feature is determined, and the cosine similarity is used as a similarity score between the first answer text and the second answer text.

[0014] In some embodiments, the multimodal large model includes a question-answering model and an answer similarity scorer. The multimodal large model is trained by question samples. The question samples include real question texts and annotated answers to the real question texts. The training process of the multimodal large model includes:

[0015] Inputting the question sample into the question-answering model for a first question-answering, obtaining a predicted answer, and constructing a first cross-entropy loss value of the question-answering model based on the predicted answer and the labeled answer;

[0016] Constructing a second cross entropy loss value of the question-answering model according to the real question text and the annotated answer, and taking the sum of the first cross entropy loss value and the second cross entropy loss value as the final loss value of the question-answering model;

[0017] Constructing a third cross entropy loss value of the answer similarity scorer according to the predicted answer;

[0018] The final loss value is back-propagated in the question-answering model, and the third cross-entropy loss value is back-propagated in the answer similarity scorer to update the parameters of the multimodal large model.

[0019] In some embodiments, the constructing a second cross entropy loss value of the question-answering model according to the real question text and the annotated answer includes:

[0020] Obtaining a marking prompt word for the marked answer, wherein the marking prompt word is used to prompt the question-answering model to ask a question based on the marked answer;

[0021] Inputting the marked answer and the marked prompt word into the question-answering model to perform a second question-answering to obtain a predicted question text;

[0022] A second cross entropy loss value of the question-answering model is constructed based on the actual question text and the predicted question text.

[0023] In some embodiments, constructing a third cross entropy loss value of the answer similarity scorer according to the predicted answer includes:

[0024] Constructing a corresponding similar answer for each predicted answer, wherein the similarity score between the similar answer and the predicted answer is greater than a preset score threshold;

[0025] For each predicted answer, randomly select one answer from the similar answers as another answer to the predicted answer;

[0026] Determine a predicted label of the predicted answer according to the similar answers and the other answers;

[0027] A third cross entropy loss value of the answer similarity scorer is constructed according to the predicted label.

[0028] The present invention also provides a credible multimodal large model reasoning device based on self-questioning and answering, the device comprising the following modules:

[0029] An acquisition module is used to acquire multimodal data and call an encoder of a corresponding modality to perform feature extraction processing on the multimodal data to obtain corresponding multimodal token features;

[0030] A first question-answering module, configured to obtain questions for the multimodal data, and input the questions and the multimodal token features into a multimodal macro model to perform a first question-answering operation;

[0031] A second question-and-answer module, configured to input a first answer text and a prompt word obtained by the multimodal large model through the first question-and-answer process into the multimodal large model for the second question-and-answer process, thereby obtaining a second answer text, wherein the prompt word is used to prompt the multimodal large model to ask a question based on the first answer text;

[0032] The determination module is used to determine the similarity score between the first answer text and the second answer text, and when the similarity score is greater than a similarity threshold, the first answer text is used as the answer text of the question.

[0033] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements any of the above-described trusted multimodal large model reasoning methods based on self-questioning and answering.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described trusted multimodal large model reasoning methods based on self-questioning and answering.

[0035] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned trusted multimodal large model reasoning methods based on self-questioning and answering.

[0036] The trustworthy multimodal large model inference method and device based on self-questioning and answering provided by the present invention obtains multimodal data, and inputs the corresponding multimodal token features and the questions for the multimodal data into the multimodal large model for the first question and answer, thereby generating a large number of first answer texts, reducing the dependence of the multimodal large model on the question and answer data of the question and answer knowledge base. Next, the obtained first answer text and prompt words are input into the multimodal large model for the second question and answer, and the second answer text is obtained. The similarity score between the first answer text and the second answer text is determined. When the similarity score is greater than the similarity threshold, the first answer text is used as the answer text for the question. The answer text is generated by repeatedly asking questions and answering and scoring answers in this way, which improves the accuracy and credibility of the answer text generated by the multimodal large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is a flowchart of the credible multimodal large model reasoning method based on self-questioning and answering provided by the present invention.

[0039] Figure 2 It is a schematic diagram of the principle of the credible multimodal large model reasoning method based on self-questioning and answering provided by the present invention.

[0040] Figure 3 It is a schematic diagram of the model question-answering training data combination provided by the present invention.

[0041] Figure 4 It is a structural schematic diagram of a credible multimodal large model reasoning device based on self-questioning and answering provided by the present invention.

[0042] Figure 5 It is a structural schematic diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Combine the following Figure 1-Figure 4 The present invention describes the trustworthy multimodal large model reasoning method and device based on self-questioning and answering.

[0045] Figure 1 is a flow chart of the credible multimodal large model reasoning method based on self-questioning and answering provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 104, which are described in detail below.

[0046] Step 101: Acquire multimodal data, and call the encoder of the corresponding modality to perform feature extraction processing on the multimodal data to obtain corresponding multimodal token features.

[0047] In the self-questioning scenario based on a multimodal large model, a multimodal large model refers to a large language model that can process data in multiple modalities. The self-questioning scenario here involves question-answering tasks in multiple scenarios, such as text question-answering tasks, image question-answering tasks, audio question-answering tasks, and video question-answering tasks, which involve data in multiple modalities such as text, image, audio, and video. For details, please refer to Figure 2 .

[0048] like Figure 2As shown, the multimodal data obtained in the embodiment of the present invention includes at least one of text, image, video, and audio. In other embodiments, it also involves a combination of multimodal data, such as a combination of text and image, which corresponds to image retrieval, text retrieval or image positioning or image-text question-answering tasks, and a combination of video and audio, which corresponds to audio and video slicing, audio and video summarization or audio and video question-answering tasks. In other words, multimodal data can be input as a combination of any modal data.

[0049] After obtaining the multimodal data, the encoder of the corresponding modality is called to perform feature extraction on the multimodal data to obtain the corresponding multimodal token features. Here, for each modality of data, the encoder of the corresponding modality needs to be called to perform feature extraction.

[0050] like Figure 2 As shown, text calls the text encoder for encoding, the image calls the visual encoder (the visual encoder here is also called the image encoder when processing the image) for encoding, the video calls the visual encoder for encoding, and the audio calls the audio encoder for encoding. It should be noted that the use of the encoder here is not limited. For example, the text encoder can use any model that can extract text features, such as the commonly used CLIP model or the Transformer model.

[0051] For multimodal data combinations, the corresponding modal encoder combination can be used to encode them separately, and then the token features of different modalities extracted are fused as multimodal token features. For example, if the multimodal data is a combination of images and texts, the corresponding text encoder and visual encoder are used to encode them separately, and finally the extracted image features and text features are fused as multimodal token features.

[0052] After the multimodal data is encoded by the encoder of the corresponding modality, the corresponding multimodal token features are obtained, that is, Figure 2 The token features shown. It can be understood that the multimodal token features include at least one of text token features, image token features, video token features, and audio token features. One modal data corresponds to a token feature of one modality, and the multimodal data combination corresponds to the fused multimodal token features.

[0053] Step 102: Obtain questions for the multimodal data, and input the questions and multimodal token features into the multimodal large model to perform the first question and answer.

[0054] When performing a question-answering task in a question-answering scenario, the user needs to ask questions about the multimodal data so that the multimodal large model L can answer the questions according to the user's questions. , and will ask questions The multimodal token features are input into the multimodal large model L for the first question and answer, and the corresponding first answer text is generated .

[0055] like Figure 2 As shown in the figure, the input user question and the extracted token features are directly input into the large language model for model answering and generating the corresponding answer. For example, when the multimodal data only has images, the image question answering task is performed. The user question can be "What is the red object in the image?" The image token features corresponding to the image and the user question are directly input into the large language model for answering, and the large language model will answer the red object in the image.

[0056] In the embodiment of the present invention, multimodal data is obtained, and the corresponding multimodal token features and questions asked about the multimodal data are input into the multimodal big model to perform the first question and answer. Thus, a large number of first answer texts can be generated according to the multimodal token features without the need to input other knowledge texts, thereby reducing the reliance of the multimodal big model on the question and answer data of the question and answer knowledge base.

[0057] Step 103: input the first answer text and the prompt word obtained by the multimodal large model for the first question and answer into the multimodal large model for the second question and answer to obtain the second answer text.

[0058] Generate the first answer text through step 102 After that, the answer text may be inaccurate or have a large deviation from the actual result, and the answer is not credible and cannot be used directly as a question. Therefore, the embodiment of the present invention is based on the first answer text Construct the corresponding prompt word, which is used to prompt the multimodal large model L based on the first answer text To ask a question, the prompt template prompt can be "Please answer according to the answer { The multimodal large model L will respond based on the first answer text. Generate the corresponding prompt word prompt.

[0059] Next, the first answer text obtained by the multimodal large model L for the first question and answer And the prompt word prompt is input into the multimodal large model L for the second question and answer, and the second answer text is obtained The multimodal large model L is based on the first answer text of the input , multimodal token features and the prompt word prompt will generate the corresponding second answer text again .like Figure 2As shown in the figure, after the large language model generates the first answer, the first answer, token features, and prompt words are re-input into the large language model for model questioning to generate the second answer.

[0060] Step 104: determine the similarity score between the first answer text and the second answer text, and when the similarity score is greater than a similarity threshold, use the first answer text as the answer text to the question.

[0061] The multimodal large model L performs two responses in steps 102 and 103 respectively, generating the corresponding first response text. And the second answer text After that, we need to compare whether there is a huge difference between the two answers of the multimodal large model L, so as to judge whether the first answer generated according to the question is accurate and credible. Therefore, here, we need to determine whether the first answer text With the second answer text The similarity score of .

[0062] Specifically, the first answer text With the second answer text Input to the answer similarity scorer T (i.e. Figure 2 The similarity judgment is performed in the multiple answer similarity scorer shown in FIG. 1 to judge the consistency of the content of the two answers, such as Figure 2 In , the two answers are input into the multiple answer similarity scorer to calculate the corresponding answer similarity score.

[0063] The answer similarity scorer can be implemented by a language model including a text encoder T. The specific structure can be an encoding layer, a fully connected layer or a multi-layer perceptron including the text encoder T, which is not limited here. In the answer similarity scorer, the text encoder T is called to extract the first answer text respectively. The first text feature , and the second answer text The second text feature , then determine the first text feature With the second text feature The cosine similarity of Finally, the cosine similarity As first answer text With the second answer text The similarity score s.

[0064] Next, we need to preset a similarity threshold S, for example, S is set to 95%. With the second answer text After the similarity score s is obtained, it is necessary to determine whether the similarity score s exceeds the similarity threshold S (i.e. 95%). When the similarity score s is greater than the similarity threshold S, the first answer text As a question The answer text.

[0065] Here, when the similarity score s is greater than the similarity threshold S, it means that the text content of the two answers given by the multimodal large model L is similar. The generated first answer text With a certain degree of credibility, the answer is correct. Therefore, the first answer text can be As a question ’s answer text, thus ending the self-questioning and answering task in the question-answering scenario.

[0066] When the similarity score s is less than or equal to the similarity threshold S, it means that the text content of the two answers given by the multimodal large model L is quite different. Generated first answer text The credibility is low and the answer may be wrong. Therefore, the first answer text can be Avoid or prompt the multimodal large model L to ask questions Generate the first answer text again , and then repeat the above steps 103 and 104 to continue verification, generate corresponding answers and perform scoring judgments.

[0067] In an embodiment of the present invention, the multimodal large model L is answered twice, and the similarity of the two answers is calculated to determine the final answer text. Thus, the corresponding answer text is generated by conducting multiple question-answering and answer-scoring methods, which can effectively improve the accuracy and credibility of the answer text generated by the multimodal large model.

[0068] However, in some other embodiments, the answer texts generated by the multimodal large model L twice may be similar, but both answer texts are incorrect answer texts for the question asked. This is because the accuracy of the multimodal large model L itself in generating answers when asking questions is low. In order to further improve the accuracy of the multimodal large model in asking and answering questions, the embodiment of the present invention also trains the multimodal large model L by collecting corresponding question samples, so that the accuracy of the answer text generated by the multimodal large model L is higher. The training process of the multimodal large model L is described below.

[0069] When training the multimodal large model L, the multimodal large model L is divided into two parts: a question-answering model and an answer similarity scorer according to the corresponding generated results. The question-answering model is specifically used to generate answer text or construct prompt words to ask questions to generate question text, while the answer similarity scorer is specifically used to score the generated answer text.

[0070] First, we need to collect question samples and use them to train the multimodal large model L. The question samples include real question texts and annotated answers to real question texts. Real question texts are also called annotated questions. They are texts that ask questions based on multimodal data (including at least one of text, image, video, and audio). They are denoted as , the annotated answer is recorded as y_a. During training, the pre-trained encoder of the corresponding modality is first called to extract the multimodal features E of these multimodal data, and then the question sample is input into the question-answering model for the first question-answering to obtain the predicted answer .

[0071] Specifically, the question sample containing the labeled answer y_a and the multimodal feature E are input into the large language model for text answering to generate the corresponding predicted answer , and then based on the predicted answer The first cross entropy loss value of the question-answering model constructed with the labeled answer y_a , the construction method is expressed as the following formula (1):

[0072] (1)

[0073] In the above formula (1), Indicates the predicted label value of the i-th word in the generated predicted answer, It indicates the probability value of the i-th word in the predicted answer, and N indicates the total number of words in the predicted answer.

[0074] Next, based on the real question text And the second cross entropy loss value of the labeled answer y_a to construct the question-answering model Specifically, we first need to obtain the annotation prompt word of the annotation answer y_a. The annotation prompt word is used to prompt the question-answering model to ask questions based on the annotation answer y_a. The annotation prompt word also uses a prompt template, such as "Please ask questions based on the answer {y_a}". Then, the annotation answer y_a and the annotation prompt word are input into the question-answering model for the second question and answer, and the predicted question text is obtained. When the second question is asked, the annotated answer y_a, the annotated prompt word, and the multimodal feature E are input into the question model for the second question. During the second question, the question model generates the corresponding predicted question text according to the annotated prompt word. , then according to the real question text With predicted question text Construct the second cross entropy loss value of the question-answering model , the construction method is expressed as the following formula (2):

[0075] (2)

[0076] In the above formula (2), Indicates generating predicted question text based on the qth word of the labeled prompt word The predicted label value of the i-th word, Indicates generating predicted question text based on the qth word of the labeled prompt word The probability value of the i-th word, N represents the predicted question text The total number of words in the text.

[0077] Through the above steps, we can calculate the first cross entropy loss value of the question-answering model to generate the predicted answer And the second cross entropy loss value for generating the predicted prompt text , these two loss values ​​are used to train the model parameters of the question model. Therefore, the first cross entropy loss value is And the second cross entropy loss value The sum of is taken as the final loss value L of the question-answering model.

[0078] In addition, the multimodal large model L also includes an answer similarity scorer that is specifically used to score the generated answer text. In order to ensure the accuracy of judging the similarity between the two answer texts, this answer similarity scorer also needs to be trained. Therefore, the predicted answer generated by the question model is Constructing the third cross entropy loss value of the answer similarity scorer .

[0079] Specifically, since the predicted answer generated by the question model There can be multiple, that is, multiple predicted answers are generated, for example, 30, and these predicted answers constitute an answer set. Construct a similar answer to the corresponding , where similar answers With predicted answer The similarity score of is greater than a preset score threshold, such as 95%. In this way, the 30 predicted answers 30 corresponding similar answers can be determined , a predicted answer corresponds to a similar answer to be matched.

[0080] For each predicted answer, randomly select one answer from similar answers as the other answers of the predicted answer, recorded as For example, predicting the answer There are 30, for each predicted answer , all from 30 similar answers Randomly select an answer as the predicted answer Other answers . Thus, each predicted answer There is another answer for this to match, and this other answer may be a similar answer to the predicted answer.

[0081] Next, based on similar answers And other answers , determine the predicted answer Here we will predict the answer With other answers Input into the answer similarity scorer to calculate the similarity score. If the similarity score is greater than the score threshold (for example, 95%), it means that the input is exactly the predicted answer. Similar answers that match the corresponding , then the corresponding training label If the similarity score is less than or equal to the score threshold (for example, 95%), it means that the input is a predicted answer. With other answers , then the corresponding training label .

[0082] Finally, according to the predicted label Constructing the third cross entropy loss value of the answer similarity scorer , the construction method is expressed as the following formula (3):

[0083] (3)

[0084] In the above formula (3), Represents the i-th predicted answer The predicted label of Represents the i-th predicted answer The corresponding similarity score. N represents the predicted answer The total quantity.

[0085] When training the multimodal large model L, the question-answering model and the answer similarity scorer are trained and updated independently, and the two do not interfere with each other. In each training round, the final loss value L is back-propagated in the question-answering model, and the third cross entropy loss value is used. Backpropagation is performed in the answer similarity scorer to update the parameters of the multimodal large model L.

[0086] Here, during the training process, the final loss value L is used to perform back propagation in the question-answering model, the gradient is calculated during the back propagation process, and the gradient is optimized by the gradient optimization algorithm to update the parameters of the question model. When training the answer similarity scorer, the parameters of the question model are fixed and only the third cross entropy loss value is used. Back propagation is performed on the answer similarity scorer, and the gradient is also calculated during the back propagation process. The gradient is optimized through the gradient optimization algorithm to update the parameters of the answer similarity scorer.

[0087] The loss function is used to update the parameters of the question model and the answer similarity scorer in the multimodal large model, thereby continuously updating the corresponding loss function. When the final loss value L no longer decreases or reaches the set training round, the update of the question model parameters is stopped. When the third cross entropy loss value When the probability of no longer decreasing or the set number of training rounds is reached, the update of the parameters of the answer similarity scorer stops. That is, the parameter updates of the question model and the answer similarity scorer are performed independently, and the two do not interfere with each other. In this way, the training of the multimodal large model is finally completed. After the training, the multimodal large model can directly perform self-questioning and answering on multimodal data.

[0088] The schematic diagram of the training process of the above multimodal large model can be found in Figure 3 As shown in (a), the labeled question (i.e. the real question text) ) and the multimodal feature E are input into the large language model to obtain the predicted answer , and construct the first cross entropy loss value by annotating the answer y_a Perform supervised training. Then Figure 3 As shown in (b), we first construct the corresponding annotation prompt word input into the large language model, and also input the annotation answer y_a and the multimodal feature E into the large language model to generate the corresponding prediction question (i.e. the prediction question text mentioned above). ) and by annotating the questions (i.e. the real question text ) Construct the second cross entropy loss value Conduct supervised training.

[0089] In some other embodiments, see Figure 3 As shown in (c), we first input the question sample into the large language model and answer it twice, and then generate the predicted question (i.e. the predicted question text above) by the second answer. ) and multimodal features E are input into the large language model to generate the corresponding predicted answer , and then construct the first cross entropy loss value by annotating the answer y_a Perform supervised training. Then Figure 3 As shown in (d), first predict the answer Construct the corresponding annotation prompt word input into the large language model, and then predict the answer and multimodal features E are input into the large language model to generate the corresponding prediction question (i.e. the prediction question text mentioned above ), and by asking the labeled questions in the sample (i.e. the real question text ) Construct the second cross entropy loss value Conduct supervised training.

[0090] The embodiment of the present invention obtains multimodal data, and inputs the corresponding multimodal token features and questions for the multimodal data into the multimodal large model for the first question and answer, thereby generating a large number of first answer texts, reducing the reliance of the multimodal large model on the question and answer data of the question and answer knowledge base. Next, the obtained first answer text and prompt words are input into the multimodal large model for the second question and answer, and the second answer text is obtained. The similarity score between the first answer text and the second answer text is determined. When the similarity score is greater than the similarity threshold, the first answer text is used as the answer text for the question. In this way of multiple question and answer and answer scoring, the accuracy and credibility of the answer text generated by the multimodal large model are improved.

[0091] The following is a description of the trusted multimodal large model reasoning device based on self-questioning and answering provided by the present invention. The trusted multimodal large model reasoning device based on self-questioning and answering described below and the trusted multimodal large model reasoning method based on self-questioning and answering described above can be referenced to each other.

[0092] like Figure 4As shown, the trusted multimodal large model reasoning device based on self-questioning and answering includes the following modules, an acquisition module 401, a first question-answering module 402, a second question-answering module 403, and a determination module 404. Among them, the acquisition module 401 is used to acquire multimodal data, and call the encoder of the corresponding modality to perform feature extraction processing on the multimodal data to obtain the corresponding multimodal token features; the first question-answering module 402 is used to acquire questions for the multimodal data, and input the questions and the multimodal token features into the multimodal large model for the first question-answering; the second question-answering module 403 is used to input the first answer text and the prompt word obtained by the multimodal large model for the first question-answering into the multimodal large model for the second question-answering to obtain the second answer text, wherein the prompt word is used to prompt the multimodal large model to ask questions according to the first answer text; the determination module 404 is used to determine the similarity score between the first answer text and the second answer text, and when the similarity score is greater than the similarity threshold, the first answer text is used as the answer text of the question.

[0093] It should be noted that the beneficial effects of the trusted multimodal large model reasoning device based on self-questioning and answering here and the trusted multimodal large model reasoning method based on self-questioning and answering mentioned above can correspond to each other, so the beneficial effects of the trusted multimodal large model reasoning device based on self-questioning and answering will not be repeated here.

[0094] Figure 5 is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute a trusted multimodal large model reasoning method based on self-questioning and answering, the method comprising: obtaining multimodal data, and calling the encoder of the corresponding modality to perform feature extraction processing on the multimodal data to obtain corresponding multimodal token features; obtaining questions for the multimodal data, and inputting the questions and the multimodal token features into the multimodal large model to perform a first question and answer; inputting a first answer text and a prompt word obtained by the multimodal large model for the first question and answer into the multimodal large model for a second question and answer to obtain a second answer text, wherein the prompt word is used to prompt the multimodal large model to ask questions based on the first answer text; determining a similarity score between the first answer text and the second answer text, and when the similarity score is greater than a similarity threshold, using the first answer text as the answer text for the question.

[0095] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0096] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the trusted multimodal large model inference method based on self-questioning and answering provided by the above-mentioned methods, and the method includes: obtaining multimodal data, and calling the encoder of the corresponding modality to perform feature extraction processing on the multimodal data to obtain corresponding multimodal token features; obtaining questions for the multimodal data, and inputting the questions and the multimodal token features into the multimodal large model for a first question and answer; inputting a first answer text and a prompt word obtained by the multimodal large model for the first question and answer into the multimodal large model for a second question and answer to obtain a second answer text, wherein the prompt word is used to prompt the multimodal large model to ask questions based on the first answer text; determining a similarity score between the first answer text and the second answer text, and when the similarity score is greater than a similarity threshold, using the first answer text as the answer text for the question.

[0097] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the trusted multimodal large model inference method based on self-questioning and answering provided by the above-mentioned methods, the method comprising: obtaining multimodal data, and calling an encoder of the corresponding modality to perform feature extraction processing on the multimodal data to obtain corresponding multimodal token features; obtaining a question for the multimodal data, and inputting the question and the multimodal token features into the multimodal large model to perform a first question and answer; inputting a first answer text and a prompt word obtained by the multimodal large model performing the first question and answer into the multimodal large model to perform a second question and answer to obtain a second answer text, wherein the prompt word is used to prompt the multimodal large model to ask a question based on the first answer text; determining a similarity score between the first answer text and the second answer text, and when the similarity score is greater than a similarity threshold, using the first answer text as the answer text for the question.

[0098] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0099] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A credible multimodal large model reasoning method based on self-questioning and answering, characterized in that: The method comprises: Acquire multimodal data, and call an encoder of a corresponding modality to perform feature extraction processing on the multimodal data to obtain corresponding multimodal token features; Obtaining a question for the multimodal data, and inputting the question and the multimodal token feature into a multimodal macro model to perform a first question-answering operation; Inputting a first answer text and a prompt word obtained by the multimodal large model for the first question and answer into the multimodal large model for the second question and answer to obtain a second answer text, wherein the prompt word is used to prompt the multimodal large model to ask a question according to the first answer text; A similarity score between the first answer text and the second answer text is determined, and when the similarity score is greater than a similarity threshold, the first answer text is used as an answer text to the question.

2. The credible multimodal large model reasoning method based on self-questioning and answering according to claim 1 is characterized in that: The multimodal data includes at least one of the following: text; image; video; audio; the multimodal token features include at least one of the following: text token features; image token features; video token features; audio token features.

3. The credible multimodal large model reasoning method based on self-questioning and answering according to claim 1 is characterized in that: The determining of the similarity score between the first answer text and the second answer text includes: Calling a text encoder to respectively extract a first text feature of the first answer text and a second text feature of the second answer text; The cosine similarity between the first text feature and the second text feature is determined, and the cosine similarity is used as a similarity score between the first answer text and the second answer text.

4. The credible multimodal large model reasoning method based on self-questioning and answering according to claim 1 is characterized in that: The multimodal large model includes a question-answering model and an answer similarity scorer. The multimodal large model is trained by question samples. The question samples include real question texts and annotated answers to the real question texts. The training process of the multimodal large model includes: Inputting the question sample into the question-answering model for a first question-answering, obtaining a predicted answer, and constructing a first cross-entropy loss value of the question-answering model based on the predicted answer and the labeled answer; Constructing a second cross entropy loss value of the question-answering model according to the real question text and the annotated answer, and taking the sum of the first cross entropy loss value and the second cross entropy loss value as the final loss value of the question-answering model; Constructing a third cross entropy loss value of the answer similarity scorer according to the predicted answer; The final loss value is back-propagated in the question-answering model, and the third cross-entropy loss value is back-propagated in the answer similarity scorer to update the parameters of the multimodal large model.

5. The credible multimodal large model reasoning method based on self-questioning and answering according to claim 4 is characterized in that: The constructing a second cross entropy loss value of the question-answering model according to the real question text and the annotated answer includes: Obtaining a marking prompt word for the marked answer, wherein the marking prompt word is used to prompt the question-answering model to ask a question based on the marked answer; Inputting the marked answer and the marked prompt word into the question-answering model to perform a second question-answering to obtain a predicted question text; A second cross entropy loss value of the question-answering model is constructed based on the actual question text and the predicted question text.

6. The credible multimodal large model reasoning method based on self-questioning and answering according to claim 4 is characterized in that: The constructing a third cross entropy loss value of the answer similarity scorer according to the predicted answer includes: Constructing a corresponding similar answer for each predicted answer, wherein the similarity score between the similar answer and the predicted answer is greater than a preset score threshold; For each predicted answer, randomly select one answer from the similar answers as another answer to the predicted answer; Determine a predicted label of the predicted answer according to the similar answers and the other answers; A third cross entropy loss value of the answer similarity scorer is constructed according to the predicted label.

7. A credible multimodal large model reasoning device based on self-questioning and answering, characterized in that: The device comprises: An acquisition module is used to acquire multimodal data and call an encoder of a corresponding modality to perform feature extraction processing on the multimodal data to obtain corresponding multimodal token features; A first question-answering module, configured to obtain questions for the multimodal data, and input the questions and the multimodal token features into a multimodal macro model to perform a first question-answering operation; A second question-and-answer module, configured to input a first answer text and a prompt word obtained by the multimodal large model through the first question-and-answer process into the multimodal large model for a second question-and-answer process, thereby obtaining a second answer text, wherein the prompt word is used to prompt the multimodal large model to ask a question based on the first answer text; The determination module is used to determine the similarity score between the first answer text and the second answer text, and when the similarity score is greater than a similarity threshold, the first answer text is used as the answer text of the question.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the credible multimodal large model reasoning method based on self-questioning and answering is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for trustworthy multimodal large model reasoning based on self-questioning and answering is implemented as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for trustworthy multimodal large model reasoning based on self-questioning and answering is implemented as described in any one of claims 1 to 6.

Images