Question and answer method, device and equipment based on multi-modal large model and medium

Abstract

The invention discloses a question and answer method, device and equipment based on a multi-modal large model and a medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a target image and a target natural language query request inputted by a user, and determining visual features corresponding to the target image and text features corresponding to the target natural language query request, determining corresponding multi-modal fusion features based on the visual features and the text features; reasoning a target problem corresponding to the target natural language query request based on a chain thinking strategy, a preset tool set and a multi-modal fusion feature by using a pre-trained target large model and a preset intelligent agent to obtain a corresponding target reasoning result; and generating candidate answers based on the target reasoning result, determining target confidence degrees corresponding to the candidate answers, and determining and outputting target answers corresponding to the target questions based on the target confidence degrees and the candidate answers. According to the method, multi-modal information and reasoning verification can be integrated to improve the accuracy and reliability of questions and answers.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a question and answer method and device based on a multi-modal large model, equipment and a medium. BACKGROUND

[0002] In recent years, with the development of deep learning and large models, multi-modal processing technologies such as image recognition, speech recognition, OCR (Optical Character Recognition), and the construction of large-scale knowledge graph databases provide a new foundation for intelligent question and answer systems. However, the existing question and answer systems based on large language models still have defects such as outdated knowledge, lack of reasoning explanation, and easy to produce illusory answers. Even if the retrieval enhancement generation technology is used to let the model refer to external documents, there are still limitations such as poor quality of search results and knowledge conflicts when dealing with complex problems. And single modal input cannot meet the diversification needs of real-world scenarios, limiting the performance of the system. Although chain thinking prompts can improve the model's ability to handle complex problems, existing systems rarely combine multi-modal input and active tool invocation. In addition, using structured knowledge bases or denial mechanisms can improve the reliability of answers, but a unified multi-modal cross-domain question and answer framework has not been formed.

[0003] In summary, how to integrate multi-modal information and reasoning verification to improve the accuracy and reliability of question and answer is a technical problem that needs to be solved at present. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a question and answer method, device, equipment and medium based on a multi-modal large model, which can integrate multi-modal information and reasoning verification to improve the accuracy and reliability of question and answer. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a question and answer method based on a multi-modal large model, comprising:

[0006] obtaining a target image and a target natural language query request input by a user, and determining the visual features corresponding to the target image and the text features corresponding to the target natural language query request, and determining the corresponding multi-modal fusion features based on the visual features and the text features;

[0007] using a pre-trained target large model and a pre-set intelligent agent to reason the target problem corresponding to the target natural language query request based on a chain thinking strategy, a pre-set tool set and the multi-modal fusion features, and obtaining the corresponding target reasoning result; the target large model is a multi-modal large model;

[0008] generate a corresponding candidate answer based on the target reasoning result, determine a target confidence degree corresponding to the candidate answer, and determine and output a target answer corresponding to the target question based on the target confidence degree and the candidate answer.

[0009] Optionally, the target image and the target natural language query request input by the user are obtained, including:

[0010] The target image is obtained based on a preset image input interface, the first target text is obtained based on a preset text input interface, and the target voice input by the user is converted into corresponding second target text based on a preset voice recognition interface;

[0011] The target natural language query request is determined based on the first target text and the second target text.

[0012] Optionally, the visual feature corresponding to the target image and the text feature corresponding to the target natural language query request are determined, and a corresponding multi-modal fusion feature is determined based on the visual feature and the text feature, including:

[0013] The visual feature corresponding to the target image is extracted by using a preset visual encoder, and the text feature corresponding to the target natural language query request is extracted by using a preset text encoder;

[0014] The visual feature and the text feature are aligned, and the aligned visual feature and the aligned text feature are fused to obtain the multi-modal fusion feature.

[0015] Optionally, the target problem corresponding to the target natural language query request is reasoned based on a chain-of-thought strategy, a preset tool set and the multi-modal fusion feature by using a pre-trained target large model and a preset intelligent agent to obtain a corresponding target reasoning result, including:

[0016] The target large model and the preset intelligent agent are used to identify a user intent corresponding to the target question;

[0017] The target question is split into a plurality of target sub-questions based on the chain-of-thought strategy, the multi-modal fusion feature and the user intent, and an execution order of the target sub-questions is determined;

[0018] According to the execution order of the target sub-questions, a target tool corresponding to the target sub-questions is called from the preset tool set based on a ReACT paradigm, and the target tool is used to reason the target sub-questions to determine a reasoning result corresponding to each target sub-question;

[0019] determine the target reasoning result corresponding to the target problem based on reasoning results corresponding to each of the target sub-problems;

[0020] Correspondingly, after determining the reasoning result corresponding to each of the target sub-problems, the method further includes:

[0021] storing each of the target sub-problems, the target tool corresponding to each of the target sub-problems, and the reasoning result corresponding to each of the target sub-problems to a preset context storage space, so as to generate the candidate answer based on storage content in the preset context storage space and the target reasoning result.

[0022] Optionally, the calling of the target tool corresponding to the target sub-problem from the preset tool set based on the ReACT paradigm and the reasoning of the target sub-problem by the target tool include:

[0023] analyzing a reasoning step corresponding to the target sub-problem based on the ReACT paradigm;

[0024] calling a preset visual analysis tool from the preset tool set based on the reasoning step to reason the target sub-problem;

[0025] or, calling a preset multi-modal retrieval enhancement generation tool from the preset tool set based on the reasoning step to reason the target sub-problem;

[0026] or, calling a preset external interface from the preset tool set based on the reasoning step to obtain corresponding reference information through the preset external interface, and reasoning the target sub-problem based on the reference information.

[0027] Optionally, the determination of the target confidence corresponding to the candidate answer includes:

[0028] determining a target entity in the candidate answer and an attribute corresponding to the target entity, and calling a target knowledge graph to verify the target entity and the attribute corresponding to the target entity to obtain a corresponding verification result;

[0029] calling a preset full-text retrieval interface to retrieve a target document corresponding to the candidate answer, and calculating a target matching degree between the candidate answer and the target document;

[0030] determining a model confidence corresponding to the target large model, and performing weighted fusion on the model confidence and the target matching degree to obtain a corresponding fusion result;

[0031] determining the target confidence corresponding to the candidate answer based on the verification result and the fusion result.

[0032] Optionally, the determining and outputting of the target answer corresponding to the target question based on the target confidence and the candidate answer comprises:

[0033] If the target confidence is greater than or equal to a preset confidence threshold, the target answer corresponding to the target question is directly determined and output based on the candidate answer;

[0034] If the target confidence is less than the preset confidence threshold, corresponding rejection information is output.

[0035] Or, if the target confidence is less than the preset confidence threshold, corresponding supplementary inquiry information is output, and supplementary information corresponding to the supplementary inquiry information is obtained, so that the target answer corresponding to the target question is determined and output based on the supplementary information and the candidate answer.

[0036] In a second aspect, the present application provides a question and answer device based on a multi-modal large model, comprising:

[0037] A multi-modal fusion feature determination module is configured to obtain a target image and a target natural language query request input by a user, determine a visual feature corresponding to the target image and a text feature corresponding to the target natural language query request, and determine a corresponding multi-modal fusion feature based on the visual feature and the text feature.

[0038] A target reasoning result determination module is configured to use a pre-trained target large model and a preset intelligent agent to perform reasoning on a target question corresponding to the target natural language query request based on a chain thinking strategy, a preset tool set and the multi-modal fusion feature, and obtain a corresponding target reasoning result; the target large model is a multi-modal large model.

[0039] A target answer determination module is configured to generate a corresponding candidate answer based on the target reasoning result, determine a target confidence corresponding to the candidate answer, and determine and output a target answer corresponding to the target question based on the target confidence and the candidate answer.

[0040] In a third aspect, the present application provides an electronic device, comprising:

[0041] A memory is configured to save a computer program.

[0042] A processor is configured to execute the computer program to implement the question and answer method based on a multi-modal large model as described above.

[0043] In a fourth aspect, the present application provides a computer readable storage medium configured to save a computer program; when the computer program is executed by a processor, the question and answer method based on a multi-modal large model as described above is implemented.

[0044] In this application, the target image and target natural language query request input by the user are first obtained, and the visual features corresponding to the target image and the text features corresponding to the target natural language query request are determined, and the corresponding multimodal fusion features are determined based on the visual features and the text features; then, a pre-trained target large model and a preset intelligent agent are used to reason about the target question corresponding to the target natural language query request based on the chain thinking strategy, the preset tool set and the multimodal fusion features to obtain the corresponding target reasoning result; the target large model is a multimodal large model; then, based on the target reasoning result, the corresponding candidate answer is generated, and the target confidence corresponding to the candidate answer is determined, and the target answer corresponding to the target question is determined and output based on the target confidence and the candidate answer. As can be seen from the above, in this application, the target image and the target natural language query request are first obtained, and then the visual features corresponding to the target image and the text features corresponding to the target natural language query request are determined, and the visual features and text features are fused to obtain the multimodal fusion features. Then, using the target large model and the preset intelligent agent, based on the chain thinking strategy and multimodal fusion features, the preset tool set is called to infer the target question corresponding to the target natural language query request to obtain the target reasoning result. Subsequently, the candidate answer corresponding to the target reasoning result is generated, and the target answer is determined based on the target confidence corresponding to the candidate answer. In this way, this application can automatically fuse multiple modal inputs such as images and texts, use large models and intelligent agents for joint reasoning, and obtain information by actively calling tools in the preset tool set. At the same time, this application verifies the candidate answers, determines the final target answer based on the confidence, thereby significantly improving the accuracy and reliability of questions and answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0046] Figure 1 A flowchart of a question-answering method based on a multimodal large model provided in this application;

[0047] Figure 2 An architecture diagram of a specific question-answering system provided for this application;

[0048] Figure 3 A specific workflow diagram of the user input module and multimodal perception and fusion module provided in this application;

[0049] Figure 4A specific intelligent agent module workflow diagram provided for the present application;

[0050] Figure 5 A specific multi-modal large model-based question answering method flowchart provided for the present application;

[0051] Figure 6 A specific fact verification and rejection module workflow diagram provided for the present application;

[0052] Figure 7 A multi-modal large model-based question answering device structure schematic diagram provided for the present application;

[0053] Figure 8 An electronic device structure diagram provided for the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0055] In recent years, with the development of deep learning and large models, image recognition, speech recognition, OCR and other multi-modal processing technologies, as well as the construction of large-scale knowledge graph databases, provide a new foundation for intelligent question answering systems. However, the existing question answering systems based on large language models still have the defects of outdated knowledge, lack of reasoning explanation, and easy to produce illusory answers. Even if the retrieval enhancement generation technology is used to let the model refer to external documents, there are still limitations such as poor quality of retrieval results and knowledge conflicts when dealing with complex problems. And single modal input cannot meet the diversification needs of real-world scenarios, limiting the performance of the system. Chain thinking prompts can improve the model's ability to handle complex problems, but existing systems rarely combine with multi-modal input and active tool invocation. In addition, using structured knowledge bases or rejection mechanisms can improve the reliability of answers, but a unified multi-modal cross-domain question answering framework has not been formed. Therefore, the present application provides a multi-modal large model-based question answering scheme that can integrate multi-modal information and reasoning verification to improve the accuracy and reliability of question answering.

[0056] Referring to Figure 1 The embodiments of the present application disclose a multi-modal large model-based question answering method, which can include:

[0057] Step S11: Obtain a target image and a target natural language query request input by a user, determine visual features corresponding to the target image and text features corresponding to the target natural language query request, and determine corresponding multimodal fusion features based on the visual features and the text features.

[0058] It should be noted that, see Figure 2 As shown, the question-answering system constructed in this embodiment includes a user input module, a multimodal perception and fusion module, an intelligent agent module, a candidate answer generation module, and a fact verification and rejection module.

[0059] In this embodiment, see Figure 3 As shown, the user input module has input interfaces such as image recognition, speech recognition, and text parsing. Specifically, in order to obtain the target image and target natural language query request input by the user, the target image can first be obtained based on the preset image input interface, and the first target text can be obtained based on the preset text input interface, and the target speech input by the user can be converted into the corresponding second target text based on the preset speech recognition interface; then the target natural language query request can be determined based on the first target text and the second target text. It should be noted that the multimodal perception and fusion module can pre-process various types of inputs and convert visual and text information into feature representations in a unified format for downstream processing. The above-mentioned determination of the visual features corresponding to the target image and the text features corresponding to the target natural language query request, and the determination of the corresponding multimodal fusion features based on the visual features and the text features can include: first using a preset visual encoder to extract the visual features corresponding to the target image, and using a preset text encoder to extract the text features corresponding to the target natural language query request; then aligning the visual features and the text features, and fusion of the aligned visual features and the aligned text features to obtain the multimodal fusion features. Specifically, the preset visual encoder can be used to extract the visual features corresponding to the target image, such as object contours, colors, and textures, and the OCR engine can be called to extract text information in the image. For the target natural language query request, the preset text encoder can be used to perform word segmentation and semantic analysis to generate text features. The modal alignment and fusion device then aligns the visual features and text features to find the semantic correspondence between different modal information and avoid information disconnection. Based on the modal alignment, the modal alignment and fusion device performs feature fusion to generate a unified cross-modal representation, namely a multimodal fusion feature, for subsequent reasoning. In this way, multimodal input fusion enables the question-answering system to understand richer user intentions and contexts, and has higher flexibility than a single modal system.

[0060] Step S12: Using the pre-trained target big model and the preset intelligent agent, reasoning is performed on the target question corresponding to the target natural language query request based on the chain thinking strategy, the preset tool set and the multimodal fusion features to obtain the corresponding target reasoning result; the target big model is a multimodal big model.

[0061] In this embodiment, see Figure 4 As shown, the intelligent agent module can perform reasoning based on a pre-trained target macro model using a CoT (Chain of Thought) strategy. This method of using the pre-trained target macro model and a preset intelligent agent to reason about the target question corresponding to the target natural language query request based on the CoT strategy, a preset tool set, and the multimodal fusion features to obtain the corresponding target reasoning result can include: first, using the target macro model and the preset intelligent agent to identify the user intent corresponding to the target question; then, based on the CoT strategy, the multimodal fusion features, and the user intent, splitting the target question into several target sub-problems and determining the execution order for the target sub-problems; then, based on the execution order for the target sub-problems and based on the ReACT paradigm, calling the target tools corresponding to the target sub-problems from the preset tool set and reasoning about the target sub-problems using the target tools to determine the reasoning results for each target sub-problem; and finally, determining the target reasoning result for the target question based on the reasoning results for each target sub-problem. Specifically, the intelligent agent module first receives the multimodal fusion features and uses the target macro model to determine the user intent based on the natural language query request. The agent (i.e., intelligent agent) then uses the task understanding and planning tool to break the target problem into multiple subproblems and plan the execution sequence to gradually generate reasoning steps, recording the thought process at each step. The tool selector and invoker then dynamically selects and triggers the appropriate tool from the tool set based on the target subproblem and context, thereby reasoning on the subproblem and obtaining the inference result.

[0062] It should be noted that after the above-mentioned determination of the reasoning results corresponding to each of the target sub-problems, the process may also include: storing each of the target sub-problems, the target tools corresponding to each of the target sub-problems, and the reasoning results corresponding to each of the target sub-problems into a preset context storage space, so as to generate the candidate answers based on the storage content in the preset context storage space and the target reasoning results. Specifically, the context management and memory module can continuously record and update information during the reasoning process. The reasoning results corresponding to the target sub-problems can be stored in a preset context storage space. The context storage space can use a dialogue tree or a vector memory structure to store historical reasoning steps, tool call records, and user preferences to support the coherence and information reuse of multiple rounds of questions and answers.

[0063] In the reasoning process of the problem, the intelligent agent can dynamically call external tools as needed. The above calling of the target tool corresponding to the target sub-problem from the preset tool set based on the ReACT paradigm and using the target tool to reason the target sub-problem can include: first analyzing the reasoning step corresponding to the target sub-problem based on the ReACT paradigm; calling a preset visual analysis tool from the preset tool set based on the reasoning step to reason the target sub-problem; or, calling a preset multi-modal retrieval enhancement generation tool from the preset tool set based on the reasoning step to reason the target sub-problem; or, calling a preset external interface from the preset tool set based on the reasoning step to obtain corresponding reference information through the preset external interface, and reasoning the target sub-problem based on the reference information. Specifically, the question and answer system is configured with a tool interface module, which integrates search engines, knowledge graph queries, and OCR recognition and other tools. When the reasoning step involves open domain knowledge or image text information, the agent automatically triggers the corresponding tool. The tool set includes a domain-specific visual analysis module, a multi-modal retrieval enhancement generation module, an external API (Application Programming Interface, i.e. application programming interface) calling module, and other general visual tools. The domain-specific visual analysis module is used to perform professional visual analysis such as license plate recognition and target detection; the multi-modal retrieval enhancement generation module is used for cross-modal retrieval based on a vector database and auxiliary generation; the external API calling module is used to encapsulate Web search, professional database query, and computing services and other open interfaces; and the other general visual tools are used to perform basic visual processing such as OCR, image segmentation, and color analysis. In a specific implementation, after detecting the license plate number in the image, the external API module is preferentially called to perform information retrieval through a search engine or a license plate database; for problems involving specific entity relationships, the knowledge graph query related entities and attributes can be called. The active tool calling expands the knowledge source and capability range of the question and answer system, effectively reduces the influence of outdated knowledge in the large model, and improves the coverage and accuracy of the answer.

[0064] Step S13, generating a corresponding candidate answer based on the target reasoning result, determining a target confidence degree corresponding to the candidate answer, and determining and outputting a target answer corresponding to the target problem based on the target confidence degree and the candidate answer.

[0065] In the embodiment, the candidate answer generation module can generate a candidate answer according to the inference result of the intelligent agent module and the tool call output. Then, the question and answer system is provided with a fact verification and rejection module to evaluate the credibility of the intermediate conclusion and the final answer generated by the intelligent agent module. The fact verification and rejection module verifies the factuality of each step of inference and the answer by searching external evidence and checking internal logical consistency, checks the candidate answer according to the target confidence corresponding to the candidate answer, to determine and output the target answer corresponding to the target question.

[0066] As can be seen from the above, in the embodiment, first, the target image and the target natural language query request input by the user are acquired, and the visual features corresponding to the target image and the text features corresponding to the target natural language query request are determined, and the corresponding multi-modal fusion features are determined based on the visual features and the text features; then, based on the chain thinking strategy, the preset tool set and the multi-modal fusion features, the target question corresponding to the target natural language query request is inferred by using the pre-trained target large model and the preset intelligent agent to obtain the corresponding target inference result; the target large model is a multi-modal large model; subsequently, the corresponding candidate answer is generated based on the target inference result, and the target confidence corresponding to the candidate answer is determined, and the target answer corresponding to the target question is determined and output based on the target confidence and the candidate answer. As can be seen from the above, in the embodiment, first, the target image and the target natural language query request are acquired, then the visual features corresponding to the target image and the text features corresponding to the target natural language query request are determined, and the multi-modal fusion features are obtained by fusing the visual features and the text features. Then, based on the chain thinking strategy and the multi-modal fusion features, the target question corresponding to the target natural language query request is inferred by using the target large model and the preset intelligent agent to obtain the target inference result. Subsequently, the candidate answer corresponding to the target inference result is generated, and the target answer is determined based on the target confidence corresponding to the candidate answer. In this way, in the embodiment, the image, text and other multi-modal inputs can be automatically fused, the large model and the intelligent agent are used for joint inference, and the tools in the preset tool set are actively called to obtain information. At the same time, the candidate answer is verified in the embodiment, and the final target answer is determined in combination with the confidence, so as to significantly improve the accuracy and reliability of the question and answer.

[0067] Referring to Figure 5 As shown in the figure, in order to improve the accuracy and reliability of the answer result, the embodiment of the application further discloses a question and answer method based on a multi-modal large model, which can include:

[0068] Step S21, obtaining a target image and a target natural language query request input by a user, determining visual features corresponding to the target image and text features corresponding to the target natural language query request, and determining corresponding multi-modal fusion features based on the visual features and the text features.

[0069] Step S22, reasoning a target problem corresponding to the target natural language query request based on a chain thinking strategy, a preset tool set and the multi-modal fusion features by using a target large model pre-trained and a preset intelligent agent, to obtain a corresponding target reasoning result; the target large model is a multi-modal large model.

[0070] Step S23, generating a corresponding candidate answer based on the target reasoning result, determining a target confidence corresponding to the candidate answer, and determining and outputting a target answer corresponding to the target problem based on the target confidence and the candidate answer.

[0071] In this embodiment, the fact verifier can perform entity extraction and evidence retrieval on the candidate answer, and compare the answer elements with external information sources such as knowledge graphs, network documents, etc. Specifically, first, the target entity in the candidate answer and the attribute corresponding to the target entity can be determined, and the target knowledge graph is called to verify the target entity and the attribute corresponding to the target entity, to obtain a corresponding verification result; then a preset full-text retrieval interface is called to retrieve the target document corresponding to the candidate answer, and a target matching degree between the candidate answer and the target document is calculated; then the model confidence corresponding to the target large model is determined, and the model confidence and the target matching degree are weighted and fused to obtain a corresponding fusion result; finally, the target confidence corresponding to the candidate answer is determined based on the verification result and the fusion result. That is, the fact verifier can perform entity extraction on the candidate answer and determine the attribute corresponding to the target entity, and then call the knowledge graph query interface to compare the entity-attribute pair to obtain the verification result. Then the preset full-text retrieval interface can be called to retrieve the related target document, and the target matching degree between the candidate answer and the target document is calculated. The target matching degree and the model confidence of the target large model are weighted and fused to obtain a fusion result, and the reliability of the candidate answer is evaluated according to the fusion result and the fact verification result, to obtain the target confidence corresponding to the candidate answer.

[0072] It should be noted that, referring to Figure 6As shown, in the embodiment, the target confidence corresponding to the candidate answer is required to determine the target answer, and the target answer corresponding to the target question is determined and output based on the target confidence and the candidate answer, which can include: if the target confidence is greater than or equal to a preset confidence threshold, the target answer corresponding to the target question is directly determined and output based on the candidate answer; if the target confidence is less than the preset confidence threshold, corresponding rejection information is output; or, if the target confidence is less than the preset confidence threshold, corresponding supplementary inquiry information is output, and supplementary information corresponding to the supplementary inquiry information is obtained, so as to determine and output the target answer corresponding to the target question based on the supplementary information and the candidate answer. Specifically, when the target confidence is greater than or equal to the preset confidence threshold, it is determined that the candidate answer is reliable, and the question and answer system presents the target answer to the user through "natural language answer". When the target confidence is less than the preset confidence threshold, the rejection or supplementary inquiry is triggered by the rejection controller, thereby guiding the user to provide supplementary information or re-ask.

[0073] More specific processing procedures for steps S21 and S22 can be referred to the corresponding content disclosed in the foregoing embodiments, which will not be described here.

[0074] As can be seen, in the embodiment, the target confidence corresponding to the candidate answer can be determined by the fact verifier, and when the target confidence is greater than or equal to the preset confidence threshold, it is determined that the candidate answer is reliable, and the candidate answer is output. When the target confidence is less than the preset confidence threshold, the rejection or supplementary inquiry is triggered by the rejection controller, thereby abandoning the output of the possible wrong answer, or re-searching the supplementary information, thereby avoiding providing unreliable information to the user, and improving the accuracy and reliability of the answer result. In this way, the embodiment can provide the user with reliable information question and answer service.

[0075] Correspondingly, referring to Figure 7 As shown, the application embodiment also provides a question and answer device based on a multi-modal large model, which can include:

[0076] The multi-modal fusion feature determination module 11 is configured to obtain a target image and a target natural language query request input by a user, determine a visual feature corresponding to the target image and a text feature corresponding to the target natural language query request, and determine a corresponding multi-modal fusion feature based on the visual feature and the text feature;

[0077] The target reasoning result determination module 12 is configured to use a pre-trained target large model and a preset intelligent agent to reason a target question corresponding to the target natural language query request based on a chain thinking strategy, a preset tool set and the multi-modal fusion feature, and obtain a corresponding target reasoning result; the target large model is a multi-modal large model.

[0078] The target answer determination module 13 is configured to generate a corresponding candidate answer based on the target reasoning result, determine a target confidence corresponding to the candidate answer, and determine and output a target answer corresponding to the target question based on the target confidence and the candidate answer.

[0079] As can be seen from the above, in the present application, first, a target image and a target natural language query request input by a user are acquired, visual features corresponding to the target image and text features corresponding to the target natural language query request are determined, and a corresponding multi-modal fusion feature is determined based on the visual features and the text features; then, a target large model and a preset intelligent agent that have been trained in advance are used to infer a target question corresponding to the target natural language query request based on a chain thinking strategy, a preset tool set, and the multi-modal fusion feature, to obtain a corresponding target reasoning result; the target large model is a multi-modal large model; subsequently, a corresponding candidate answer is generated based on the target reasoning result, a target confidence corresponding to the candidate answer is determined, and a target answer corresponding to the target question is determined and output based on the target confidence and the candidate answer. As can be seen from the above, in the present application, first, a target image and a target natural language query request are acquired, then visual features corresponding to the target image and text features corresponding to the target natural language query request are determined, and the visual features and the text features are fused to obtain a multi-modal fusion feature. Then, a target large model and a preset intelligent agent are used to infer a target question corresponding to the target natural language query request based on a chain thinking strategy and a multi-modal fusion feature, to obtain a target reasoning result. Subsequently, a candidate answer corresponding to the target reasoning result is generated, and a target answer is determined based on a target confidence corresponding to the candidate answer. In this way, in the present application, images, texts, and other multi-modal inputs can be automatically fused, a large model and an intelligent agent are used for joint reasoning, and information is obtained by actively calling tools in a preset tool set. At the same time, the present application verifies a candidate answer, determines a final target answer in combination with a confidence, thereby significantly improving the accuracy and reliability of question answering.

[0080] In some embodiments, the multi-modal fusion feature determination module 11 can include:

[0081] A target image acquisition unit is configured to acquire the target image based on a preset image input interface, acquire a first target text based on a preset text input interface, and convert a target voice input by a user into a corresponding second target text based on a preset voice recognition interface.

[0082] A natural language query request acquisition unit is configured to determine the target natural language query request based on the first target text and the second target text.

[0083] In some embodiments, the multi-modal fusion feature determination module 11 can include:

[0084] a feature extraction unit configured to extract the visual features corresponding to the target image using a preset visual encoder and extract the text features corresponding to the target natural language query request using a preset text encoder;

[0085] a multi-modal fusion feature determination unit configured to perform feature alignment on the visual features and the text features and perform feature fusion on the aligned visual features and the aligned text features to obtain the multi-modal fusion features.

[0086] In some embodiments, the target reasoning result determination module 12 can include:

[0087] a user intent identification sub-module configured to identify the user intent corresponding to the target question using the target large model and the preset intelligent agent;

[0088] a target question splitting sub-module configured to split the target question into a plurality of target sub-questions based on the chain-of-thought strategy, the multi-modal fusion features, and the user intent and determine an execution order of the target sub-questions;

[0089] a question reasoning sub-module configured to invoke target tools corresponding to the target sub-questions from the preset tool set based on the ReACT paradigm according to the execution order of the target sub-questions, and perform reasoning on the target sub-questions using the target tools to determine reasoning results corresponding to the target sub-questions;

[0090] a target reasoning result determination sub-module configured to determine the target reasoning result corresponding to the target question based on the reasoning results corresponding to the target sub-questions;

[0091] Correspondingly, the question and answer device based on a multi-modal large model can further include:

[0092] a candidate answer generation module configured to store each of the target sub-questions, the target tools corresponding to each of the target sub-questions, and the reasoning results corresponding to each of the target sub-questions to a preset context storage space, so as to generate the candidate answers based on the stored content in the preset context storage space and the target reasoning result.

[0093] In some embodiments, the question reasoning sub-module can include:

[0094] a reasoning step analysis unit configured to analyze the reasoning steps corresponding to the target sub-questions based on the ReACT paradigm;

[0095] a question reasoning unit configured to invoke a preset visual analysis tool from the preset tool set to reason the target sub-question based on the reasoning step, or invoke a preset multi-modal retrieval enhancement generation tool from the preset tool set to reason the target sub-question based on the reasoning step, or invoke a preset external interface from the preset tool set to obtain corresponding reference information through the preset external interface and reason the target sub-question based on the reference information.

[0096] In some embodiments, the target answer determination module 13 can include:

[0097] a verification result determination unit configured to determine a target entity in the candidate answer and an attribute corresponding to the target entity, and invoke a target knowledge graph to verify the target entity and the attribute corresponding to the target entity to obtain a corresponding verification result;

[0098] a target matching degree determination unit configured to invoke a preset full-text retrieval interface to retrieve a target document corresponding to the candidate answer, and calculate a target matching degree between the candidate answer and the target document;

[0099] a fusion result determination unit configured to determine a model confidence corresponding to the target large model, and perform weighted fusion on the model confidence and the target matching degree to obtain a corresponding fusion result;

[0100] a target confidence determination unit configured to determine the target confidence corresponding to the candidate answer based on the verification result and the fusion result.

[0101] In some embodiments, the target answer determination module 13 can include:

[0102] a first target answer determination unit configured to, if the target confidence is greater than or equal to a preset confidence threshold, directly determine and output the target answer corresponding to the target question based on the candidate answer;

[0103] a second target answer determination unit configured to, if the target confidence is less than the preset confidence threshold, output corresponding rejection information, or, if the target confidence is less than the preset confidence threshold, output corresponding supplementary inquiry information, and obtain supplementary information corresponding to the supplementary inquiry information, to determine and output the target answer corresponding to the target question based on the supplementary information and the candidate answer.

[0104] Further, the embodiments of the present application also disclose an electronic device, Figure 8is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the diagram should not be considered as any limitation on the use range of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the related steps in the multi-modal large model-based question answering method disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the present embodiment can be an electronic computer.

[0105] In the present embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited here; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited here.

[0106] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk, or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0107] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the multi-modal large model-based question answering method executed by the electronic device 20 disclosed in any of the preceding embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0108] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the multi-modal large model-based question answering method disclosed in the preceding embodiments. For the specific steps of the method, please refer to the corresponding content disclosed in the preceding embodiments, which will not be repeated here.

[0109] The embodiments in the present specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. For the same or similar parts between embodiments, please refer to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and please refer to the method part for the relevant part.

[0110] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality, without referring to a specific sequence of operations for implementing the functions. The order of various illustrative blocks, modules, circuits, and steps may be re-arranged or otherwise implemented without departing from the spirit of the application, which is

[0111] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, hard disk can be used as a non-transitory storage medium to store software modules.

[0112] Finally, it should be noted that the terms "comprises", "comprising", or other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element. The terms "comprising", "comprises", "including", "includes", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0113] The above detailed description has set forth various embodiments of the methods and devices disclosed herein. The descriptions of the embodiments are not intended to be exhaustive or to be construed as limiting the application. Many modifications and variations will be apparent to those of ordinary skill in the art, in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of the application and its best mode of operation to others skilled in the art.

Claims

1. A question-answering method based on a multimodal large model, characterized in that: include: Obtaining a target image and a target natural language query request input by a user, determining visual features corresponding to the target image and text features corresponding to the target natural language query request, and determining corresponding multimodal fusion features based on the visual features and the text features; Using a pre-trained target large model and a preset intelligent agent, reasoning on the target question corresponding to the target natural language query request based on a chain thinking strategy, a preset tool set, and the multimodal fusion features, to obtain a corresponding target reasoning result; The target large model is a multimodal large model; Generate corresponding candidate answers based on the target reasoning result, determine the target confidence corresponding to the candidate answers, and determine and output the target answer corresponding to the target question based on the target confidence and the candidate answers.

2. The question-answering method based on a multimodal large model according to claim 1, characterized in that: The step of obtaining a target image and a target natural language query request input by a user includes: Acquire the target image based on a preset image input interface, acquire the first target text based on a preset text input interface, and convert the target voice input by the user into a corresponding second target text based on a preset voice recognition interface; The target natural language query request is determined based on the first target text and the second target text.

3. The question-answering method based on a multimodal large model according to claim 1, characterized in that: The determining of the visual features corresponding to the target image and the text features corresponding to the target natural language query request, and determining corresponding multimodal fusion features based on the visual features and the text features, includes: Extracting the visual features corresponding to the target image using a preset visual encoder, and extracting the text features corresponding to the target natural language query request using a preset text encoder; Feature alignment is performed on the visual features and the text features, and feature fusion is performed on the aligned visual features and the aligned text features to obtain the multimodal fusion features.

4. The question-answering method based on a multimodal large model according to claim 1, characterized in that: The method utilizes the pre-trained target large model and the preset intelligent agent to reason about the target question corresponding to the target natural language query request based on the chain thinking strategy, the preset tool set, and the multimodal fusion feature, and obtains the corresponding target reasoning result, including: Using the target macro model and the preset intelligent agent, identifying the user intent corresponding to the target question; Splitting the target problem into a plurality of target sub-problems based on the chain thinking strategy, the multimodal fusion feature, and the user intention, and determining the execution order corresponding to the target sub-problems; According to the execution order corresponding to the target subproblems and based on the ReACT paradigm, the target tools corresponding to the target subproblems are called from the preset tool set, and the target tools are used to reason about the target subproblems to determine the reasoning results corresponding to the target subproblems; Determine the target reasoning result corresponding to the target problem based on the reasoning results corresponding to each of the target subproblems; Accordingly, after determining the reasoning results corresponding to each target sub-problem, the method further includes: Each of the target sub-problems, the target tool corresponding to each of the target sub-problems, and the reasoning result corresponding to each of the target sub-problems are stored in a preset context storage space, so as to generate the candidate answer based on the storage content in the preset context storage space and the target reasoning result.

5. The question-answering method based on a multimodal large model according to claim 4, characterized in that: The calling of the target tool corresponding to the target sub-problem from the preset tool set based on the ReACT paradigm, and reasoning about the target sub-problem using the target tool, includes: Analyze the reasoning steps corresponding to the target sub-problem based on the ReACT paradigm; Based on the reasoning step, a preset visual analysis tool is called from the preset tool set to perform reasoning on the target sub-problem; Or, based on the reasoning step, calling a preset multimodal retrieval enhancement generation tool from the preset tool set to reason about the target sub-problem; Alternatively, a preset external interface is called from the preset tool set based on the reasoning step to obtain corresponding reference information through the preset external interface, and reasoning is performed on the target sub-problem based on the reference information.

6. The question-answering method based on a multimodal large model according to claim 1, characterized in that: Determining the target confidence corresponding to the candidate answer includes: Determine the target entity in the candidate answer and the attributes corresponding to the target entity, and call the target knowledge graph to verify the target entity and the attributes corresponding to the target entity to obtain a corresponding verification result; Calling a preset full-text search interface to retrieve the target document corresponding to the candidate answer, and calculating the target matching degree between the candidate answer and the target document; Determining the model confidence corresponding to the target large model, and performing weighted fusion on the model confidence and the target matching degree to obtain a corresponding fusion result; The target confidence corresponding to the candidate answer is determined based on the verification result and the fusion result.

7. The question-answering method based on a multimodal large model according to any one of claims 1 to 6, characterized in that: The determining and outputting a target answer corresponding to the target question based on the target confidence and the candidate answers includes: If the target confidence is greater than or equal to a preset confidence threshold, directly determining and outputting the target answer corresponding to the target question based on the candidate answers; If the target confidence is less than the preset confidence threshold, outputting corresponding rejection information; Alternatively, if the target confidence is less than the preset confidence threshold, the corresponding supplementary inquiry information is output, and supplementary information corresponding to the supplementary inquiry information is obtained to determine and output the target answer corresponding to the target question based on the supplementary information and the candidate answers.

8. A question-answering device based on a multimodal large model, characterized in that: include: a multimodal fusion feature determination module, configured to obtain a target image and a target natural language query request input by a user, determine visual features corresponding to the target image and text features corresponding to the target natural language query request, and determine corresponding multimodal fusion features based on the visual features and the text features; A target reasoning result determination module is used to use a pre-trained target large model and a preset intelligent agent to reason about the target question corresponding to the target natural language query request based on a chain thinking strategy, a preset tool set, and the multimodal fusion feature to obtain a corresponding target reasoning result; The target large model is a multimodal large model; The target answer determination module is used to generate corresponding candidate answers based on the target reasoning results, determine the target confidence corresponding to the candidate answers, and determine and output the target answer corresponding to the target question based on the target confidence and the candidate answers.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the question-answering method based on a multimodal large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed by a processor, implements the question-answering method based on a multimodal large model as described in any one of claims 1 to 7.

Images