Question and answer processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention discloses a question and answer processing method and device, equipment, a storage medium and a program product, which are used for improving the reasoning ability and generalization ability of visual reasoning while the reasoning result of a question is not easy to deviate. The method comprises the following steps: acquiring a to-be-processed image and a to-be-answered question; the to-be-processed image and the to-be-answered question are processed based on the target large language model, target decision information is generated, the target decision information is used for indicating an analysis condition related to a first keyword in the to-be-answered question, and the first keyword is used for describing key information of a to-be-processed object; processing the target decision information and the to-be-processed image based on a multi-modal large language model to generate supplementary information of the first keyword; and processing the target decision information, the supplementary information of the first keyword, the to-be-processed image and the to-be-answered question based on the target large language model, wherein the target question and answer result comprises a target answer of the to-be-answered question and specific analysis information of the target answer.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a method, apparatus, device, storage medium, and program product for question-and-answer processing. Background Technology

[0002] Visual reasoning is a crucial computer vision task that aims to enable models to understand visual content and reason based on questions. Visual reasoning tasks require models not only to identify objects in images but also to understand the content and reason about complex questions. With the rise of multimodal computing, the multimodal chain of thought (MCoT) approach is increasingly being used to solve visual reasoning tasks.

[0003] In the process of handling visual reasoning tasks using existing multimodal thinking chain methods, it is usually limited to reasoning on text data. It is necessary to decompose the problem into multiple related sub-tasks and call various external visual tools such as code programs to process each sub-task in turn. Then, a single-modal large language model is used to answer each sub-task one by one to form the thinking chain of the problem.

[0004] However, current solutions only consider the text content of the input question, which can easily lead to incorrect decisions by the unimodal large language model due to insufficient visual information. This affects the model's reasoning ability, reduces its visual reasoning capacity, and consequently causes biased reasoning results for visual questions. Furthermore, current solutions require additional low-level perceptual tools to process each subtask. Due to the limitations of these low-level perceptual tools, unimodal large language models cannot handle reasoning in complex scenarios such as long texts, thus reducing the generalization ability of visual reasoning. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and program product for question-answering processing, which aims to improve the reasoning ability and generalization ability of visual reasoning while making the reasoning results of questions less prone to deviation.

[0006] Firstly, embodiments of this application provide a question-and-answer processing method. The method includes:

[0007] The process involves obtaining an image to be processed and a question to be answered. The image to be processed includes the object to be processed and a text description of the object. The question to be answered is a question set for the object to be processed.

[0008] Based on the target large language model, the image to be processed and the question to be answered are processed to generate target decision information. The target decision information is used to indicate the parsing situation related to the first keyword in the question to be answered. The first keyword is used to describe the key information of the object to be processed.

[0009] The target decision information and the image to be processed are processed based on a multimodal large language model to generate supplementary information of the first keyword. The supplementary information of the first keyword is used to indicate the auxiliary parsing of the first keyword.

[0010] Based on the target large language model, the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered are processed. The target question answering result includes the target answer to the question to be answered and the specific analysis information of the target answer.

[0011] Secondly, embodiments of this application provide a question-and-answer processing apparatus. This apparatus includes an acquisition unit, a processing unit, and a determination unit.

[0012] The acquisition unit is used to acquire the image to be processed and the question to be answered. The image to be processed includes the object to be processed and the text description content used to describe the object to be processed. The question to be answered is a question set for the object to be processed.

[0013] The processing unit is used to process the image to be processed and the question to be answered based on the target large language model, and generate target decision information. The target decision information is used to indicate the parsing situation related to the first keyword in the question to be answered.

[0014] The processing unit is used to process the target decision information and the image to be processed based on the multimodal large language model, and generate supplementary information of the first keyword. The supplementary information of the first keyword is used to indicate the auxiliary parsing of the first keyword.

[0015] The determination unit is used to process the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered based on the target large language model, and to determine the target question answering result. The target question answering result includes the target answer to the question to be answered and the specific analysis information of the target answer.

[0016] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing unit is specifically used for:

[0017] Extract the primary keyword from the unanswered question;

[0018] The image to be processed and the first keyword are input into the target large language model to obtain target principle information and target task information. The target principle information includes the basic principle of the first keyword and the extraction principle of each second keyword. The extraction principle of each second keyword is used to indicate the feature extraction method of the corresponding second keyword. The first keyword is obtained from multiple second keywords and the correlation between multiple second keywords. The target task information is used to indicate the task allocation of each second keyword. The task allocation is related to the feature extraction method of the corresponding second keyword.

[0019] Based on the target principle information and target task information, target decision information is generated.

[0020] In another possible design, in another implementation of another aspect of the embodiments of this application, the processing unit is specifically used for:

[0021] Analyze the lexical meaning of each second keyword to determine the feature extraction type for the corresponding second keyword;

[0022] Based on the feature extraction type of each second keyword, a target expert module that matches the feature extraction type of the corresponding second keyword is selected from the candidate expert modules. The multimodal large language model includes candidate expert modules.

[0023] The target principle information, target task information, and the image to be processed are input into the target expert module corresponding to each second keyword to obtain supplementary information for each second keyword.

[0024] Based on the supplementary information of multiple second keywords, supplementary information of the first keyword is generated.

[0025] In another possible design, in another implementation of another aspect of the embodiments of this application, the processing unit is specifically used for:

[0026] In cases where the target large language model includes a multimodal large language model, the image to be processed and the question to be answered are input into the multimodal large language model to generate target decision information.

[0027] In another possible design, in another implementation of another aspect of the embodiments of this application, the processing unit is specifically used for:

[0028] When the target large language model includes a single-modal large language model, extract image description information from the image to be processed;

[0029] Image description information and the question to be answered are input into a single-modal large language model to generate target decision information.

[0030] In another possible design, in another implementation of another aspect of the embodiments of this application,

[0031] The acquisition unit is further configured to: before processing the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered based on the target large language model, and before determining the target question answering result, acquire the first prompt information, which is used to indicate the first output format corresponding to the target question answering result;

[0032] The determination unit is specifically used to process the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered according to the first prompt information, and to determine the target question and answer result.

[0033] In another possible design, in another implementation of another aspect of the embodiments of this application,

[0034] The acquisition unit is further configured to: acquire second prompt information, which indicates the second output format corresponding to the target question-and-answer result, wherein the first output format is different from the second output format;

[0035] The determination unit is specifically used to process the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered based on the target large language model according to the second prompt information, and to determine the target question and answer result.

[0036] In another possible design, in another implementation of another aspect of the embodiments of this application,

[0037] The acquisition unit is further used to: acquire a question-and-answer service request, which includes the field identifier of the prompt field in the first prompt information and the target field value of the prompt field;

[0038] The processing unit is further configured to adjust the current field value identified by the field identifier to the target field value of the corresponding prompt field in order to obtain the updated first prompt information;

[0039] The determination unit is specifically used to process the target decision information, supplementary information of the first keyword, the image to be processed, and the question to be answered based on the updated first prompt information and the target large language model to determine the target question-answering result.

[0040] A third aspect of this application provides a question-and-answer processing device, including: a memory, an input / output (I / O) interface, and a processor. The memory stores program instructions. The processor executes the program instructions in the memory to perform the question-and-answer processing method corresponding to the embodiment of the first aspect described above.

[0041] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method corresponding to the embodiments of the first aspect described above.

[0042] The fifth aspect of this application provides a computer program product containing instructions that, when run on a computer or processor, causes the computer or processor to execute the method described above for performing the implementation method of the first aspect.

[0043] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0044] In this embodiment, since the image to be processed includes the object to be processed and textual descriptions of the object, and the question to be answered is a question specifically designed for the object, after acquiring the image to be processed and the question to be answered, the image to be processed and the question to be answered are processed based on a target large language model to generate target decision information. The target decision information can be used to indicate the parsing status related to the first keyword in the question to be answered. After obtaining the target decision information, the image to be processed is simultaneously used as input to a multimodal large language model to process the target decision information and the image to be processed, thereby obtaining supplementary information for the first keyword. This supplementary information for the first keyword allows for the understanding of the auxiliary parsing status of the first keyword. Then, the target decision information, the supplementary information for the first keyword, the image to be processed, and the question to be answered are processed again based on the target large language model to determine the target question-and-answer result. The target question-and-answer result includes the target answer to the question to be answered and the specific parsing information of the target answer.

[0045] Through the above methods, this application considers not only the textual description content in the target large language model's decision-making but also the image to be processed in other visual dimensions. Furthermore, this application does not rely on external tools but directly uses a multimodal large language model to determine supplementary information about keywords in the question to be answered, thus fully combining textual description content, the image to be processed, target decision information, and supplementary information to comprehensively analyze the question. This not only enables the model to promote visual reasoning based on correct decision-making but also allows it to directly utilize the multimodal large language model to handle reasoning in complex scenarios such as long texts, making the reasoning results less prone to bias while improving the reasoning ability and generalization ability of visual reasoning. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 The illustration shows an application scenario provided by an embodiment of this application;

[0048] Figure 2 A flowchart illustrating a question-and-answer processing method provided in an embodiment of this application is shown.

[0049] Figure 3 A schematic diagram of a question-and-answer processing flow provided in an embodiment of this application is shown;

[0050] Figure 4 A schematic diagram of another question-and-answer processing flow provided in an embodiment of this application is shown;

[0051] Figure 5 This paper illustrates a flowchart of a process for generating target decision information according to an embodiment of this application.

[0052] Figure 6 This paper illustrates a flowchart of a process for generating supplementary information according to an embodiment of this application.

[0053] Figure 7 A schematic diagram of one embodiment of the question-and-answer processing apparatus provided in this application is shown;

[0054] Figure 8 A schematic diagram of one embodiment of the question-and-answer processing device provided in this application is shown. Detailed Implementation

[0055] This application provides a method, apparatus, device, storage medium, and program product for question-answering processing, which aims to improve the reasoning ability and generalization ability of visual reasoning while making the reasoning results of questions less prone to deviation.

[0056] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that implementations of the application described herein can be implemented, for example, in sequences other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0059] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, autonomous driving, smart healthcare, intelligent customer service, and question-and-answer scenarios. It is believed that with further technological development, AI will be applied in even more fields and play an increasingly important role.

[0060] For example, in question-and-answer scenarios, combining artificial intelligence technology can analyze users' question-and-answer intentions and identify the underlying needs, thereby enabling fast and accurate information retrieval and answers, and improving user experience. Through continuous optimization and expansion, AI question-and-answer systems will play an important role in more fields.

[0061] Visual reasoning is a crucial computer vision task that aims to enable models to understand visual content and reason about questions. Visual reasoning tasks require models not only to recognize objects in images but also to understand image content and possess the reasoning ability to answer complex visual questions, such as understanding causal relationships within a scene, relationships between objects, and the temporal sequence of events. However, while existing large language models have made some progress in handling visual reasoning tasks, they still have many shortcomings. For example, fine-tuning models for specific tasks requires a large amount of labeled data to achieve good performance, which leads to a time-consuming and labor-intensive labeling process. Furthermore, although models can recognize objects in images, their performance in understanding complex relationships between objects and reasoning about question answers remains less than ideal, making it difficult to accurately determine the answer to the question.

[0062] To address this, researchers proposed using the chain of thought (CoT) method to simulate the steps of human problem-solving. By breaking down complex reasoning processes into a series of coherent steps, this helps models better understand and answer questions, rather than directly providing answers. Using chain of thought to handle complex problems can improve the model's reasoning ability; however, this method is more effective in pure text-based, single-modal tasks than in complex reasoning tasks. With technological advancements and the rise of multimodal fields, solutions using multimodal chain of thought methods to solve complex problems have been proposed.

[0063] However, existing multimodal reasoning chain methods for problem-solving go beyond just text-based reasoning. They require calling external visual tools, such as code programs, to sequentially process the sub-tasks derived from the problem's division, and then using a unimodal large language model to answer each sub-task to form the problem's reasoning chain. In other words, current multimodal solutions only consider the text content of the input problem, making the unimodal large language model prone to erroneous decisions due to insufficient visual information. This affects the model's reasoning, reduces its visual reasoning ability, and leads to biased reasoning results for visual problems. Furthermore, current solutions require additional low-level perceptual tools to process each sub-task. Due to the limitations of these low-level tools, the unimodal large language model cannot handle reasoning in complex scenarios such as long texts, resulting in biased reasoning results for visual problems and reducing the generalization ability of visual reasoning.

[0064] Therefore, to address the aforementioned technical problems, this application provides a question-answering processing method. The question-answering processing method provided by this application can be applied to question-answering scenarios, including but not limited to scenarios involving answer reasoning. Using the method provided by this application, there is no need to rely on external tools; instead, it directly utilizes a target large language model to synchronously process text descriptions and other visual dimensions of the image to be processed, and uses a multimodal large language model to determine supplementary information of keywords in the question to be answered, thereby comprehensively analyzing the question. This not only enables the model to promote visual reasoning of the multimodal large language model based on correct decision-making, but also allows direct use of the multimodal large language model to process reasoning in complex scenarios such as long texts, making the reasoning results less prone to deviation while improving the reasoning ability and generalization ability of visual reasoning. The question-answering processing method provided by this application can be applied in at least one of the following scenarios.

[0065] Applicable Scenario 1: Education Field

[0066] In one example, in different subjects such as mathematics and physics, students and teachers often need to solve different questions through logical reasoning to determine the answers and reasoning processes behind those answers. In this case, users can use their question-and-answer processing device to acquire the image to be processed and the question to be answered. They then use a target large language model to process the image and the question, and further use a multimodal large language model to process the obtained target decision information and the image to obtain supplementary information about the keywords in the question. Finally, by combining the target decision information, supplementary information, the image to be processed, and the question to be answered, the target question-and-answer result is determined, thus obtaining the target answer to the question and its detailed explanation. This allows for personalized learning guidance based on the user's questions, helping them understand complex concepts and problem-solving strategies.

[0067] Applicable Scenario 2: Customer Service

[0068] In one example, during the product purchase, use, and after-sales processes, consumers and other users frequently have questions about product functions, usage methods, returns and exchanges, and complaints. For this scenario, the intelligent customer service system can, after acquiring the images and questions uploaded by consumers and other users, process them using a target large language model. Then, it can further process the obtained target decision information and images using a multimodal large language model to obtain supplementary information about keywords in the question. Finally, by combining the target decision information, supplementary information, images, and questions, it determines the target question-and-answer result, thus obtaining the target answer to the question and its detailed explanation. This allows for a rapid assessment of the nature of the problem and the provision of corresponding solutions and approaches.

[0069] Applicable Scenario 3: Search Engines

[0070] In one example, such as in search question-answering scenarios like result optimization or semantic understanding, a search user can obtain the image to be processed and the question to be answered through their question-answering processing device. The device then uses a target large language model to process the image and the question, and further uses a multimodal large language model to process the obtained target decision information and the image to obtain supplementary information about the keywords in the question. Subsequently, the target decision information, supplementary information, the image to be processed, and the question to be answered are combined to determine the target question-answering result, thus obtaining the target answer to the question and the specific analytical information of the target answer. This allows the system to understand the search user's query intent, semantic information, and other visual information, thereby providing the search user with more accurate question-answering results.

[0071] It should be noted that, in practical applications, the question-and-answer processing method provided in this application can also be applied to question-and-answer scenarios such as smart assistants, smart homes, daily life decision-making, scientific research, and social media.

[0072] For example, the method provided in this application can be applied to Figure 1 The illustrated implementation environment uses the question-and-answer processing device 100 to perform operations. For example, in... Figure 1The illustrated implementation environment includes a question-and-answer processing device 100. In step S1, the user inputs an image to be processed and a question to be answered into the question-and-answer processing device 100, thereby enabling the device to obtain the image and the question. In step S2, the device processes the image and the question using a target large language model to generate target decision information, which indicates the parsing of the first keyword in the question. In step S3, the device further processes the target decision information and the image based on a multimodal large language model to generate supplementary information for the first keyword, which indicates the auxiliary parsing of the first keyword. In step S4, the device processes the target decision information, the supplementary information for the first keyword, the image, and the question based on the target large language model to determine the target question-and-answer result, which includes the target answer to the question and the specific parsing information of the target answer. Optionally, in step S5, the question-and-answer processing device 100 may also display the target question-and-answer results through a client or other visual interface for users to view.

[0073] It should be noted that the question-and-answer processing device 100 mentioned above includes, but is not limited to, terminal devices, servers, or other robots. For example, terminal devices include, but are not limited to, mobile phones, tablets, laptops, desktop computers, intelligent voice interaction devices, smart home appliances, in-vehicle terminals, aircraft, and wearable devices. The client is deployed on the terminal device and can run on the terminal device via a browser or as a standalone application (APP). For example, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence (AI) platforms.

[0074] For example, in the above Figure 1In the illustrated implementation environment, the question-and-answer processing device 100 can also be connected to the database 200. The database 200, in short, can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, shared by multiple users, with minimal redundancy, and independent of application programs. A Database Management System (DBMS) is a computer software system designed to manage the database 200, generally possessing basic functions such as storage, retrieval, security, and backup. Database management systems can be classified according to the database model they support, such as relational or Extensible Markup Language (XML); or according to the type of computer they support, such as server clusters or mobile phones; or according to the query language used, such as Structured Query Language (SQL) or XQuery; or according to performance priorities, such as maximum scale or highest operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, simultaneously supporting multiple query languages. In this application, database 200 can be used to store models such as the target large language model, multimodal large language model, and unimodal large language model, and can also be used to store prompt templates and other prompt information. Optionally, the storage location of the large language model and prompt information involved in this application is not limited to database 200; for example, it can also be stored in the distributed file system of the question-answering processing device 100.

[0075] Based on the above introduction, the question-and-answer processing method in this application will be described below. Please refer to [link / reference]. Figure 2 The question-and-answer processing method in this application embodiment can be performed by a question-and-answer processing device as the execution subject. The method of this application includes:

[0076] 201. Obtain the image to be processed and the question to be answered. The image to be processed includes the object to be processed and the text description content used to describe the object to be processed. The question to be answered is a question set for the object to be processed.

[0077] In one or more embodiments, a user can input a question to be answered and an image to be processed into the question-and-answer processing device based on their query needs, question-and-answer requirements, etc. For example, a user can input the question to be answered and the image to be processed into the agent or other editable area in the question-and-answer processing device. Thus, the question-and-answer processing device can obtain the corresponding image to be processed and the question to be answered.

[0078] The image to be processed in this application includes the object to be processed and textual description content. The described textual description content is used to describe the content of the object to be processed from the perspective of text. Optionally, the image to be processed mentioned in this application may include one image, multiple images, or an image composed of multiple sub-images, etc., and is not limited in this application. In addition, the question to be answered is a question set for the object to be processed. Or, the question to be answered includes a first keyword, which can be used to describe the key information of the object to be processed.

[0079] 202. Based on the target large language model, process the image to be processed and the question to be answered to generate target decision information. The target decision information is used to indicate the parsing status of the first keyword in the question to be answered.

[0080] In one or more examples, the question to be answered includes a first keyword, and this first keyword can be used to describe key information about the object to be processed. The image to be processed and the question to be answered described here can be understood with reference to the content described in step 201 above, and will not be repeated here.

[0081] Figure 3 This diagram illustrates a framework of a question-and-answer processing flow provided in an embodiment of this application. Figure 3 As shown, after obtaining the image to be processed and the question to be answered in step 201, the image to be processed and the question to be answered can also be used as the model input of the target large language model, so as to process the image to be processed and the question to be answered through the target large language model, thereby generating target decision information.

[0082] The target large language models described include, but are not limited to, multimodal large language models and unimodal large language models. The multimodal large language models (MLLMs) described include, but are not limited to, Gemini, GPT-4V (generative pre-trained transformer-4Vision) models, LLaVA (large language and vision assistant) models, etc.; unimodal large language models (LLMs) include, but are not limited to, GPT-3.5 (generative pre-trained transformer 3.5) models, GPT-4 (generative pre-trained transformer4) models, etc., and are not limited in this application.

[0083] Optionally, taking a target large language model that includes a multimodal large language model as an example, since the multimodal large language model can directly process images, the image to be processed and the question to be answered can be directly input into the multimodal large language model. The multimodal large language model then processes the image and the question, generating corresponding target decision information. By inputting only the image and the question into the multimodal large language model, it is possible to integrate and understand data from different modalities, such as images and text, within the image and the question, thereby achieving cross-modal understanding and generation, and ultimately providing more comprehensive and accurate target decision information.

[0084] Optionally, taking a target large language model that includes a unimodal large language model as an example, since a unimodal large language model can only process text-level information, when the target large language model includes a unimodal large language model, after obtaining the image to be processed, image description information can also be extracted from the image to be processed, such as subsequent... Figure 4 Information such as the number and location of particles in object b1 to be processed in image a1 is extracted. After extracting the image description information from the image to be processed, this image description information and the question to be answered are used as input to a single-modal large language model. The single-modal large language model then processes the image description information and the question to be answered to generate target decision information. In this way, after inputting the image to be processed, the single-modal large language model can focus on processing a single type of text data input from the image to be processed and the question to be answered. This not only expands its applicability to image scenarios but also enables a deeper understanding of the content in the image, providing rich and accurate target decision information.

[0085] The aforementioned target decision information can indicate the analysis of the first keyword in the question to be answered, such as the basic principles of the first keyword and the feature extraction method for analyzing the first keyword.

[0086] For example, Figure 4 A schematic diagram of another question-and-answer processing flow provided in an embodiment of this application is shown. For example... Figure 4 As shown in Figure A301, images to be processed are displayed, such as images a1 and a2. Image a1 includes the object to be processed, b1 (e.g., solution A), and text description b2, for example, the text description b2 is "solvent volume: 40ml, solution A"; image a2 includes the object to be processed, c1 (e.g., solution B), and text description c2, for example, the text description c2 is "solvent volume: 40ml, solution B".

[0087] In addition, from Figure 4It can also be seen that the unsolved questions are shown in A302, such as "Which solution has a higher concentration? (A) Solution A, (B) Solution B, (C) Same". Furthermore, in the unsolved questions shown in A302, the first keyword is "solution concentration", which reflects relevant information about the concentration of the solution.

[0088] Therefore, after obtaining the images to be processed shown in A301 (i.e., images a1 and a2) and the questions to be answered shown in A302 (i.e., "Which solution has a higher concentration? (A) Solution A, (B) Solution B, (C) Same"), these two are used as the aforementioned Figure 3 The target language model is used as input to determine the target decision information.

[0089] It should be noted that the above Figure 4 The image and question shown are merely illustrative. In practical applications, the image and question can be other images and questions. Furthermore, the method of acquiring the image and question is not specifically limited; for example, they can be acquired synchronously or according to preset acquisition rules such as sequential time order.

[0090] 203. Based on a multimodal large language model, the target decision information and the image to be processed are processed to generate supplementary information of the first keyword. The supplementary information of the first keyword is used to indicate the auxiliary parsing of the first keyword.

[0091] In one or more embodiments, as described above Figure 3 As shown, after obtaining the target decision information through step 202, the image to be processed and the target decision information are combined as input to the multimodal large language model. This allows the multimodal large language model to be used as an expert module to process the target decision information and the image to be processed, thereby generating supplementary information for the first keyword. The supplementary information for the first keyword, as described above, is understood to indicate the auxiliary parsing of the first keyword.

[0092] It should be noted that the multimodal large language model mentioned here can be understood by referring to the content described in step 202 above, and will not be repeated here.

[0093] 204. Based on the target large language model, the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered are processed to determine the target question answering result. The target question answering result includes the target answer to the question to be answered and the specific analysis information of the target answer.

[0094] In one or more embodiments, as described above Figure 3As shown, after obtaining the target decision information and the supplementary information of the first keyword, it is also necessary to combine the image to be processed and the question to be answered to determine the target question-answering result. That is, by using the target decision information, the supplementary information of the first keyword, and the image to be processed as the model input of the target large language model, the target answer to the question to be answered and the specific parsing information of the target answer are derived through the target large language model, i.e., the target question-answering result.

[0095] For example, in the future Figure 4 Taking the example shown, the resulting question-and-answer result includes two parts: the target answer and the specific explanation information. For example, the target answer is: same; the specific explanation information is: the two solutions have the same volume and the same number of particles, therefore the two solutions have the same concentration.

[0096] Optionally, in one or more embodiments, a first prompt message may be obtained before performing step 204. This first prompt message indicates the first output format corresponding to the target question-and-answer result. After obtaining the first prompt message, in the process of determining the target question-and-answer result using the target large language model based on the target decision information, supplementary information of the first keyword, the image to be processed, and the question to be answered, the target decision information, supplementary information of the first keyword, the image to be processed, and the question to be answered are specifically processed according to the first prompt message using the target large language model to determine the target question-and-answer result.

[0097] For example, the first prompt above could be: "You are a knowledgeable and skilled information integration science expert. Please think through and answer the question step by step based on the given unanswered question and supplementary information. Please note that we not only need the answer, but more importantly, we need the reasoning behind the answer (i.e., specific analytical information). Please combine your knowledge and supplementary information to arrive at your reasoning and answer. Please prioritize using your knowledge to answer the question. If you cannot answer, please maintain critical thinking and select effective information to help you choose the most correct option as the answer. Furthermore, please do not rely solely on supplementary information, as the provided supplementary information may not always be effective."

[0098] It should be noted that in practical applications, the first prompt message and subsequent second prompt messages can also be other information that can provide a prompting function, and this application does not make specific limitations.

[0099] Optionally, in one or more embodiments, the question-and-answer processing method may further include: obtaining second prompt information, the second prompt information being used to indicate a second output format corresponding to the target question-and-answer result, wherein the first output format is different from the second output format. Thus, after obtaining the second prompt information, in the process of determining the target question-and-answer result by processing the target decision information, supplementary information of the first keyword, the image to be processed, and the question to be answered using the target large language model according to the first prompt information, the method further specifically processes the target decision information, supplementary information of the first keyword, the image to be processed, and the question to be answered based on the target large language model according to the second prompt information to determine the target question-and-answer result.

[0100] Optionally, in one or more embodiments, the question-and-answer processing method may further include updating the first prompt information to derive the target question-and-answer result using the updated first prompt information. Specifically, the question-and-answer processing method further includes obtaining a question-and-answer service request, which includes a field identifier of a prompt field in the first prompt information and a target field value of the prompt field; after obtaining the question-and-answer service request, the current field value identified by the field identifier is adjusted to the target field value of the corresponding prompt field to obtain the updated first prompt information. Thus, in the process of processing the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered according to the first prompt information to determine the target question-and-answer result, specifically, the updated first prompt information can be used to process the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered based on the target large language model to determine the target question-and-answer result.

[0101] Providing first, second, or updated prompts in the aforementioned manner enables the target language model to perform answer reasoning according to the output format required by the prompts. This not only improves reasoning efficiency but also reduces output errors, enhancing readability and the maintainability of the reasoning logic. Furthermore, the flexible use of these prompts allows the target language model to adapt to different question-and-answer requirements without necessitating model training or large-scale updates and modifications, significantly improving the flexibility and scalability of the reasoning process.

[0102] In some optional examples, in the foregoing Figure 2 Based on the described embodiments, the specific implementation process for generating target decision information in step 202 above can be referred to below. Figure 5 The flowchart shown is used for understanding.

[0103] like Figure 5 As shown, the method includes steps S2021 to S2023, as detailed below:

[0104] S2021. Extract the first keyword from the unanswered question.

[0105] In one or more embodiments, after obtaining the question to be answered, a first keyword can be extracted from the question. The first keyword mentioned herein can be referred to as described above. Figure 2 The content described in step 201 is for your understanding and will not be repeated here. For example, as mentioned above... Figure 4 As shown, the first keyword A1 can include "the concentration of the solution".

[0106] S2022. Input the image to be processed and the first keyword into the target large language model to obtain target principle information and target task information.

[0107] In one or more embodiments, after the first keyword is extracted, the first keyword and the image to be processed are used together as input to the target large language model, so that the target large language model processes the image to be processed and the first keyword, thereby obtaining target principle information and target task information. The target principle information mentioned in this application includes not only the basic principle of the first keyword, but also the extraction principle of each second keyword.

[0108] The fundamental principle of the first keyword can be understood as its definition, concept, and other underlying principles. Furthermore, this fundamental principle may be derived through the definitions of different basic keywords (i.e., the second keywords of this application) and the relationships between them. In other words, the first keyword is derived from multiple second keywords and the relationships between them. Additionally, the extraction principle for each second keyword can be used to indicate the feature extraction method for that second keyword. Moreover, the target task information mentioned in this application can be used to indicate the task allocation for each second keyword, and the task allocation is related to the feature extraction method for the corresponding second keyword.

[0109] For example, as mentioned above Figure 4 Taking the image to be processed A301 and the question to be answered A302 shown as examples, Figure 4 It also shows the corresponding target decision information. For example... Figure 4 As shown, the target decision information includes at least the target basic principle a401 and the target task information b402. The target basic principle a401 includes the basic principle of the first keyword A1, and the extraction principles of the second keyword B1 and the second keyword B2. The target task information b402 includes the task allocation information of the second keyword B1 and the task allocation information of the second keyword B2.

[0110] For example, as mentioned above Figure 4 Taking the first keyword A1, "concentration of solution," as an example, its basic principle is that "the concentration in a solution refers to the amount of substance (solute) present in one solvent and another substance in a specific solution; therefore, concentration = amount of solute / volume of solution." Thus, it can be determined that the concentration of a solution is derived from both the amount of solute and the volume of the solution. In this case, "amount of solute" and "volume of solution" can be considered as the second keyword.

[0111] Taking "amount of solute" as the second keyword B1 and "solution volume" as the second keyword B2 as examples, the extraction principle of the second keyword B1 is "to understand the number of particles in the solution, an object quantitative locator is needed for detection," and the extraction principle of the second keyword B2 is "to know the solution volume, a text intelligent extractor is needed to extract information about the solvent volume from the image." In other words, the number of particles in the solution can be detected by an object quantitative locator, and the text intelligent extractor can extract text information related to "solution volume" from the image to be processed. That is to say, the extraction principle of the second keyword B1 can be understood as the feature extraction method of the second keyword B1, that is, extraction through an object quantitative locator; and the extraction principle of the second keyword B2 can be understood as the feature extraction method of the second keyword B2, that is, extraction through a text intelligent extractor.

[0112] In addition, the task allocation information of the second keyword B1 can be understood as assigning the subtask of extracting the "amount of solute" to the object quantitative locator; the task allocation information of the second keyword B2 can be understood as assigning the other subtask of extracting the "solution volume" to the text intelligent extractor.

[0113] In some optional examples, during the generation of decision information, prompts can be provided to the target large language model, such as "provide a basic principle for this problem, explaining how you would mine image information and use expert modules to form a comprehensive answer." This prompts the target large language model to comprehensively evaluate and modularly analyze the problem to be solved, generating more relevant target decision information. This approach can improve reasoning efficiency.

[0114] In some optional examples, during task allocation, prompts can be provided to the target large language model, such as "Assign specific tasks to each module as needed based on its capabilities to collect additional information required for accurate question answers." This prompts the target large language model to select necessary expert modules and assign appropriate tasks to the corresponding expert modules. This approach can improve inference efficiency.

[0115] S2023. Generate target decision information based on target principle information and target task information.

[0116] In one or more embodiments, after determining the target principle information and the target task information, target decision information can be generated based on the target principle information and the target task information, as detailed in the following examples. Figure 4 To understand the content.

[0117] In some optional examples, in the foregoing Figure 2 and Figure 5 Based on the described embodiments, the specific implementation process for generating supplementary information for the first keyword in step 203 can be referred to below. Figure 6 The method flowchart shown is used for understanding. Figure 6 As shown, the method includes steps S2031 to S2034, as detailed below:

[0118] S2031. Analyze the lexical meaning of each second keyword to determine the feature extraction type of the corresponding second keyword.

[0119] In one or more embodiments, from the foregoing Figure 5 As shown in step S2022, the first keyword is derived from multiple second keywords and the relationships between them. Therefore, in determining the supplementary information of the first keyword, this supplementary information can be broken down into components constructed from the supplementary information of multiple second keywords. Based on this, the lexical meaning of each second keyword can be analyzed to determine the feature extraction type of the corresponding second keyword.

[0120] For example, as mentioned above Figure 4 Taking the example of the second keyword B1 being "amount of solute" and the second keyword B2 being "solution volume", by analyzing the meaning of the words "amount of solute" and "solution volume", we can know that the feature extraction type of the second keyword B1 is quantity extraction and the feature extraction type of the second keyword B2 is text extraction.

[0121] S2032. Based on the feature extraction type of each second keyword, select the target expert module that matches the feature extraction type of the corresponding second keyword from the candidate expert modules. The multimodal large language model includes candidate expert modules.

[0122] In one or more embodiments, the multimodal large language model includes different candidate expert modules, such as, but not limited to, a text intelligent extractor, an object quantitative locator, a visual intelligent analyzer, and a chart-aware parser. The described text intelligent extractor can extract text content from an image to be processed and perform targeted extraction of the required text content as needed. The object quantitative locator is responsible for identifying and locating various objects in the image to be processed, supporting functions such as quantity comparison and spatial relationship recognition. The visual intelligent analyzer is designed to process and parse visual data, capable of answering any questions related to the image content of the image to be processed, and supports image analysis functions. The chart-aware parser focuses on the analysis and interpretation of chart information, supporting functions such as extracting data points and identifying trends, and supporting the search for important components in the chart such as titles, axes, labels, and legends. As an illustrative description, the candidate expert modules can also be updated in this application, for example, by adding modules supporting different functions such as a "latest data searcher" and a "document comprehension" to enrich the multimodal large language model and facilitate its application in different question-and-answer scenarios.

[0123] After determining the feature extraction type for each second keyword, the feature extraction type of each keyword is matched with the feature extraction functions supported by the candidate expert modules. Then, a target expert module that matches the feature extraction type of the corresponding second keyword is selected from the candidate expert modules. For example, using the aforementioned... Figure 4 Taking the example of second keyword B1 being "amount of solute" and second keyword B2 being "solution volume", by analyzing the meaning of the second keyword B1 and second keyword B2 and by matching, it can be determined that the target expert module corresponding to second keyword B1 is "object quantitative locator" and the target expert module corresponding to second keyword B2 is "text intelligent extractor".

[0124] S2033. Input the target principle information, target task information, and the image to be processed into the target expert module corresponding to each second keyword to obtain supplementary information for each second keyword.

[0125] In one or more embodiments, after determining the target expert module corresponding to each second keyword, the target principle information, target task information, and the image to be processed are input into the target expert module corresponding to each second keyword to obtain supplementary information for each second keyword. As an illustrative description, specifically, the target principle information, the task allocation information for each second keyword, and the image to be processed can be input into the target expert module corresponding to the corresponding second keyword, and the corresponding target expert module can output the supplementary information for the corresponding second keyword. For example, as described above... Figure 4For example, the target principle information, the task allocation information of the second keyword B1, and the image to be processed can be input into the target expert module (e.g., "object quantitative locator") corresponding to the second keyword B1 to determine the supplementary information of the second keyword B1, such as "the number of particles in solution A is the same as the number of particles in solution B". Similarly, the target principle information, the task allocation information of the second keyword B2, and the image to be processed can be input into the target expert module (e.g., "text intelligent extractor") corresponding to the second keyword B2 to determine the supplementary information of the second keyword B2, such as "solution A: 40ml, solution B: 40ml".

[0126] S2034. Based on the supplementary information of multiple second keywords, generate supplementary information for the first keyword.

[0127] In one or more embodiments, after obtaining supplementary information for multiple second keywords, supplementary information for the first keyword is generated based on this supplementary information. As an illustrative description, the supplementary information for the first keyword in this application may include more advanced information such as positional relationships and quantitative relationships between objects to be processed.

[0128] For example, as mentioned above Figure 4 Taking the first keyword A1 as "concentration of solution" as an example, its supplementary information can include "the number of particles in solution A is the same as the number of particles in solution B" and "solution A: 40ml, solution B: 40ml".

[0129] By using the above method, multimodal large language models can be used to call different expert modules to complete the various tasks assigned in the decision-making stage. This allows for the acquisition of higher-level cognitive information (i.e., the aforementioned supplementary information). Combining this supplementary information with reasoning can greatly improve the model's reasoning ability, enhance the performance of visual reasoning tasks, and form a better thought chain.

[0130] In this embodiment, the decision-making process of the target large language model considers not only the textual description content of the text dimension but also the image to be processed in other visual dimensions. Furthermore, this application does not rely on external tools but directly uses a multimodal large language model to determine supplementary information of keywords in the question to be answered, thus fully combining the textual description content, the image to be processed, the target decision information, and the supplementary information to comprehensively analyze the question. This not only enables the model to promote visual reasoning based on correct decision-making but also allows for direct use of the multimodal large language model to handle reasoning in complex scenarios such as long texts. This makes the reasoning results less prone to deviation while improving the reasoning ability and generalization ability of visual reasoning, and possesses strong versatility and convenience.

[0131] Experiments show that, compared with existing methods that utilize multimodal thinking chains to process visual reasoning tasks in question-answering scenarios, the question-answering processing method provided in this application can effectively improve the reasoning ability of large models and enhance the accuracy of reasoning results on the same question-answering dataset.

[0132] For example, Table 1 compares the performance of the proposed solutions and this application on the ScienceQA question-answering dataset, and Table 2 compares the performance of the proposed solutions and this application on the MathVista visual mathematical reasoning dataset. As shown in Tables 1 and 2:

[0133] Table 1

[0134] Model IMG NAT SOC LAN txt NO G1-6 G7-12 Avg Related solution (GPT-3.5) 67.92 78.82 70.98 83.18 77.37 86.13 80.72 74.03 78.31 Ours (GPT-3.5) 77.54 80.37 85.49 84.00 77.27 86.83 85.61 76.60 82.39 Ours (Gemini) 82.40 84.24 87.85 84.09 82.11 86.97 88.18 79.17 84.96

[0135] As shown in Table 1, the multimodal large language model in this application, taking Gemini as an example and the single-modal large language model, taking GPT-3.5 as an example, achieves a Cantor accuracy of 82.39% by using the Gemini model as a decision generator and the GPT-3.5 model as an expert module to determine the target question-answering result for the question to be answered. This is a 4.08% improvement in visual reasoning ability compared to the related scheme that only uses the GPT-3.5 model.

[0136] Table 2

[0137] Model FQA GPS MWP TQA VQA ALG ARI GEO LOG NUM SCI STA ALL Related solutions 37.1 29.3 38.1 57.5 36.3 36.0 35.7 31.4 24.3 25.7 50.0 41.9 38.8 This application 50.2 39.4 39.8 49.4 43.8 42.0 41.5 41.4 10.8 30.8 46.7 59.5 44.7

[0138] As shown in Table 2, in this application, the multimodal large language model, taking Gemini as an example, and the single-modal large language model, taking GPT-3.5 as an example, in the visual mathematical reasoning dataset MathVista, by using the Gemini model as a decision generator and combining the GPT-3.5 model as an expert module to determine the target question answering result, the Cantor accuracy is improved by 5.9% compared with the related schemes that only use Gemini.

[0139] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. It is understood that to achieve the above functions, corresponding hardware structures and / or software modules are included to execute each function. Those skilled in the art should readily recognize that, based on the modules and algorithm steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0140] This application embodiment can divide the device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0141] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0142] The question-and-answer processing device in the embodiments of this application will be described in detail below. Figure 7 This is a schematic diagram of one embodiment of the question-and-answer processing device provided in this application. Figure 7 As shown, the question-and-answer processing device includes an acquisition unit 701, a processing unit 702, and a determination unit 703. Among them,

[0143] The acquisition unit 701 is used to acquire an image to be processed and a question to be answered. The image to be processed includes an object to be processed and text descriptions of the object to be processed. The question to be answered is a question set for the object to be processed.

[0144] The processing unit 702 is used to process the image to be processed and the question to be answered based on the target large language model, and generate target decision information. The target decision information is used to indicate the parsing situation related to the first keyword in the question to be answered.

[0145] The processing unit 702 is used to process the target decision information and the image to be processed based on a multimodal large language model, and generate supplementary information of the first keyword. The supplementary information of the first keyword is used to indicate the auxiliary parsing of the first keyword.

[0146] The determination unit 703 is used to process the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered based on the target large language model, and determine the target question answering result. The target question answering result includes the target answer to the question to be answered and the specific analysis information of the target answer.

[0147] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing unit 702 is specifically used for:

[0148] Extract the primary keyword from the unanswered question;

[0149] The image to be processed and the first keyword are input into the target large language model to obtain target principle information and target task information. The target principle information includes the basic principle of the first keyword and the extraction principle of each second keyword. The extraction principle of each second keyword is used to indicate the feature extraction method of the corresponding second keyword. The first keyword is obtained from multiple second keywords and the correlation between multiple second keywords. The target task information is used to indicate the task allocation of each second keyword. The task allocation is related to the feature extraction method of the corresponding second keyword.

[0150] Based on the target principle information and target task information, target decision information is generated.

[0151] In another possible design, in another implementation of another aspect of the embodiments of this application, the processing unit 702 is specifically used for:

[0152] Analyze the lexical meaning of each second keyword to determine the feature extraction type for the corresponding second keyword;

[0153] Based on the feature extraction type of each second keyword, a target expert module that matches the feature extraction type of the corresponding second keyword is selected from the candidate expert modules. The multimodal large language model includes candidate expert modules.

[0154] The target principle information, target task information, and the image to be processed are input into the target expert module corresponding to each second keyword to obtain supplementary information for each second keyword.

[0155] Based on the supplementary information of multiple second keywords, supplementary information of the first keyword is generated.

[0156] In another possible design, in another implementation of another aspect of the embodiments of this application, the processing unit 702 is specifically used for:

[0157] In cases where the target large language model includes a multimodal large language model, the image to be processed and the question to be answered are input into the multimodal large language model to generate target decision information.

[0158] In another possible design, in another implementation of another aspect of the embodiments of this application, the processing unit 702 is specifically used for:

[0159] When the target large language model includes a single-modal large language model, extract image description information from the image to be processed;

[0160] Image description information and the question to be answered are input into a single-modal large language model to generate target decision information.

[0161] In another possible design, in another implementation of another aspect of the embodiments of this application,

[0162] The acquisition unit 701 is further configured to: before processing the target decision information, the supplementary information of the first keyword, the image to be processed and the question to be answered based on the target large language model, and before determining the target question answering result, acquire the first prompt information, which is used to indicate the first output format corresponding to the target question answering result;

[0163] The determination unit 703 is specifically used to process the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered according to the first prompt information, and to determine the target question and answer result.

[0164] In another possible design, in another implementation of another aspect of the embodiments of this application,

[0165] The acquisition unit 701 is further configured to: acquire second prompt information, the second prompt information being used to indicate the second output format corresponding to the target question-and-answer result, the first output format being different from the second output format;

[0166] The determination unit 703 is specifically used to process the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered based on the target large language model according to the second prompt information, and to determine the target question and answer result.

[0167] In another possible design, in another implementation of another aspect of the embodiments of this application,

[0168] The acquisition unit 701 is further used to: acquire a question-and-answer service request, which includes the field identifier of the prompt field in the first prompt information and the target field value of the prompt field;

[0169] The processing unit 702 is further configured to adjust the current field value identified by the field identifier to the target field value of the corresponding prompt field, so as to obtain the updated first prompt information;

[0170] The determination unit 703 is specifically used to process the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered based on the target large language model according to the updated first prompt information, and to determine the target question answering result.

[0171] Through the above methods, this application considers not only the textual description content in the target large language model's decision-making but also the image to be processed in other visual dimensions. Furthermore, this application does not rely on external tools but directly uses the multimodal large language model through processing unit 702 to determine supplementary information of keywords in the question to be answered. This allows determination unit 703 to fully combine the textual description content, the image to be processed, the target decision information, and the supplementary information to comprehensively analyze the question to be answered. This not only enables the model to promote visual reasoning based on correct decision-making but also allows it to directly utilize the multimodal large language model to handle reasoning in complex scenarios such as long texts. This makes the reasoning results less prone to bias while improving the reasoning ability and generalization ability of visual reasoning.

[0172] The question-and-answer processing device in the embodiments of this application has been described above from the perspective of modular functional entities. The question-and-answer processing device in the embodiments of this application will now be described below from the perspective of hardware processing. Figure 8 This is a schematic diagram of the question-and-answer processing device provided in an embodiment of this application. The question-and-answer processing device 300 can vary considerably due to differences in configuration or performance, and may include, but is not limited to, the aforementioned... Figure 7 The question-and-answer processing device described in the text, etc.

[0173] like Figure 8 As shown, the question-and-answer processing device 300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and a memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. The memory 332 and storage media 330 may be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the classification processing device. Furthermore, the CPU 322 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the classification processing device 300. Exemplarily, the CPU 322 is used to execute the application program 342 stored in the storage media 330, thereby implementing the question-and-answer processing method provided in the above embodiments of this application.

[0174] The question-and-answer processing device 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0175] For example, Figure 8 The central processing unit 322 can invoke computer execution instructions stored in memory 332 to cause the question-and-answer processing device to perform actions such as... Figures 2 to 6 The method in the corresponding method embodiment.

[0176] Specifically, Figure 7 The functions / implementation processes of the processing unit 702 and the determining unit 703 can be achieved through... Figure 8 The central processing unit 322 in the memory calls computer execution instructions stored in the memory 332 to achieve this. Figure 7 The function / implementation process of the acquisition unit 701 can be achieved through... Figure 8 The input / output interface 358 is used to implement this.

[0177] The steps performed by the question-and-answer processing device in the above embodiments can be based on this. Figure 8 The structure of the question-and-answer processing device is shown.

[0178] This application also provides a computer-readable storage medium storing a computer program or instructions thereon, which, when executed by a processor, implements the steps of the methods described in the foregoing embodiments.

[0179] This application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the methods described in the foregoing embodiments.

[0180] In the above embodiments, the implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0182] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0183] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0184] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0185] A computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, they generate, in whole or in part, the processes or functions according to embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., SSDs), etc.

[0186] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A question-and-answer processing method, characterized in that, include: Obtain an image to be processed and a question to be answered. The image to be processed includes an object to be processed and text descriptions of the object to be processed. The question to be answered is a question set for the object to be processed. The image to be processed and the question to be answered are processed based on the target large language model to generate target decision information. The target decision information is used to indicate the parsing situation related to the first keyword in the question to be answered. The first keyword is used to describe the key information of the object to be processed. The target decision information and the image to be processed are processed based on a multimodal large language model to generate supplementary information of the first keyword. The supplementary information of the first keyword is used to indicate the auxiliary parsing of the first keyword. Based on the target large language model, the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered are processed to determine the target question-answering result. The target question-answering result includes the target answer to the question to be answered and the specific analysis information of the target answer.

2. The method according to claim 1, characterized in that, The image to be processed and the question to be answered are processed based on the target large language model to generate target decision information, including: Extract the first keyword from the question to be answered; The image to be processed and the first keyword are input into the target large language model to obtain target principle information and target task information. The target principle information includes the basic principle of the first keyword and the extraction principle of each second keyword. The extraction principle of each second keyword is used to indicate the feature extraction method of the corresponding second keyword. The first keyword is obtained from multiple second keywords and the correlation between multiple second keywords. The target task information is used to indicate the task allocation of each second keyword. The task allocation is related to the feature extraction method of the corresponding second keyword. Based on the target principle information and the target task information, target decision information is generated.

3. The method according to claim 2, characterized in that, The target decision information and the image to be processed are processed based on a multimodal large language model to generate supplementary information for the first keyword, including: The lexical meaning of each second keyword is analyzed to determine the feature extraction type corresponding to the second keyword; Based on the feature extraction type of each second keyword, a target expert module that matches the feature extraction type of the corresponding second keyword is selected from the candidate expert modules, and the multimodal large language model includes the candidate expert modules; The target principle information, the target task information, and the image to be processed are input into the target expert module corresponding to each second keyword to obtain supplementary information for each second keyword; Supplementary information for the first keyword is generated based on supplementary information for multiple second keywords.

4. The method according to any one of claims 1 to 3, characterized in that, The image to be processed and the question to be answered are processed based on the target large language model to generate target decision information, including: In the case where the target large language model includes the multimodal large language model, the image to be processed and the question to be answered are input into the multimodal large language model to generate target decision information.

5. The method according to any one of claims 1 to 3, characterized in that, The image to be processed and the question to be answered are processed based on the target large language model to generate target decision information, including: In the case where the target large language model includes a single-modal large language model, image description information is extracted from the image to be processed; The image description information and the question to be answered are input into the single-modal large language model to generate target decision information.

6. The method according to any one of claims 1 to 5, characterized in that, Before processing the target decision information, supplementary information of the first keyword, the image to be processed, and the question to be answered based on the target large language model to determine the target question-answering result, the method further includes: Obtain first prompt information, which is used to indicate the first output format corresponding to the target question-and-answer result; Based on the target large language model, the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered are processed to determine the target question-answering result, including: Based on the first prompt information, the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered are processed according to the target large language model to determine the target question answering result.

7. The method according to claim 6, characterized in that, The method further includes: Obtain a second prompt message, which indicates the second output format corresponding to the target question-and-answer result, wherein the first output format is different from the second output format; Based on the first prompt information, the target decision information, supplementary information of the first keyword, the image to be processed, and the question to be answered are processed according to the target large language model to determine the target question-answering result, including: According to the second prompt information, the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered are processed based on the target large language model to determine the target question answering result.

8. The method according to claim 6, characterized in that, The method further includes: Obtain a question-and-answer service request, wherein the question-and-answer service request includes the field identifier of the prompt field in the first prompt information and the target field value of the prompt field; The current field value identified by the field identifier is adjusted to the target field value corresponding to the prompt field to obtain the updated first prompt information; Based on the first prompt information, the target decision information, supplementary information of the first keyword, the image to be processed, and the question to be answered are processed according to the target large language model to determine the target question-answering result, including: Based on the updated first prompt information, the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered are processed according to the target large language model to determine the target question answering result.

9. A question-and-answer processing device, characterized in that, include: An acquisition unit is used to acquire an image to be processed and a question to be answered. The image to be processed includes an object to be processed and text description content for describing the object to be processed. The question to be answered is a question set for the object to be processed. The processing unit is used to process the image to be processed and the question to be answered based on the target large language model, and generate target decision information, which is used to indicate the parsing status of the first keyword in the question to be answered; The processing unit is used to process the target decision information and the image to be processed based on a multimodal large language model, and generate supplementary information of the first keyword. The supplementary information of the first keyword is used to indicate the auxiliary parsing status of the first keyword. The determining unit is used to process the target decision information, the supplementary information of the first keyword, the image to be processed, and the question to be answered based on the target large language model, and to determine the target question-answering result. The target question-answering result includes the target answer to the question to be answered and the specific parsing information of the target answer.

10. A question-and-answer processing device, characterized in that, include: Input / output interface, processor, and memory, wherein the memory stores program instructions; The processor is configured to execute program instructions stored in the memory to perform the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 8.

Images