Answer generation method, device and system, electronic equipment and nonvolatile storage medium

Abstract

The invention discloses an answer generation method, device and system, electronic equipment and a nonvolatile storage medium, and is applied to the technical field of artificial intelligence. The method comprises the following steps: inputting a multi-modal question of a logical reasoning task to be processed and a task cue word into an answer generation model, and extracting question characteristics of the multi-modal question by utilizing a visual language processing layer; splitting the question features into a plurality of to-be-processed questions by using an answering sub-model, for each to-be-processed question, calculating a single-step answer corresponding to the current to-be-processed question based on the task cue word, and identifying the accuracy of the single-step answer; and when determining that the current correct single-step answer is the final answer of the logical reasoning task to be processed, outputting all the correct single-step answers as the answers of the logical reasoning task to be processed. According to the method and the device, the problem that the execution result of the mathematical logic class task cannot meet the actual demand of a user in the related technology can be solved, the mathematical logic class answering task can be accurately executed, and the answering process with high logicality is output.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an answer generation method, device, system, electronic device and non-volatile storage medium. Background Art

[0002] NLP (Natural Language Processing) can help computers understand and process human language, enabling computers to read and understand natural language. With the rapid development of artificial intelligence technology, the accuracy and explainability requirements for processing language tasks are becoming increasingly higher.

[0003] The comprehension and generation capabilities of language models in related technologies have been greatly improved. However, their mathematical and logical reasoning capabilities are relatively poor. Not only do they lack detailed step-by-step solution processes for tasks that require step-by-step reasoning and explanation, but the accuracy of the answers they provide is also relatively low, which cannot meet user needs.

[0004] In view of this, accurately executing mathematical logic-solving tasks and outputting a highly logical solution process is a technical problem that technical personnel in this field need to solve.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present invention provides an answer generation method, device, system, electronic device and non-volatile storage medium, which can accurately execute mathematical logic solution tasks and output a solution process with strong logic and strong interpretability.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] One aspect of the present invention provides an answer generation method, comprising:

[0009] Obtain a multimodal problem and task prompt words of a pending logical reasoning task; the multimodal problem is a mathematical problem described in visual form and / or language form; input the multimodal problem and task prompt words into a pre-trained answer generation model; the answer generation model is a solution sub-model including a visual language processing layer; the solution sub-model is a pre-trained language model; the visual language processing layer extracts the problem features of the multimodal problem; the solution sub-model is used to split the problem features into a plurality of pending problems, and for each pending problem, a single-step answer corresponding to the current pending problem is calculated based on the task prompt words, and the accuracy of the single-step answer is identified; when it is determined that the current correct single-step answer is the final answer to the pending logical reasoning task, all correct single-step answers are output as the answer to the pending logical reasoning task.

[0010] In a first exemplary embodiment, the multimodal question and task prompt word are input into a pre-trained answer generation model, including: obtaining a pre-trained single-step reasoning sub-model, a step verification sub-model and an answer output sub-model; the single-step reasoning sub-model, the step verification sub-model and the answer output sub-model are language models with a visual language processing layer; obtaining a logical reasoning sample set, an answer supervision sample set and an answer recognition sample set; each logical reasoning sample of the logical reasoning data sample set is a mathematical problem with a label that is parsed step by step to obtain a correct answer, and the answer supervision sample set is a mathematical problem with a label that is parsed step by step to obtain a correct answer. Each answer supervision sample in the set is a math problem-answer pair, and has a correct answer or an incorrect answer label, and each answer recognition sample in the answer recognition sample set is a math problem-answer pair, and has a parsing completed label or a parsing incomplete label; the single-step reasoning sub-model is fine-tuned based on the logical reasoning data sample set, the step verification sub-model is fine-tuned based on the answer supervision sample set, and the answer output sub-model is fine-tuned based on the answer recognition sample set; according to the fine-tuned single-step reasoning sub-model, step verification sub-model and answer output sub-model, a trained answer generation model is obtained.

[0011] In a second exemplary embodiment, the task prompt words include single-step reasoning prompt words, step verification prompt words and answer output prompt words; after the multimodal problem and the task prompt words are input into a pre-trained answer generation model, it also includes: using the single-step reasoning sub-model to split the problem features corresponding to the multimodal problem into multiple problems to be processed, parsing the current problem to be processed according to the single-step reasoning prompt words, and outputting a single-step answer to the step verification sub-model; using the step verification sub-model to identify the accuracy of the single-step answer based on the step verification prompt words and the multimodal problem, and output an answer accuracy identification label; using the answer output sub-model to identify whether the multimodal problem has been parsed based on the answer output prompt words, the multimodal problem and the current correct single-step answer, and when it is determined that the multimodal problem has been parsed, all single-step answers identified as correct by the step verification sub-model are output as answers to the pending logical reasoning task.

[0012] In a third exemplary embodiment, each answer supervision sample in the answer supervision sample set also includes an incorrect answer label and an incorrect reason label; when the step verification sub-model identifies the single-step answer as an incorrect answer based on the step verification prompt word and the multimodal question, the incorrect answer label and the corresponding incorrect reason label are output to the single-step reasoning sub-model, so that the single-step reasoning sub-model can parse the current problem to be processed again based on the incorrect reason label and the step verification prompt word.

[0013] In a fourth exemplary embodiment, the inputting of the multimodal question and the task prompt word into a pre-trained answer generation model includes: generating a numerical operation data set by performing numerical operations on a mathematical relationship including at least one numerical operator to obtain a corresponding numerical calculation result; obtaining a language sample data set matching the logical reasoning task to be processed, wherein each language sample data of the language sample data set includes at least a mathematical problem, a correct answer corresponding to the mathematical problem, and an answer parsing process; performing data enhancement processing on the language sample data set from a mathematical logic perspective to obtain a logic enhanced sample set; performing a first stage fine-tuning processing on the answer generation model using the logic enhanced sample set to obtain an initial answer generation model; performing a second stage fine-tuning processing on the initial answer generation model using the numerical operation data set to obtain an optimized answer generation model; performing a third stage fine-tuning processing on the optimized answer generation model using the language sample data set to obtain an answer generation model.

[0014] In a fifth exemplary implementation, the data enhancement processing is performed on the language sample data set from a mathematical logic perspective to obtain a logic enhanced sample set, including: for at least one target language sample data of the language sample data set, restating the original question of the target language sample data to obtain a new question without changing the original meaning; restating the original answer parsing process of the target language sample data to obtain a first type of new answer parsing process without changing the original meaning; resolving the original question and / or the new question to obtain a candidate answer parsing process, and using the candidate answer parsing process that has been verified for correctness as the second type of new answer parsing process; recombining the original question, the original answer parsing process, the rewritten question and / or the rewritten answer parsing process to generate multiple new language sample data. The invention relates to a method for generating a mathematical logic reasoning model according to the present invention. The method comprises the following steps: for at least one target language sample data of the language sample data set, taking the target known condition of the original question of the target language sample data as the content to be sought, and rewriting the original question based on the content to be sought, so that the content to be sought is a reverse new answer, and the original correct answer is a known condition in the reverse new question; obtaining a trained first mathematical logic reasoning model, wherein the first mathematical logic reasoning model is used to output a corresponding answer and a parsing process according to an input question; inputting the reverse new question into the first mathematical logic reasoning model to obtain the reverse new answer and a candidate reverse parsing process; if the reverse new answer is the same as the target known condition, taking the candidate reverse parsing process as the reverse parsing process; and generating new language sample data according to the reverse new answer, the reverse parsing process and the reverse new question.

[0015] Another aspect of the present invention provides an answer generating device, comprising:

[0016] The problem acquisition module is used to acquire multimodal problems and task prompt words of the logical reasoning task to be processed; the multimodal problem is a mathematical problem described in visual form and / or language form.

[0017] The answer generation module is used to input the multimodal question and task prompt words into a pre-trained answer generation model; the answer generation model is a solution sub-model including a visual language processing layer; the solution sub-model is a pre-trained language model; the visual language processing layer extracts the problem features of the multimodal question; the problem features are split into multiple pending questions using the solution sub-model, for each pending question, the single-step answer corresponding to the current pending question is calculated based on the task prompt words, and the accuracy of the single-step answer is identified; when it is determined that the current correct single-step answer is the final answer to the pending logical reasoning task, all correct single-step answers are output as the answer to the pending logical reasoning task.

[0018] The present invention also provides an electronic device, comprising a processor, wherein the processor is configured to implement the steps of the answer generation method as described in any of the preceding items when executing a computer program stored in a memory.

[0019] The present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the answer generation method as described in any of the preceding items are implemented.

[0020] Finally, the present invention also provides an answer generation system, including a user interface and an answer task processor;

[0021] The user interface at least includes a task triggering area, a task processing result display area, a data input area and / or a file importing area, so that the user can describe the logic reasoning task to be processed through the data input area, input the multimodal question and the task prompt word through the data input area or the file importing area, and send the multimodal question and the task prompt word to the task answering processor through the task triggering area;

[0022] The answer task processor, based on the multimodal question and the task prompt word, parses the multimodal question using the steps of the answer generation method as described in any of the previous items, generates a correct answer and an answer parsing process, and sends the correct answer and the answer parsing process to the task processing result display area of ​​the user interface.

[0023] The advantage of the technical solution provided by the present invention is that the answer generation model uses the visual language processing layer to extract features from the visual and language perspectives for the problems to be processed in the logical reasoning task, which can more comprehensively understand the diverse content contained in the multimodal problem and improve the accuracy of the analysis of the question. Then the answer sub-model is used to generate the answer step by step. After each step of generating the answer, its correctness needs to be judged to ensure the accuracy and logic of the generated answer. If the final answer is reached, all the step-by-step processes are output together with the answer. It is necessary to not only provide the correct answer, but also to gradually generate a logical problem-solving process according to the problem-solving steps, thereby meeting the user's needs for a detailed step-by-step solution process and step-by-step reasoning and explanation.

[0024] In addition, the present invention also provides corresponding implementation devices, systems, electronic devices and non-volatile storage media for the answer generation method, which further makes the method more practical, and the devices, systems, electronic devices and non-volatile storage media have corresponding advantages.

[0025] The technical features mentioned above, the technical features to be mentioned below, and the technical features shown separately in the drawings can be arbitrarily combined with each other, as long as the combined technical features are not contradictory. All feasible feature combinations are technical contents clearly recorded in this article. Any of the multiple sub-features contained in the same sentence can be applied independently, and does not have to be applied together with other sub-features. It should be understood that the above general description and the detailed description below are only exemplary and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of a flow chart of an answer generation method provided by the present invention;

[0028] Figure 2 A schematic flow chart of another answer generation method provided by the present invention;

[0029] Figure 3 A schematic diagram of a flow chart of an answer generation method provided by the present invention in an exemplary application scenario;

[0030] Figure 4 A structural diagram of an exemplary embodiment of an answer generating device provided by the present invention;

[0031] Figure 5 A structural diagram of an exemplary embodiment of an electronic device provided by the present invention;

[0032] Figure 6 A structural diagram of an exemplary implementation of the answer generation system provided by the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Among them, the terms "first", "second", "third", "fourth", etc. in the specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations of the two are intended to cover non-exclusive inclusions. The term "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described here as "exemplary" is not necessarily interpreted as being superior or better than other embodiments.

[0034] With the development of natural language processing technology, more and more users use artificial intelligence systems to solve problems encountered in daily life and work, such as the automatic answering system that is widely used to assist students in learning. Traditional automatic answering systems rely on regular matching technology or statistical models, such as learning machines and learning software. This type of method will build a question bank, match questions in the question bank according to the input pictures or questions, and find the corresponding answers to the questions. This method cannot effectively understand and process mathematical reasoning problems, and cannot meet the needs of step-by-step reasoning, such as elementary school math questions. The final generated answers are not intuitive enough, cannot cope with a variety of question types, and are not interpretable. In order to solve the shortcomings of traditional automatic answering systems, related technologies are based on text language modules to perform answering tasks. However, this method only supports the input of pure text questions, cannot solve problems with attached pictures, lacks the ability to recognize images, tables or handwritten questions, resulting in poor processing capabilities for multimodal information in the input questions, and cannot fully understand the multimodal information in the questions, affecting the completeness and accuracy of the answers. Similarly, the related technology directly inputs the answer to the question, and does not simultaneously provide step-by-step solution steps, which cannot meet the needs of decomposing and explaining the problem-solving steps. Furthermore, the related technology lacks an effective quality control mechanism, resulting in low accuracy and rationality of the generated answers and poor user experience.

[0035] In view of this, the present invention uses the visual language processing layer to extract features from the visual and language perspectives for the problems to be processed in the logical reasoning tasks through the answer generation model, which can more comprehensively understand the diverse content contained in the multimodal problems and improve the accuracy of the analysis of the questions. Then the answer sub-model is used to generate the answer step by step. After each step of generating the answer, its correctness needs to be judged to ensure the accuracy and logic of the generated answer. If the final answer is reached, all the step-by-step processes are output together with the answer. After introducing the technical scheme of the present invention, various non-limiting embodiments of the present invention are described in detail below. In order to better illustrate the present invention, numerous specific details are given in the specific embodiments below. Those skilled in the art should understand that the present invention can also be implemented without these specific details. In other examples, the methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present invention.

[0036] First see Figure 1 , Figure 1 A flowchart of an answer generation method provided in this embodiment may include the following contents:

[0037] S101 obtains a multimodal question and task prompt words of a logic reasoning task to be processed.

[0038] In this step, the logical reasoning task to be processed is a related question-and-answer task that requires mathematical logical reasoning, such as an elementary school math problem solving task, or an application problem solving task. Multimodal problems are math problems described in visual form and / or language form. The problems in this embodiment support any form of expression, such as text form, image form, or voice form. The question stem and the question picture may appear at the same time. In other words, text, tables, graphics, and even handwritten content may exist in multimodal problems at the same time. Task prompts are prompts used to show the answer generation model how to perform tasks, and can be flexibly set according to actual application scenarios.

[0039] S102: Input the multimodal question and task prompt words into a pre-trained answer generation model.

[0040] In this embodiment, in order to support the input of multimodal questions, the answer generation model is a solution sub-model including a visual language processing layer; the visual language processing layer extracts the problem features of the multimodal question, which includes at least an image encoder and a text encoder, an image and text fusion layer, and a language model generation layer. The image and text encoder is responsible for generating embeddings, the image and text fusion layer performs fusion, and finally generates the final visual perception text through the language model generation layer. By combining visual and language information through the visual language processing layer, the system can more comprehensively understand the diverse content contained in the question, such as the question picture or complex typesetting format. Of course, if voice input is supported, a network model for voice-to-text or voice-to-image conversion needs to be added after the input layer of the answer generation model. The answer sub-model is a pre-trained language model, such as LLaVA-OneVision (open source multimodal model), InternVL2 (open source multimodal large language model), Qwen2VL (open source visual reasoning model), which does not affect the implementation of the present invention.

[0041] S103: Use the answer sub-model to split the problem features into multiple pending problems. For each pending problem, calculate the single-step answer corresponding to the current pending problem based on the task prompt word, and identify the accuracy of the single-step answer; when it is determined that the current correct single-step answer is the final answer to the pending logical reasoning task, output all correct single-step answers as the answers to the pending logical reasoning task.

[0042] In order to realize the step-by-step decomposition and calculation of the input logical reasoning tasks, the present invention needs to first divide the multimodal problem into multiple problems according to the problem-solving steps, and calculate the corresponding results for each problem in turn. In order to ensure the accuracy of the final output answer, an answer is generated for each problem to be processed. For the convenience of description, it is defined as a single-step answer. The answer needs to be verified for accuracy. If the answer is inaccurate, it needs to be regenerated. If the answer is accurate, it is determined whether it is the final answer, that is, whether the task has been solved. If not, the next problem to be processed is obtained for the same processing. If completed, all single-step answers are output. In this way, the correct answer corresponding to the logical reasoning task to be processed and the step-by-step analysis process of the answer are obtained.

[0043] In the technical solution provided in this embodiment, the answer generation model uses the visual language processing layer to extract features from the visual and language perspectives for the problems to be processed in the logical reasoning task, which can more comprehensively understand the diverse content contained in the multimodal problem and improve the accuracy of the analysis of the question. Then the answer sub-model is used to generate the answer step by step. After each step of generating the answer, its correctness needs to be judged to ensure the accuracy and logic of the generated answer. If the final answer is reached, all the step-by-step processes are output together with the answer. It is necessary to not only provide the correct answer, but also to gradually generate a logical problem-solving process according to the problem-solving steps, so as to meet the user's needs for a detailed step-by-step solution process and step-by-step reasoning and explanation.

[0044] In the above embodiment, there is no limitation on how to train the answer generation model. Based on the above embodiment, the present invention also provides an exemplary training method for the answer generation model, which may include the following contents:

[0045] Obtain a pre-trained single-step reasoning sub-model, a step verification sub-model and an answer output sub-model; obtain a logic reasoning sample set, an answer supervision sample set and an answer recognition sample set; each logic reasoning sample in the logic reasoning data sample set is a math problem with a label of a correct answer obtained by parsing according to the steps, each answer supervision sample in the answer supervision sample set is a math problem-answer pair with a correct answer or an incorrect answer label, and each answer recognition sample in the answer recognition sample set is a math problem-answer pair with a parsing completion label or an parsing incomplete label; fine-tune the single-step reasoning sub-model based on the logic reasoning data sample set, fine-tune the step verification sub-model based on the answer supervision sample set, and fine-tune the answer output sub-model based on the answer recognition sample set; obtain a trained answer generation model based on each fine-tuned single-step reasoning sub-model, step verification sub-model and answer output sub-model. In order to ensure that the task is executed efficiently, each answer supervision sample in the above-mentioned answer supervision sample set also includes an incorrect answer label and an error reason label; when the step verification sub-model identifies a single-step answer as an incorrect answer based on the step verification prompt words and the multimodal question, it outputs the incorrect answer label and the corresponding error reason label to the single-step reasoning sub-model, so that the single-step reasoning sub-model can parse and process the current problem to be processed again based on the error reason label and the step verification prompt words.

[0046] Among them, the single-step reasoning sub-model, the step verification sub-model and the answer output sub-model are language models with a visual language processing layer; the single-step reasoning sub-model, the step verification sub-model and the answer output sub-model are all multimodal models, and the currently open-source pre-trained models, such as LLaVA-OneVision, InternVL2, and Qwen2VL, can be used to initialize the parameters, and then the corresponding fine-tuning can be performed to meet the corresponding task requirements. The single-step reasoning sub-model is used for the reasoning generation of the single-step solution steps of mathematical problems. The corresponding data can be used to fine-tune the single-step reasoning sub-model. The true value label of the data is the correct single-step answer to the mathematical problem. For the convenience of description, the fine-tuning sample set used is defined as the logical reasoning sample set. In other words, each sample of the logical reasoning sample set is at least a question-answer pair. For complex problems, especially complex mathematical problems, it is difficult for the answer generation model to directly give the correct answer. In order to enable the answer generation model to handle complex tasks and improve the interpretability of the answer generation model reasoning, the language sample data includes the answer parsing process in addition to the question and the corresponding correct answer, that is, the question-answer pair also includes the parsing process from the question to the correct answer. In this way, the answer generation model trained with the language sample data set can explicitly output the intermediate step-by-step reasoning steps before outputting the final answer, which is conducive to improving the arithmetic, common sense and reasoning ability of the answer generation model, thereby improving its performance on complex reasoning tasks. The step verification sub-model judges the correctness of the single-step problem-solving steps generated by the single-step reasoning sub-model. Therefore, it is necessary to use the corresponding data to fine-tune the step verification sub-model. The true value label of the data is the correctness judgment of the single-step answer, that is, "correct" or "wrong and the reason for the error". For the convenience of description, the fine-tuning sample set used is defined as the answer supervision sample set and the answer recognition sample set. The answer output sub-model judges whether the problem is solved based on the current problem-solving step. Therefore, it is necessary to use the corresponding data to fine-tune the answer output sub-model. The true value label of the data is "the problem is solved" or "the problem is not solved". For the convenience of description, the fine-tuning sample set used is defined as the answer recognition sample set.

[0047] The process of executing the pending logical reasoning task based on the answer generation model obtained through the above training may include:

[0048] The answer generation model of this embodiment is completed by the collaborative work of three sub-models, so the task prompt words include single-step reasoning prompt words, step verification prompt words, and answer output prompt words. The single-step reasoning sub-model is used to split the problem features corresponding to the multimodal problem into multiple problems to be processed, and the current problem to be processed is parsed according to the single-step reasoning prompt words, and the single-step answer is output to the step verification sub-model; the step verification sub-model is used to identify the accuracy of the single-step answer based on the step verification prompt words and the multimodal problem, and the answer accuracy identification label is output; the answer output sub-model is used to identify whether the multimodal problem has been parsed based on the answer output prompt words, the multimodal problem and the current correct single-step answer, and when it is determined that the multimodal problem has been parsed, all single-step answers identified as correct by the step verification sub-model are output as the answers to the pending logical reasoning tasks. Figure 2 As shown, for example, the multimodal problem is a picture of a primary school question. The picture of the primary school question is input into the single-step reasoning sub-model, and the single-step reasoning sub-model gives the solution steps of the current step. Then the current step solution steps and the picture of the primary school question are sent to the step verification sub-model. The step verification sub-model determines whether the current step is correct. If not, the error reason is given. Then the single-step reasoning sub-model regenerates the solution steps based on the picture of the primary school question according to the error reason, and then the step verification sub-model determines whether the current step is correct again. If not, it is regenerated again until the step verification sub-model determines that the current step is correct, then the current step is added to the step list, and finally the step list and the picture of the primary school question are input into the answer output sub-model together. The answer output sub-model determines whether the picture of the primary school question has been parsed. If not, the single-step reasoning sub-model continues to generate the next step of the solution. If completed, the step list is output as the correct answer to the picture of the primary school question and the step-by-step solution steps.

[0049] In order to make the implementation process of the technical solution of the present invention more clear to those skilled in the art, the present invention also provides an illustrative example, such as Figure 3 As shown, Figure 3 11 represents a single-step reasoning sub-model, 12 represents a single-step reasoning sub-model, and 13 represents an answer output sub-model, which may include the following contents:

[0050] Get the single-step reasoning prompt words, which may be, for example: "Please generate the next problem-solving step based on the existing problem-solving steps. Existing steps: {None}", input the question picture and the single-step reasoning prompt words into the single-step reasoning sub-model, and the single-step reasoning sub-model outputs: S1 "The plane in the picture is 16 yuan, the clock is 9 yuan, and the thermometer is 7 yuan". Get the step verification prompt words, which may be, for example, "Please determine whether the current problem-solving step is correct based on the existing problem-solving steps". Insert the current step S1 obtained by the single-step reasoning sub-model into the step verification prompt words, and obtain "Please determine whether the current problem-solving step is correct based on the existing problem-solving steps. Existing steps: {None}. Current problem-solving step: {The plane in the picture is 16 yuan, the clock is 9 yuan, and the thermometer is 7 yuan}", and input it together with the question picture into the step verification sub-model. The step verification sub-model determines that this step is correct and adds this step to the list of existing steps. Get the answer output prompt, such as "Please judge whether the final answer is obtained based on the existing problem-solving steps", and insert the current step list content to get "Please judge whether the final answer is obtained based on the existing problem-solving steps. Existing steps: {(1) The airplane in the picture is 16 yuan, the clock is 9 yuan, and the thermometer is 7 yuan}", and input it and the question picture into the answer output sub-model. The answer output sub-model determines that the problem-solving is not complete. Therefore, the single-step reasoning sub-model will continue to reason about the next step of the problem-solving step. Input the question picture and the single-step reasoning prompt to the single-step reasoning sub-model. The single-step reasoning prompt is: "Please generate the next step of the problem-solving step based on the existing problem-solving steps. Existing steps: {(1) The airplane in the picture is 16 yuan, the clock is 9 yuan, and the thermometer is 7 yuan}", and the output of the single-step reasoning sub-model is: S2 "The cost of the clock is more expensive than the thermometer by 9 yuan - 7 yuan = 1 yuan". The picture of the question and the step verification prompt are input into the step verification sub-model. The step verification prompt is: "Based on the existing problem-solving steps, please determine whether the current problem-solving step is correct. Existing steps: {(1) The airplane in the picture costs 16 yuan, the clock costs 9 yuan, and the thermometer costs 7 yuan}. Current problem-solving step: {The clock is more expensive than the thermometer by 9 yuan - 7 yuan = 1 yuan}". The step verification sub-model determines that this step is wrong. The error is caused by a numerical calculation error. Therefore, the current step is regenerated by the single-step reasoning sub-model. The picture of the question and the single-step reasoning prompt are input into the single-step reasoning sub-model. The single-step reasoning prompt is: "Based on the existing problem-solving steps, please generate the next problem-solving step. Existing steps: {(1) The airplane in the picture costs 16 yuan, the clock costs 9 yuan, and the thermometer costs 7 yuan}". The output of the single-step reasoning sub-model is: New S2 "The clock is more expensive than the thermometer by 9 yuan - 7 yuan = 2 yuan". The question picture and step verification prompts are input into the step verification sub-model. The step verification prompts may be: "Please judge whether the current problem-solving steps are correct based on the existing problem-solving steps. Existing steps: {(1) The airplane in the picture costs 16 yuan, the clock costs 9 yuan, and the thermometer costs 7 yuan}. Current problem-solving steps: {The clock costs more than the thermometer by 9 yuan - 7 yuan = 2 yuan}".The step verification sub-model determines that this step is correct and adds this step to the existing step list. The answer output prompt word, the question picture and the existing step list are input to the answer output sub-model, which is: "Please judge whether the final answer is obtained based on the existing problem-solving steps. Existing steps: {(1) The airplane in the picture costs 16 yuan, the clock costs 9 yuan, and the thermometer costs 7 yuan; (2) The clock is more expensive than the thermometer by 9 yuan - 7 yuan = 2 yuan}". The answer output sub-model determines that the problem is solved and outputs the existing step list as the final result.

[0051] As can be seen from the above, this embodiment can combine visual and language information, and significantly improve the depth and accuracy of understanding of the content of the question through multimodal input; the step-checking sub-model performs process supervision and reward feedback mechanism for each problem-solving step, so that the answer generation model can generate answers step by step; the answer generated at each step is judged by the step-checking sub-model, and only the steps that meet the logic and accuracy will enter the next step of reasoning, thereby ensuring the rationality and accuracy of the final output answer. The three models are used for collaborative interaction to achieve efficient and high-accuracy generation of correct answers and step-by-step problem-solving ideas without relying on established rules and question banks.

[0052] It should be noted that there is no strict order of execution between the steps in the present invention. As long as they comply with the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 1-Figure 3 This is just a schematic and does not mean that this is the only execution order.

[0053] In order to further improve the performance of the answer generation model, based on the above embodiment, the present invention also provides another model training method, which may include the following contents:

[0054] A numerical operation data set is generated by performing numerical operations on mathematical relations containing at least one numerical operator to obtain corresponding numerical calculation results; a language sample data set matching the logical reasoning task to be processed is obtained, and data enhancement processing is performed on the language sample data set from the perspective of mathematical logic to obtain a logic enhancement sample set; the answer generation model is fine-tuned in the first stage using the logic enhancement sample set to obtain an initial answer generation model; the initial answer generation model is fine-tuned in the second stage using the numerical operation data set to obtain an optimized answer generation model; the optimized answer generation model is fine-tuned in the third stage using the language sample data set to obtain an answer generation model.

[0055] In this step, the numerical operation data set can improve the accuracy of the mathematical operations of the answer generation model. The mathematical operations include but are not limited to basic addition, subtraction, multiplication and division operations and complex mixed operations. The mathematical relations may include unary operation relations, binary operation relations and mixed operation relations. The numerical operators in the binary operation relations include but are not limited to addition, subtraction, multiplication, division, exponentiation, logarithm, and remainder. The numerical operators in the unary operation relations include but are not limited to negation, sine, cosine, tangent, secant, cosecant, and factorial. The numerical operators in the mixed operation relations include but are not limited to the numerical operators in the binary operation relations and the numerical operators in the unary operation relations. For each mathematical relation, a value is randomly assigned to the mathematical relation, and the mathematical relation is calculated according to the operation law, rule and calculation priority of the corresponding operator. The mathematical relation with the assignment, the answer and the calculation relationship are taken as a numerical operation data. By increasing the category of the mathematical relation and changing the assignment of the same mathematical relation, multiple numerical operation data can be obtained. These multiple numerical operation data are constructed into a numerical operation data set. The numerical operation data set is used as the training sample data of the answer generation model, so that the answer generation model can learn the mapping relationship from the input question to the output answer, and improve the mathematical reasoning ability of the answer generation model. The language sample data set is an existing training data set for training the answer generation model, or a fine-tuning data set used by the pre-trained answer generation model in the fine-tuning stage. Each language sample data of the language sample data set at least includes a mathematical problem, the correct answer corresponding to the mathematical problem and the answer analysis process; the mathematical logic perspective refers to expanding the language sample data of the language sample data set from the perspective of improving the logical reasoning ability of the data, so as to enrich the data volume and data richness of the language sample data set.

[0056] The numerical operation dataset, logic enhancement sample set and language sample dataset have their own unique characteristics, and each of them plays a different role in the answer generation model. The language sample dataset directly faces the direct needs of the answer generation model, and its data accuracy is the highest, which is the highest quality training sample data; the role of the numerical operation dataset is to improve the correctness of the answer generation model for basic addition, subtraction, multiplication and division operations and complex mixed operations, and is used to ensure the correct calculation of the answer generation model. It does not directly meet the needs of the answer generation model. The logic enhancement sample set can improve the richness of the training samples, but the data quality is somewhat different from the language sample dataset. The richness of the language sample logic enhancement data has been greatly improved, but the quality may not be as good as the language sample dataset, but the data scale is much larger than the language sample dataset, so the language sample logic enhancement data can be used to fine-tune the answer generation model in the first stage. Considering the impact of data scale on natural model training, some high-quality data can be obtained from the language sample logic enhancement data and added to the language sample dataset to participate in the fine-tuning of the answer generation model in the third stage. In order to improve the accuracy of the answer generation model for basic addition, subtraction, multiplication and division operations and complex mixed operations, considering that the numerical calculation data does not directly meet the needs of the answer generation model, but is to ensure the correctness of the calculation, this embodiment can use the numerical calculation data in the second stage of fine-tuning of the answer generation model.

[0057] From the above, it can be seen that this embodiment can maximize the reasoning effect that can be stimulated by data logic data through three-stage training, and effectively improve the mathematical reasoning ability of the answer generation model.

[0058] The above embodiment does not limit how to perform data enhancement processing on a language sample data set from a mathematical logic perspective to obtain a logic enhanced sample set. Based on the above embodiment, the present invention also provides a logic enhanced sample set generation process:

[0059] For at least one target language sample data of the language sample data set, the original question of the target language sample data is restated to obtain a new question without changing the original meaning; the original answer parsing process of the target language sample data is restated to obtain a first type of new answer parsing process without changing the original meaning; the original question and / or the new question are re-solved to obtain a candidate answer parsing process, and the candidate answer parsing process that has been verified for correctness is used as a second type of new answer parsing process; the original question, the original answer parsing process, the rewritten question and / or the rewritten answer parsing process are recombined to generate multiple new language sample data. In actual application, open source deep learning models with language style conversion capabilities can be obtained to restate the original question and answer parsing process, such as Qwen (open source Tongyi Qianwen large model), Baichuan (open source Baichuan large model), and Chatglm (the name of the open source dialogue pre-training model). For the sake of convenience, the deep learning model with language style conversion capabilities is defined as a language style conversion model. The language style conversion model is used to convert the input original natural language into a natural language of the target style; the original question is input into the language style conversion model to obtain the corresponding new question; the original answer parsing process is input into the language style conversion model to obtain the corresponding first-class new answer parsing process.

[0060] For at least one target language sample data of the language sample data set, the target known condition of the original question of the target language sample data is used as the content to be sought, and the original question is rewritten based on the content to be sought, so that the content to be sought is the reverse new answer, and the original correct answer is the known condition in the reverse new question. For example, a known number N can be randomly selected from the original question and replaced with an unknown number x. N is the correct answer corresponding to the reverse question, and the correct answer to the original question is M. The original question can be rewritten by adding a description at the end of the original question. If the answer to the question is M, what is the value of the unknown number x? Obtain a trained first mathematical logic reasoning model, which is used to output the corresponding answer and parsing process according to the input question; input the reverse new question into the first mathematical logic reasoning model to obtain the reverse new answer and the candidate reverse parsing process; if the reverse new answer is the same as the target known condition, the candidate reverse parsing process is used as the reverse parsing process; according to the reverse new answer, the reverse parsing process and the reverse new question, generate new language sample data. After the problem is determined, an open source deep learning model with mathematical logic reasoning capabilities, such as Qwen, Baichuan, and Chatglm, can be obtained to solve the new reverse problem. For the sake of description, the deep learning model with mathematical logic reasoning capabilities is defined as the first mathematical logic reasoning model, and the trained first mathematical logic reasoning model is obtained. The first mathematical logic reasoning model is used to output the corresponding answer and analysis process according to the input problem; the new reverse problem is input into the first mathematical logic reasoning model, and the first mathematical logic reasoning model solves the new reverse problem and generates a detailed analysis, thereby obtaining a new reverse answer and a candidate reverse analysis process; the solution process of the first mathematical logic reasoning model is correctness-checked. If the new reverse answer is the same as the target known condition, the candidate reverse analysis process is used as the reverse analysis process. If the new reverse answer is not the same as the target known condition, the new reverse answer and the candidate reverse analysis process are discarded.

[0061] From the above, it can be seen that, under the premise of ensuring data accuracy, the present invention, based on the original training sample data set, obtains a logic-enhanced sample set from the question and answer parsing process through restatement and regeneration as well as enhanced processing from the perspective of reverse data enhancement. The logic-enhanced sample set is used to greatly expand the data volume and richness of high-quality training samples of mathematical logic of the answer generation model, which is beneficial to improving the execution efficiency and accuracy of the logical reasoning tasks to be processed.

[0062] The above embodiment does not limit how to construct a numerical operation data set. Based on the above embodiment, the present invention also provides an exemplary generation process of a numerical operation data set:

[0063] By changing the numerical value and operator of the target numerical operation relation, multiple original question data are generated; for the mixed operation in each original question data, the operator with the highest current priority is calculated from left to right for each operation, and the calculation results of each original question data are gradually obtained; the original question data is taken as the question, the calculation result of the original question data is taken as the correct answer, and the calculation process of the original question data is taken as the answer parsing process, as a group of numerical operation data; based on each group of numerical operation data, a numerical operation data set is constructed.

[0064] In this embodiment, the target numerical operation expression is any acquired or existing numerical operation relational expression, which may be a unary operation relational expression, a binary operation relational expression or a mixed operation relational expression. The value is any number in the target numerical operation expression. The operation of changing the operator and the value includes modification, addition or deletion. There are many target numerical operation relations. The original problem data is the operation relational expression obtained after changing the value and / or operator of the target data operation expression. As an efficient and simple method for generating original question data, multiple binary operation relational expressions can be obtained, such as A+B, AB, A+BC, A*B+C, A*(B+C), and the letters of at least one binary operation relational expression are replaced with random values, which can be random integers, decimals, or fractions, which do not affect the implementation of the present invention, to generate binary operation data; a unary operator is randomly set on at least one value of each binary operation data, that is, a unary operator is added before the integer, decimal, or fraction of the binary operation data to obtain multiple original question data. For example, the binary operation data is A+BC, and a unary operator sin is added before B, and the generated original question data is A+sin(B)-C; the binary operation data is A*B+C, and a unary operator is added after C, and the generated original question data is A*B+C! . More conveniently, a binary operation template can be pre-constructed, and the binary operation relational expression can be directly obtained from the binary operation template. In order to further improve the efficiency of constructing numerical calculation data sets, technicians in this field can also expand and process current open source mathematical logic databases, such as bella (database name), gsm8k (database name), ape (database name), and blossom (database name), according to the above method.

[0065] After the original problem data is generated, according to the operation laws, rules and calculation priorities of the operators, detailed solution steps need to be listed. The generation of detailed solution steps can be achieved by combining the infix-postfix expression conversion method. Each calculation only calculates the operation relationship with the highest priority in the current step, and gradually advances until the calculation obtains the final result. For unary operations, the results can be calculated directly. For binary operations, the calculation results of each original problem data are gradually obtained by calculating the operator with the highest priority from left to right in each operation. For example, the calculation process of the calculation result is: 5×7+3+9 / 3+2.5**(3-1)=35+3+9 / 3+2.5**(3-1)=38+9 / 3+2.5**(3-1)=38+3+2.5**(3-1)=41+2.5**(3-1)=41+2.5**2=41+6.25=47.25.

[0066] From the above, it can be seen that this embodiment constructs the original question data first, and then implements detailed solution steps by converting the original question data into infix-postfix expressions to construct a numerical operation data set, so that the answer generation model learns the mapping relationship from input questions to output answers, thereby having the ability to perform numerical calculation tasks, improving the correctness of the answer generation model for basic addition, subtraction, multiplication and division operations and complex mixed operations, and is conducive to improving the accuracy of more complex logical reasoning.

[0067] The present invention also provides a corresponding device for the answer generation method, which further makes the method more practical. Among them, the device can be described from the perspective of functional modules and hardware. The answer generation device provided by the present invention is introduced below. The device is used to implement the answer generation method provided by the present invention. In this embodiment, the answer generation device may include or be divided into one or more program modules, and the one or more program modules are stored in a storage medium and executed by one or more processors to complete the answer generation method disclosed in Example 1. The program module referred to in this embodiment refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the execution process of the answer generation device in the storage medium than the program itself. The following description will specifically introduce the functions of each program module of this embodiment. The answer generation device described below and the answer generation method described above can be referenced to each other.

[0068] From the perspective of functional modules, see Figure 4 , Figure 4 The structure diagram of the answer generation device provided in this embodiment in a specific implementation manner, the device may include:

[0069] The question acquisition module 401 is used to acquire the multimodal question and task prompt words of the logical reasoning task to be processed.

[0070] The answer generation module 402 is used to input the multimodal question and the task prompt words into a pre-trained answer generation model; the answer generation model is a solution sub-model including a visual language processing layer; the solution sub-model is a pre-trained language model; the visual language processing layer extracts the problem features of the multimodal question; the solution sub-model is used to split the problem features into multiple pending problems, and for each pending problem, the single-step answer corresponding to the current pending problem is calculated based on the task prompt words, and the accuracy of the single-step answer is identified; when it is determined that the current correct single-step answer is the final answer to the pending logical reasoning task, all correct single-step answers are output as the answers to the pending logical reasoning task.

[0071] Exemplarily, in some implementations of the present embodiment, the above-mentioned device may also include a model training module, which is used to obtain a pre-trained single-step reasoning sub-model, a step verification sub-model and an answer output sub-model; the single-step reasoning sub-model, the step verification sub-model and the answer output sub-model are language models with a visual language processing layer; a logical reasoning sample set, an answer supervision sample set and an answer recognition sample set are obtained; each logical reasoning sample of the logical reasoning data sample set is a mathematical problem with a label that the correct answer is obtained by step-by-step analysis, each answer supervision sample of the answer supervision sample set is a mathematical problem-answer pair, and has a correct answer or an incorrect answer label, and each answer recognition sample of the answer recognition sample set is a mathematical problem-answer pair, and has a parsing completion label or an parsing incomplete label; the single-step reasoning sub-model is fine-tuned based on the logical reasoning data sample set, the step verification sub-model is fine-tuned based on the answer supervision sample set, and the answer output sub-model is fine-tuned based on the answer recognition sample set; according to each fine-tuned single-step reasoning sub-model, step verification sub-model and answer output sub-model, a trained answer generation model is obtained.

[0072] As an exemplary implementation method of the above embodiment, the above answer generation module 402 can also be used to: use the single-step reasoning sub-model to split the problem features corresponding to the multimodal problem into multiple problems to be processed, parse the current problem to be processed according to the single-step reasoning prompt words, and output the single-step answer to the step verification sub-model; use the step verification sub-model to identify the accuracy of the single-step answer based on the step verification prompt words and the multimodal problem, and output the answer accuracy identification label; use the answer output sub-model to identify whether the multimodal problem has been parsed based on the answer output prompt words, the multimodal problem and the current correct single-step answer, and when it is determined that the multimodal problem has been parsed, output all single-step answers identified as correct by the step verification sub-model as the answers to the logical reasoning task to be processed.

[0073] As an exemplary implementation method of the above embodiment, the above answer generation module 402 can also be used for: when a single-step answer is identified as an incorrect answer based on the step verification prompt words and the multimodal question, an incorrect answer label and a corresponding error cause label are output to the single-step reasoning sub-model, so that the single-step reasoning sub-model can parse and process the current problem to be processed again based on the error cause label and the step verification prompt words.

[0074] Exemplarily, in some other implementations of the present embodiment, the above-mentioned device may also include a model training module, which may also be used to: generate a numerical operation data set by performing numerical operations on mathematical relations containing at least one numerical operator to obtain corresponding numerical calculation results; obtain a language sample data set that matches the logical reasoning task to be processed, each language sample data of the language sample data set at least including a mathematical problem, a correct answer corresponding to the mathematical problem, and an answer analysis process; perform data enhancement processing on the language sample data set from the perspective of mathematical logic to obtain a logic enhanced sample set; use the logic enhanced sample set to perform a first stage fine-tuning processing on the answer generation model to obtain an initial answer generation model; use the numerical operation data set to perform a second stage fine-tuning processing on the initial answer generation model to obtain an optimized answer generation model; use the language sample data set to perform a third stage fine-tuning processing on the optimized answer generation model to obtain an answer generation model.

[0075] As an exemplary implementation of the above embodiment, the above model training model can be further used to: for at least one target language sample data of the language sample data set, restate the original question of the target language sample data to obtain a new question without changing the original meaning; restate the original answer parsing process of the target language sample data to obtain a first type of new answer parsing process without changing the original meaning; re-solve the original question and / or the new question to obtain a candidate answer parsing process, and use the candidate answer parsing process that has been verified for correctness as the second type of new answer parsing process; re-combine the original question, the original answer parsing process, the rewritten question and / or the rewritten answer parsing process to generate multiple new language sample data ; For at least one target language sample data of the language sample data set, the target known conditions of the original problem of the target language sample data are taken as the content to be sought, and the original problem is rewritten based on the content to be sought, so that the content to be sought is the reverse new answer, and the original correct answer is the known condition in the reverse new problem; obtain the trained first mathematical logic reasoning model, and the first mathematical logic reasoning model is used to output the corresponding answer and parsing process according to the input question; input the reverse new problem into the first mathematical logic reasoning model to obtain the reverse new answer and the candidate reverse parsing process; if the reverse new answer is the same as the target known condition, then the candidate reverse parsing process is used as the reverse parsing process; generate new language sample data according to the reverse new answer, the reverse parsing process and the reverse new problem.

[0076] The answer generating device mentioned above is described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, which is described from the perspective of hardware. Figure 5 The following is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention in one implementation manner. Figure 5 As shown, the electronic device includes a memory 50 for storing a computer program; and a processor 51 for implementing the steps of the answer generation method mentioned in any of the above embodiments when executing the computer program.

[0077] Among them, the processor 51 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 51 may also be a controller, a microcontroller, a microprocessor or other data processing chip. The processor 51 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 51 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 51 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 51 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0078] The memory 50 may include one or more computer non-volatile storage media, which may be non-transitory. The memory 50 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the memory 50 may be an internal storage unit of an electronic device, such as a hard disk of a server. In other embodiments, the memory 50 may also be an external storage device of an electronic device, such as a plug-in hard disk equipped on a server, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 50 may also include both an internal storage unit and an external storage device of an electronic device. The memory 50 may not only be used to store application software and various types of data installed in the electronic device, such as: the code of the program in the process of executing the answer generation method, etc., but also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 50 is at least used to store the following computer program 501, wherein, after the computer program is loaded and executed by the processor 51, the relevant steps of the answer generation method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 50 may also include an operating system 502 and data 503, etc., and the storage method may be temporary storage or permanent storage. The operating system 502 may include Windows, Unix, Linux, etc. The data 503 may include but is not limited to data corresponding to the answer generation result, etc.

[0079] In some embodiments, the electronic device may further include a display screen 52, an input / output interface 53, a communication interface 54 or a network interface, a power supply 55 and a communication bus 56. Among them, the display screen 52 and the input / output interface 53, such as a keyboard, belong to the user interface, and exemplary user interfaces may also include standard wired interfaces, wireless interfaces, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The communication interface 54 may exemplarily include a wired interface and / or a wireless interface, such as a WI-FI interface, a Bluetooth interface, etc., which are generally used to establish a communication connection between the electronic device and other electronic devices. The communication bus 56 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0080] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than those shown in the figure, for example, it may also include a sensor 57 for realizing various functions.

[0081] It is understandable that if the answer generation method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes but is not limited to: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes. Based on this, the present invention also provides a non-volatile storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the answer generation method recorded in any of the above embodiments are executed.

[0082] It is understandable that if the answer generation method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, the computer software product does not need to be stored in a physical storage medium. For example, it can be directly transmitted to a computer or other device with information processing capabilities through a wired network or a wireless network to execute all or part of the steps of the methods of each embodiment of the present invention. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product. Based on this, the present invention also provides a computer program product that stores a computer program. When the computer program is executed by a processor, it performs the steps of the answer generation method recorded in any of the above embodiments.

[0083] Finally, the present invention also provides an answer generation system, see Figure 6The answer generation system may include a user interface 601 located at the front end and a task answering processor 602 located at the back end. The user interface 601 includes at least a task triggering area, a task processing result display area, a data input area and / or a file importing area, so that the user can describe the pending logical reasoning task through the data input area, input the multimodal question and the task prompt word through the data input area or the file importing area, and send the pending logical reasoning task, the multimodal question and the task prompt word to the task answering processor through the task triggering area; the task answering processor 602 calculates the multimodal question based on the task prompt word based on the pending logical reasoning task, using the steps of the answer generation method recorded in any of the above embodiments, generates the correct answer and the answer parsing process, and sends the correct answer and the answer parsing process to the task processing result display area of ​​the user interface. The correct answer and the answer parsing process can be directly displayed in text or image, or can be a file, and the identification information for the user to download is displayed at the same time. The file importing area can import the corresponding multimodal question by uploading a file, such as uploading a picture. The data input area may include a text input area and a voice input area, and the user can choose to input the corresponding multimodal question or the task prompt word in text or voice form.

[0084] In this embodiment, the answer generation system can be, for example, a web page or an application package. The user end can pre-deploy the answer generation system and display the user page of the answer generation system with the help of the human-computer interaction page of the user end. The user operates on the user page and the final correct answer and the answer parsing process are fed back on the user page.

[0085] The above is a detailed introduction to an answer generation method, device, electronic device, non-volatile storage medium and computer program product provided by the present invention. The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can refer to each other. Whether the units and algorithm steps of each example described in each disclosed embodiment are executed in electronic hardware or computer software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods for each specific application to implement the described functions, and such implementation should not be considered to exceed the scope of the present invention. Without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for generating an answer, characterized in that: include: Obtain multimodal questions and task prompt words for the logical reasoning task to be processed; The multimodal problem is a mathematical problem described in a visual form and / or a verbal form; Inputting the multimodal question and the task prompt words into a pre-trained answer generation model; the answer generation model is a solution sub-model including a visual language processing layer; the solution sub-model is a pre-trained language model; the visual language processing layer extracts the problem features of the multimodal question; The problem features are split into multiple problems to be processed using the answer sub-model. For each problem to be processed, the single-step answer corresponding to the current problem to be processed is calculated based on the task prompt word, and the accuracy of the single-step answer is identified; when it is determined that the current correct single-step answer is the final answer to the pending logical reasoning task, all correct single-step answers are output as the answer to the pending logical reasoning task.

2. The answer generation method according to claim 1, characterized in that: The step of inputting the multimodal question and the task prompt word into a pre-trained answer generation model comprises: Obtaining a pre-trained single-step reasoning sub-model, a step verification sub-model, and an answer output sub-model; the single-step reasoning sub-model, the step verification sub-model, and the answer output sub-model are language models with a visual language processing layer; Obtain a logic reasoning sample set, an answer supervision sample set, and an answer recognition sample set; each logic reasoning sample of the logic reasoning data sample set is a math problem with a label of a correct answer obtained by step-by-step analysis, each answer supervision sample of the answer supervision sample set is a math problem-answer pair, and has a correct answer or an incorrect answer label, and each answer recognition sample of the answer recognition sample set is a math problem-answer pair, and has a parsing completion label or a parsing incomplete label; Fine-tune the single-step reasoning sub-model based on the logical reasoning data sample set, fine-tune the step verification sub-model based on the answer supervision sample set, and fine-tune the answer output sub-model based on the answer recognition sample set; According to the fine-tuned single-step reasoning sub-model, step verification sub-model and answer output sub-model, a trained answer generation model is obtained.

3. The answer generation method according to claim 2, characterized in that: The task prompt words include single-step reasoning prompt words, step verification prompt words and answer output prompt words; After the multimodal question and the task prompt word are input into the pre-trained answer generation model, the method further includes: using the single-step reasoning sub-model to split the problem features corresponding to the multimodal question into a plurality of problems to be processed, parsing the current problem to be processed according to the single-step reasoning prompt word, and outputting a single-step answer to the step checking sub-model; Using the step verification sub-model to identify the accuracy of the single-step answer based on the step verification prompt word and the multimodal question, and outputting an answer accuracy identification label; The answer output submodel is used to identify whether the multimodal question has been parsed based on the answer output prompt word, the multimodal question and the current correct single-step answer. When it is determined that the multimodal question has been parsed, all single-step answers identified as correct by the step verification submodel are output as answers to the pending logical reasoning task.

4. The answer generation method according to claim 2, characterized in that: Each answer supervision sample in the answer supervision sample set also includes an incorrect answer label and an error reason label; when the step verification sub-model identifies the single-step answer as an incorrect answer based on the step verification prompt word and the multimodal question, it outputs the incorrect answer label and the corresponding error reason label to the single-step reasoning sub-model, so that the single-step reasoning sub-model can parse and process the current problem to be processed again based on the error reason label and the step verification prompt word.

5. The answer generation method according to any one of claims 1 to 4, characterized in that: The step of inputting the multimodal question and the task prompt word into a pre-trained answer generation model comprises: Generate a numerical operation data set by performing numerical operations on a mathematical relational expression containing at least one numerical operator to obtain a corresponding numerical calculation result; Acquire a language sample data set that matches the logic reasoning task to be processed, wherein each language sample data of the language sample data set includes at least a math problem, a correct answer corresponding to the math problem, and an answer parsing process; Performing data enhancement processing on the language sample data set from a mathematical logic perspective to obtain a logic enhanced sample set; Performing a first-stage fine-tuning process on the answer generation model using the logic-enhanced sample set to obtain an initial answer generation model; Using the numerical operation data set to perform a second-stage fine-tuning process on the initial answer generation model to obtain an optimized answer generation model; The optimized answer generation model is fine-tuned in the third stage using the language sample data set to obtain an answer generation model.

6. The answer generation method according to claim 5, characterized in that: The performing data enhancement processing on the language sample data set from the perspective of mathematical logic to obtain a logic enhanced sample set includes: For at least one target language sample data of the language sample data set, restate the original question of the target language sample data to obtain a new question without changing the original meaning; restate the original answer parsing process of the target language sample data to obtain a first type of new answer parsing process without changing the original meaning; re-solve the original question and / or the new question to obtain a candidate answer parsing process, and use the candidate answer parsing process that has been verified for correctness as a second type of new answer parsing process; re-combine the original question, the original answer parsing process, the rewritten question and / or the rewritten answer parsing process to generate multiple new language sample data; For at least one target language sample data of the language sample data set, the target known condition of the original question of the target language sample data is used as the content to be sought, and the original question is rewritten based on the content to be sought, so that the content to be sought is a reverse new answer, and the original correct answer is a known condition in the reverse new question; a trained first mathematical logic reasoning model is obtained, and the first mathematical logic reasoning model is used to output a corresponding answer and a parsing process according to an input question; the reverse new question is input into the first mathematical logic reasoning model to obtain the reverse new answer and a candidate reverse parsing process; if the reverse new answer is the same as the target known condition, the candidate reverse parsing process is used as the reverse parsing process; and new language sample data is generated according to the reverse new answer, the reverse parsing process and the reverse new question.

7. An answer generation device, characterized in that: include: The question acquisition module is used to obtain multimodal questions and task prompt words for the logical reasoning task to be processed; The multimodal problem is a mathematical problem described in a visual form and / or a verbal form; An answer generation module is used to input the multimodal question and the task prompt word into a pre-trained answer generation model; the answer generation model is a solution sub-model including a visual language processing layer; the solution sub-model is a pre-trained language model; the visual language processing layer extracts the problem features of the multimodal question; The problem features are split into multiple problems to be processed using the answer sub-model. For each problem to be processed, the single-step answer corresponding to the current problem to be processed is calculated based on the task prompt word, and the accuracy of the single-step answer is identified; when it is determined that the current correct single-step answer is the final answer to the pending logical reasoning task, all correct single-step answers are output as the answer to the pending logical reasoning task.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is configured to implement the steps of the answer generation method according to any one of claims 1 to 6 when executing a computer program stored in the memory.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the answer generation method according to any one of claims 1 to 6 are implemented.

10. An answer generation system, comprising a user interface and an answer task processor; The user interface at least includes a task triggering area, a task processing result display area, a data input area and / or a file importing area, so that the user can describe the logic reasoning task to be processed through the data input area, input the multimodal question and the task prompt word through the data input area or the file importing area, and send the multimodal question and the task prompt word to the task answering processor through the task triggering area; The answer task processor, based on the multimodal question and the task prompt word, parses the multimodal question using the steps of the answer generation method as described in any one of claims 1 to 6, generates a correct answer and an answer parsing process, and sends the correct answer and the answer parsing process to the task processing result display area of ​​the user interface.

Images