Question and answer processing method and device, equipment and program product

Abstract

The invention provides a question and answer processing method and device, equipment and a program product, and the method comprises the steps: inputting a target question into a question and answer model, so as to enable the question and answer model to generate an answer result corresponding to the target question; wherein in the training process of the question and answer model, a comment model provides comment feedback information corresponding to the reasoning step of the question and answer model for the question and answer model, so that the question and answer model optimizes the reasoning process based on the comment feedback information; the comment model is obtained based on comment data sample training, the comment data sample comprises a reasoning path and a comment sample corresponding to a wrong reasoning step in the reasoning path, and the reasoning path and the comment sample are generated by the question and answer model in the process of answering based on a question sample. The method can improve the correctness and reliability of questions and answers.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a question and answer processing method and device, equipment and program product. BACKGROUND

[0002] With the continuous evolution of large models in tasks such as image-text understanding and visual question answering, they have shown broad application prospects in complex answering tasks in education, scientific research and other scenarios, especially in mathematical reasoning problems. Such tasks not only require the model to have image-text understanding ability, but also to generate a complete structure, logical reasoning process, and explainable reasoning process, and have human-like answering and deductive reasoning ability. However, the existing model still faces many challenges when dealing with such tasks, often appearing phenomena such as reasoning chain skipping, information redundancy or logical illusion, and once an error occurs, the model lacks a self-correction mechanism, making it difficult to effectively back up and repair, affecting the correctness and reliability of the final answering result. SUMMARY

[0003] Based on the above technical problems, the present application provides a question and answer processing method, device, equipment and program product, which can improve the correctness and reliability of the question and answer.

[0004] The first aspect of the present application provides a question and answer processing method, comprising:

[0005] inputting a target question into a question and answer model to make the question and answer model generate an answering result corresponding to the target question;

[0006] wherein, in the training process of the question and answer model, a comment model provides comment feedback information corresponding to the reasoning steps of the question and answer model to the question and answer model, so that the question and answer model optimizes the reasoning process based on the comment feedback information; the comment model is trained based on comment data samples, the comment data samples include a reasoning path and a comment sample corresponding to an error reasoning step in the reasoning path, and the reasoning path and the comment sample are generated by the question and answer model in the process of answering based on a question sample.

[0007] In some implementations, the training process of the question and answer model includes:

[0008] training the comment model according to the comment data samples generated in the process of the question and answer model answering based on a question sample;

[0009] jointly training the trained comment model and the question and answer model.

[0010] In some implementations, training the comment model according to the comment data samples generated in the process of the question and answer model answering based on a question sample includes:

[0011] obtain an inference path sample generated in a process in which the question and answer model answers a question sample, and a comment sample corresponding to the inference path sample; wherein, in the process in which the question and answer model answers the question sample, the inference path is searched in a Monte Carlo tree search manner, and an incorrect inference path is selected from the searched inference path as the inference path sample, and the comment sample corresponding to the inference path sample is generated by comparing the difference between the inference path sample and a correct inference path;

[0012] train the comment model based on the inference path sample generated by the question and answer model and the comment sample corresponding to the inference path sample.

[0013] In some implementations, the training of the comment model based on the inference path sample generated by the question and answer model and the comment sample corresponding to the inference path sample includes:

[0014] cause the question and answer model to correct the inference path sample based on the comment sample, and generate a candidate answer corresponding to the question sample based on the corrected inference path;

[0015] in a case where the candidate answer represents the corrected inference path as being correct, use the inference path sample and the comment sample corresponding to the inference path sample to train the comment model.

[0016] In some implementations, the search process in the Monte Carlo tree search manner includes a selection stage, an expansion stage, a simulation stage, and a backtracking stage, in the simulation stage, the question and answer model generates multiple answers by multiple times of deduction on a candidate step, and determines an evaluation value of the candidate step by comparing the generated answers with a standard answer.

[0017] In some implementations, the joint training of the trained comment model and the question and answer model includes:

[0018] input a question sample into the question and answer model, obtain a candidate inference path generated by the question and answer model, and determine a correctness evaluation value of the candidate inference path;

[0019] input the candidate inference path into the comment model, cause the comment model to generate a comment corresponding to the candidate inference path, and determine a correctness evaluation value of the comment;

[0020] optimize the question and answer model based on the correctness evaluation value of the candidate inference path, the comment, and the correctness evaluation value of the comment, and optimize the comment model based on the comment and the correctness evaluation value of the comment.

[0021] In some implementations, the determining the correctness evaluation value of the comment includes:

[0022] inputting the comment into the question-answering model, so that the question-answering model corrects the candidate reasoning path based on the comment to obtain a corrected reasoning path, and determines a corrected answer corresponding to the corrected reasoning path;

[0023] determining a correctness evaluation value of the corrected reasoning path by comparing the corrected answer with a standard answer;

[0024] determining the correctness evaluation value of the comment based on the correctness evaluation value of the corrected reasoning path.

[0025] In some implementations, the number of candidate reasoning paths is multiple, and the inputting the candidate reasoning path into the comment model so that the comment model generates a comment corresponding to the candidate reasoning path includes:

[0026] selecting a candidate reasoning path from the multiple candidate reasoning paths and inputting the candidate reasoning path into the comment model so that the comment model generates multiple comments corresponding to the candidate reasoning path.

[0027] In some implementations, the optimizing the question-answering model based on the correctness evaluation value of the candidate reasoning path, the comment, and the correctness evaluation value of the comment, and the optimizing the comment model based on the comment and the correctness evaluation value of the comment include:

[0028] determining a standardized advantage value of each comment based on a respective correctness evaluation value of the comment; wherein the standardized advantage value of the comment is used to represent a relative optimization effect of the comment relative to other comments;

[0029] optimizing the question-answering model based on the correctness evaluation value of the candidate reasoning path, each comment, and the respective standardized advantage value of each comment, and optimizing the comment model based on each comment and the standardized advantage value of each comment.

[0030] The second aspect of the present application provides a question-answering processing device, which includes:

[0031] a question-answering processing unit configured to input a target question into a question-answering model, so that the question-answering model generates an answer result corresponding to the target question;

[0032] In the training process of the question and answer model, the comment model provides comment feedback information corresponding to the inference steps of the question and answer model to the question and answer model, so that the question and answer model optimizes the inference process based on the comment feedback information; the comment model is trained based on comment data samples, the comment data samples include an inference path and a comment sample corresponding to an error inference step in the inference path, and the inference path and the comment sample are generated by the question and answer model in the process of answering based on a question sample.

[0033] The third aspect of the present application provides an electronic device, comprising a memory and a processor;

[0034] The memory is connected with the processor, and is used for storing programs;

[0035] The processor is used for realizing the question and answer processing method by running the programs in the memory.

[0036] The fourth aspect of the present application provides a computer program product, comprising computer program instructions, which make the processor realize the question and answer processing method when the processor runs.

[0037] The question and answer processing method provided by the present application provides step-level comment feedback information for the inference path of the question and answer model by the comment model when training the question and answer model, so that the question and answer model obtains step-level comment feedback guidance and assistance in the training process, and the question and answer model can more accurately correct the error inference steps in the inference path and more quickly and efficiently generate the correct inference path and the correct answer.

[0038] Meanwhile, the comment data samples used for training the comment model in the above question and answer model training process are composed of step-level comments generated by the question and answer model in the answering process, these comment data samples can truly reflect the common error types in the actual inference process of the question and answer model, improve the coverage and pertinence of the comment data samples, and at the same time, get rid of the dependence on external large models and artificial annotation, which can significantly improve the construction efficiency and quality of the comment data samples, and further improve the training effect and efficiency of the comment model and the question and answer model, and improve the question and answer performance of the question and answer model. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0040] Figure 1A comment model training process schematic diagram provided by an embodiment of the present application.

[0041] Figure 2 A question and answer model training process schematic diagram provided by an embodiment of the present application.

[0042] Figure 3 A question and answer processing method process schematic diagram provided by an embodiment of the present application.

[0043] Figure 4 A question and answer processing device structure schematic diagram provided by an embodiment of the present application.

[0044] Figure 5 A structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solution of the embodiment of the present application is applicable to the application scenario of the question and answer model performing a question and answer task. By using the technical solution of the embodiment of the present application, the accuracy and reliability of the answer of the question and answer model can be improved. In addition, the training efficiency and training effect of the question and answer model can be improved by using the technical solution of the embodiment of the present application, so as to further improve the performance of the question and answer model.

[0046] With the continuous evolution of large models in tasks such as image-text understanding and visual question answering, they have shown broad application prospects in complex answering tasks in education, scientific research and other scenarios, especially in mathematical reasoning problems. Such tasks not only require the model to have image-text understanding ability, but also need to generate a reasoning process that is complete in structure, logical and interpretable, and has human-like answering and deductive reasoning ability. However, existing models still face many challenges when dealing with such tasks, often appearing phenomena such as reasoning chain skipping, information redundancy or logical illusion, and once an error occurs, the model lacks a self-correction mechanism, making it difficult to effectively back up and repair, affecting the correctness and reliability of the final answering result.

[0047] Currently, in view of the stability and error correction ability of large language models in complex answering tasks such as mathematical reasoning, existing research has attempted to introduce a comment feedback mechanism to guide the model to identify and correct the incorrect reasoning path through external evaluation in natural language form, in order to improve the reasoning accuracy and robustness.

[0048] The scheme of the prior art for guiding the model through the comment feedback mechanism mainly includes two stages of data construction and model training:

[0049] In the data construction stage, the mainstream method usually relies on an external large model to generate comments, mainly including two forms: one is to fake reasoning errors by a large model, and then generate corresponding comments by multiple smaller models, and score and select by a large model; the second is to obtain multiple reasoning paths by high-temperature repeated sampling, and then compare and analyze by a large model to generate corresponding comments.

[0050] In the model training stage, the review model is usually supervised or reinforced trained by the constructed data to improve its ability to generate review feedback, and then the review model is used to guide the question and answer model to perform question and answer training.

[0051] The above-mentioned large model inference optimization method based on review feedback mechanism has certain potential in theory, but there are still many deficiencies in practical application. For example, the review data construction of the above-mentioned scheme depends on external large models, which generates review by calling large models to fake inference errors or generating inference paths by repeated sampling and then comparing them by large models. These methods not only highly depend on external models and have high construction cost, but also lack systematic mining of real intermediate answer steps of the model, making it difficult to cover fine-grained inference defects and limiting the supervision effect of the review model. In addition, the review model training and the answer model training are decoupled in the above-mentioned scheme, and the review model and the answer model are usually optimized separately, lacking of interaction mechanism, resulting in weak linkage between review generation and answer improvement, the feedback effect of the review model cannot be fully utilized, and then the training effect of the answer model is affected.

[0052] In view of the above technical problems, the embodiment of the present application provides a question and answer processing method, which trains the question and answer model based on review feedback through a new training scheme, so as to improve the model performance and improve the question and answer accuracy and reliability.

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0054] The embodiment of the present application first provides a question and answer model training method, which trains the question and answer model based on review feedback by means of a review model, so as to improve the performance of the question and answer model.

[0055] The above-mentioned question and answer model training method can be applied to any device or apparatus with data processing function, such as server, computer, learning machine, intelligent terminal, wearable device, etc. The question and answer model training method can also be applied to a device or apparatus running the question and answer model to realize intelligent question and answer function, such as learning machine, intelligent dialogue interaction device, intelligent robot, etc. The performance of the question and answer model trained by the above-mentioned question and answer model training method is better, and the question and answer function based on the question model is more accurate and efficient.

[0056] In the process of training the question and answer model by the question and answer model training method provided in the embodiment, the comment model provides the question and answer model with comment feedback information corresponding to the inference steps of the question and answer model, so that the question and answer model optimizes the inference steps in the inference process based on the comment feedback information.

[0057] The question and answer model described above refers to a model for reasoning and answering input questions and finally outputting answers. The question and answer model can be a large language model, specifically an arbitrary single-modal large language model or a multi-modal large language model, thereby realizing the answering of single-modal questions or multi-modal questions respectively. The comment model described above refers to a model capable of evaluating the output of the question and answer model and generating comments. In the embodiment, the comment model is a model capable of generating step-level comments for the inference steps of the question and answer model. The model can be a large model or a smaller model.

[0058] In the embodiment, the comment model for assisting in training the question and answer model is trained based on comment data samples, which include an inference path generated by the question and answer model in the process of reasoning and answering based on a question sample, and a comment sample generated for an erroneous inference step in the inference path. That is, in the question and answer model training method provided in the embodiment, the comment model is trained according to the comment data samples generated in the process of answering based on the question sample by the question and answer model.

[0059] In the process, the question and answer model further identifies erroneous inference steps in the inference path generated based on the question sample, and generates corresponding comments for the erroneous inference steps in the inference path to obtain comment samples. The inference path and the corresponding comment samples are used as a set of comment data samples for comment generation training of the comment model. It can be understood that the comment samples are comments generated for specific inference steps in the inference path generated by the question and answer model, and are therefore step-level comment samples. Training the comment model based on the comment data samples described above enables the comment model to generate step-level comments for the inference path.

[0060] In some embodiments, when there are multiple erroneous inference steps in an inference path, the question and answer model can generate a comment sample for each erroneous inference step, and these comment samples are used together as the corresponding comment sample for the inference path.

[0061] Referring to Figure 1 In another embodiment, training of the comment model is implemented by the following steps S101-S102:

[0062] S101, obtain an inference path sample generated in a process in which the question and answer model answers based on a question sample, and a comment sample corresponding to the inference path sample.

[0063] In the process in which the question and answer model answers based on a question sample, an error inference path is selected as an inference path sample from an inference path searched in a Monte Carlo tree search manner, and a comment sample corresponding to the inference path sample is generated by comparing differences between the inference path sample and a correct inference path.

[0064] Specifically, given a multi-modal question input:

[0065] Q={x, I} where x represents a text input designed through a prompt word, and I represents a visual input. The answer model π θ will search a reasoning path in a self-regenerative manner according to a Monte Carlo tree search manner:

[0066] (Q, s1, s2, s3,..., s M ) ~ π θ (·|Q)

[0067] Each reasoning step s i is limited to no more than L tokens and serves as a basic unit of a Monte Carlo search tree. In the Monte Carlo search tree, each node is represented as:

[0068] s i ={P(s i ), N(s i ), V(s i )} where P(s i ) represents a partial reasoning path from a root node to s i , N(s i ) is the number of visits, and V(s i ) is the estimated value of the node. This structure facilitates fine-grained tracking and analysis of changes in the step-by-step reasoning path during the entire search process, enabling the model to systematically explore diverse reasoning paths while recording intermediate states and final results.

[0069] In some embodiments, it is disclosed that the search process of the question and answer model in the manner of Monte Carlo tree search includes four stages of selection, expansion, simulation and backtracking. In the simulation stage, the question and answer model generates multiple answers by multiple inferences for the candidate step, and determines the evaluation value of the candidate step by comparing the generated answers with the standard answers. Specifically, for each candidate step, the generated answers by multiple inferences are compared with the standard answers to determine the proportion of the answers by inferences consistent with the standard answers, that is, to determine the success rate of inference, and the proportion is taken as the evaluation value of the candidate step. The above simulation method can simplify the simulation process of the question and answer model, thereby improving the search efficiency of the inference path.

[0070] In the above manner, the question and answer model can search multiple inference paths for the question sample, and based on each inference path, a corresponding inference answer can be obtained.

[0071] In the present embodiment, the question and answer model is triggered by the prompt in the above inference path search and answer process, and also generates corresponding comments for the inference path. That is, the question and answer model selects a correct inference path that obtains a correct answer from the multiple inference paths as a reference path based on each inference path and its corresponding inference answer, and selects an incorrect inference path that obtains an incorrect answer from the multiple inference paths as an inference path sample. Further, the question and answer model generates a comment sample for the inference path sample by comparing the difference between the inference path sample and the correct inference path.

[0072] Specifically, the question and answer model compares the inference path sample with the correct inference path, determines the last common node of the two, and compares the difference between the inference path sample and the correct inference path from the common node. Based on the difference between the two, a comment corresponding to the difference inference step is generated as a comment sample.

[0073] In some embodiments, the above comment sample is a comment in natural language form. The question and answer model can comment on the inference process corresponding to the difference inference step in the inference path sample based on the difference inference step between the inference path sample and the correct inference path, taking the inference process corresponding to the difference inference step in the correct inference path as a reference, such as commenting on the error reason, providing a comment basis, providing an inference correction suggestion, etc., thereby generating a comment corresponding to the difference inference step.

[0074] For the correct inference path, the question and answer model can directly output a set comment, such as "no need to modify".

[0075] According to the above manner, for each incorrect reasoning path in the reasoning path generated by the question and answer model based on the question sample, a comment sample in the form of step-level natural language corresponding to the incorrect reasoning path can be generated. Each incorrect reasoning path is a reasoning path sample, and the comment sample corresponding to the reasoning path sample forms a set of comment data samples for comment generation training of the comment model.

[0076] S102, based on the reasoning path sample generated by the question and answer model and the comment sample corresponding to the reasoning path sample, the comment model is trained.

[0077] Specifically, after obtaining the comment data sample generated by the question and answer model through the processing of the above step S101, the comment data sample can be used for comment generation training of the comment model.

[0078] Specifically, for example, the reasoning path sample is input into the comment model, the comment corresponding to the reasoning path sample is generated by the comment model, and then the parameter optimization of the comment model is performed to make the comment generated by the comment model consistent with the comment sample corresponding to the reasoning path sample. Repeating the above process can continuously optimize the comment model, and the corresponding comment can be generated based on any reasoning path.

[0079] In another embodiment, it is disclosed that the comment sample generated by the question and answer model can also be incorrect, or the comment sample is not good enough. If these comment samples are used to train the comment model, it can cause negative optimization of the comment model, thereby affecting the training effect of the comment model.

[0080] To solve the above problem, in this embodiment, after obtaining the reasoning path sample generated by the question and answer model and the corresponding comment sample, the comment sample is screened, and the high-quality comment sample is selected for training the comment model.

[0081] Specifically, the screening of the comment sample and the training of the comment model can be realized through the following steps S1021-S1022:

[0082] S1021, the question and answer model corrects the reasoning path sample based on the comment sample, and generates a candidate answer of the corresponding question sample based on the corrected reasoning path.

[0083] Specifically, for any set of parallel data of reasoning path samples and comment samples, the comment sample is input as a prompt into the question and answer model, the question and answer model corrects the reasoning path sample based on the comment sample, and generates a reasoning answer based on the corrected reasoning path as a candidate answer of the corresponding question sample.

[0084] In some embodiments, the question and answer model can be caused to correct the inference path sample based on the comment sample multiple times, and multiple candidate answers can be generated based on the inference path after each correction.

[0085] S1022, if the candidate answer represents the corrected inference path is correct, the inference path sample and the comment sample corresponding to the inference path sample are used to train the comment model.

[0086] Specifically, if the candidate answer generated based on the corrected inference path is consistent with the standard answer, it means that the inference path sample is corrected to a correct inference path after the comment sample is used to correct the inference path sample, that is, the comment sample is objective and correct. At this time, the comment sample and the inference path sample corresponding to the comment sample can be used to train the comment model.

[0087] In some embodiments, the question and answer model can be caused to correct the inference path sample based on the comment sample multiple times, and multiple candidate answers can be generated based on the inference path after each correction. In this case, the proportion of correct answers contained in the multiple candidate answers is calculated to determine whether the comment sample is correct. For example, if more than 30% of the generated multiple candidate answers are consistent with the standard answer, it can be considered that the inference path sample can be corrected to a correct inference path based on the comment sample, that is, it can be considered that the comment sample is objective and correct. At this time, the comment sample and the inference path sample corresponding to the comment sample can be used to train the comment model.

[0088] The specific training method of training the comment model using the comment sample and the inference path sample corresponding to the comment sample can be referred to the introduction of the above embodiments or the following embodiments.

[0089] The present embodiment filters the comment sample to ensure that the comment data sample used to train the comment model has true error correction ability and guiding value, thereby facilitating to improve the training effect and training efficiency of the comment model.

[0090] The comment model trained according to the above embodiments can be used to assist in training the question and answer model. That is, during the training of the question and answer model, the comment model trained according to the above embodiments provides the question and answer model with comment feedback information corresponding to the inference step of the question and answer model, so that the question and answer model optimizes the inference process based on the comment feedback information.

[0091] For example, in the question and answer training process of the question and answer model, the question sample is input into the question and answer model, so that the question and answer model outputs an inference path based on the question sample and an inference answer obtained based on the inference path. At the same time, the comment model generates step-level comments for the inference path output by the question and answer model, that is, the comment model identifies the incorrect inference steps in the inference path, and generates comments for the incorrect inference steps. Then, the question and answer model takes the comments generated by the comment model as a reference, and combines the difference between the inference answer and the standard answer to optimize and adjust the inference steps and correct the inference path, so that the inference answer obtained based on the corrected inference path is consistent with the standard answer.

[0092] Through the above training, the question and answer model can continuously optimize the inference process and the operation parameters based on the comments of the comment model, so that the correctness of the generated inference path is higher, and the correctness of the generated inference answer is higher.

[0093] It can be understood that the question and answer model training method introduced in the above embodiment can train the question and answer model, so that the comment model provides step-level comment feedback information for the inference path of the question and answer model, so that the question and answer model obtains step-level comment feedback guidance and assistance in the training process, and further enables the question and answer model to more accurately correct the incorrect inference steps in the inference path and more quickly and efficiently generate a correct inference path.

[0094] At the same time, the comment data samples used by the above training method to train the comment model are composed of step-level comments generated by the question and answer model in the answering process. These comment data samples can truly reflect the common error types in the actual inference process of the question and answer model, improve the coverage and pertinence of the comment data samples, and at the same time, get rid of the dependence on external large models and artificial annotation, which can significantly improve the construction efficiency and quality of the comment data samples, and further improve the training effect and efficiency of the comment model and the question and answer model, and improve the question and answer performance of the question and answer model.

[0095] In another embodiment of the question and answer model training method, after training the comment model according to the comment data samples generated in the answering process of the question and answer model based on the question sample, the trained comment model and the question and answer model are jointly trained.

[0096] Referring to Figure 2 The question and answer model training method provided in the embodiment includes:

[0097] S201, training the comment model according to the comment data samples generated in the answering process of the question and answer model based on the question sample.

[0098] Specifically, the specific implementation process of training the comment model based on the comment data samples generated in the answering process of the question and answer model based on the question samples can be referred to the comment model training process introduced in the above embodiments, which will not be repeated here.

[0099] S202, jointly train the trained comment model and the question and answer model.

[0100] Specifically, after training the comment model based on the comment data samples generated in the answering process of the question and answer model based on the question samples, the comment model can have the basic function of generating comments for the inference path of the question and answer model.

[0101] If the above trained comment model is used to train the question and answer model, and the comment model is no longer trained and optimized, it is possible that as the question and answer model is continuously trained and optimized, the comment model and the question and answer model do not match in ability, the comment model is difficult to provide effective guidance and assistance for the question and answer model, or the comment model is difficult to generate accurate comments, thereby affecting the training of the question and answer model.

[0102] In order to avoid the above situation, in this embodiment, after the comment model is preliminarily trained to have the basic comment function, the comment model is jointly trained with the question and answer model in the process of training the question and answer model with the comment model, so that the question and answer model and the comment model are optimized in the training process.

[0103] In the above joint training process, the question and answer model generates an inference path based on a question sample, the comment model generates a comment based on the inference path generated by the question and answer model, and then the question and answer model corrects the inference path based on the comment output by the comment model to verify the effectiveness of the comment. Finally, in combination with the effectiveness of the comment output by the comment model, the comment output by the comment model is used to optimize the inference process of the question and answer model and the comment generation process of the comment model. Repeat the above processing, so that the question and answer model and the comment model are optimized in the training process.

[0104] In another embodiment, by performing the following steps A1-A3, the joint training of the question and answer model and the comment model can be realized:

[0105] A1, input the question sample into the question and answer model to obtain the candidate inference path generated by the question and answer model, and determine the correctness evaluation value of the candidate inference path.

[0106] Specifically, input the question sample Q into the question and answer model π θ , so that the question and answer model π θ generates N candidate inference paths based on the question sample Q:

[0107] A = {A1, A2, ..., A} N},A i ~π θ (A|Q)

[0108] Among them, A i Representing the question-answering model π θ The i-th candidate reasoning path is generated based on the problem sample Q.

[0109] For each of the above candidate reasoning paths A i Based on this candidate reasoning path A i The corresponding reasoning answer and the standard answer A * By comparing the results, the correctness evaluation value r of the reasoning answer is determined. i And the correctness evaluation value r i As the correctness evaluation value of this candidate reasoning path:

[0110] r i =I[A i =A * ]

[0111] Among them, I[A i =A * ] indicates that by comparing candidate reasoning paths A i The corresponding reasoning answer and the standard answer A * Determine the candidate reasoning path A i The correctness assessment value.

[0112] After obtaining the correctness evaluation value for each candidate reasoning path, the question sample, candidate reasoning path, and the correctness evaluation value of the candidate reasoning path are treated as a set of parallel data, which can be used to train the question answering model. For example, (Q, A) i r i As a set of parallel data, it can be used to train the question-answering model to generate candidate reasoning paths with higher correctness evaluation values.

[0113] A2. Input the candidate reasoning path into the comment model, so that the comment model generates comments for the corresponding candidate reasoning path and determines the correctness evaluation value of the comment.

[0114] Specifically, after training as described in the above embodiments, the comment model possesses basic comment generation capabilities. During the training of the question-answering model, the candidate reasoning paths generated by the question-answering model based on question samples are input into the comment model, enabling the comment model to generate comments corresponding to those candidate reasoning paths.

[0115] The comment model can generate one or more comments for the input candidate reasoning path.

[0116] In some embodiments, the question and answer model can generate multiple candidate reasoning paths for an input question sample, in which case, the multiple candidate reasoning paths can be input into the comment model, so that the comment model generates one or more comments corresponding to each candidate reasoning path, or a candidate reasoning path can be randomly selected from the multiple candidate reasoning paths and input into the comment model, so that the comment model generates one or more comments corresponding to the candidate reasoning path.

[0117] For example, from the question and answer model π θ The generated multiple candidate reasoning paths A = {A1, A2, …, A N A candidate reasoning path A j is randomly selected from the multiple candidate reasoning paths A φ and input into the comment model π φ , so that the comment model π M generates M comments corresponding to the candidate reasoning path:

[0118] C = {C1, C2, …, C k}, C φ ~ π j (C|Q, A

[0119] wherein C k represents the kth comment generated by the comment model π φ according to the candidate reasoning path A j .

[0120] After obtaining the comments corresponding to the candidate reasoning path, the correctness evaluation value of the comments is determined by performing the following steps A21-A23:

[0121] A21, input the comment into the question and answer model, so that the question and answer model corrects the candidate reasoning path based on the comment to obtain a corrected reasoning path, and determines a corrected answer corresponding to the corrected reasoning path.

[0122] Specifically, the comment generated by the comment model is input into the question and answer model as a prompt, so that the question and answer model corrects the candidate reasoning path based on the comment to obtain a corrected reasoning path, and infers a corrected answer based on the corrected reasoning path.

[0123] In some embodiments, the comment model generates multiple comments for a candidate reasoning path, and for each comment, the comment is input into the question and answer model as a prompt, so that the question and answer model corrects the candidate reasoning path L times based on the comment to obtain L corrected reasoning paths, and determines a corrected answer corresponding to each corrected reasoning path.

[0124] A22, by comparing the corrected answer with the standard answer, the correctness evaluation value of the corrected reasoning path is determined.

[0125] Specifically, the corrected answer obtained from the corrected reasoning path is compared with the standard answer to determine the correctness evaluation value of the corrected answer. This correctness evaluation value can be directly used as the correctness evaluation value of the corrected reasoning path.

[0126] For example, given any comment generated by the comment model based on the candidate inference path generated by the question-answering model, the answering model performs L inference corrections under certain conditions, resulting in L corrected inference paths:

[0127] A′ k ={A′ k,1 ,A′ k,2 ,…,A′ k,L},A′ k,l ~π θ (A′|Q,A j C k )

[0128] Among them, A′ k,l Representing the question-answering model π θ According to comment C k For candidate inference path A j The l-th revised reasoning path obtained by performing the l-th revision.

[0129] Then, for each corrected reasoning path, the correctness evaluation value of the corrected reasoning path is determined by comparing the reasoning answer determined based on the corrected reasoning path with the standard answer. For example, for the l-th corrected reasoning path, the correctness evaluation value r′k,l of the corrected reasoning path can be calculated according to the following formula:

[0130] r′ k,l =I[A′ k,l =A * ]

[0131] Wherein, it represents comparing the l-th modified candidate reasoning path A′. k,l The corresponding reasoning answer and the standard answer A * Determine the modified candidate reasoning path A′ for the l-th path. k,l The correctness assessment value.

[0132] A23. Determine the correctness assessment value of the comment based on the correctness assessment value of the corrected reasoning path.

[0133] Specifically, if there is only one corrected reasoning path generated by the question-answering model based on the comments, the correctness evaluation value of the comments is the correctness evaluation value of the corrected reasoning path.

[0134] In the case that there are multiple modified inference paths generated by the comment, the correctness evaluation value of the comment is the average of the correctness evaluation values of the multiple modified inference paths generated by the comment. For example, for the kth comment C k The correctness evaluation value of the comment C k can be calculated in the following manner

[0135]

[0136] A3, based on the correctness evaluation value of the candidate inference path, the comment and the correctness evaluation value of the comment, optimizing the question and answer model, and based on the comment and the correctness evaluation value of the comment, optimizing the comment model.

[0137] Specifically, when optimizing the question and answer model, the candidate inference path A i generated by the question and answer model based on the question sample Q, and the correctness evaluation value r i of the candidate inference path A i are combined as a set of parallel data (Q, A i , r i ) to serve as the basis for optimizing the question and answer model, so that the question and answer model generates a candidate inference path with a higher correctness evaluation value. For example, by measuring the difference between the correctness evaluation value of the candidate inference path generated this time based on the question sample and the correctness evaluation value of the candidate inference path generated last time based on the question sample, it is determined whether the generated candidate inference path is more correct, and then it is determined whether the optimization of the question and answer model is appropriate, and based on the determination result, the model is optimized in the appropriate model optimization direction.

[0138] At the same time, based on the comment generated by the comment model for the candidate inference path A i and the correctness evaluation value of the comment, the question and answer model is guided and assisted to optimize the inference process, so that the comment on the candidate inference path generated by the question and answer model tends to indicate that the candidate inference path is correct, that is, to improve the correctness of the candidate inference path generated by the question and answer model. For example, by measuring the difference between the comment corresponding to the candidate inference path generated this time based on the question sample and the comment indicating that the candidate inference path is correct, and the difference between the comment corresponding to the candidate inference path generated last time based on the question sample and the comment indicating that the candidate inference path is correct, it is determined whether the optimization of the question and answer model is appropriate, and based on the determination result, the model is optimized in the appropriate model optimization direction.

[0139] When optimizing a question-answering model using both the correctness evaluation values ​​and comments of candidate inference paths, the optimization contribution coefficients of these two factors can be determined. For example, a higher correctness evaluation value for a comment results in a greater optimization contribution coefficient, meaning the question-answering model refers to the comment more when optimizing inference, and vice versa. Similarly, a higher correctness evaluation value for a candidate inference path results in a greater optimization contribution coefficient, meaning the question-answering model refers to the candidate inference path more when optimizing inference parameters, such as making the next inference path generated based on a question sample as similar as possible to the candidate inference path, and vice versa.

[0140] When optimizing a comment model based on comments and their accuracy evaluation values, the goal is to increase the accuracy evaluation values ​​of the comments generated by the model. For example, by measuring whether the accuracy evaluation value of the comments generated this time is greater than the previous one, it can be determined whether the optimization of the comment model is appropriate. Furthermore, based on the judgment results, a suitable optimization direction can be determined for further optimization of the comment model.

[0141] In another embodiment, when the comment model generates multiple comments for a candidate reasoning path generated by the question-answering model, the standardized advantage value of each comment is first determined based on the correctness evaluation value corresponding to each comment.

[0142] The standardized advantage value of the comments is used to represent the relative optimization effect of the comments compared to other comments.

[0143] Specifically, the multiple comments generated for a given candidate reasoning path are grouped together, and the average correctness evaluation value of the comments within the group is calculated first. Assume that a total of G comments are generated for a given candidate reasoning path, and the correctness evaluation value of each comment is r′. k The average accuracy score of the group comments can then be calculated using the following formula.

[0144]

[0145] Then, the standardized odds value for each comment within the group is calculated using the following formula.

[0146]

[0147] in, denotes the standardized advantage value of the kth comment, and denotes the standard deviation of the correctness evaluation value in the group.

[0148] After determining the standardized advantage value of each comment, the question-answer model is optimized based on the correctness evaluation value of the candidate reasoning path, each comment, and the standardized advantage value of each comment, and the comment model is optimized based on the comments and the standardized advantage value of each comment.

[0149] When optimizing the question-answer model, the influence of different comments on the optimization of the question-answer model is adjusted based on the standardized advantage value of the comments, such as making the comments with high standardized advantage values have a greater influence on the optimization of the question-answer model, and making the comments with low standardized advantage values have a smaller influence on the optimization of the question-answer model. For specific implementation of training and optimization of the question-answer model, refer to the above embodiment introduction.

[0150] In order to make the comment model output comments with higher standardized advantage values, the embodiment of the application adopts a group relative policy optimization (GRPO) mechanism to construct a target function:

[0151]

[0152] wherein, is the probability ratio of the new comment generation strategy of the comment model to the old comment generation strategy, and ∈ is a clipping coefficient. The clip in the above formula is a clipping function, which is used to clip when the standardized advantage value of the comment exceeds the set range, so as to avoid the optimized model deviating too much from the original model due to the standardized advantage value of the comment exceeding the range.

[0153] In order to further improve the training stability, the embodiment also introduces rejection sampling (RS) to construct a target function:

[0154]

[0155] wherein, λ is a weight coefficient, denotes that the standardized advantage value is positive, and logπ φ (C|Q,A) denotes the comment generation strategy of the comment model, denotes that the weight of the comment generation strategy corresponding to the comment with a positive standardized advantage value is further improved.

[0156] According to the above target function, the comment model π φ is trained, which can more efficiently and quickly improve the performance of the comment model.

[0157] In the above embodiment, the question and answer model is jointly trained with the comment model, which can make the question and answer model and the comment model evolve together, can break through the limitation of decoupled training of the question and answer model and the comment model, avoid the limitation of the training and optimization of the comment model on the question and answer model, thereby significantly improving the training effect of the question and answer model, and improving the answering accuracy and correction and modification ability of the question and answer model.

[0158] Based on the question and answer model trained by the above question and answer model training method, another embodiment of the present application further provides a question and answer processing method, which can be applied to a device or terminal equipped with a question and answer model, such as a learning machine, an intelligent human-computer interaction device, an intelligent robot, etc.

[0159] Referring to Figure 3 The question and answer processing method provided by the embodiment includes:

[0160] S301, inputting a target question into a question and answer model, so that the question and answer model generates an answering result corresponding to the target question.

[0161] The target question can be a question in text form, a question in voice form, or a question in image form, or can be a multi-modal question, such as a question containing text, image and voice.

[0162] The question and answer model is adapted to the data type of the target question. When the target question is a single-modal question, such as a question in text form, a question in voice form or a question in image form, the corresponding question and answer model is a text question and answer model, a voice question and answer model or an image question and answer model. When the target question is a multi-modal question, the corresponding question and answer model is a multi-modal question and answer model. In some embodiments, the question and answer model can use a large model.

[0163] The question and answer model used in the above question and answer processing method is trained according to the question and answer model training method introduced in any of the above embodiments.

[0164] Based on the above training, the question and answer model has more accurate and reliable question and answer performance, so that the question and answer processing method provided by the embodiment can generate more accurate and reliable answering results for the target question with the help of the above question and answer model.

[0165] The training process of the question and answer model in the question and answer processing method provided by the embodiment, and the beneficial effects brought by the training of the question and answer model, can be referred to the corresponding introduction of the above embodiments.

[0166] Corresponding to the above question and answer processing method, the embodiment of the present application further provides a question and answer processing device, referring to Figure 4 The device includes:

[0167] The question and answer processing unit 100 is configured to input a target question into the question and answer model, so that the question and answer model generates an answer result corresponding to the target question.

[0168] In the training process of the question and answer model, the comment model provides comment feedback information corresponding to the inference steps of the question and answer model, so that the question and answer model optimizes the inference process based on the comment feedback information; the comment model is trained based on comment data samples, the comment data samples include an inference path and a comment sample corresponding to an error inference step in the inference path, and the inference path and the comment sample are generated by the question and answer model in the process of answering based on a question sample.

[0169] In some implementations, the question and answer processing apparatus further includes a model training unit configured to train the question and answer model, and the training process of the question and answer model includes:

[0170] training the comment model based on the comment data samples generated in the process of the question and answer model answering based on the question sample;

[0171] jointly training the trained comment model and the question and answer model.

[0172] In some implementations, the model training unit trains the comment model based on the comment data samples generated in the process of the question and answer model answering based on the question sample, including:

[0173] obtaining an inference path sample generated in the process of the question and answer model answering based on the question sample, and a comment sample corresponding to the inference path sample; wherein, in the process of the question and answer model answering based on the question sample, the inference path is searched in a Monte Carlo tree search manner, and an error inference path is selected as the inference path sample from the searched inference path, and the comment sample corresponding to the inference path sample is generated by comparing the difference inference steps between the inference path sample and a correct inference path;

[0174] training the comment model based on the inference path sample generated by the question and answer model and the comment sample corresponding to the inference path sample.

[0175] In some implementations, the model training unit trains the comment model based on the inference path sample generated by the question and answer model and the comment sample corresponding to the inference path sample, including:

[0176] making the question and answer model correct the inference path sample based on the comment sample, and generating a candidate answer corresponding to the question sample based on the corrected inference path.

[0177] In a case where the candidate answer indicates that the modified reasoning path is correct, the reasoning path sample and the comment sample corresponding to the reasoning path sample are used to train the comment model.

[0178] In some implementations, the search process in the manner of the Monte Carlo tree search includes a selection stage, an expansion stage, a simulation stage, and a backtracking stage. In the simulation stage, the question and answer model generates multiple answers by multiple reasoning of a candidate step, and determines an evaluation value of the candidate step by comparing the generated answers with a standard answer.

[0179] In some implementations, the model training unit jointly trains the comment model and the question and answer model after the training of the comment model is completed, including:

[0180] inputting a question sample into the question and answer model to obtain a candidate reasoning path generated by the question and answer model, and determining a correctness evaluation value of the candidate reasoning path;

[0181] inputting the candidate reasoning path into the comment model to make the comment model generate a comment corresponding to the candidate reasoning path, and determining a correctness evaluation value of the comment;

[0182] optimizing the question and answer model based on the correctness evaluation value of the candidate reasoning path, the comment, and the correctness evaluation value of the comment, and optimizing the comment model based on the comment and the correctness evaluation value of the comment.

[0183] In some implementations, the model training unit determines the correctness evaluation value of the comment, including:

[0184] inputting the comment into the question and answer model to make the question and answer model modify the candidate reasoning path based on the comment to obtain a modified reasoning path, and determining a modified answer corresponding to the modified reasoning path;

[0185] determining a correctness evaluation value of the modified reasoning path by comparing the modified answer with a standard answer;

[0186] determining the correctness evaluation value of the comment based on the correctness evaluation value of the modified reasoning path.

[0187] In some implementations, there are multiple candidate reasoning paths, and the model training unit inputs the candidate reasoning paths into the comment model to make the comment model generate comments corresponding to the candidate reasoning paths, including:

[0188] The comment model is input with a candidate reasoning path selected from a plurality of candidate reasoning paths, so that the comment model generates a plurality of comments corresponding to the candidate reasoning path.

[0189] In some implementations, the model training unit optimizes the question and answer model based on the correctness evaluation value of the candidate reasoning path, the comments, and the correctness evaluation value of the comments, and optimizes the comment model based on the comments and the correctness evaluation value of the comments, including:

[0190] Based on the respective correctness evaluation value of each comment, a standardized advantage value of each comment is determined, wherein the standardized advantage value of a comment is used to represent the relative optimization effect of the comment relative to other comments;

[0191] Based on the correctness evaluation value of the candidate reasoning path, each of the comments, and the respective standardized advantage value of each of the comments, the question and answer model is optimized, and the comment model is optimized based on each of the comments and the standardized advantage value of each of the comments.

[0192] The question and answer processing apparatus provided by the embodiments of the present application belongs to the same application concept as the question and answer processing method provided by the above embodiments of the present application, can execute the question and answer processing method provided by any of the above embodiments of the present application, and has the corresponding function modules and beneficial effects of executing the question and answer processing method. Technical details not described in detail in the embodiments can be referred to the specific processing content of the question and answer processing method provided by the above embodiments of the present application, which will not be described here.

[0193] The functions implemented by each unit of the question and answer processing apparatus above can be respectively implemented by the same or different processors, which is not limited in the embodiments of the present application.

[0194] It should be understood that the units in the above apparatus can be implemented in the form of processor calling software. For example, the apparatus includes a processor connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of the units of the apparatus, wherein the processor can be a general processor such as CPU or microprocessor, and the memory can be an internal memory or an external memory of the apparatus. Alternatively, the units in the apparatus can be implemented in the form of hardware circuit, and the functions of part or all of the units can be realized by the design of the hardware circuit, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is ASIC, and the functions of part or all of the units are realized by the design of the logical relationship of the elements in the circuit. For another example, in another implementation, the hardware circuit can be realized by PLD, and taking FPGA as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of part or all of the units. All the units of the above apparatus can be realized in the form of processor calling software, or realized in the form of hardware circuit, or part of them is realized in the form of processor calling software, and the remaining part is realized in the form of hardware circuit.

[0195] In the embodiments of the present application, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as CPU, microprocessor, GPU, or DSP, etc. In another implementation, the processor can realize certain functions through the logical relationship of hardware circuit, which is fixed or can be reconfigured, such as ASIC or PLD implemented hardware circuit, such as FPGA, etc. In the reconfigurable hardware circuit, the process of the processor loading configuration document to realize hardware circuit configuration can be understood as the process of the processor loading instructions to realize the functions of part or all of the units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as a kind of ASIC, such as NPU, TPU, DPU, etc.

[0196] It can be seen that each unit in the above apparatus can be one or more processors (or processing circuits) configured to implement the above method, such as CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0197] In addition, all or part of each unit in the above apparatus can be integrated together or can be independently implemented. In one implementation, the units are integrated together to be implemented in the form of a SOC. The SOC can include at least one processor for implementing any of the above methods or functions of the units of the apparatus, and the at least one processor can be of different types, such as including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, and the like.

[0198] Another embodiment of the present application also provides an electronic device, as shown in Figure 5 The device includes:

[0199] a memory 200 and a processor 210;

[0200] The memory 200 is connected with the processor 210, and is configured to store a program.

[0201] The processor 210 is configured to realize the question and answer processing method or the question and answer model training method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0202] Specifically, the electronic device can further include a bus, a communication interface 220, an input device 230, and an output device 240.

[0203] The processor 210, the memory 200, the communication interface 220, the input device 230, and the output device 240 are connected with each other through the bus. Among them:

[0204] The bus can include a path for transmitting information between various components of the computer system.

[0205] The processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or can be an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready-to-use programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0206] The processor 210 can include a main processor, and can also include a baseband chip, a modem, etc.

[0207] The memory 200 stores programs for implementing the technical solutions of the present application, and can also store operating systems and other key services. Specifically, the programs can include program codes, and the program codes include computer operation instructions. More specifically, the memory 200 can include read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (RAM), other types of dynamic storage devices that can store information and instructions, disk storage, flash memory, and the like.

[0208] The input device 230 can include devices that receive data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, a gravity sensor, and the like.

[0209] The output device 240 can include devices that allow information to be output to a user, such as a display screen, a printer, a speaker, and the like.

[0210] The communication interface 220 can include devices of the transceiver type or the like for communicating with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), and the like.

[0211] The processor 210 executes programs stored in the memory 200 and invokes other devices, which can be used to implement each step of any of the question and answer processing methods or question and answer model training methods provided by the above-described embodiments.

[0212] In addition to the above methods and devices, the embodiments of the present application can also be computer program products that include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the question and answer processing method or the question and answer model training method described in any of the above-described embodiments.

[0213] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of the present application, including object-oriented programming languages such as Java, C++, and the like, and conventional procedural programming languages such as "C" language or similar programming languages. The program code can be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0214] In addition, the embodiments of the present application can also be storage media having a computer program stored thereon, the computer program causing the processor to execute the steps of the question and answer processing method or the question and answer model training method described in any of the embodiments of the present application when executed by the processor.

[0215] For each of the foregoing method embodiments, for the sake of simple description, each is described as a combination of a series of actions, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0216] It should be noted that each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For device embodiments, since they are basically similar to method embodiments, they are described more simply, and the relevant parts are referred to the part of the method embodiment.

[0217] The steps in the method of each embodiment of the present application can be adjusted, combined and reduced in sequence according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0218] The modules and sub-modules in the device and terminal of each embodiment of the present application can be combined, divided and reduced according to actual needs.

[0219] In several embodiments provided by the present application, it should be understood that the disclosed terminal, device and method can be implemented by other ways. For example, the terminal embodiments described above are only schematic, and the division of the modules or sub-modules is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of sub-modules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutually can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0220] The modules or sub-modules described as separate components can or can not be physically separated, and the components of the modules or sub-modules can or can not be physical modules or sub-modules, that is, they can be located in one place, or can be distributed to a plurality of network modules or sub-modules. Some or all of the modules or sub-modules can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0221] In addition, each functional module or sub-module in each embodiment of the present application can be integrated in one processing module, or each module or sub-module can exist physically alone, or two or more modules or sub-modules can be integrated in one module. The integrated module or sub-module can be realized in the form of hardware or in the form of a software functional module or sub-module.

[0222] The skilled person can further realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, each example has been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0223] The steps of the method or algorithm described in connection with the embodiments disclosed herein can be directly implemented by hardware, a software unit executed by a processor, or a combination of both. The software unit can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0224] Finally, it should be noted that, in this document, relational terms such as first and second, and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0225] The above description of disclosed embodiments enables one of ordinary skill in the art to make and use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A question and answer processing method characterized by, The method comprises the following steps: inputting a target question into a question and answer model to enable the question and answer model to generate an answer corresponding to the target question; wherein, in the training process of the question and answer model, a comment model provides comment feedback information corresponding to the inference steps of the question and answer model to the question and answer model to enable the question and answer model to optimize the inference process based on the comment feedback information; the comment model is trained based on comment data samples, the comment data samples comprising an inference path and a comment sample corresponding to an erroneous inference step in the inference path, the inference path and the comment sample being generated by the question and answer model in the process of answering based on a question sample.

2. The method of claim 1, wherein, The training process of the question and answer model comprises: training the comment model based on the comment data samples generated in the process of the question and answer model answering based on a question sample; jointly training the trained comment model and the question and answer model.

3. The method of claim 2, wherein, The training of the comment model based on the comment data samples generated in the process of the question and answer model answering based on a question sample comprises: obtaining an inference path sample generated in the process of the question and answer model answering based on a question sample, and a comment sample corresponding to the inference path sample; wherein, in the process of the question and answer model answering based on a question sample, the question and answer model searches for an inference path in a Monte Carlo tree search manner and selects an erroneous inference path from the searched inference path as an inference path sample, and generates a comment sample corresponding to the inference path sample by comparing the difference inference steps between the inference path sample and a correct inference path; training the comment model based on the inference path sample generated by the question and answer model and the comment sample corresponding to the inference path sample.

4. The method of claim 3, wherein, The training of the comment model based on the inference path sample generated by the question and answer model and the comment sample corresponding to the inference path sample comprises: enabling the question and answer model to correct the inference path sample based on the comment sample, and generating a candidate answer corresponding to the question sample based on the corrected inference path; in the case where the candidate answer represents the corrected inference path is correct, using the inference path sample and the comment sample corresponding to the inference path sample to train the comment model.

5. The method of claim 3, wherein, The search process in the Monte Carlo tree search manner comprises a selection stage, an expansion stage, a simulation stage and a backtracking stage, in the simulation stage, the question and answer model generates multiple answers by multiple times of deduction on a candidate step, and determines the evaluation value of the candidate step by comparing the generated answers with a standard answer.

6. The method of claim 2, wherein, The joint training of the trained comment model and the question and answer model comprises: inputting a question sample into the question and answer model to obtain a candidate inference path generated by the question and answer model, and determining the correctness evaluation value of the candidate inference path; inputting the candidate inference path into the comment model to enable the comment model to generate a comment corresponding to the candidate inference path, and determine the correctness evaluation value of the comment; The question and answer model is optimized based on the correctness evaluation value of the candidate reasoning path, the comment, and the correctness evaluation value of the comment, and the comment model is optimized based on the comment and the correctness evaluation value of the comment.

7. The method of claim 6, wherein, The correctness evaluation value of the comment is determined by: inputting the comment into the question and answer model to enable the question and answer model to correct the candidate reasoning path based on the comment to obtain a corrected reasoning path and determine a corrected answer corresponding to the corrected reasoning path; determining the correctness evaluation value of the corrected reasoning path by comparing the corrected answer with a standard answer; and determining the correctness evaluation value of the comment based on the correctness evaluation value of the corrected reasoning path.

8. The method according to claim 6 or 7, characterized in that, The number of candidate reasoning paths is a plurality, and inputting the candidate reasoning paths into the comment model to enable the comment model to generate comments corresponding to the candidate reasoning paths includes: selecting one candidate reasoning path from the plurality of candidate reasoning paths to input into the comment model to enable the comment model to generate a plurality of comments corresponding to the candidate reasoning path.

9. The method of claim 8, wherein, The question and answer model is optimized based on the correctness evaluation value of the candidate reasoning path, the comment, and the correctness evaluation value of the comment, and the comment model is optimized based on the comment and the correctness evaluation value of the comment, including: determining a standardized advantage value of each comment based on the respective correctness evaluation value of each comment; wherein the standardized advantage value of a comment is used to represent the relative optimization effect of the comment relative to other comments; optimizing the question and answer model based on the correctness evaluation value of the candidate reasoning path, each comment, and the respective standardized advantage value of each comment, and optimizing the comment model based on each comment and the standardized advantage value of each comment.

10. A question and answer processing apparatus characterized by comprising: including: a question and answer processing unit configured to input a target question into a question and answer model to enable the question and answer model to generate an answer result corresponding to the target question; wherein, in the training process of the question and answer model, a comment model provides comment feedback information corresponding to the reasoning steps of the question and answer model for the question and answer model to enable the question and answer model to optimize the reasoning process based on the comment feedback information; the comment model is trained based on comment data samples, the comment data samples include reasoning paths and comment samples corresponding to incorrect reasoning steps in the reasoning paths, and the reasoning paths and the comment samples are generated by the question and answer model in the process of answering based on question samples.

11. An electronic device, comprising: including a memory and a processor; the memory is connected to the processor and is used to store programs; the processor is used to realize the question and answer processing method of any one of claims 1 to 9 by running the programs in the memory.

12. A computer program product, characterised in that, including computer program instructions that, when executed by a processor, cause the processor to implement the question and answer processing method of any one of claims 1 to 9.

Images