Question and answer model training method and device, electronic equipment and storage medium

Abstract

The invention provides a question and answer model training method and device, electronic equipment and a storage medium. The method comprises the steps that multiple pieces of training data are acquired; in the process of training the question and answer model according to the sample question, performing repeated fragment identification on a reasoning process generated by performing multiple sampling on the sample question to determine a repeated penalty value of a repeated fragment, and performing reward value calculation on an answer generated by performing sampling on the sample question; performing fusion based on the repeated penalty value and the reward value to form a reward signal, performing normalization processing on the reward signal to obtain a dominant value of current sampling, and introducing the dominant value of the current sampling into a grouping strategy optimization algorithm to construct a loss function; and updating model parameters through gradient back propagation to inhibit repeated thinking behaviors of the question and answer model in the reasoning process, and continuing to perform iterative training on the updated question and answer model. More attention is paid to'effective thinking 'in the training process, and the ability of the question and answer model to solve complex problems is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a question and answer model training method and device, electronic equipment and storage medium. BACKGROUND

[0002] In recent years, reinforcement learning has made significant progress in large language models (LLM), especially in solving complex problems such as programming problems and mathematical problems. For example, some representative RL optimization-based LLM models include OpenAI-o3, DeepSeek-R1, Kimi-2, Claude-3.7, etc. Although the current LLM adopts the "think first and output answer later" paradigm in reinforcement learning training, which improves the reasoning ability, it also brings problems such as slow reasoning speed, sparse reward signal, and redundant reasoning process. The existing technology has obvious deficiencies in training data construction, reasoning length control, and reward mechanism design. Therefore, how to improve the reasoning efficiency and training efficiency of the model has become a technical problem that cannot be underestimated. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a question and answer model training method and device, electronic equipment and storage medium, by introducing a penalty term for repeated thinking process in advantage value calculation, which can effectively reduce unnecessary repeated thinking of the question and answer model, use limited token space to explore other solution ideas, and pay more attention to "effective thinking" in the training process, improving the ability of the question and answer model to solve complex problems.

[0004] The present application provides a question and answer model training method, which comprises: obtaining a plurality of training data; wherein the training data comprises a sample question, a reference reasoning process, and a sample answer; In the training process of the question and answer model according to the sample question, the repeated penalty value of the repeated segment is determined by repeated segment identification of the reasoning process generated by multiple sampling of the sample question, and the reward value calculation is performed on the answer generated by sampling of the sample question; wherein the question and answer model is an artificial intelligence model, and its task is to answer the question asked by the user; The reward signal is formed by fusion based on the repeated penalty value and the reward value, the current sampling advantage value is obtained by normalizing the reward signal, and the current sampling advantage value is introduced into the grouping strategy optimization algorithm to construct a loss function; The gradient of the loss function to the model parameters of the question and answer model is determined, the model parameters are updated by gradient backpropagation to suppress the repeated thinking behavior of the question and answer model in the reasoning process, and the updated question and answer model is iteratively trained.

[0005] In one possible implementation, the training data is determined by the following steps: For each initial sample question, the initial sample question is input into a plurality of closed-source models to output an initial answer and an initial reasoning process for each initial sample question, and it is verified whether the initial answer is consistent with the standard answer of the initial sample question. If consistent, the shortest length of the initial reasoning process with consistent answers is selected as the reference reasoning process of the initial sample question; Based on the multiple sampling results of the same initial sample question by the plurality of closed-source models, the task difficulty of each initial sample question is determined; Based on the task difficulty of the initial sample question, the initial sample questions are processed for question screening to determine the training data.

[0006] In one possible implementation, the task difficulty of each initial sample question is determined based on the multiple sampling results of the same initial sample question by the plurality of closed-source models, including: The probability of generating a correct answer at least once for the multiple sampling results of each initial sample question is calculated; Based on the probability of correct answer of each initial sample question, the task difficulty of each initial sample question is determined.

[0007] In one possible implementation, the repeated segment identification of the sampling generated reasoning process of the sample question determines the repeated penalty value of the repeated segment, including: The reasoning process is segmented into segments, and the similarity between each two segments is determined based on a distance algorithm. The segments with a similarity greater than a threshold value are determined as repeated segments, and the position index of each repeated segment in the reasoning process is recorded; The deviation value between the length of the reasoning process and the length of the corresponding reference reasoning process is determined; The deviation value is multiplied by the penalty weight of each repeated segment to determine the repeated penalty value of the repeated segment; wherein each repeated segment is applied with a penalty weight based on its position index, and the penalty weight corresponding to the later position is greater.

[0008] In one possible implementation, the reward value calculation of the sampling generated answer of the sample question includes: The answer of the sample question generated by sampling is semantically scored based on a scoring model to determine the reward value; Or, the answer and the corresponding sample answer are processed for multi-dimensional matching, and the reward value is determined based on the matching result.

[0009] In one possible implementation, the advantage value is determined by the following formula:

[0010] in, The dominant value of the current sample. a For sample answers, The first corresponding to the sample problem i Each sample output contains the complete reasoning process and the answer, where G is the total number of samples. For the reward function, for The reward value, The length is n , t for The t One token, for The repeated penalty value, mean Calculated for the mean. std Calculate the standard deviation.

[0011] In one possible implementation, after determining the task difficulty of each initial sample problem based on multiple sampling results of the same initial sample problem using multiple closed-source models, the training method further includes: The maximum length limit of the reasoning process generated by the question-answering model is dynamically adjusted based on the task difficulty of the sample question.

[0012] This application embodiment also provides a training device for a question-answering model, the training device comprising: The acquisition module is used to acquire multiple training data sets; wherein, the training data sets include sample questions, reference reasoning processes, and sample answers. The model training module is used to identify repeated segments and determine the repeated penalty value of the repeated segments in the reasoning process generated by sampling the sample questions multiple times during the training process of the question answering model based on the sample questions; and to calculate the reward value of the answers generated by sampling the sample questions; wherein, the question answering model is an artificial intelligence model whose task is to answer questions asked by users; The loss function construction module is used to fuse the repeated penalty value and the reward value to form a reward signal, normalize the reward signal to obtain the advantage value of the current sample, and introduce the advantage value of the current sample into the grouping strategy optimization algorithm to construct the loss function. The model parameter adjustment module is configured to determine the gradient of the loss function with respect to the model parameter of the question and answer model, update the model parameter through gradient back propagation to suppress the repeated thinking behavior of the question and answer model in the inference process, and continue to iteratively train the updated question and answer model.

[0013] The embodiments of the present application also provide an electronic device, which comprises a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, the processor and the memory communicate through the bus when the electronic device is running, and the machine readable instructions are executed by the processor to perform the steps of the training method of the question and answer model.

[0014] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to perform the steps of the training method of the question and answer model.

[0015] The embodiments of the present application provide a training method and device of a question and answer model, an electronic device and a storage medium. The training method comprises: obtaining a plurality of training data; wherein the training data comprises a sample question, a reference inference process and a sample answer; in a training process of a question and answer model according to the sample question, a repeated segment identification is performed on an inference process generated by multiple samplings of the sample question to determine a repeated penalty value of a repeated segment, and an answer generated by sampling of the sample question is subjected to reward value calculation; wherein the question and answer model is an artificial intelligence model, and a task of the question and answer model is to answer a question raised by a user; a reward signal is formed by fusion of the repeated penalty value and the reward value, a current sampling advantage value is obtained by normalization processing of the reward signal, the current sampling advantage value is introduced into a grouping strategy optimization algorithm to construct a loss function; a gradient of the loss function with respect to a model parameter of the question and answer model is determined, the model parameter is updated through gradient back propagation to suppress a repeated thinking behavior of the question and answer model in an inference process, and the updated question and answer model is iteratively trained. By introducing a penalty term of a repeated thinking process into advantage value calculation, unnecessary repeated thinking of the question and answer model can be effectively reduced, a limited token space is used to explore other solution ideas, and in the training process, more attention is paid to “effective thinking”, and the ability of the question and answer model to solve complex problems is improved.

[0016] In order to make the above objectives, characteristics and advantages of the present application more apparent, more comprehensible, and more clear, the following preferred embodiments are specifically described below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor under the premise of these drawings.

[0018] Figure 1 The flow chart of the method for training a question and answer model provided by the embodiments of the present application; Figure 2 The structural schematic diagram of the training device for a question and answer model provided by the embodiments of the present application; Figure 3 The structural schematic diagram of the training device for a question and answer model provided by the embodiments of the present application; Figure 4 The structural schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, every other embodiment obtained by those skilled in the art without creative labor belongs to the scope of the present application.

[0020] Firstly, the application scenarios applicable to the present application are introduced. The present application can be applied to the technical field of machine learning.

[0021] It is found through research that in recent years, reinforcement learning has made significant progress in the application of large language models (LLM), especially in solving complex problems such as programming problems and mathematical problems. For example, some representative RL optimization-based LLM models include OpenAI-o3, DeepSeek-R1, Kimi-2, Claude-3.7, etc. Although the current LLM adopts the "think first and output answer later" paradigm in reinforcement learning training, which improves the reasoning ability, it also brings problems such as slow reasoning speed, sparse reward signal, and redundant reasoning process. The existing technology has obvious shortcomings in training data construction, reasoning length control, reward mechanism design, etc. Therefore, how to improve the reasoning efficiency of the model and the training efficiency has become a technical problem that cannot be underestimated.

[0022] Based on this, the embodiment of the present application provides a training method of a question and answer model. By introducing a penalty term for repeated thinking process in advantage value calculation, unnecessary repeated thinking of the question and answer model can be effectively reduced, limited token space can be used to explore other solution ideas, and more attention can be paid to "effective thinking" in the training process, thereby improving the ability of the question and answer model to solve complex problems.

[0023] Please refer to Figure 1 , Figure 1 The flowchart of the training method of the question and answer model provided by the embodiment of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the training method provided by the embodiment of the present application comprises the following steps. S101: Obtain a plurality of training data; wherein the training data comprises a sample question, a reference reasoning process, and a sample answer.

[0024] It should be noted that the reference reasoning process is the thinking process of the closed-source model when processing the sample question. The sample question can be a data reasoning problem, a logical reasoning problem, a program generation problem, and a natural language understanding and reasoning task, or other types of problems.

[0025] Wherein, because there is only question and reference answer in the training data set, there is no explicit supervision signal (such as thought chain / reasoning track) in the thinking process of the model during the training process, and only the final answer (not the thinking process) output by the model can be given to the model as a feedback signal, that is, a terminal reward. When the intermediate thinking process of the model is long, the reward delay causes difficulty in gradient transmission, so that the reward signal of a single token becomes very sparse, thereby affecting the training efficiency of the model. Because the training data lacks the thinking process, the model lacks supervision signal in thinking, which leads to uncontrolled model thinking process and more redundant thinking. In order to solve this problem, the present application includes the thinking process in the training data, the purpose is not to let the model simulate the standard thinking process, but to measure whether the thinking length of the sampled sample in the training process is obviously longer than the reasonable reasoning length, and to punish the token level of the super-long thinking process, so as to suppress the tendency of redundant thinking of the model.

[0026] In one possible implementation, the training data is determined by the following steps: A: For each initial sample question, input the initial sample question into a plurality of closed source models to output the initial answer and the initial reasoning process of each initial sample question, verify whether the initial answer is consistent with the standard answer of the initial sample question, and if consistent, select the shortest initial reasoning process from the initial reasoning process with consistent answer as the reference reasoning process of the initial sample question.

[0027] Here, the initial sample question is input into a plurality of closed source models for processing, each closed source model outputs the initial answer and the initial reasoning process of each initial sample question, and verifies whether the initial answer is consistent with the standard answer of the initial sample question. If consistent, select the shortest initial reasoning process from the initial reasoning process with consistent answer as the reference reasoning process of the initial sample question.

[0028] Wherein, the shortest initial reasoning process is selected because the thinking process generated by the model is smaller in length and the answer is correct, which indicates that this reasoning idea is optimal.

[0029] It should be noted that the closed source model includes penAI-o3, Claude-3.7, and Gemini-2.5 model.

[0030] B: Based on the multiple sampling results of the same initial sample question by a plurality of closed source models, the task difficulty of each initial sample question is determined.

[0031] Here, according to the multiple sampling results of the same initial sample question by a plurality of closed source models, the task difficulty of each initial sample question is determined.

[0032] It should be noted that the task difficulty is a quantitative index for measuring the difficulty of a question for a question and answer model.

[0033] In one possible implementation, the task difficulty of each initial sample question is determined based on the multiple sampling results of the same initial sample question by the multiple closed-source models. The probability of generating a correct answer at least once in the multiple sampling results of each initial sample question is calculated, and the task difficulty of each initial sample question is determined based on the probability of the correct answer of each initial sample question.

[0034] Here, the probability of generating a correct answer at least once in the multiple sampling results of each initial sample question is calculated, and the task difficulty of each initial sample question is determined based on the probability of the correct answer of each initial sample question.

[0035] Wherein, the higher the probability of generating a correct answer in the multiple sampling results corresponds to the lower task difficulty of the question.

[0036] C: Based on the task difficulty of the initial sample question, the multiple initial sample questions are subjected to question screening processing to determine the training data.

[0037] Here, the initial sample questions with a task difficulty less than a first preset threshold and greater than a second preset threshold are removed, and the remaining initial sample questions are used as sample questions in the training data.

[0038] In one possible implementation, after the task difficulty of each initial sample question is determined based on the multiple sampling results of the same initial sample question by the multiple closed-source models, the training method further includes: The maximum length limit of the inference process generated by the question and answer model is dynamically adjusted based on the task difficulty of the sample question.

[0039] Here, the maximum length limit of the inference process generated by the question and answer model is dynamically adjusted according to the task difficulty of the sample question. By designing a task difficulty evaluation index, different complexity tasks use different inference length limits. The maximum inference length of a simple task is set to a small value (such as 1k), so that the model can quickly give an answer. The maximum inference length of a complex task is set to a larger value (such as 32k), so that the model has enough space to explore. By limiting the inference length according to the task difficulty, unnecessary thinking of the model can be avoided, the model is forced to find a simple solution under the reasonable token number limit, the token consumption during sampling and inference is reduced, and the inference speed is improved.

[0040] It should be noted that when the sample question is particularly simple or particularly difficult, the model is difficult to learn effectively, because when the sample question is particularly simple for the current model, the model can completely answer the reward signal to 0 in the group sampling represented by GRPO, and for the same reason, when the sample question is particularly difficult for the current model, the reward signal is still 0 in the group sampling. The reward signal of 0 further leads to the advantage value of 0, and the advantage value of 0 leads to the gradient signal of 0, so that the model parameters cannot be updated, and the model cannot be trained, but the training resources are wasted. Therefore, before the model training, the simple and difficult samples need to be removed from the training set when constructing the training data set.

[0041] S102: In the process of training the question and answer model according to the sample question, the repeated penalty value of the repeated segment of the inference process generated by multiple sampling of the sample question is determined by repeated segment identification, and the reward value of the answer generated by sampling of the sample question is calculated; wherein the question and answer model is an artificial intelligence model, and its task is to answer the question asked by the user.

[0042] In this step, in the process of training the question and answer model according to the sample question, the repeated penalty value of the repeated segment of the inference process generated by multiple sampling of the sample question is determined by repeated segment identification, and the reward value of the answer generated by sampling of the sample question is calculated to determine the reward value for the answer.

[0043] The question and answer model is an artificial intelligence model, and its task is to answer the question asked by the user. The question and answer model can be a fine-tuned model obtained by fine-tuning a generative large language model using data in a specific field. The generative large language model is an intelligent model capable of generating natural language text, such as a large language model (LLM) and a GPT-3 model. The question and answer model can be a model applied to the field of mathematics, which can help users answer various mathematical problems, provide users with problem-solving ideas and steps, and help users check errors. Specifically, the question and answer model can be obtained by fine-tuning a generative large language model using question and answer pairs in the field of mathematics. The question and answer model can also be a model in the field of logical reasoning, which is not limited in this part.

[0044] Here, the question and answer model adopts the mainstream LLM model architecture, and the main training process is as follows: the text question is input into the TextEncoder for encoding. The TextEncoder result is input into the LLM model, and the model generates a single token step by step through the autoregressive method until the end symbol <eos>, to get the complete output OUTPUT. OUTPUT contains two parts, one is the <think>< / think> thinking process wrapped by the label, and the other is the final answer of the model output wrapped by the label. <answer>< / answer>

[0045] In one possible implementation, the repeated segment identification of the inference process generated by sampling the sample question determines the repeated penalty value of the repeated segment, including: a: segmenting the inference process, determining the similarity between each two segments based on the distance algorithm, determining the repeated segment whose similarity is greater than the threshold value, and recording the position index of each repeated segment in the inference process.

[0046] Here, according to the punctuation, the inference process is segmented into multiple segments for a single sampling result, the similarity between each two segments is determined according to the distance algorithm, the segment whose similarity is greater than the threshold value is determined as the repeated segment, and the position index of each repeated segment in the inference process is recorded.

[0047] It should be noted that considering that multiple content repetitions may occur in a thinking process, i.e., content A repeats 1 time, content B repeats 2 times, etc. The position index of each content repetition needs to be recorded, and smaller penalty is given to the early repeated segment and larger penalty weight is given to the later repeated segment.

[0048] b: determining the deviation value between the length of the inference process and the length of the corresponding reference inference process.

[0049] Here, the deviation value between the length of the inference process and the length of the corresponding reference inference process is determined by the following formula:

[0050] wherein, is the length of the inference process for single sampling of the question q, is the length of the reference inference process. By this method, the larger the length deviation, the stronger the penalty, the maximum penalty is 1, and the minimum penalty is 0, is the deviation value, is the coefficient.

[0051] c: multiplying the deviation value and the penalty weight of each repeated segment to determine the repeated penalty value of the repeated segment; wherein the penalty weight based on the position index is applied to each repeated segment, and the later the position, the greater the corresponding penalty weight.

[0052] ​Here, the bias value is multiplied by the penalty weight of each repeated segment to determine the repetition penalty value of the repeated segment. Wherein, a penalty weight based on the position index of each repeated segment is applied, and the later the position, the greater the corresponding penalty weight.

[0053] Wherein, the final penalty value of a single token in the inference process is the penalty value of the entire sampling multiplied by the repetition penalty weight of the single token :

[0054] In one possible implementation, the reward value calculation of the answer generated by sampling the sample question includes: Based on the scoring model, the semantic score of the answer generated by sampling the sample question is determined to determine the reward value; or, the answer and the corresponding sample answer are processed by multi-dimensional matching, and the reward value is determined based on the matching result.

[0055] Here, the difference between the final answer output by the comparison model and the sample answer in the training sample is compared, and the reward value is calculated by the reward function.

[0056] S103: Based on the fusion of the repetition penalty value and the reward value, a reward signal is formed, the reward signal is normalized to obtain the advantage value of the current sampling, and the advantage value of the current sampling is introduced into the grouping policy optimization algorithm to construct a loss function.

[0057] In this step, the repetition penalty value and the reward value are fused to form a reward signal, the reward signal is normalized to obtain the advantage value of the current sampling, and the advantage value of the current sampling is introduced into the grouping policy optimization algorithm to construct a loss function.

[0058] It should be noted that the advantage value includes the reward value of the answer and the repetition penalty value of the repeated segment.

[0059] In one possible implementation, the advantage value is determined by the following formula:

[0060] Wherein, is the advantage value of the current sampling, a is the sample answer, is the first i sampling output corresponding to the sample question, the content of the sampling output includes the complete inference process and the answer, and G is the total number of samplings, is the reward function, is the reward value of , the length of is n , t is the t token, is a repetition penalty value, mean is std a mean calculation,

[0061] wherein the loss function is:

[0062] wherein, is a loss function, D is a training set, (q, r, a) is a single training sample, q is a sample question, r is a reference reasoning process, and a is a sample answer. is an old policy function representing the policy function of the last training iteration, is a target policy function that needs to be trained, represents the output obtained by sampling q G times using the old policy function (including the reasoning process + answer). G is the number of samplings for a single sample. is used to measure the difference in prediction of the next token by the target policy function and the old policy function, is a truncation function, indicating that when is greater than or less than truncation is performed.

[0063] S104: Determine the gradient of the loss function with respect to the model parameters of the question and answer model, update the model parameters by gradient backpropagation to suppress the repetitive thinking behavior of the question and answer model in the reasoning process, and continue to iteratively train the updated question and answer model.

[0064] In this step, the gradient of the loss function with respect to the model parameters of the question and answer model is determined, the model parameters are updated by gradient backpropagation to suppress the repetitive thinking behavior of the question and answer model in the reasoning process, and the updated question and answer model is iteratively trained until the question and answer model reaches a predetermined standard to stop training, so that the trained question and answer model is used to answer the user's question.

[0065] Among the specific embodiments, when the question and answer model completes the reinforcement learning training, it receives user questions and generates answers in actual applications (such as intelligent customer service, mathematical problem solving assistants, and the like). Before the question and answer model generates answers for the questions, the question and answer model first evaluates the task difficulty of the current question (this value has been pre-calculated in the training data construction stage and can be cached or estimated online). The maximum thinking length limit is dynamically set according to the task difficulty, which prevents the model from over-expanding the reasoning on simple problems and forces it to converge quickly. A repetition penalty mechanism is introduced in the reasoning process (implicit influence behavior), although the repetition penalty term mainly affects the calculation of the advantage function in the training stage, but its effect has been internalized into the model strategy. During reasoning, the model has learned to avoid generating reasoning steps with repeated or circular semantics, resulting in more compact and efficient thinking paths. When the question and answer model considers that the reasoning is sufficient or reaches the length limit, the final answer to the question is generated.

[0066] In this application, by introducing a repeated thinking process penalty term in the advantage value calculation, the model can be effectively prevented from going back, unnecessary repeated thinking can be reduced, and limited token space can be used to explore other solution ideas. Through task difficulty identification, a shorter maximum generation length is set for simple tasks, and a larger maximum generation length is set for difficult tasks, which can effectively prevent the model from over-thinking on simple tasks, while giving the model sufficient thinking space on difficult tasks.

[0067] The training method of the question and answer model provided in the embodiments of the present application comprises: obtaining a plurality of training data; wherein the training data comprises a sample question, a reference reasoning process, and a sample answer; during training of the question and answer model according to the sample question, a repeated penalty value of a repeated segment of a reasoning process generated by multiple samplings of the sample question is determined, and a reward value calculation is performed on an answer generated by sampling of the sample question; wherein the question and answer model is an artificial intelligence model, and its task is to answer user questions; a reward signal is formed by fusing the repeated penalty value and the reward value, a current sampling advantage value is obtained by normalizing the reward signal, and the current sampling advantage value is introduced into a grouping policy optimization algorithm to construct a loss function; the gradient of the loss function to the model parameters of the question and answer model is determined, the model parameters are updated through gradient backpropagation to suppress the repeated thinking behavior of the question and answer model in the reasoning process, and the question and answer model after updating is iteratively trained. By introducing a penalty term for repeated thinking process in the advantage value calculation, unnecessary repeated thinking of the question and answer model can be effectively reduced, limited token space can be used to explore other solution ideas, more attention is paid to "effective thinking" during training, and the ability of the question and answer model to solve complex problems is improved.

[0068] Please refer to Figure 2 、 Figure 3 , Figure 2 Figure 1 is a structural schematic diagram of a question and answer model training device provided by an embodiment of the present application; Figure 3 Figure 2 is another structural schematic diagram of a question and answer model training device provided by an embodiment of the present application. As shown in Figure 2, the question and answer model training device 200 comprises: Figure 2 an acquisition module 210, configured to acquire a plurality of training data; wherein the training data comprises a sample question, a reference reasoning process and a sample answer; a model training module 220, configured to, in a training process of a question and answer model according to the sample question, perform repeated segment identification on a reasoning process generated by multiple samplings of the sample question to determine a repeated penalty value of a repeated segment, and perform reward value calculation on an answer generated by sampling of the sample question; wherein the question and answer model is an artificial intelligence model, and a task of the question and answer model is to answer a question asked by a user; a loss function construction module 230, configured to fuse the repeated penalty value and the reward value to form a reward signal, perform normalization processing on the reward signal to obtain an advantage value of a current sampling, and introduce the advantage value of the current sampling into a grouping policy optimization algorithm to construct a loss function; a model parameter adjustment module 240, configured to determine a gradient of the loss function on a model parameter of the question and answer model, update the model parameter through gradient back propagation to suppress a repeated thinking behavior of the question and answer model in a reasoning process, and continue to iteratively train the question and answer model after updating.

[0069] Further, the acquisition module 210 is configured to determine the training data through the following steps: for each initial sample question, input the initial sample question into a plurality of closed source models to output an initial answer and an initial reasoning process of each initial sample question, verify whether the initial answer is consistent with a standard answer of the initial sample question, and if so, select a shortest one from the initial reasoning processes with consistent answers as a reference reasoning process of the initial sample question; based on multiple sampling results of the same initial sample question by the plurality of closed source models, determine a task difficulty of each initial sample question; based on the task difficulty of the initial sample question, perform question screening processing on a plurality of initial sample questions to determine the training data.

[0070] Further, the acquisition module 210 is configured to determine the task difficulty of each initial sample question based on the multiple sampling results of the same initial sample question by the plurality of closed source models: ​The probability of at least one sampling generating a correct answer is calculated for the multiple sampling results of each initial sample question; The task difficulty of each initial sample question is determined based on the probability of the correct answer of each initial sample question.

[0071] Further, the model training module 220 is configured to identify repeated segments in the repeated segment of the inference process of the sample question, and determine a repeated penalty value of the repeated segment: The inference process is segmented, the similarity between each two segments is determined based on a distance algorithm, and the segments with a similarity greater than a threshold value are determined as repeated segments and the position index of each repeated segment in the inference process is recorded; The deviation value between the length of the inference process and the length of the corresponding reference inference process is determined; The deviation value is multiplied by the penalty weight of each repeated segment to determine the repeated penalty value of the repeated segment; wherein each repeated segment is applied with a penalty weight based on its position index, and the penalty weight corresponding to the later position is greater.

[0072] Further, the model training module 220 is configured to calculate a reward value of the answer generated by sampling the sample question: The answer generated by sampling the sample question is semantically scored based on a scoring model to determine a reward value; Or, the answer and the corresponding sample answer are processed by multi-dimensional matching, and the reward value is determined based on the matching result.

[0073] Further, the loss function construction module 230 determines the advantage value by the following formula:

[0074] Wherein, is the advantage value of the current sampling, a is the sample answer, is the i th sampling output corresponding to the sample question, the content of the sampling output includes the complete inference process and the answer, G is the total number of samplings, is the reward function, is the reward value of is the length of , n , t is the th token of t , is the repeated penalty value of , mean is the mean value calculation, std is calculated as a standard deviation.

[0075] Further, as shown in Figure 3 , the question and answer model training device 200 further comprises a length adjustment module 250, which is configured to: dynamically adjust the maximum length limit of the inference process generated by the question and answer model based on the task difficulty of the sample question.

[0076] The question and answer model training device provided in the embodiments of the present application comprises: an acquisition module configured to acquire a plurality of training data; wherein the training data comprises a sample question, a reference inference process and a sample answer; a model training module configured to, in a training process of a question and answer model according to the sample question, perform repeated segment identification on an inference process generated by multiple samplings of the sample question to determine a repeated penalty value of a repeated segment, and perform reward value calculation on an answer generated by sampling of the sample question; wherein the question and answer model is an artificial intelligence model, and the task of the question and answer model is to answer a question asked by a user; a loss function construction module configured to fuse the repeated penalty value and the reward value to form a reward signal, perform normalization processing on the reward signal to obtain an advantage value of a current sampling, and introduce the advantage value of the current sampling into a grouping strategy optimization algorithm to construct a loss function; and a model parameter adjustment module configured to determine a gradient of the loss function on model parameters of the question and answer model, update the model parameters through gradient back propagation to inhibit a repeated thinking behavior of the question and answer model in an inference process, and continue to iteratively train the question and answer model after updating. By introducing a penalty term of a repeated thinking process into advantage value calculation, unnecessary repeated thinking of the question and answer model can be effectively reduced, limited token space is used to explore other solution ideas, and more attention is paid to "effective thinking" in the training process, thereby improving the ability of the question and answer model to solve complex problems.

[0077] Please refer to Figure 4 , Figure 4 The electronic device provided in the embodiments of the present application is shown in a structural schematic diagram. As shown in Figure 4 , the electronic device 400 comprises a processor 410, a memory 420 and a bus 430.

[0078] The memory 420 stores machine readable instructions executable by the processor 410, and when the electronic device 400 is running, the processor 410 and the memory 420 communicate through the bus 430. The machine readable instructions are executed by the processor 410, and can execute the steps of the question and answer model training method in the method embodiment shown in Figure 1 . For specific implementation manners, please refer to the method embodiments, which will not be described here.

[0079] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program can execute the method as described above when the computer program is run by a processor. Figure 1 The steps of the training method of the question and answer model in the method embodiment are not repeated here, and the specific implementation can be referred to the method embodiment.

[0080] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0081] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and other division manners can be used in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0082] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment.

[0083] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0084] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application essentially or the parts of the prior art that make contributions or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0085] Finally, it should be noted that: the above-described embodiments are merely specific embodiments of the present application, used to illustrate the technical solutions of the present application, and not to limit them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.< / eos>

Claims

1. A training method for a question-answering model, characterized in that, The training method comprises: obtaining a plurality of training data; wherein the training data comprises a sample question, a reference reasoning process and a sample answer; during the training of the question and answer model according to the sample question, repeatedly identifying repeated fragments of the reasoning process generated by multiple samplings of the sample question to determine the repeated penalty value of the repeated fragments, and calculating the reward value of the answer generated by sampling the sample question; wherein the question and answer model is an artificial intelligence model, and the task of the question and answer model is to answer the question asked by the user; fusing the repeated penalty value and the reward value to form a reward signal, normalizing the reward signal to obtain the advantage value of the current sampling, and introducing the advantage value of the current sampling into the grouping strategy optimization algorithm to construct a loss function; determining the gradient of the loss function to the model parameters of the question and answer model, updating the model parameters through gradient back propagation to suppress the repeated thinking behavior of the question and answer model in the reasoning process, and continuing to iteratively train the updated question and answer model.

2. The training method of claim 1, wherein, The training data is determined by the following steps: for each initial sample question, inputting the initial sample question into a plurality of closed source models to output an initial answer and an initial reasoning process for each initial sample question, verifying whether the initial answer is consistent with the standard answer of the initial sample question, and if so, selecting the shortest initial reasoning process from the initial reasoning processes with consistent answers as the reference reasoning process of the initial sample question; based on the multiple sampling results of the same initial sample question by the plurality of closed source models, determining the task difficulty of each initial sample question; based on the task difficulty of the initial sample question, performing question screening processing on a plurality of initial sample questions to determine the training data.

3. The training method of claim 2, wherein, The task difficulty of each initial sample question is determined based on the multiple sampling results of the same initial sample question by the plurality of closed source models, comprising: calculating the probability of at least one sampling generating a correct answer for each initial sample question; based on the probability of the correct answer of each initial sample question, determining the task difficulty of each initial sample question.

4. The training method of claim 1, wherein, The repeated penalty value of the repeated fragments of the reasoning process generated by sampling the sample question is determined by repeatedly identifying the repeated fragments, comprising: segmenting the reasoning process, determining the similarity between each two segments based on a distance algorithm, determining the repeated fragments with a similarity greater than a threshold value and recording the position index of each repeated fragment in the reasoning process; determining the deviation value between the length of the reasoning process and the length of the corresponding reference reasoning process; multiplying the deviation value by the penalty weight of each repeated fragment to determine the repeated penalty value of the repeated fragment; wherein each repeated fragment is subjected to a penalty weight based on its position index, and the later the position, the greater the corresponding penalty weight.

5. The training method of claim 1, wherein, The reward value calculation of the answer generated by sampling the sample question comprises: The answer generated by sampling the sample question based on the scoring model is semantically scored to determine a reward value. Alternatively, the answer and the corresponding sample answer are subjected to multi-dimensional matching processing, and the reward value is determined based on the matching result.

6. The training method of claim 1, wherein, The advantage value is determined by the following formula: wherein, is the advantage value of the current sample, a is the sample answer, is the first i sample output corresponding to the sample question, the content of the sample output includes the complete inference process and the answer, G is the total number of sampling, is the reward function, is the reward value of is the length of n , t is the first t token of is the repetition penalty value of mean is the mean calculation, std is the standard deviation calculation.

7. The training method of claim 2, wherein, After determining the task difficulty of each initial sample question based on the multiple sampling results of the same initial sample question by the multiple closed-source models, the training method further comprises: Dynamically adjusting the maximum length limit of the reasoning process generated by the question and answer model based on the task difficulty of the sample question.

8. A training device for a question-answering model, characterized in that, The training device comprises: An acquisition module configured to acquire multiple training data; wherein the training data comprises sample questions, reference reasoning processes, and sample answers; A model training module configured to, in a training process of a question and answer model according to the sample questions, determine a repeated penalty value of a repeated segment of a reasoning process generated by multiple sampling of the sample questions, and calculate a reward value of an answer generated by sampling of the sample questions; wherein the question and answer model is an artificial intelligence model, and the task of the question and answer model is to answer questions raised by users; A loss function construction module configured to fuse the repeated penalty value and the reward value to form a reward signal, normalize the reward signal to obtain an advantage value of the current sampling, and introduce the advantage value of the current sampling into a grouping strategy optimization algorithm to construct a loss function; A model parameter adjustment module configured to determine a gradient of the loss function to model parameters of the question and answer model, update the model parameters through gradient backpropagation to suppress repeated thinking behavior of the question and answer model in the reasoning process, and continue to iteratively train the updated question and answer model.

9. An electronic device, comprising: It comprises: A processor, a memory, and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to perform the steps of the question and answer model training method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the question and answer model training method according to any one of claims 1 to 7.

Images