Inference model training method and device, inference problem processing method and device

Abstract

This disclosure provides a training method and apparatus for a reasoning model, and a processing method and apparatus for reasoning problems, relating to the field of artificial intelligence, specifically to the fields of natural language processing, large-scale models, and deep learning. The method involves obtaining a training sample set, wherein the training samples in the training sample set include at least sample reasoning problems, corresponding reference reasoning chains, and reference answers. Based on a first training sample in the training sample set, a pre-trained model is subjected to supervised fine-tuning (SFT) to obtain a fine-tuned first reasoning model. Based on a second training sample in the training sample set, the first reasoning model is subjected to reinforcement learning (RL) training to obtain a second reasoning model, thereby improving the accuracy and stability of the second large-scale reasoning model during the reasoning process.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, specifically to the fields of natural language processing, large models and deep learning technologies, and particularly to a method and apparatus for training a reasoning model and a method and apparatus for processing reasoning problems. Background Technology

[0002] In related technologies, the reasoning ability of large language models (LLMs) is often improved through methods such as supervised fine-tuning (SFT) and chain of thought (CoT). Although these methods improve the reasoning ability of LLMs to a certain extent, they also have problems such as unstable activation of reasoning ability, dependence on prompting engineering, and poor adaptability, resulting in poor accuracy and stability in the LLM reasoning process. Summary of the Invention

[0003] This disclosure provides a method for training a reasoning model, a method for processing reasoning problems, a training device for a reasoning model, a device for processing reasoning problems, an electronic device, a storage medium, and a computer program product.

[0004] According to a first aspect of this disclosure, a method for training a reasoning model is provided, comprising: acquiring a training sample set, wherein the training samples in the training sample set include at least a sample reasoning problem, a reference reasoning chain corresponding to the sample reasoning problem, and a reference answer; performing supervised fine-tuning (SFT) on a pre-trained model based on a first training sample in the training sample set to obtain a fine-tuned first reasoning model; and performing reinforcement learning (RL) training on the first reasoning model based on a second training sample in the training sample set to obtain a second reasoning model.

[0005] According to a second aspect of this disclosure, a method for processing reasoning problems is provided, comprising: receiving a target problem to be reasoned and inputting the target problem into a second reasoning model; generating one or more candidate reasoning chains for the target problem through the second reasoning model, and reasoning about the target problem based on the candidate reasoning chains to obtain a target answer to the target problem; wherein the second reasoning model is a model trained using the training method described in the first aspect.

[0006] According to a third aspect of this disclosure, a training apparatus for a reasoning model is provided, comprising: an acquisition module for acquiring a training sample set, wherein the training samples in the training sample set include at least a sample reasoning problem, a reference reasoning chain corresponding to the sample reasoning problem, and a reference answer; a first training module for performing supervised fine-tuning (SFT) on a pre-trained model based on a first training sample in the training sample set to obtain a fine-tuned first reasoning model; and a second training module for performing reinforcement learning (RL) training on the first reasoning model based on a second training sample in the training sample set to obtain a second reasoning model.

[0007] According to a fourth aspect of this disclosure, a processing apparatus for reasoning problems is provided, comprising: a receiving module for receiving a target problem to be reasoned and inputting the target problem into a second reasoning model; and a reasoning module for generating one or more candidate reasoning chains for the target problem through the second reasoning model and reasoning about the target problem based on the candidate reasoning chains to obtain a target answer to the target problem; wherein the second reasoning model is a model trained using the training method described in the first aspect.

[0008] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a training method for a reasoning model according to a first aspect of this disclosure or a processing method for a reasoning problem according to a second aspect.

[0009] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to execute the training method for the inference model described in the first aspect of this disclosure or the processing method for the inference problem described in the second aspect.

[0010] According to a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements a training method for a reasoning model according to a first aspect of this disclosure or a processing method for a reasoning problem according to a second aspect.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is a flowchart illustrating a training method for an inference model according to an embodiment of the present disclosure.

[0014] Figure 2 This is a flowchart illustrating a training method for an inference model according to an embodiment of the present disclosure.

[0015] Figure 3 This is a flowchart illustrating a method for processing reasoning problems according to an embodiment of this disclosure;

[0016] Figure 4 This is a flowchart illustrating a method for training a reasoning model and a method for processing reasoning problems according to an embodiment of this disclosure.

[0017] Figure 5 This is a schematic diagram of the structure of a training device for a reasoning model according to an embodiment of the present disclosure;

[0018] Figure 6 This is a schematic diagram of the structure of a reasoning problem processing apparatus according to an embodiment of the present disclosure;

[0019] Figure 7 This is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] The following is a brief description of the technical field involved in the solution disclosed herein:

[0022] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. AI hardware technologies generally include computer vision, speech recognition, natural language processing, and related technologies such as deep learning, big data processing, and knowledge graphs.

[0023] Natural Language Processing (NLP) is an important research direction in the field of artificial intelligence. It integrates knowledge from multiple disciplines such as linguistics, computer science, machine learning, mathematics, and cognitive psychology. It is an interdisciplinary field that combines computer science, artificial intelligence, and linguistics, encompassing two main aspects: natural language understanding and natural language generation. Its research content includes multiple levels such as characters, words, phrases, sentences, paragraphs, and texts, serving as a bridge between machine language and human language. Its aim is to enable machines to understand, interpret, and generate human language, achieving effective communication between humans and machines, and enabling computers to perform tasks such as language translation, sentiment analysis, and text summarization.

[0024] Large models refer to machine learning models with a large number of parameters and high complexity. They require a lot of computing resources and storage space for training and storage, and often require distributed computing and special hardware acceleration technologies. Large models have stronger generalization and expressive capabilities.

[0025] Deep learning (DL) is a new research direction in the field of machine learning (ML), bringing it closer to its original goal—artificial intelligence. Deep learning learns the inherent laws and hierarchical representations of sample data; the information gained during this learning process greatly aids in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to possess analytical and learning capabilities like humans, capable of recognizing data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition far exceeding previous related technologies.

[0026] It should be noted that in related technologies, the reasoning ability of LLM is often improved through methods such as Supervised Fine Tuning (SFT) and Chain of Thought (CoT).

[0027] Supervised fine-tuning of SFT is a traditional and widely used technique that introduces a fine-tuning process with labeled data on top of an existing pre-trained language model to make the model perform more accurately on a specific task.

[0028] Supervised fine-tuning of SFT typically involves the following steps: Training data preparation: By collecting or generating high-quality datasets containing detailed reasoning steps, these datasets include not only the question and the final answer, but also the intermediate reasoning steps. Each reasoning step must be logically coherent with other steps and correctly derive the final answer. Fine-tuning objective: Using this data labeled with the reasoning process, the model is fine-tuned so that the model learns the overall pattern from the question to the reasoning steps to the answer. Specifically, the goal of fine-tuning is to minimize the cross-entropy loss between the inference step sequence and the answer sequence generated by the model and the labeled data. However, the above approach has the following drawbacks: (1) It depends on the number of high-quality labels, that is, SFT requires a large amount of data labeled with detailed reasoning steps, which often requires manual labeling or relies on high-quality generated data in practice, which increases the training cost and the difficulty of data acquisition. (2) Poor adaptability, that is, the fine-tuned model may only perform well on specific tasks and has poor adaptability to different question types or new tasks. (3) Over-reliance on manual intervention, that is, in many cases, especially in zero-shot learning, the model still needs to be designed and prompted manually, which limits its application in a wider range of tasks.

[0029] Chain reasoning (CoT) is a method that has been widely used in large language model reasoning in recent years. The core idea of ​​CoT is to break down complex reasoning tasks into a series of simple intermediate steps, and then reach the final reasoning result through step-by-step thinking (i.e., chain reasoning).

[0030] The implementation of CoT can be divided into the following aspects: Generating intermediate inference steps: During the inference process, the model not only generates the final answer but also a series of intermediate inference steps. These steps demonstrate the model's thought process, making the inference process more transparent and easier to understand. Step-by-step inference: The basic strategy of CoT is to guide the model to unfold the inference step by step, generating an inference node at each step (for example, in solving a math problem, first derive the formula, then substitute the values, and finally draw a conclusion). These steps are derived by the model step by step based on known information, and each step should be consistent with the previous inference results. Hint templates: In practical applications, CoT usually relies on carefully designed hint templates to stimulate the model to perform chained inference. For example, given a math problem, the hint template could be "Please derive the answer step by step, starting with…". These prompts can help the model generate the reasoning process, but this method still relies on specific prompt design. However, the above method has the following defects: (1) It depends on prompt engineering. Although CoT can effectively promote the model to make step-by-step reasoning, its reasoning ability still depends heavily on the design of the prompts. If the prompts are not appropriate, the model may not be able to successfully carry out the reasoning, resulting in the inability to guarantee the accuracy and consistency of the reasoning results. (2) It has a large computational cost. Since each reasoning requires the model to generate multiple reasoning steps, compared with directly generating the answer, the CoT method will increase the additional computational cost. Especially when facing complex problems, the reasoning process may become very long. (3) The training phase is insufficient. The existing CoT method mainly relies on prompt engineering to guide the reasoning. However, during the training process, the model does not specifically learn how to automatically expand the reasoning chain. Therefore, although the prompts can activate the reasoning ability to a certain extent, the model still lacks the inherent and automatic reasoning ability.

[0031] In summary, related technologies often rely excessively on prompts. Both SFT and CoT depend to varying degrees on manually designed prompts or data, resulting in a lack of broad adaptability and flexibility in their reasoning capabilities. Furthermore, their reasoning depth and complexity are limited. Current technologies still struggle to enable models to autonomously and deeply engage in multi-step reasoning, especially in complex tasks. The depth of the model's reasoning and the accuracy of its inference results cannot be effectively guaranteed. The inference ability has not been systematically internalized during the training phase, causing the model to passively rely on external prompts when faced with new tasks, unable to proactively engage in complex reasoning.

[0032] The following describes a method for training a reasoning model and a method for processing reasoning problems according to embodiments of the present disclosure, with reference to the accompanying drawings.

[0033] Figure 1This is a flowchart illustrating a training method for an inference model according to an embodiment of the present disclosure. It should be noted that the execution entity of the training method for the inference model in this embodiment is a training device for the inference model, which can specifically be a hardware device or software within a hardware device. The hardware device can be, for example, a terminal device or a server.

[0034] like Figure 1 As shown, the training method for the inference model proposed in this embodiment includes the following steps:

[0035] S101. Obtain a training sample set. The training samples in the training sample set shall include at least the sample reasoning problem, the reference reasoning chain corresponding to the sample reasoning problem, and the reference answer.

[0036] It should be noted that, in order to give the reasoning model stronger logical reasoning ability, the sample reasoning problem can be of the following types: multi-step reasoning, causal reasoning, logical reasoning (such as inductive reasoning and deductive reasoning).

[0037] For example, sample reasoning question 1 could be "What is 5+3?", and sample reasoning question 2 could be "If X increases, how will Y change?".

[0038] In this embodiment of the disclosure, after obtaining the sample reasoning question, training samples can be determined based on the sample reasoning question, the corresponding reference reasoning chain, and the reference answer.

[0039] For example, for training sample 1, the sample reasoning question is "What is 5+3?", the corresponding reference reasoning chain is "5+3=8", and the reference answer is "8".

[0040] It should be noted that the training sample can be in the format X→Z=(z1,z2,z3…zk)→a, where X is the sample reasoning problem, Z=(z1,z2,z3…zk) is the reference reasoning chain, z1,z2,z3…zk are the steps of the reference reasoning chain, and A is the reference answer.

[0041] S102. Based on the first training sample in the training sample set, perform supervised fine-tuning SFT on the pre-trained model to obtain the fine-tuned first inference model.

[0042] Optionally, X training samples can be randomly selected from the training sample set as the first training samples.

[0043] The goal of supervised fine-tuning of SFT is to enable the first inference model to generate the correct inference chain and answer corresponding to the inference question.

[0044] In this embodiment of the disclosure, a supervised fine-tuning SFT can be performed on the pre-trained model based on the first training sample, so that the first inference model can generate the correct inference chain and answer corresponding to the inference question. The loss function of the pre-trained model during the supervised fine-tuning SFT process is obtained, and it is determined whether the pre-trained model meets the fine-tuning termination condition. In response to the pre-trained model meeting the fine-tuning termination condition, the fine-tuned first inference model is obtained.

[0045] S103. Based on the second training sample in the training sample set, perform reinforcement learning (RL) training on the first inference model to obtain the second inference model.

[0046] It should be noted that although supervised fine-tuning can enable the first inference model to generate the correct inference chain corresponding to the inference problem, it may not be able to handle all situations when faced with more complex inference tasks. Therefore, reinforcement learning (RL) training can be performed on the first inference model to further improve its inference ability and obtain the second inference model.

[0047] Optionally, the training samples in the training sample set other than the first training samples can be used as the second training samples; alternatively, Y training samples can be randomly selected from the first training samples, and the training samples in the training sample set other than the first training samples and the Y training samples can be used together as the second training samples.

[0048] In this embodiment of the disclosure, a second training sample can be input into the first inference model. The first inference model predicts the second inference chain and the second answer corresponding to the sample inference question. Based on the reference inference chain and the reference answer, as well as the second inference chain and the second answer, the reward value of the first inference model is determined. With the maximization of the reward value as the optimization objective, the model parameters of the first inference model are adjusted until the RL training meets the termination condition, and the second inference model is obtained.

[0049] According to the training method of the inference model in the embodiments of this disclosure, a training sample set is obtained, wherein the training samples in the training sample set include at least a sample inference problem, a reference inference chain corresponding to the sample inference problem, and a reference answer. Based on the first training sample in the training sample set, a pre-trained model is subjected to supervised fine-tuning (SFT) to obtain a fine-tuned first inference model. Based on the second training sample in the training sample set, the first inference model is subjected to reinforcement learning (RL) training to obtain a second inference model. Thus, by combining supervised fine-tuning, reinforcement learning, and thought chains, this disclosure effectively improves the reasoning ability of the second inference model in complex inference tasks, eliminates excessive reliance on prompt templates, and enables the second inference model to automatically carry out an efficient and reliable reasoning process without complex prompts, thereby improving the accuracy and stability of the second inference model in the reasoning process.

[0050] Figure 2 It is a schematic flowchart of a training method for an inference model according to an embodiment of the present disclosure.

[0051] As Figure 2 shown, the training method for the inference model proposed in this embodiment includes the following steps:

[0052] S201. Obtain a training sample set, where the training samples in the training sample set at least include sample inference questions, reference inference chains corresponding to the sample inference questions, and reference answers.

[0053] For the relevant content of step S201, reference may be made to the above embodiment, which will not be elaborated here.

[0054] S102 in the above embodiment, "Based on the first training sample in the training sample set, perform supervised fine-tuning SFT on the pre-trained model to obtain the first inference model after fine-tuning" may specifically include S202 - S204.

[0055] S202. Input the first training sample into the pre-trained model, and the pre-trained model predicts the first inference chain and the first answer corresponding to the sample inference question.

[0056] S203. Determine the loss function of the pre-trained model according to the reference inference chain and the reference answer, as well as the first inference chain and the first answer.

[0057] In the embodiment of the present disclosure, after obtaining the first inference chain and the first answer, the loss function of the pre-trained model can be determined according to the reference inference chain and the reference answer, as well as the first inference chain and the first answer. The expression of the loss function can be:

[0058]

[0059] where, L SFT [[ID=^{33}]](θ) is the loss function of the inference model, θ is the model parameter of the pre-trained model, X is the sample inference question, Z is the reference inference chain, A is the reference answer, D is the distribution of the first training sample, Y is the concatenation of the first inference chain and the first answer, y t is the currently generated token (inference step of the first inference chain or the first answer), P θ (y t ∣X,y<t) is the conditional probability that the pre-trained model generates the inference step of the first inference chain or the first answer at position t.

[0060] S204. Under the constraint of minimizing the loss function, adjust the model parameters of the pre-trained model until the SFT training meets the end condition to obtain the first inference model.

[0061] In this embodiment of the disclosure, after obtaining the loss function of the pre-trained model, under the constraint of minimizing the loss function, the model parameters of the pre-trained model can be continuously adjusted according to the loss function until the SFT training meets the termination condition, and the pre-trained model after the last adjustment of the model parameters is determined as the first inference model after training is completed.

[0062] Optionally, the model parameters of the pre-trained model can be adjusted based on a standard optimization algorithm, such as the Adam optimizer, and the pre-trained model with adjusted parameters can be trained to minimize the loss function. This allows the pre-trained model to learn a stepwise inference strategy until the loss function is minimized, thus determining that the SFT training meets the termination condition and obtaining the first inference model.

[0063] It should be noted that after obtaining the first reasoning model, the first reasoning model can generate a suitable reasoning chain based on the reasoning question and finally arrive at the correct answer.

[0064] S103 in the above embodiment, "Based on the second training sample in the training sample set, perform reinforcement learning (RL) training on the first inference model to obtain the second inference model", may specifically include S205-S207.

[0065] S205. Input the second training sample into the first inference model, and the first inference model predicts the second inference chain and the second answer corresponding to the sample inference question.

[0066] S206. Based on the reference reasoning chain and the reference answer, as well as the second reasoning chain and the second answer, determine the reward value of the first reasoning model.

[0067] The reward value of the first inference model can be used to evaluate the quality of the predicted second inference chain and the second answer. The reward value is the sum of reward values ​​from multiple dimensions.

[0068] In this embodiment of the disclosure, the first reward value Rans of the answer can be determined based on the reference answer and the second answer, the second reward value Rproc and the third reward value Rlength of the inference chain can be determined based on the reference inference chain and the second inference chain, and the reward value R(Y) can be determined based on the first reward value, the second reward value and the third reward value.

[0069] Optionally, after obtaining the first reward value, the second reward value, and the third reward value, the reward value R(Y) of the first inference model can be determined according to the following formula:

[0070] R(Y) = Rans + αRproc + βRlength

[0071] Where R(Y) is the reward value, Rans is the first reward value, Rproc is the second reward value, α is the hyperparameter of the second reward value, Rlength is the third reward value, and β is the hyperparameter of the third reward value, which are used to control the weights of the second and third reward values ​​in the reward value of the first inference model.

[0072] In this embodiment of the disclosure, after obtaining the second answer, the first reward value Rans of the answer can be determined based on the degree of matching between the reference answer and the second answer.

[0073] For example, if the second answer matches the reference answer exactly, a positive reward value is given; if the second answer matches the reference answer partially, a partial positive reward value can be given according to the degree of matching and a pre-built strategy; if the second answer does not match the reference answer at all, a negative reward value is given.

[0074] In this embodiment of the disclosure, a first step sequence of a reference inference chain is determined, a second step sequence of a second inference chain is determined, the first step sequence and the second step sequence are matched, logically abnormal steps and logically correct steps are determined from the second step sequence, and reward processing is performed on the logically abnormal steps and the logically correct steps respectively to obtain their respective reward values, and a second reward value is determined based on their respective reward values.

[0075] Optionally, the first step sequence and the second step sequence can be matched. From the second step sequence, steps with logical errors and steps with logical correctness can be identified. Steps with logical errors are given negative reward values, and steps with logical correctness are given positive reward values. Based on their respective reward values, a second reward value is determined.

[0076] In this embodiment of the disclosure, in order to train and generate a concise and effective inference chain, redundant steps can be determined from the second step sequence of the second inference chain based on the first step sequence of the reference inference chain, and the redundant steps can be rewarded to obtain the corresponding reward value. Then, a third reward value can be determined based on the corresponding reward value.

[0077] Optionally, redundant steps (irrelevant and redundant steps) can be identified from the second step sequence of the second inference chain based on the first step sequence of the reference inference chain, and negative rewards can be given to the redundant steps.

[0078] For example, the first information entropy of a step in the first step sequence of a reference inference chain and the second information entropy of a step in the second step sequence of a second inference chain can be obtained. For the same step, irrelevant steps can be determined from the second step sequence of the second inference chain based on the corresponding first and second information entropies. Redundant steps can be determined from the second step sequence of the second inference chain based on a pre-trained expert small model and the first step sequence of the reference inference chain.

[0079] S207. With the goal of maximizing the reward value, adjust the model parameters of the first inference model until the RL training meets the termination condition to obtain the second inference model.

[0080] Optionally, a reinforcement learning algorithm can be used to adjust the model parameters of the first inference model with the goal of maximizing the reward value, and continue training the first inference model after parameter adjustment until the RL training meets the termination condition to obtain the second inference model.

[0081] It should be noted that this disclosure does not limit the type of reinforcement learning algorithm. For example, the reinforcement learning algorithm can be the policy gradient method (Proximal Policy Optimization, or PPO for short).

[0082] It should be noted that the training method of the reasoning model proposed in this disclosure can internalize logical reasoning ability into the basic skills of the second reasoning model. The trained second reasoning model can easily cope with problems in different fields, such as the medical field, the legal field, and the engineering field. It can be applied across fields and can be transferred from one reasoning task to another. It can quickly adapt to new reasoning scenarios and improve the flexibility and efficiency of the reasoning model in the reasoning process.

[0083] According to the training method of the inference model according to the embodiments of this disclosure, a training sample set is obtained, wherein the training samples in the training sample set include at least a sample inference problem, a reference inference chain corresponding to the sample inference problem, and a reference answer. A first training sample is input into a pre-trained model, and the pre-trained model predicts the first inference chain and the first answer corresponding to the sample inference problem. Based on the reference inference chain and the reference answer, as well as the first inference chain and the first answer, the loss function of the pre-trained model is determined. Under the constraint of minimizing the loss function, the model parameters of the pre-trained model are adjusted until the SFT training meets the termination condition, thus obtaining a first inference model. A second training sample is input into the first inference model, and the first inference model predicts the first inference chain and the first answer corresponding to the sample inference problem. The second inference chain and the second answer are used to determine the reward value of the first inference model based on the reference inference chain and the reference answer, as well as the second inference chain and the second answer. The model parameters of the first inference model are adjusted with the goal of maximizing the reward value, until the RL training meets the termination condition, thus obtaining the second inference model. Therefore, this disclosure obtains the first inference model by adjusting the model parameters of the pre-trained model until the SFT training meets the termination condition, enabling the first inference model to learn the correct inference chain and answer, and by adjusting the model parameters of the first inference model with the goal of maximizing the reward value, until the RL training meets the termination condition, thus obtaining the second inference model. This greatly improves the reasoning ability of the second inference model and enhances the efficiency and stability of the second inference model in reasoning. It can be applied to complex reasoning tasks, cross-domain applications, and multi-step reasoning tasks.

[0084] Figure 3 This is a flowchart illustrating a method for processing reasoning problems according to an embodiment of this disclosure. It should be noted that the execution entity of this method is a reasoning problem processing device, which can specifically be a hardware device or software within a hardware device. The hardware device can be, for example, a terminal device or a server.

[0085] like Figure 3 As shown, the method for handling reasoning problems proposed in this embodiment includes the following steps:

[0086] S301. Receive the target question to be reasoned and input the target question into the second reasoning model.

[0087] The target problem can be any problem to be reasoned about.

[0088] It should be noted that when the target problem is input into the second reasoning model, the second reasoning model analyzes the target problem. According to the reasoning strategy learned in the training stage, if multi-step reasoning is required, the second reasoning model automatically starts the process of the reasoning chain.

[0089] S302. Generate one or more candidate reasoning chains for the target problem through the second reasoning model, and perform reasoning on the target problem based on the candidate reasoning chains to obtain the target answer of the target problem.

[0090] It should be noted that in order to improve the reliability and stability in the process of reasoning problems, a self-consistency strategy is adopted, that is, for the same target problem, multiple different candidate reasoning chains Z(1), Z(2),..., Z(M) are generated, and the target answer of the target problem is determined from the candidate answers corresponding to the candidate reasoning chains.

[0091] Optionally, for the same target problem, the second reasoning model generates multiple reasoning chains Z(1), Z(2),..., Z(N) from different random seeds or sampling strategies.

[0092] In the embodiments of the present disclosure, for the i-th step in any candidate reasoning chain, determine the sequence of previous steps of the candidate reasoning chain that has been generated by the second reasoning model currently, where i is an integer and 2 ≤ i ≤ M, and M is the number of steps in the candidate reasoning chain. Generate the i-th step in the candidate reasoning chain according to the target problem and the sequence of previous steps until the candidate reasoning chain is output.

[0093] It should be noted that the second reasoning model generates a candidate reasoning chain Z = (z1, z2, z3…zk) according to the input target problem Q, and each reasoning step is generated based on the previous step and the conditions of the target problem Q.

[0094] Optionally, for the target problem Q, the conditional probability of generating a candidate reasoning chain is:

[0095]

[0096] where, P θ (Z∣Q) is the conditional probability of generating the candidate reasoning chain Z, and P θ (Z i ∣Q,z<i) is the conditional probability of generating the i-th reasoning zi.

[0097] It should be noted that when each step is generated, the second reasoning model generates the next step according to the currently generated step and the target problem Q until the complete candidate reasoning chain is output.

[0098] In this embodiment of the disclosure, after obtaining the candidate inference chain, the target question can be reasoned based on the candidate inference chain to obtain the candidate answer corresponding to the candidate inference chain. Consistency voting is performed on the candidate answer, and the candidate answer with the most votes is determined from the candidate answers based on the voting results as the target answer.

[0099] For example, after obtaining the candidate answers A(1), A(2), ..., A(M) corresponding to the candidate inference chain, the candidate answers corresponding to the candidate inference chain can be evaluated through a self-consistent voting mechanism to obtain the voting results and determine the candidate answer with the most votes as the target answer.

[0100] The second inference model is the inference model obtained using the training method of the first aspect.

[0101] Optionally, after obtaining the target answer to the target question, a correction instruction message for the target answer is received. In response to the correction instruction message, the candidate inference chain with the highest confidence is selected from the candidate inference chains, and the candidate answer corresponding to the candidate inference chain with the highest confidence is determined as the target answer.

[0102] It should be noted that when facing problems with high uncertainty, traditional large language models are prone to generating incorrect answers due to a single reasoning failure. This disclosure can generate multiple reasoning chains through a self-consistency strategy and select the most likely answer through voting, thereby improving the stability of the reasoning results. For example, in complex reasoning tasks, when inconsistencies occur in the reasoning chains, the model can effectively filter out erroneous reasoning chains through multiple sampling and verification, thereby reducing the possibility of errors in the second reasoning model. When the input problem is noisy or ambiguous, the second reasoning model can generate a more stable reasoning process through self-correction and multiple verifications, thereby significantly reducing the sensitivity to input details and improving the robustness of the target answer.

[0103] It should be noted that after obtaining the target answer to the target question, if the target answer is inconsistent with the most common answer, a correction instruction message for the target answer will be generated.

[0104] Optionally, after obtaining the target answer to the target question, in order to improve the interpretability of the reasoning process and increase the transparency of the reasoning process, candidate reasoning chains associated with the target answer are obtained as the target reasoning chain and displayed; or, the candidate reasoning chain with the highest confidence among the candidate reasoning chains is determined as the target reasoning chain and displayed.

[0105] For example, in fields requiring high credibility, such as law, medicine, and finance, visualization technology can be used to display the target reasoning chain. Users or experts can examine the reasoning process of the second reasoning model based on the displayed target reasoning chain, thereby confirming the rationality and accuracy of the target answer.

[0106] According to the reasoning problem processing method of the present disclosure, a target problem to be reasoned is received and input into a second reasoning model. The second reasoning model generates one or more candidate reasoning chains for the target problem, and reasoning is performed on the target problem based on the candidate reasoning chains to obtain the target answer. Thus, the present disclosure generates one or more candidate reasoning chains for the target problem through the second reasoning model and reasons on the target problem based on the candidate reasoning chains to obtain the target answer. This eliminates the excessive reliance on prompt templates. The second reasoning model can autonomously carry out an efficient and reliable reasoning process without complex prompts, significantly improving the accuracy and reliability of the target answer.

[0107] The following explains the specific process of training the reasoning model and processing the reasoning problem proposed in this disclosure.

[0108] For example, such as Figure 4 As shown, for the training phase of the inference model, a training sample set is obtained. The training samples in the training sample set include at least a sample inference question, a reference inference chain corresponding to the sample inference question, and a reference answer. Based on the first training sample in the training sample set, the pre-trained model is subjected to supervised fine-tuning of SFT with the goal of minimizing the inference chain generation loss, resulting in the fine-tuned first inference model. Based on the second training sample in the training sample set, the first inference model is subjected to reinforcement learning (RL) training with the goal of maximizing the reward value, resulting in the second inference model. For the inference phase, the target question to be inferred is received and input into the second inference model. Multiple candidate inference chains are generated through a self-consistency strategy, and the candidate answers corresponding to the candidate inference chains are voted on to obtain the target answer to the target question.

[0109] In summary, the training method for the reasoning model and the method for handling reasoning problems proposed in this disclosure (1) enhance the logical reasoning ability of the second reasoning model. That is, through the design of the training phase and the reasoning (activation) phase, the second reasoning model has a stronger logical reasoning ability. Especially when facing complex tasks such as multi-step reasoning, mathematical reasoning, and causal reasoning, by explicitly and gradually learning the reasoning ability in the training phase, the second reasoning model can actively carry out a multi-step reasoning process in the reasoning phase, thereby effectively reducing erroneous reasoning and unfounded reasoning. (2) Improve the reasoning stability and robustness of the second reasoning model. That is, by guiding the model to actively generate multiple reasoning chains in the reasoning process, and by enhancing the reliability of the target answer through methods such as self-consistent voting. (3) Applicable to complex tasks and cross-domain reasoning, improving the model's performance in a single task, and also enabling the model to have cross-domain reasoning ability, which is more widely applicable. It is applicable to various complex tasks and fields. That is, this disclosure can internalize logical reasoning ability into the basic skills of the model. The trained second reasoning model can easily cope with problems in different fields. By systematically cultivating the model's reasoning ability during the training phase, this disclosure enables the second reasoning model to migrate from one task to another and quickly adapt to new reasoning scenarios. It can solve fixed types of tasks and is more flexible and efficient when facing new reasoning problems. (4) It improves the interpretability of the model. While the second reasoning model generates the answer, it can provide the intermediate steps of the reasoning chain, which is especially important for fields that require high credibility. The reasoning process of the model can be checked according to the reasoning chain, so as to ensure the rationality and accuracy of the answer. Combined with visualization technology, each step of the reasoning chain can be displayed to help users understand the model's decision-making process and further improve the trust and acceptance of the model.

[0110] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0111] Corresponding to the training methods of the inference models provided in the above embodiments, one embodiment of this disclosure also provides a training device for the inference model. Since the training device for the inference model provided in this embodiment corresponds to the training methods of the inference models provided in the above embodiments, the implementation methods of the inference model training methods are also applicable to the training device for the inference model provided in this embodiment, and will not be described in detail in this embodiment.

[0112] Figure 5 This is a schematic diagram of the structure of a training device for an inference model according to an embodiment of the present disclosure.

[0113] like Figure 5 As shown, the training device 500 for the inference model includes: an acquisition module 510, a first training module 520, and a second training module 530.

[0114] The acquisition module 510 is used to acquire a training sample set, wherein the training samples in the training sample set include at least a sample reasoning problem, a reference reasoning chain corresponding to the sample reasoning problem, and a reference answer;

[0115] The first training module 520 is used to perform supervised fine-tuning of the pre-trained model SFT based on the first training sample in the training sample set to obtain the fine-tuned first inference model.

[0116] The second training module 530 is used to perform reinforcement learning (RL) training on the first inference model based on the second training samples in the training sample set to obtain the second inference model.

[0117] The second training module 530 is further configured to: input the second training sample into the first inference model, and have the first inference model predict the second inference chain and the second answer corresponding to the sample inference question; determine the reward value of the first inference model based on the reference inference chain and the reference answer, as well as the second inference chain and the second answer; and adjust the model parameters of the first inference model with the goal of maximizing the reward value, until the RL training meets the termination condition, thereby obtaining the second inference model.

[0118] The second training module 530 is further configured to: determine a first reward value for an answer based on the reference answer and the second answer; determine a second reward value and a third reward value for an inference chain based on the reference inference chain and the second inference chain; and determine the reward value based on the first reward value, the second reward value, and the third reward value.

[0119] The second training module 530 is further configured to: determine a first step sequence of the reference inference chain; determine a second step sequence of the second inference chain; match the first step sequence and the second step sequence, and determine logically abnormal steps and logically correct steps from the second step sequence; perform reward processing on the logically abnormal steps and logically correct steps respectively to obtain their respective reward values, and determine the second reward value based on their respective reward values.

[0120] The second training module 530 is further configured to: determine redundant steps from the second step sequence of the second inference chain based on the first step sequence of the reference inference chain; perform reward processing on the redundant steps to obtain corresponding reward values, and determine the third reward value based on the corresponding reward values.

[0121] The first training module 520 is further configured to: input the first training sample into the pre-trained large model, and have the pre-trained large model predict the first inference chain and the first answer corresponding to the sample inference question; determine the loss function of the inference model based on the reference inference chain and the reference answer, as well as the first inference chain and the first answer; and adjust the model parameters of the inference model under the constraint of minimizing the loss function until the SFT training meets the termination condition to obtain the first inference model.

[0122] According to the training apparatus of the inference model of the present disclosure, by acquiring a training sample set, wherein the training samples in the training sample set include at least a sample inference problem, a reference inference chain corresponding to the sample inference problem, and a reference answer, a pre-trained model is subjected to supervised fine-tuning (SFT) based on a first training sample in the training sample set to obtain a fine-tuned first inference model, and a second inference model is obtained by performing reinforcement learning (RL) training on the first inference model based on a second training sample in the training sample set. Thus, by combining supervised fine-tuning, reinforcement learning, and thought chains, the present disclosure effectively improves the reasoning ability of the second inference model in complex reasoning tasks, eliminates excessive reliance on prompt templates, and enables the second inference model to automatically carry out an efficient and reliable reasoning process without complex prompts, thereby improving the accuracy and stability of the second inference model in the reasoning process.

[0123] Corresponding to the reasoning problem processing methods provided in the above embodiments, one embodiment of this disclosure also provides a reasoning problem processing apparatus. Since the reasoning problem processing apparatus provided in this embodiment corresponds to the reasoning problem processing methods provided in the above embodiments, the implementation methods for reasoning problem processing methods are also applicable to the reasoning problem processing apparatus provided in this embodiment, and will not be described in detail in this embodiment.

[0124] Figure 6 This is a schematic diagram of the structure of a reasoning problem processing apparatus according to an embodiment of the present disclosure.

[0125] like Figure 6 As shown, the processing device 600 for the reasoning problem includes a receiving module 610 and a reasoning module 620.

[0126] in:

[0127] The receiving module 610 is used to receive the target question to be reasoned and input the target question into the second reasoning model;

[0128] The reasoning module 620 is used to generate one or more candidate reasoning chains for the target problem through the second reasoning model, and to reason about the target problem based on the candidate reasoning chains to obtain the target answer to the target problem;

[0129] The second inference model is an inference model obtained using the training method described in the first aspect.

[0130] The reasoning module 620 is further configured to: reason about the target question based on the candidate reasoning chain to obtain the candidate answer corresponding to the candidate reasoning chain; perform a consensus vote on the candidate answer, and determine the candidate answer with the most votes from the candidate answers based on the voting results, as the target answer.

[0131] The reasoning module 620 is further configured to: for the i-th step in any candidate reasoning chain, determine the sequence of preceding steps currently generated by the second reasoning model that belongs to the candidate reasoning chain, where i is an integer and 2≤i≤M, and M is the number of steps in the candidate reasoning chain; generate the i-th step in the candidate reasoning chain according to the target problem and the sequence of preceding steps, until the candidate reasoning chain is output.

[0132] After obtaining the target answer to the target question, the device 600 is further configured to: receive correction instruction information for the target answer; in response to the correction instruction information, select the candidate inference chain with the highest confidence from the candidate inference chains; and determine the candidate answer corresponding to the candidate inference chain with the highest confidence as the target answer.

[0133] After obtaining the target answer to the target question, the device 600 is further configured to: acquire the candidate inference chain associated with the target answer as the target inference chain and display the target inference chain; or, determine the candidate inference chain with the highest confidence among the candidate inference chains as the target inference chain and display the target inference chain.

[0134] The reasoning problem processing apparatus according to embodiments of the present disclosure receives a target problem to be reasoned about and inputs the target problem into a second reasoning model. The second reasoning model generates one or more candidate reasoning chains for the target problem, and reasones about the target problem based on these candidate reasoning chains to obtain the target answer. Thus, the present disclosure generates one or more candidate reasoning chains for the target problem through the second reasoning model and reasons about the target problem based on these candidate reasoning chains to obtain the target answer. This eliminates excessive reliance on prompt templates. The second reasoning model can autonomously conduct an efficient and reliable reasoning process without complex prompts, significantly improving the accuracy and reliability of the target answer.

[0135] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0136] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0137] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0138] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0139] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as methods for training inference models or methods for processing inference problems. For example, in some embodiments, methods for training inference models or methods for processing inference problems can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the methods for training inference models or processing inference problems described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured by any other suitable means (e.g., by means of firmware) to perform a method for training an inference model or a method for processing inference problems.

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0144] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0145] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0146] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the training method for the inference model or the processing method for the inference problem as described above.

[0147] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0148] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method of training an inference model, wherein, The method further includes the following steps: Obtain a training sample set, wherein the training samples in the training sample set include at least a sample reasoning question, a reference reasoning chain corresponding to the sample reasoning question, and a reference answer, wherein the sample reasoning question is text data; Based on the first training sample in the training sample set, supervised fine-tuning SFT is performed on the pre-trained model to obtain the fine-tuned first inference model. Based on the second training sample in the training sample set, the first inference model is trained by reinforcement learning (RL) to obtain the second inference model. The step of performing reinforcement learning (RL) training on the first inference model based on the second training samples in the training sample set to obtain the second inference model includes: The second training sample is input into the first inference model, and the first inference model predicts the second inference chain and the second answer corresponding to the inference question of the sample. The reward value of the first reasoning model is determined based on the reference reasoning chain and the reference answer, as well as the second reasoning chain and the second answer; With the goal of maximizing reward value, the model parameters of the first inference model are adjusted until the RL training meets the termination condition, thus obtaining the second inference model. The step of determining the reward value of the first inference model based on the reference inference chain and the reference answer, as well as the second inference chain and the second answer, includes: Based on the reference answer and the second answer, determine the first reward value for the answer; Based on the reference inference chain and the second inference chain, determine the second reward value and the third reward value of the inference chain; The reward value is determined based on the first reward value, the second reward value, and the third reward value.

2. The method of claim 1, wherein, Based on the reference inference chain and the second inference chain, a second reward value for the inference chain is determined, including: Determine the first step sequence of the reference inference chain; Determine the second step sequence of the second inference chain; The first step sequence and the second step sequence are matched, and logically abnormal steps and logically correct steps are determined from the second step sequence. The steps with logical errors and the steps with logical correctness are rewarded separately to obtain their respective reward values, and the second reward value is determined based on the respective reward values.

3. The method of claim 1, wherein, The step of determining the third reward value of the inference chain based on the reference inference chain and the second inference chain includes: Based on the first step sequence of the reference inference chain, determine the redundant steps from the second step sequence of the second inference chain; The redundant steps are processed to obtain corresponding reward values, and the third reward value is determined based on the corresponding reward values.

4. The method according to any one of claims 1-3, wherein, The step of performing supervised fine-tuning of the pre-trained model using a first training sample from the training sample set to obtain a fine-tuned first inference model includes: The first training sample is input into the pre-trained large model, and the pre-trained large model predicts the first reasoning chain and the first answer corresponding to the reasoning question of the sample. Based on the reference inference chain and the reference answer, as well as the first inference chain and the first answer, determine the loss function of the inference model; Under the constraint of minimizing the loss function, the model parameters of the inference model are adjusted until the SFT training meets the termination condition, thus obtaining the first inference model.

5. A method for handling reasoning problems, wherein, The method includes the following steps: Receive the target question to be reasoned and input the target question into the second reasoning model; One or more candidate reasoning chains for the target problem are generated using the second reasoning model, and the target answer for the target problem is obtained based on the candidate reasoning chains. Wherein, the second inference model is the inference model obtained by the training method as described in any one of claims 1-4.

6. The method according to claim 5, wherein, The step of reasoning about the target question based on the candidate reasoning chain to obtain the target answer to the target question includes: Based on the candidate reasoning chain, the target question is reasoned to obtain the candidate answer corresponding to the candidate reasoning chain; A consensus vote is conducted on the candidate answers, and the candidate answer with the most votes is determined from the candidate answers based on the voting results, and is taken as the target answer.

7. The method according to claim 6, wherein, The step of generating one or more candidate reasoning chains for the target problem using the second reasoning model includes: For any candidate inference chain, the first... i The steps include determining the sequence of preceding steps that the second inference model has currently generated and that belong to the candidate inference chain, wherein... i It is an integer, and 2≤ i ≤ M M is the number of steps in the candidate inference chain; Based on the target problem and the sequence of preceding steps, the i-th step in the candidate inference chain is generated until the candidate inference chain is output.

8. The method according to any one of claims 5-7, wherein, After obtaining the target answer to the target question, the process also includes: Receive the correction instruction information for the target answer; In response to the correction instruction information, the candidate inference chain with the highest confidence is selected from the candidate inference chains; The candidate answer corresponding to the candidate inference chain with the highest confidence is determined as the target answer.

9. The method according to any one of claims 5-7, wherein, After obtaining the target answer to the target question, the process also includes: Obtain the candidate reasoning chain associated with the target answer, use it as the target reasoning chain, and display the target reasoning chain; or, The candidate inference chain with the highest confidence among the candidate inference chains is determined as the target inference chain, and the target inference chain is displayed.

10. A training device for a reasoning model, wherein, The device includes: The acquisition module is used to acquire a training sample set, wherein the training samples in the training sample set include at least a sample reasoning question, a reference reasoning chain corresponding to the sample reasoning question, and a reference answer, wherein the sample reasoning question is text data; The first training module is used to perform supervised fine-tuning of the pre-trained model using the first training sample in the training sample set to obtain the fine-tuned first inference model. The second training module is used to perform reinforcement learning (RL) training on the first inference model based on the second training samples in the training sample set to obtain the second inference model. The second training module is used for: The second training sample is input into the first inference model, and the first inference model predicts the second inference chain and the second answer corresponding to the inference question of the sample. The reward value of the first reasoning model is determined based on the reference reasoning chain and the reference answer, as well as the second reasoning chain and the second answer; With the goal of maximizing reward value, the model parameters of the first inference model are adjusted until the RL training meets the termination condition, thus obtaining the second inference model. The second training module is used for: Based on the reference answer and the second answer, determine the first reward value for the answer; Based on the reference inference chain and the second inference chain, determine the second reward value and the third reward value of the inference chain; The reward value is determined based on the first reward value, the second reward value, and the third reward value.

11. A processing apparatus for reasoning problems, wherein, The device includes: A receiving module is used to receive the target question to be reasoned and input the target question into the second reasoning model; The reasoning module is used to generate one or more candidate reasoning chains for the target problem through the second reasoning model, and to reason about the target problem based on the candidate reasoning chains to obtain the target answer to the target problem; Wherein, the second inference model is the inference model obtained by the training method as described in any one of claims 1-4.

12. An electronic device, characterized in that, Including processor and memory; The processor runs a program corresponding to the executable program code stored in the memory to implement the method as described in any one of claims 1-4 or 5-9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-4 or 5-9.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-4 or 5-9.

Images