Method and device for training reasoning model and method and device for processing reasoning problem

Abstract

The invention provides a training method and device of an inference model and a processing method and device of an inference problem, and relates to the field of artificial intelligence, in particular to the technical field of natural language processing, large models and deep learning. Obtaining a training sample set, wherein training samples in the training sample set at least comprise sample reasoning questions, reference reasoning chains corresponding to the sample reasoning questions and reference answers; on the basis of a first training sample in the training sample set, performing supervised fine tuning SFT on a pre-training model to obtain a first inference model after fine tuning; and performing reinforcement learning RL training on the first reasoning model based on the second training sample in the training sample set to obtain a second reasoning model, thereby improving the accuracy and stability of the second reasoning model in the reasoning process.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, specifically to the fields of natural language processing, large models and deep learning technology, and in particular to a method and device for training an inference model, and a method and device for processing inference problems. Background Art

[0002] In related technologies, the reasoning ability of large language models (LLMs) is often improved through methods such as supervised fine tuning (SFT) and chain of thought (CoT). Although the above methods have improved the reasoning ability of LLMs to a certain extent, there are problems such as unstable activation of reasoning ability, dependence on prompt engineering, and poor adaptability, resulting in poor accuracy and stability in the LLM reasoning process. Summary of the Invention

[0003] The present disclosure provides a method for training an inference model, a method for processing an inference problem, a device for training an inference model, a device for processing an inference problem, an electronic device, a storage medium, and a computer program product.

[0004] According to a first aspect of the present disclosure, a method for training an inference model is provided, comprising: obtaining a training sample set, wherein the training samples in the training sample set include at least a sample inference problem, a reference inference chain corresponding to the sample inference problem, and a reference answer; based on a first training sample in the training sample set, performing supervised fine-tuning (SFT) on a pre-trained model to obtain a fine-tuned first inference model; and based on a second training sample in the training sample set, performing reinforcement learning (RL) training on the first inference model to obtain a second inference model.

[0005] According to a second aspect of the present disclosure, a method for processing reasoning problems is provided, comprising: receiving a target problem to be reasoned, and inputting the target problem into a second reasoning model; generating one or more candidate reasoning chains for the target problem through the second reasoning model, and reasoning the target problem based on the candidate reasoning chains to obtain a target answer to the target problem; wherein the second reasoning model is a model trained using the training method described in the first aspect.

[0006] According to a third aspect of the present disclosure, a training device for an inference model is provided, comprising: an acquisition module for acquiring a training sample set, wherein the training samples in the training sample set include at least a sample inference problem, a reference inference chain corresponding to the sample inference problem, and a reference answer; a first training module for performing supervised fine-tuning (SFT) on a pre-trained model based on a first training sample in the training sample set to obtain a fine-tuned first inference model; and a second training module for performing reinforcement learning (RL) training on the first inference model based on a second training sample in the training sample set to obtain a second inference model.

[0007] According to a fourth aspect of the present disclosure, a device for processing reasoning problems is provided, comprising: a receiving module for receiving a target problem to be reasoned and inputting the target problem into a second reasoning model; a reasoning module for generating one or more candidate reasoning chains for the target problem through the second reasoning model, and reasoning the target problem based on the candidate reasoning chains to obtain a target answer to the target problem; wherein the second reasoning model is a model trained using the training method described in the first aspect.

[0008] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the inference model described in the first aspect of the present disclosure or the method for processing the inference problem described in the second aspect.

[0009] According to the sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the training method of the inference model described in the first aspect of the present disclosure or the method for processing the inference problem described in the second aspect.

[0010] According to the seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the training method of the inference model according to the first aspect of the present disclosure or the method for processing the inference problem according to the second aspect.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1 A flowchart of a method for training an inference model according to an embodiment of the present disclosure is provided;

[0014] Figure 2 A flowchart of a method for training an inference model according to an embodiment of the present disclosure is provided;

[0015] Figure 3 A flowchart of a method for processing an inference problem according to an embodiment of the present disclosure;

[0016] Figure 4 A flowchart of a method for training an inference model and a method for processing an inference problem according to an embodiment of the present disclosure;

[0017] Figure 5 A schematic diagram of the structure of a training device for an inference model according to an embodiment of the present disclosure;

[0018] Figure 6 A schematic diagram of the structure of a device for processing reasoning problems according to an embodiment of the present disclosure;

[0019] Figure 7 A schematic block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0020] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] The following is a brief description of the technical field involved in the solution of the present disclosure:

[0022] Artificial Intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). This discipline encompasses both hardware and software technologies. AI hardware technologies generally include computer vision, speech recognition, natural language processing, as well as deep learning / learning, big data processing, and knowledge graphs.

[0023] Natural Language Processing (NLP) is a key research area in the field of artificial intelligence. It integrates knowledge from multiple disciplines, including linguistics, computer science, machine learning, mathematics, and cognitive psychology. It is an interdisciplinary subject that integrates computer science, artificial intelligence, and linguistics. It encompasses two main areas: natural language understanding and natural language generation. Its research covers multiple levels, including characters, words, phrases, sentences, paragraphs, and texts. It serves as a bridge between machine language and human language. It aims to enable machines to understand, interpret, and generate human language, enabling effective communication between humans and machines and enabling computers to perform tasks such as language translation, sentiment analysis, and text summarization.

[0024] A large model refers to a machine learning model with a large parameter scale and complexity. It requires a large amount of computing resources and storage space to train and store, and often requires distributed computing and special hardware acceleration technology. Large models have stronger generalization and expression capabilities.

[0025] Deep learning (DL) is a new research direction in the field of machine learning (ML). It was introduced to bring ML closer to its original goal: artificial intelligence. Deep learning studies the inherent laws and representational hierarchies of sample data. The information gained from this learning process is highly helpful in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to acquire human-like analytical learning capabilities and recognize data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far surpass previous related technologies.

[0026] It should be noted that in related technologies, the reasoning ability of LLM is often improved through methods such as Supervised Fine Tuning (SFT) and Chain of Thought (CoT).

[0027] Regarding supervised fine-tuning (SFT), supervised fine-tuning (SFT) is a traditional and widely used technology that introduces a fine-tuning process with labeled data based on an existing pre-trained language model to make the model perform more accurately in specific tasks.

[0028] Among them, supervised fine-tuning SFT usually involves the following steps: Training data preparation: by collecting or generating high-quality datasets containing detailed reasoning steps, these datasets contain not only questions and final answers, but also intermediate reasoning steps. Each reasoning step must be logically consistent with other steps and correctly derive the final answer. Fine-tuning goal: use these data with reasoning process annotations to fine-tune the model, the goal is to make the model learn the overall pattern from questions to reasoning steps to answers. Specifically, the goal of fine-tuning is to minimize the cross-entropy loss between the reasoning step sequence and answer sequence generated by the model and the annotated data. However, the above method has the following defects: (1) dependence on high-quality annotations, that is, SFT requires a large amount of data with detailed reasoning steps annotated, which often requires manual annotation or reliance on high-quality generated data in practice, which increases training costs and data acquisition difficulties; (2) poor adaptability, that is, the fine-tuned model may only perform well on specific tasks and has poor adaptability to different question types or new tasks; (3) over-reliance on manual intervention, that is, in many cases, especially in zero-shot learning, the model still needs manual design and prompts, which limits its application in a wider range of tasks.

[0029] Chaining (CoT) is a method that has been widely used in large language model reasoning in recent years. The core idea of ​​CoT is to decompose complex reasoning tasks into a series of simple intermediate steps, and then reach the final reasoning result through step-by-step thinking (i.e., chaining).

[0030] Among them, the implementation scheme of CoT can be divided into the following aspects: Generate intermediate reasoning steps: During the reasoning process, the model not only generates the final answer, but also generates a series of intermediate reasoning steps. These steps can show the thinking process of the model, making the reasoning process more transparent and easy to understand. Step-by-step reasoning: The basic strategy of CoT is to guide the model to unfold step by step during reasoning, and generate a reasoning node at each step (for example, when solving a math problem, first derive the formula, then substitute the numerical value, and finally draw a conclusion). These steps are deduced step by step by the model based on known information, and each step should be consistent with the previous reasoning result. Prompt template: In practical applications, CoT usually relies on carefully designed prompt templates to stimulate the model to perform chain reasoning. For example, given a math problem, the prompt template can be "Please derive the answer step by step, starting from..." These prompts can help the model generate reasoning processes, but this method still relies on specific prompt design. However, the above method has the following defects: (1) Dependence on prompt engineering, that is, although CoT can effectively promote the model to perform step-by-step reasoning, its reasoning ability is still heavily dependent on the design of prompts. If the prompts are inappropriate, the model may not be able to successfully carry out reasoning, resulting in the accuracy and consistency of the reasoning results cannot be guaranteed; (2) High computational overhead, that is, since each reasoning requires the model to generate multiple steps of reasoning, compared with directly generating answers, the CoT method will increase additional computational overhead, especially when facing complex problems, the reasoning process may become very lengthy; (3) Insufficient training stage, that is, the existing CoT method mainly relies on prompt engineering to guide reasoning, but during the training process, the model does not specifically learn how to automatically expand the reasoning chain. Therefore, although prompts can activate reasoning ability to a certain extent, the model still lacks intrinsic and automatic reasoning ability.

[0031] In summary, relevant technologies often rely too much on prompts. That is, both SFT and CoT rely on artificially designed prompts or data to varying degrees, which makes their reasoning capabilities lack broad adaptability and flexibility, and the reasoning depth and complexity are limited. That is, current technology still makes it difficult for the model to conduct multi-step reasoning autonomously and in depth. Especially in complex tasks, the model's reasoning depth and the accuracy of the reasoning results cannot be effectively guaranteed. The reasoning ability has not been systematically internalized during the training phase, resulting in the model still passively relying on external prompts when facing new tasks and unable to actively carry out complex reasoning.

[0032] The following describes a method for training an inference model and a method for processing inference problems according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0033] Figure 1This is a flowchart of a method for training an inference model according to an embodiment of the present disclosure. It should be noted that the method for training an inference model in this embodiment is performed by a training device for the inference model. The training device for the inference model can be a hardware device or software within the hardware device. Examples of hardware devices include terminal devices and servers.

[0034] like Figure 1 As shown, the training method of the inference model proposed in this embodiment includes the following steps:

[0035] S101. Obtain a training sample set, where the training samples in the training sample set at least include a sample reasoning problem, a reference reasoning chain corresponding to the sample reasoning problem, and a reference answer.

[0036] It should be noted that in order to give the reasoning model stronger logical reasoning capabilities, the types of sample reasoning problems can be multi-step reasoning, causal reasoning, logical reasoning (such as inductive reasoning, deductive reasoning), etc.

[0037] For example, sample reasoning question 1 can be "What is 5+3?", and sample reasoning question 2 can be "If X increases, how will Y change?"

[0038] In an embodiment of the present disclosure, after a sample reasoning problem is obtained, a training sample may be determined based on the sample reasoning problem, a reference reasoning chain corresponding to the sample reasoning problem, and a reference answer.

[0039] For example, for training sample 1, the sample reasoning question is "What is 5+3?", the corresponding reference reasoning chain is "5+3=8", and the reference answer is "8".

[0040] It should be noted that the format of the training sample can be X→Z=(z1,z2,z3…zk)→a, where X is the sample reasoning problem, Z=(z1,z2,z3…zk) is the reference reasoning chain, z1,z2,z3…zk are the steps of the reference reasoning chain, and A is the reference answer.

[0041] S102 : Based on the first training sample in the training sample set, perform supervised fine-tuning (SFT) on the pre-trained model to obtain a fine-tuned first inference model.

[0042] Optionally, X training samples may be randomly selected from the training sample set as the first training samples.

[0043] Among them, the fine-tuning goal of supervised fine-tuning SFT is to enable the first reasoning model to generate the correct reasoning chain and answer corresponding to the reasoning problem based on the reasoning problem.

[0044] In an embodiment of the present disclosure, the pre-trained model can be supervised fine-tuned SFT based on the first training sample, so that the first reasoning model can generate the correct reasoning chain and answer corresponding to the reasoning problem according to the reasoning problem, obtain the loss function of the pre-trained model in the process of supervised fine-tuning SFT, judge whether the pre-trained model meets the fine-tuning end condition, and obtain the fine-tuned first reasoning model in response to the pre-trained model meeting the fine-tuning end condition.

[0045] S103: Based on the second training sample in the training sample set, perform reinforcement learning (RL) training on the first inference model to obtain a second inference model.

[0046] It should be noted that although supervised fine-tuning can enable the first reasoning model to generate the correct reasoning chain corresponding to the reasoning problem, when faced with more complex reasoning tasks, the first reasoning model may not be able to handle all situations. Therefore, reinforcement learning (RL) training can be performed on the first reasoning model to further improve the reasoning ability of the first reasoning model and obtain the second reasoning model.

[0047] Optionally, the training samples in the training sample set excluding the first training samples can be used as the second training samples; optionally, Y training samples can be randomly selected from the first training samples, and the training samples in the training sample set excluding the first training samples and the Y training samples can be used together as the second training samples.

[0048] In an embodiment of the present disclosure, a second training sample can be input into the first reasoning model, and the first reasoning model predicts the second reasoning chain and the second answer corresponding to the sample reasoning problem. The reward value of the first reasoning model is determined based on the reference reasoning chain and the reference answer, as well as the second reasoning chain and the second answer. With maximizing the reward value as the optimization goal, the model parameters of the first reasoning model are adjusted until the RL training meets the termination conditions, thereby obtaining the second reasoning model.

[0049] According to the training method of the reasoning model of the embodiment of the present disclosure, a training sample set is obtained, and the training samples in the training sample set at least include sample reasoning problems, reference reasoning chains corresponding to the sample reasoning problems, and reference answers. Based on the first training sample in the training sample set, the pre-trained model is supervised fine-tuned SFT to obtain the fine-tuned first reasoning model, and based on the second training sample in the training sample set, the first reasoning model is reinforced learning RL training to obtain the second reasoning model. Therefore, the present disclosure effectively improves the reasoning ability of the second reasoning model in complex reasoning tasks by combining supervised fine-tuning, reinforcement learning and thinking chain, eliminates excessive dependence on prompt templates, and enables the second reasoning model to automatically carry out an efficient and reliable reasoning process without complex prompts, thereby improving the accuracy and stability of the second reasoning large model in the reasoning process.

[0050] Figure 2 It is a schematic flowchart of a method for training an inference model according to an embodiment of the present disclosure.

[0051] As Figure 2 shown, the method for training the inference model proposed in this embodiment includes the following steps:

[0052] S201. Obtain a training sample set, where the training samples in the training sample set at least include sample inference questions, the reference inference chain corresponding to the sample inference questions, and the reference answers.

[0053] For the relevant content of step S201, reference can be made to the above embodiment, which will not be elaborated here.

[0054] S102 in the above embodiment, "Based on the first training sample in the training sample set, perform supervised fine-tuning SFT on the pre-trained model to obtain the first inference model after fine-tuning" can specifically include S202 - S204.

[0055] S202. Input the first training sample into the pre-trained model, and the pre-trained model predicts the first inference chain and the first answer corresponding to the sample inference question.

[0056] S203. Determine the loss function of the pre-trained model according to the reference inference chain and the reference answer, as well as the first inference chain and the first answer.

[0057] In the embodiment of the present disclosure, after obtaining the first inference chain and the first answer, the loss function of the pre-trained model can be determined according to the reference inference chain and the reference answer, as well as the first inference chain and the first answer. The expression of the loss function can be:

[0058] <000014!>[[ID=)!]]Where L t , t ,

[0061] ,

[0060] ,

[0059] , θ , SFT , , , (θ) is the loss function of the inference model, θ is the model parameter of the pre-trained model, X is the sample inference question, Z is the reference inference chain, A is the reference answer, D is the distribution of the first training sample, Y is the concatenation of the first inference chain and the first answer, y t is the currently generated token (the inference step of the first inference chain or the first answer), P θ (y t ∣X,y<t) is the conditional probability that the pre-trained model generates the inference step of the first inference chain or the first answer at position t.

[0060] S204. Under the constraint of minimizing the loss function, adjust the model parameters of the pre-trained model until the SFT training meets the end condition, and obtain the first inference model.

[0061] In an embodiment of the present disclosure, after obtaining the loss function of the pre-trained model, under the constraint of minimizing the loss function, the model parameters of the pre-trained model can be continuously adjusted according to the loss function until the SFT training meets the end conditions, and the pre-trained model after the last adjustment of the model parameters is determined as the first inference model that has completed training.

[0062] Optionally, the model parameters of the pre-trained model can be adjusted based on a standard optimization algorithm, such as the Adam optimizer, and the pre-trained model with adjusted parameters can be continued to be trained to minimize the loss function, so that the pre-trained model learns a step-by-step reasoning strategy until the loss function is minimized, and it is determined that the SFT training meets the termination conditions to obtain the first inference model.

[0063] It should be noted that after obtaining the first reasoning model, the first reasoning model can generate a suitable reasoning chain according to the reasoning question and ultimately obtain a correct answer.

[0064] In the above embodiment, S103 “performing reinforcement learning (RL) training on the first inference model based on the second training sample in the training sample set to obtain a second inference model” may specifically include S205-S207.

[0065] S205: Input the second training sample into the first reasoning model, and use the first reasoning model to predict a second reasoning chain and a second answer corresponding to the sample reasoning question.

[0066] S206 : Determine the reward value of the first reasoning model based on the reference reasoning chain and the reference answer, and the second reasoning chain and the second answer.

[0067] The reward value of the first reasoning model can be used to evaluate the quality of the predicted second reasoning chain and the second answer, and the reward value is the sum of the reward values ​​of multiple dimensions.

[0068] In the embodiment of the present disclosure, the first reward value Rans of the answer can be determined based on the reference answer and the second answer, the second reward value Rproc and the third reward value Rlength of the reasoning chain can be determined based on the reference reasoning chain and the second reasoning chain, and the reward value R(Y) can be determined based on the first reward value, the second reward value and the third reward value.

[0069] Optionally, after obtaining the first reward value, the second reward value, and the third reward value, the reward value R(Y) of the first reasoning model may be determined according to the following formula:

[0070] R(Y)=Rans+αRproc+βRlength

[0071] Among them, R(Y) is the reward value, Rans is the first reward value, Rproc is the second reward value, α is the hyperparameter of the second reward value, Rlength is the third reward value, and β is the hyperparameter of the third reward value, which is used to control the weights of the second and third reward values ​​in the reward values ​​of the first inference model.

[0072] In the embodiment of the present disclosure, after the second answer is obtained, the first reward value Rans of the answer can be determined according to the degree of matching between the reference answer and the second answer.

[0073] For example, if the second answer completely matches the reference answer, a positive reward value is given. If the second answer partially matches the reference answer, a partial positive reward value can be given according to the degree of match and a pre-built strategy. If the second answer does not match the reference answer at all, a negative reward value is given.

[0074] In an embodiment of the present disclosure, a first step sequence of a reference reasoning chain is determined, a second step sequence of a second reasoning chain is determined, the first step sequence and the second step sequence are matched, and from the second step sequence, logically abnormal steps and logically correct steps are determined, and reward processing is performed on the logically abnormal steps and the logically correct steps respectively to obtain respective reward values, and based on the respective reward values, a second reward value is determined.

[0075] Optionally, the first step sequence and the second step sequence can be matched, and the logically abnormal steps and the logically correct steps can be determined from the second step sequence. A negative reward value can be given to the logically abnormal steps, and a positive reward value can be given to the logically correct steps, and a second reward value can be determined based on the respective reward values.

[0076] In an embodiment of the present disclosure, in order to train and generate a concise and effective reasoning chain, redundant steps can be determined from the second step sequence of the second reasoning chain based on the first step sequence of the reference reasoning chain, and the redundant steps can be rewarded to obtain corresponding reward values, and a third reward value can be determined based on the corresponding reward values.

[0077] Optionally, redundant steps (irrelevant steps and redundant steps) may be determined from the second step sequence of the second reasoning chain based on the first step sequence of the reference reasoning chain, and negative rewards may be given to the redundant steps.

[0078] For example, the first information entropy of the steps in the first step sequence of the reference reasoning chain and the second information entropy of the steps in the second step sequence of the second reasoning chain can be obtained. For the same step, irrelevant steps can be determined from the second step sequence of the second reasoning chain based on the corresponding first information entropy and second information entropy. Redundant steps can be determined from the second step sequence of the second reasoning chain based on the pre-trained expert small model and the first step sequence of the reference reasoning chain.

[0079] S207 : Taking maximizing the reward value as the optimization goal, adjust the model parameters of the first inference model until the RL training meets the termination condition, thereby obtaining a second inference model.

[0080] Optionally, a reinforcement learning algorithm can be used to adjust the model parameters of the first inference model with the maximization of the reward value as the optimization goal, and the first inference model with the adjusted parameters can be continuously trained until the RL training meets the termination condition to obtain the second inference model.

[0081] It should be noted that the present disclosure does not limit the type of reinforcement learning algorithm. For example, the reinforcement learning algorithm can be a proximal policy optimization (PPO) method.

[0082] It should be noted that through the training method of the reasoning model proposed in the present invention, the logical reasoning ability can be internalized as the basic skills of the second reasoning model. The trained second reasoning model can easily cope with problems in different fields, such as: medicine, law, engineering, etc., can be used across fields, and can be transferred from one reasoning task to another, can quickly adapt to new reasoning scenarios, and improve the flexibility and efficiency of the reasoning model in the reasoning process.

[0083] According to the training method of the reasoning model of the embodiment of the present disclosure, a training sample set is obtained, and the training samples in the training sample set at least include a sample reasoning problem, a reference reasoning chain corresponding to the sample reasoning problem, and a reference answer. The first training sample is input into the pre-training model, and the pre-training model predicts the first reasoning chain and the first answer corresponding to the sample reasoning problem. The first training sample is input into the pre-training model, and the pre-training model predicts the first reasoning chain and the first answer corresponding to the sample reasoning problem. According to the reference reasoning chain and the reference answer, as well as the first reasoning chain and the first answer, the loss function of the pre-training model is determined. Under the constraint condition of minimizing the loss function, the model parameters of the pre-training model are adjusted until the SFT training meets the end condition, and the first reasoning model is obtained. The second training sample is input into the first reasoning model, and the first reasoning model predicts the sample reasoning problem to be correct. The corresponding second reasoning chain and second answer are determined according to the reference reasoning chain and reference answer, as well as the second reasoning chain and second answer. The reward value of the first reasoning model is determined, and the model parameters of the first reasoning model are adjusted with the maximization of the reward value as the optimization goal until the RL training meets the end condition to obtain the second reasoning model. Therefore, the present disclosure adjusts the model parameters of the pre-trained model until the SFT training meets the end condition to obtain the first reasoning model, so that the first reasoning model learns the correct reasoning chain and answer, and takes the maximization of the reward value as the optimization goal, adjusts the model parameters of the first reasoning model until the RL training meets the end condition to obtain the second reasoning model, which greatly improves the reasoning ability of the second reasoning model, and improves the efficiency and stability of the reasoning of the second reasoning model, and can be applied to complex reasoning tasks, cross-domain applications and multi-step reasoning tasks.

[0084] Figure 3 This is a flowchart of a method for handling reasoning problems according to one embodiment of the present disclosure. It should be noted that the method for handling reasoning problems according to this embodiment is performed by a device for handling reasoning problems. Specifically, the device for handling reasoning problems can be a hardware device or software within the hardware device. Examples of hardware devices include terminal devices and servers.

[0085] like Figure 3 As shown, the method for processing the reasoning problem proposed in this embodiment includes the following steps:

[0086] S301: Receive a target problem to be inferred, and input the target problem into a second reasoning model.

[0087] The target problem can be any problem to be reasoned.

[0088] It should be noted that when the target problem is input into the second reasoning model, the second reasoning model analyzes the target problem. According to the reasoning strategy learned in the training stage, if multi-step reasoning is required, the second reasoning model automatically starts the process of the reasoning chain.

[0089] S302. Generate one or more candidate reasoning chains for the target problem through the second reasoning model, and perform reasoning on the target problem based on the candidate reasoning chains to obtain the target answer of the target problem.

[0090] It should be noted that in order to improve the reliability and stability in the process of reasoning problems, a self-consistency strategy is adopted, that is, for the same target problem, multiple different candidate reasoning chains Z(1), Z(2),..., Z(M) are generated, and the target answer of the target problem is determined from the candidate answers corresponding to the candidate reasoning chains.

[0091] Optionally, for the same target problem, the second reasoning model generates multiple reasoning chains Z(1), Z(2),..., Z(N) from different random seeds or sampling strategies.

[0092] In the embodiments of the present disclosure, for the i-th step in any candidate reasoning chain, determine the sequence of previous steps that belong to the candidate reasoning chain and have been generated by the second reasoning model currently, where i is an integer and 2 ≤ i ≤ M, and M is the number of steps in the candidate reasoning chain. Generate the i-th step in the candidate reasoning chain according to the target problem and the sequence of previous steps until the candidate reasoning chain is output.

[0093] It should be noted that the second reasoning model generates a candidate reasoning chain Z = (z1, z2, z3…zk) according to the input target problem Q, and each reasoning step is generated based on the previous step and the conditions of the target problem Q.

[0094] Optionally, for the target problem Q, the conditional probability of generating a candidate reasoning chain is:

[0095]

[0096] where, P θ (Z∣Q) is the conditional probability of generating the candidate reasoning chain Z, and P θ (Z i ∣Q,z<i) is the conditional probability of generating the i-th reasoning zi.

[0097] It should be noted that when each step is generated, the second reasoning model generates the next step according to the currently generated step and the target problem Q until the complete candidate reasoning chain is output.

[0098] In the embodiment of the present disclosure, after obtaining the candidate reasoning chain, the target question can be reasoned based on the candidate reasoning chain to obtain the candidate answers corresponding to the candidate reasoning chain, and the candidate answers can be voted on for consistency. Based on the voting results, the candidate answer with the most votes can be determined from the candidate answers as the target answer.

[0099] For example, after obtaining the candidate answers A(1), A(2), ..., A(M) corresponding to the candidate reasoning chain, the candidate answers corresponding to the candidate reasoning chain can be evaluated through the self-consistency voting mechanism, the voting results can be obtained, and the candidate answer with the most votes can be determined from the candidate answers as the target answer.

[0100] The second inference model is an inference model obtained by using the training method of the first aspect.

[0101] Optionally, after obtaining the target answer to the target question, correction indication information of the target answer is received, and in response to the correction indication information, the candidate reasoning chain with the highest confidence is selected from the candidate reasoning chains, and the candidate answer corresponding to the candidate reasoning chain with the highest confidence is determined as the target answer.

[0102] It should be noted that when faced with problems with high uncertainty, traditional large language models are prone to produce incorrect answers due to a single reasoning failure. The present disclosure can generate multiple reasoning chains through a self-consistency strategy and select the most likely answer by voting, thereby improving the stability of the reasoning results. For example, in complex reasoning tasks, when inconsistencies occur in the reasoning chain, the model can effectively filter out erroneous reasoning chains through multiple sampling and verification, thereby reducing the possibility of errors in the second reasoning model. When the input problem is noisy or ambiguous, the second reasoning model can produce a more stable reasoning process through self-correction and multiple verifications, thereby significantly reducing sensitivity to input details and improving the robustness of the target answer.

[0103] It should be noted that after obtaining the target answer to the target question, if the target answer is inconsistent with the most common answer, correction instruction information of the target answer is generated.

[0104] Optionally, after obtaining the target answer to the target question, in order to improve the interpretability of the reasoning process and increase the transparency of the reasoning process, the candidate reasoning chain associated with the target answer is obtained as the target reasoning chain, and the target reasoning chain is displayed; or, the candidate reasoning chain with the highest confidence among the candidate reasoning chains is determined as the target reasoning chain, and the target reasoning chain is displayed.

[0105] For example, in fields that require a high degree of credibility, such as law, medicine, and finance, visualization technology can be used to display the target reasoning chain. Users or experts can check the reasoning process of the second reasoning model based on the displayed target reasoning chain to confirm the rationality and accuracy of the target answer.

[0106] According to the method for processing reasoning problems in an embodiment of the present disclosure, by receiving a target problem to be reasoned and inputting the target problem into a second reasoning model, one or more candidate reasoning chains for the target problem to be reasoned are generated through the second reasoning model, and the target problem is reasoned based on the candidate reasoning chains to obtain a target answer to the target problem. Thus, the present disclosure generates one or more candidate reasoning chains for the target problem to be reasoned through the second reasoning model, and reasoned on the target problem based on the candidate reasoning chains to obtain a target answer to the target problem, thereby eliminating excessive reliance on prompt templates. The second reasoning model can independently carry out an efficient and reliable reasoning process without complex prompts, significantly improving the accuracy and reliability of the target answer.

[0107] The following is an explanation of the specific process of the method for training the reasoning model and the method for processing the reasoning problem proposed in this disclosure.

[0108] For example, if Figure 4 As shown, for the training stage of the reasoning model, a training sample set is obtained, and the training samples in the training sample set include at least a sample reasoning problem, a reference reasoning chain corresponding to the sample reasoning problem, and a reference answer. Based on the first training sample in the training sample set, with the goal of minimizing the reasoning chain generation loss, the pre-trained model is supervised fine-tuned SFT to obtain the fine-tuned first reasoning model. Based on the second training sample in the training sample set, with the optimization goal of maximizing the reward value, the first reasoning model is reinforced learning RL training to obtain the second reasoning model. For the reasoning stage, the target problem to be reasoned is received, and the target problem is input into the second reasoning model. Multiple candidate reasoning chains are generated through a self-consistency strategy, and the candidate answers corresponding to the candidate reasoning chains are voted to obtain the target answer to the target problem.

[0109] In summary, the training method of the reasoning model and the method for handling the reasoning problem proposed in the present disclosure (1) enhance the logical reasoning ability of the second reasoning model, that is, through the design of the training stage and the reasoning (activation) stage, the second reasoning model has stronger logical reasoning ability, especially when facing complex tasks such as multi-step reasoning, mathematical reasoning, causal reasoning, etc., by explicitly learning the reasoning ability step by step in the training stage, the second reasoning model can actively carry out the multi-step reasoning process in the reasoning stage, thereby effectively reducing erroneous reasoning and unfounded reasoning, (2) improve the reasoning stability and robustness of the second reasoning model, that is, by guiding the model to actively generate multiple reasoning chains in the reasoning process, and enhance the reliability of the target answer through methods such as self-consistency voting, (3) it is suitable for complex tasks and cross-domain reasoning, improves the performance of the model in a single task, and can also enable the model to have the ability of cross-domain reasoning, widely used. It is widely applicable to various complex tasks and fields, that is, the present disclosure can internalize logical reasoning ability as the basic skill of the model. The trained second reasoning model can easily cope with problems in different fields. By systematically cultivating the reasoning ability of the model in the training stage, the present disclosure enables the second reasoning model to migrate from one task to another and quickly adapt to new reasoning scenarios. It can solve fixed types of tasks and perform more flexibly and efficiently when facing new reasoning problems. (4) It improves the interpretability of the model. While the second reasoning model generates answers, it can provide the intermediate steps of the reasoning chain, which is particularly important for fields that require high credibility. The reasoning process of the model can be checked according to the reasoning chain, so as to ensure the rationality and accuracy of the answer. Combined with visualization technology, each step of the reasoning chain can be displayed to help users understand the decision-making process of the model and further improve the trust and acceptance of the model.

[0110] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0111] Corresponding to the training methods of the inference models provided in the above-mentioned embodiments, an embodiment of the present disclosure also provides a training device for an inference model. Since the training device for the inference model provided in the embodiment of the present disclosure corresponds to the training methods for the inference models provided in the above-mentioned embodiments, the implementation method of the training method for the inference model is also applicable to the training device for the inference model provided in this embodiment, and will not be described in detail in this embodiment.

[0112] Figure 5 3 is a schematic diagram of the structure of a training device for an inference model according to an embodiment of the present disclosure.

[0113] like Figure 5 As shown, the training device 500 of the inference model includes: an acquisition module 510, a first training module 520 and a second training module 530.

[0114] An acquisition module 510 is configured to acquire a training sample set, wherein the training samples in the training sample set include at least a sample reasoning problem, a reference reasoning chain corresponding to the sample reasoning problem, and a reference answer;

[0115] A first training module 520 is configured to perform supervised fine-tuning (SFT) on the pre-trained model based on a first training sample in the training sample set to obtain a fine-tuned first inference model;

[0116] The second training module 530 is configured to perform reinforcement learning (RL) training on the first inference model based on a second training sample in the training sample set to obtain a second inference model.

[0117] Among them, the second training module 530 is also used to: input the second training sample into the first reasoning model, and the first reasoning model predicts the second reasoning chain and the second answer corresponding to the sample reasoning problem; determine the reward value of the first reasoning model based on the reference reasoning chain and the reference answer, as well as the second reasoning chain and the second answer; and adjust the model parameters of the first reasoning model with maximizing the reward value as the optimization goal until the RL training meets the end conditions to obtain the second reasoning model.

[0118] Among them, the second training module 530 is also used to: determine the first reward value of the answer based on the reference answer and the second answer; determine the second reward value and the third reward value of the reasoning chain based on the reference reasoning chain and the second reasoning chain; determine the reward value based on the first reward value, the second reward value and the third reward value.

[0119] Among them, the second training module 530 is also used to: determine the first step sequence of the reference reasoning chain; determine the second step sequence of the second reasoning chain; match the first step sequence and the second step sequence, and determine the logically abnormal steps and the logically correct steps from the second step sequence; respectively reward the logically abnormal steps and the logically correct steps to obtain their respective reward values, and determine the second reward value based on the respective reward values.

[0120] Among them, the second training module 530 is also used to: determine redundant steps from the second step sequence of the second reasoning chain according to the first step sequence of the reference reasoning chain; reward the redundant steps to obtain corresponding reward values, and determine the third reward value based on the corresponding reward value.

[0121] Among them, the first training module 520 is also used to: input the first training sample into the pre-trained large model, and the pre-trained large model predicts the first reasoning chain and the first answer corresponding to the sample reasoning problem; determine the loss function of the reasoning model based on the reference reasoning chain and the reference answer, as well as the first reasoning chain and the first answer; adjust the model parameters of the reasoning model under the constraint condition of minimizing the loss function until the SFT training meets the end condition to obtain the first reasoning model.

[0122] According to the training device of the reasoning model of the embodiment of the present disclosure, by obtaining a training sample set, the training samples in the training sample set at least include sample reasoning problems, reference reasoning chains corresponding to the sample reasoning problems and reference answers, based on the first training sample in the training sample set, the pre-trained model is supervised fine-tuned SFT to obtain the fine-tuned first reasoning model, based on the second training sample in the training sample set, the first reasoning model is reinforced learning RL training to obtain the second reasoning model. Therefore, the present disclosure effectively improves the reasoning ability of the second reasoning model in complex reasoning tasks by combining supervised fine-tuning, reinforcement learning and thinking chain, eliminates excessive dependence on prompt templates, and enables the second reasoning model to automatically carry out an efficient and reliable reasoning process without complex prompts, thereby improving the accuracy and stability of the second reasoning large model in the reasoning process.

[0123] Corresponding to the methods for processing reasoning problems provided in the above-mentioned embodiments, an embodiment of the present disclosure also provides a device for processing reasoning problems. Since the device for processing reasoning problems provided in the embodiment of the present disclosure corresponds to the methods for processing reasoning problems provided in the above-mentioned embodiments, the implementation of the method for processing reasoning problems is also applicable to the device for processing reasoning problems provided in this embodiment, and will not be described in detail in this embodiment.

[0124] Figure 6 2 is a schematic diagram of the structure of a device for processing reasoning problems according to an embodiment of the present disclosure.

[0125] like Figure 6 As shown, the device 600 for processing the reasoning problem includes: a receiving module 610 and a reasoning module 620.

[0126] in:

[0127] A receiving module 610 is configured to receive a target problem to be inferred and input the target problem into a second reasoning model;

[0128] A reasoning module 620 is configured to generate one or more candidate reasoning chains for the target question using the second reasoning model, and to reason about the target question based on the candidate reasoning chains to obtain a target answer to the target question;

[0129] The second inference model is an inference model obtained using the training method described in the first aspect.

[0130] Among them, the reasoning module 620 is also used to: reason about the target question based on the candidate reasoning chain to obtain the candidate answers corresponding to the candidate reasoning chain; perform consistency voting on the candidate answers, and determine the candidate answer with the most votes from the candidate answers based on the voting results as the target answer.

[0131] Among them, the reasoning module 620 is also used to: for the i-th step in any candidate reasoning chain, determine the sequence of predecessor steps currently generated by the second reasoning model and belonging to the candidate reasoning chain, wherein i is an integer, and 2≤i≤M, M is the number of steps in the candidate reasoning chain; generate the i-th step in the candidate reasoning chain according to the target problem and the sequence of predecessor steps until the candidate reasoning chain is output.

[0132] Among them, after obtaining the target answer to the target question, the device 600 is also used to: receive correction indication information of the target answer; in response to the correction indication information, select the candidate reasoning chain with the highest confidence from the candidate reasoning chains; and determine that the candidate answer corresponding to the candidate reasoning chain with the highest confidence is the target answer.

[0133] Among them, after obtaining the target answer to the target question, the device 600 is also used to: obtain the candidate reasoning chain associated with the target answer as the target reasoning chain, and display the target reasoning chain; or, determine the candidate reasoning chain with the highest confidence among the candidate reasoning chains as the target reasoning chain, and display the target reasoning chain.

[0134] According to the device for processing reasoning problems in an embodiment of the present disclosure, by receiving a target problem to be reasoned and inputting the target problem into a second reasoning model, one or more candidate reasoning chains for the target problem to be reasoned are generated through the second reasoning model, and the target problem is reasoned based on the candidate reasoning chains to obtain a target answer to the target problem. Thus, the present disclosure generates one or more candidate reasoning chains for the target problem to be reasoned through the second reasoning model, and reasoned the target problem based on the candidate reasoning chains to obtain a target answer to the target problem, thereby eliminating excessive reliance on prompt templates. The second reasoning model can autonomously carry out an efficient and reliable reasoning process without complex prompts, significantly improving the accuracy and reliability of the target answer.

[0135] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0136] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0137] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0138] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0139] The computing unit 701 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the training method of the inference model or the method for processing the inference problem. For example, in some embodiments, the training of the inference model or the method for processing the inference problem can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the training of the inference model or the method for processing the inference problem described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute a method for training an inference model or a method for processing an inference problem in any other appropriate manner (for example, by means of firmware).

[0140] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0144] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0145] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0146] The present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned method for training an inference model or method for processing an inference problem.

[0147] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0148] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for training an inference model, wherein: The method further comprises the following steps: Acquire a training sample set, where the training samples in the training sample set at least include a sample reasoning problem, a reference reasoning chain corresponding to the sample reasoning problem, and a reference answer; Based on the first training sample in the training sample set, performing supervised fine-tuning SFT on the pre-trained model to obtain a fine-tuned first inference model; Based on the second training sample in the training sample set, reinforcement learning (RL) training is performed on the first inference model to obtain a second inference model.

2. The method according to claim 1, wherein The step of performing reinforcement learning (RL) training on the first inference model based on the second training sample in the training sample set to obtain a second inference model includes: Inputting the second training sample into the first reasoning model, and having the first reasoning model predict a second reasoning chain and a second answer corresponding to the sample reasoning question; determining a reward value of the first reasoning model based on the reference reasoning chain and the reference answer, and the second reasoning chain and the second answer; With the maximization of the reward value as the optimization goal, the model parameters of the first inference model are adjusted until the RL training meets the termination conditions, thereby obtaining a second inference model.

3. The method according to claim 2, wherein: Determining the reward value of the first reasoning model based on the reference reasoning chain and the reference answer, and the second reasoning chain and the second answer includes: Determining a first reward value of the answer based on the reference answer and the second answer; determining a second reward value and a third reward value of the reasoning chain according to the reference reasoning chain and the second reasoning chain; The reward value is determined according to the first reward value, the second reward value, and the third reward value.

4. The method according to claim 3, wherein: Determining a second reward value of the inference chain according to the reference inference chain and the second inference chain includes: determining a first step sequence of the reference reasoning chain; determining a second sequence of steps of said second chain of reasoning; Matching the first step sequence and the second step sequence, and determining logically abnormal steps and logically correct steps from the second step sequence; Reward processing is performed on the logically abnormal step and the logically correct step respectively to obtain respective reward values, and the second reward value is determined based on the respective reward values.

5. The method according to claim 3, wherein Determining a third reward value of the inference chain according to the reference inference chain and the second inference chain includes: determining redundant steps from a second sequence of steps in the second chain of reasoning based on the first sequence of steps in the reference chain of reasoning; Reward processing is performed on the redundant steps to obtain corresponding reward values, and the third reward value is determined based on the corresponding reward values.

6. The method according to any one of claims 1 to 5, wherein The method of performing supervised fine-tuning (SFT) on the pre-trained model based on the first training sample in the training sample set to obtain a fine-tuned first inference model includes: Inputting the first training sample into the pre-trained large model, and having the pre-trained large model predict a first reasoning chain and a first answer corresponding to the sample reasoning problem; Determining a loss function of the reasoning model based on the reference reasoning chain and the reference answer, and the first reasoning chain and the first answer; Under the constraint condition of minimizing the loss function, the model parameters of the inference model are adjusted until the SFT training meets the end condition, thereby obtaining the first inference model.

7. A method for dealing with reasoning problems, wherein: The method comprises the following steps: receiving a target problem to be reasoned, and inputting the target problem into a second reasoning model; generating one or more candidate reasoning chains for the target question using the second reasoning model, and reasoning about the target question based on the candidate reasoning chains to obtain a target answer to the target question; The second inference model is an inference model obtained by the training method according to any one of claims 1 to 6.

8. The method according to claim 7, wherein: The reasoning on the problem to be reasoned based on the candidate reasoning chain to obtain a target answer to the target problem includes: Reasoning the target question based on the candidate reasoning chain to obtain a candidate answer corresponding to the candidate reasoning chain; A consensus vote is performed on the candidate answers, and the candidate answer with the most votes is determined from the candidate answers according to the voting result as the target answer.

9. The method according to claim 8, wherein Generating one or more candidate reasoning chains for the target problem by using the second reasoning model includes: For the i-th step in any candidate reasoning chain, determine the sequence of preceding steps currently generated by the second reasoning model and belonging to the candidate reasoning chain, where i is an integer and 2≤i≤M, and M is the number of steps in the candidate reasoning chain; According to the target problem and the sequence of preceding steps, an i-th step in the candidate reasoning chain is generated until the candidate reasoning chain is output.

10. The method according to any one of claims 7 to 9, wherein: After obtaining the target answer to the target question, the method further includes: receiving correction instruction information of the target answer; In response to the correction instruction information, selecting a candidate reasoning chain with the highest confidence from the candidate reasoning chains; The candidate answer corresponding to the candidate reasoning chain with the highest confidence is determined as the target answer.

11. The method according to any one of claims 7 to 9, wherein: After obtaining the target answer to the target question, the method further includes: Obtaining a candidate reasoning chain associated with the target answer as a target reasoning chain, and displaying the target reasoning chain; or A candidate reasoning chain with the highest confidence among the candidate reasoning chains is determined as a target reasoning chain, and the target reasoning chain is displayed.

12. A training device for an inference model, wherein: The device comprises: An acquisition module is configured to acquire a training sample set, wherein the training samples in the training sample set include at least a sample reasoning problem, a reference reasoning chain corresponding to the sample reasoning problem, and a reference answer; A first training module is configured to perform supervised fine-tuning (SFT) on the pre-trained model based on a first training sample in the training sample set to obtain a fine-tuned first inference model; The second training module is used to perform reinforcement learning (RL) training on the first inference model based on a second training sample in the training sample set to obtain a second inference model.

13. A device for processing reasoning problems, wherein: The device comprises: A receiving module, configured to receive a target problem to be inferred and input the target problem into a second inference model; a reasoning module, configured to generate one or more candidate reasoning chains for the target question using the second reasoning model, and to reason about the target question based on the candidate reasoning chains to obtain a target answer to the target question; The second inference model is an inference model obtained by the training method according to any one of claims 1 to 6.

14. An electronic device, characterized in that: including processor and memory; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method according to any one of claims 1 to 6 or claims 7 to 11.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 or claims 7 to 11 is implemented.

16. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6 or claims 7 to 11.

Images