Large language model training method, logic problem processing method and computer equipment

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a large language model training method, a logic problem processing method and computer equipment, and the method comprises the steps: obtaining a first logic problem set, and outputting multiple groups of hyper-parameters of a formatted first large language model; any first logic problem in the first logic problem set is input into a first large language model corresponding to any group of hyper-parameters, thinking chain data comprising a plurality of reasoning steps are obtained, and the large language model generates a current reasoning step based on the first logic problem, the previous reasoning steps and evaluation information of the previous reasoning steps; adjusting model parameters of the first large language model based on the thinking chain data and the evaluation information to obtain a second large language model and an inference step scoring model; and based on the second logic problem set and the reasoning step scoring model, adjusting model parameters of the second large language model to obtain a target large language model with generalization ability. The logical reasoning problem processing effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large language model training method, a logic problem processing method and a computer device. Background Art

[0002] The Large Language Model (LLM) exhibits two thinking modes when processing information: fast thinking and slow thinking. Fast thinking relies on intuition and pattern recognition and is suitable for scenarios with quick responses, while slow thinking involves complex logical reasoning and is suitable for problems that require in-depth analysis and reasoning.

[0003] At present, LLM has made significant progress in fast thinking. However, to solve complex logical reasoning problems, LLM in related technologies often needs to rely on techniques such as hint engineering or multi-solution search, which have poor processing effects when solving complex logical reasoning problems. Summary of the invention

[0004] In view of this, the present invention provides a large language model training method, a logic problem processing method and a computer device to solve the problem in the related art that the large language model has poor processing effect on complex logic problems.

[0005] In a first aspect, the present invention provides a large language model training method, the method comprising: Get the first language model for output formatting; Obtain multiple sets of hyperparameters for a first logic problem set and a first language model; For any first logical problem in the first logical problem set, the first logical problem is input into a first large language model corresponding to any set of hyperparameters to obtain thinking chain data of the first logical problem, the thinking chain data includes a plurality of reasoning steps, and the large language model generates a current reasoning step based on the first logical problem, a previous reasoning step and evaluation information of the previous reasoning step; Based on the first logical question, the thinking chain data of the first logical question and the evaluation information of the reasoning steps in the thinking chain data, the model parameters of the first language model are adjusted to obtain the second language model and the reasoning step scoring model with preliminary logical reasoning ability; A second logical question set is obtained, and based on the second logical question set and the reasoning step scoring model, model parameters of the second large language model are adjusted to obtain a target large language model with generalization capability, wherein the amount of logical question data of the second logical question set is greater than the amount of logical question data of the first logical question set.

[0006] The large language model training method provided in this embodiment, by inputting the first logical problem into the first large language model corresponding to any set of hyperparameters for any first logical problem in the first logical problem set, obtains the thinking chain data of the first logical problem, the thinking chain data includes multiple reasoning steps, and the large language model generates the current reasoning step based on the first logical problem, the previous reasoning step and the evaluation information of the previous reasoning step, so as to ensure that the generated thinking chain data has the action of making mistakes and correcting mistakes, thereby adjusting the model parameters of the first large language model based on the first logical problem, the first logical chain data and the evaluation information of the reasoning steps in the thinking chain data, and the obtained second large language model has more accurate and reasonable logical reasoning ability. A second logical problem set with more logical problem data is obtained, and the model parameters of the second large language model are adjusted based on the second logical problem set and the reasoning step scoring model, so that the second large language model can be exposed to more abundant and diverse logical problem scenarios, learn a wider range of logical patterns and laws, thereby enhancing the generalization ability of the model when facing new and unseen logical problems. The target large language model obtained by the large language model training method of this embodiment can accurately handle complex logical problems.

[0007] In an optional implementation, obtaining the first language model for output formatting includes: Obtain a data set, where training samples in the data set include logic problems and reasoning steps corresponding to the logic problems; Based on the data set, the model parameters of the pre-trained large language model are adjusted to obtain the first large language model with formatted output.

[0008] The large language model training method provided in this embodiment adjusts the model parameters of the pre-trained large language model to obtain the first large language model with formatted output, thereby improving the interpretability and transparency of the model output, ensuring the format consistency and standardization of the model output results, and reducing misunderstandings or errors caused by inconsistent formats of output results.

[0009] In an optional implementation, based on the data set, adjusting the model parameters of the pre-trained large language model to obtain the first large language model with output formatting includes: Inputting the training sample into the pre-trained large language model to obtain a first output result, where the first output result represents a probability value of a label output by the pre-trained large language model being a label corresponding to the training sample; Based on the first output result, determining a first loss value, the first loss value including a loss value of the format symbol; Based on the first loss value, the model parameters of the pre-trained large language model are adjusted until an adjustment termination condition is reached, thereby obtaining a first large language model with formatted output.

[0010] The large language model training method provided in this embodiment obtains the loss value of the format symbol, and adjusts the model parameters of the pre-trained large language model according to the loss value of the format symbol to obtain the first large language model with formatted output, thereby improving the interpretability and transparency of the model output, ensuring the format consistency and standardization of the model output results, and reducing misunderstandings or errors caused by inconsistent formats of output results.

[0011] In an optional implementation, determining the first loss value based on the first output result includes: Determine the position of the format symbol mark based on the reasoning steps corresponding to the logical problems in the training samples; Based on the position of the format symbol mark, determining from the first output result a probability value that the mark corresponding to the position of the format symbol mark is the mark corresponding to the format symbol; A first loss value is determined based on a probability value that a mark corresponding to the position of the format symbol mark is a mark corresponding to the format symbol.

[0012] The large language model training method provided in this embodiment determines a first loss value by determining the probability value of the mark corresponding to the position of the format symbol mark determined in the first output result being the mark corresponding to the format symbol, and adjusts the model parameters of the pre-trained large language model based on the first loss value to obtain the first large language model with formatted output, thereby ensuring the formatting and consistency of the model output.

[0013] In an optional implementation, the first logical problem is input into the first language model corresponding to any set of hyperparameters to obtain the thought chain data of the first logical problem, including: Input the first logic question into the first language model corresponding to any set of hyperparameters to obtain a first reasoning step; Inputting the first logic problem and the multiple first reasoning steps corresponding to the first logic problem into the third largest language model to obtain the target first reasoning step and the evaluation information of the target first reasoning step, wherein the logic reasoning ability of the third largest language model is higher than the logic reasoning ability of the pre-trained large language model; Based on the target first reasoning step, determining a target hyperparameter corresponding to the target first reasoning step; Inputting the first logic problem, the target first reasoning step, and the evaluation information of the target first reasoning step into the first language model corresponding to the target hyperparameter to obtain the second reasoning step; Determine evaluation information of the second reasoning step based on the target first reasoning step, the second reasoning step and the first logical question by using the third language model; Inputting the target first reasoning step, the second reasoning step, the first logical question, and the evaluation information of the second reasoning step into the first language model corresponding to the target hyperparameter to obtain the third reasoning step; Iterative execution utilizes the third language model to determine evaluation information of the currently obtained reasoning step based on the first logical question, the reasoning step before the currently obtained reasoning step, and the currently obtained reasoning step, inputs the first logical question, the reasoning step before the currently obtained reasoning step, the currently obtained reasoning step, and the evaluation information of the currently obtained reasoning step into the first language model corresponding to the target hyperparameter, obtains the next reasoning step, until the logical reasoning is completed, and obtains the thinking chain data of the first logical question; Among them, the evaluation information includes scores and guidance.

[0014] The large language model training method provided in this embodiment obtains the first reasoning step by inputting the first logical problem into the first large language model corresponding to any set of hyperparameters, and then uses the third large language model with stronger logical reasoning ability to screen the first reasoning step to obtain the target first reasoning step and the evaluation information of the target first reasoning step, thereby ensuring the diversity of the generated thinking chain data and reducing the amount of calculation for generating the thinking chain data.

[0015] During each iteration, the subsequent reasoning steps will be further adjusted and optimized based on the current reasoning steps and their evaluation information, ensuring the logical consistency, coherence and accuracy of the final output thinking chain data, and ensuring that the generated thinking chain data has the actions of making and correcting errors, thereby improving the logical reasoning ability of the second largest language model and improving the processing effect of complex logical problems.

[0016] In an optional implementation, inputting the first logic question and a plurality of first reasoning steps corresponding to the first logic question into the third language model to obtain the target first reasoning step includes: Using the third language model, based on the first logical problem, multiple first reasoning steps corresponding to the first logical problem are screened from the dimensions of correctness, uncertainty and repeatability to obtain the target first reasoning step.

[0017] The large language model training method provided in this embodiment utilizes the third large language model to screen multiple first reasoning steps corresponding to the first logical problem from the dimensions of correctness, uncertainty and repeatability based on the first logical problem, obtain the target first reasoning steps, remove the first reasoning steps that do not meet the requirements, and ensure the accuracy of the obtained thinking chain data.

[0018] In an optional implementation, based on the first logical question, the thinking chain data of the first logical question, and the evaluation information of the reasoning steps in the thinking chain data, the model parameters of the first language model are adjusted to obtain a second language model with preliminary logical reasoning ability and a reasoning step scoring model, including: Inputting the first logical question and the thinking chain data of the first logical question into the first large language model to obtain a second output result, wherein the second output result represents the probability value of the mark output by the first large language model corresponding to the first logical question and the thinking chain data of the first logical question; Based on the second output result, determining a second loss value, the second loss value including a loss value of a reasoning step of the thought chain data; Based on the second loss value, adjusting the model parameters of the first language model to obtain a second language model with preliminary logical reasoning ability; Inputting the first logical question, the thinking chain data of the first logical question, and the evaluation information of the reasoning steps in the thinking chain data into the first large language model to obtain a third output result, wherein the third output result represents the probability value of the mark output by the first large language model corresponding to the first logical question, the thinking chain data of the first logical question, and the evaluation information of the reasoning steps in the thinking chain data; Based on the third output result, determine a third loss value, the third loss value comprising a loss value of the score of the reasoning step; Based on the third loss value, adjusting the model parameters of the first language model to obtain an inference step scoring model; The evaluation information includes ratings.

[0019] The large language model training method provided in this embodiment calculates the loss value of the reasoning step of the thinking chain data and the loss value of the score of the reasoning step, and adjusts the model parameters of the first large language model based on the loss value of the reasoning step of the thinking chain data and the loss value of the score of the reasoning step, thereby improving the model's logical reasoning ability and the accuracy of the reasoning steps.

[0020] In an optional implementation, based on the second logical question set and the reasoning step scoring model, the model parameters of the second large language model are adjusted to obtain a target large language model with generalization ability, including: Create multiple environments, and use the second largest language model for each environment. In each environment, for any second logical problem in the second logical problem set, the second logical problem is input into the second largest language model to obtain a first reasoning step of the second logical problem; Obtaining the probability value of the first reasoning step; Inputting the first reasoning step and the second logic question into the reasoning step scoring model to obtain a score for the first reasoning step; Inputting the second logical problem and the first reasoning step into the second largest language model to obtain a second reasoning step of the second logical problem; Obtaining the probability value of the second reasoning step; Inputting the second reasoning step, the second logical question and the first reasoning step into the reasoning step scoring model to obtain a score for the second reasoning step; Iterative execution inputs the second logical question and the reasoning step before the reasoning step to be obtained into the second largest language model, obtains the reasoning step to be obtained, obtains the probability value of the reasoning step to be obtained, inputs the second logical question, the reasoning step to be obtained, and the reasoning step before the reasoning step to be obtained into the reasoning step scoring model, obtains the score of the reasoning step to be obtained, until the environment completes the logical reasoning for the second logical question; The second logical question, the reasoning steps of the second logical question, the probability values ​​of the reasoning steps, and the scores of the reasoning steps in the logical reasoning process are saved; Based on the saved second logical problem, the reasoning steps of the second logical problem, the probability values ​​of the reasoning steps and the scores of the reasoning steps, the model parameters of the second largest language model are adjusted to obtain a target large language model with generalization ability.

[0021] The large language model training method provided in this embodiment creates multiple environments so that the model can be fully trained in different environments, thereby enhancing the generalization ability of the model, enabling the model to provide reliable and accurate answers when faced with unprecedented logical problems, thereby improving its applicability in various scenarios.

[0022] In an optional implementation, obtaining the probability value of the first reasoning step includes: The second logical question and the first reasoning step are input into the second largest language model to obtain the probability value of the first reasoning step.

[0023] The large language model training method provided in this embodiment obtains the probability value of the first reasoning step, guides the model to optimize based on the probability value of the first reasoning step, and enhances the logical reasoning ability of the model.

[0024] In an optional implementation, based on the saved second logic problem, the reasoning steps of the second logic problem, the probability values ​​of the reasoning steps, and the scores of the reasoning steps, the model parameters of the second largest language model are adjusted to obtain a target large language model with generalization ability, including: Determine the adjacent reasoning steps of the second logical problem under a reference model and the probability value of each of the adjacent reasoning steps based on the saved second logical problem, the reasoning steps of the second logical problem, and the probability values ​​of the reasoning steps, wherein the reference model is the second largest language model without model parameter adjustment; Determine a score for each of the adjacent reasoning steps based on the saved scores of the reasoning steps; determining a fourth loss value based on adjacent reasoning steps of the second logical problem under the reference model, a probability value of each reasoning step in the adjacent reasoning steps, and a score of each reasoning step in the adjacent reasoning steps; Based on the fourth loss value, the model parameters of the second largest language model are adjusted to obtain a target large language model with generalization capability.

[0025] The large language model training method provided in this embodiment determines a fourth loss value based on adjacent reasoning steps of the second logical problem under a reference model, the probability value of each reasoning step in the adjacent reasoning steps, and the score of each reasoning step in the adjacent reasoning steps, and adjusts the model parameters of the second large language model based on the fourth loss value to obtain a target large language model with generalization ability, thereby enhancing the model's logical reasoning ability and generalization ability.

[0026] In an optional implementation, determining a fourth loss value based on adjacent reasoning steps of the second logic problem under the reference model, a probability value of each reasoning step in the adjacent reasoning steps, and a score of each reasoning step in the adjacent reasoning steps includes: Obtaining a probability value of each reasoning step in adjacent reasoning steps of the second logical problem under the second largest language model corresponding to the current model parameters; For any reasoning step in adjacent reasoning steps, based on the probability value of the reasoning step in the adjacent reasoning step of the second logical problem under the second largest language model corresponding to the current model parameters and the probability value of the reasoning step in the adjacent reasoning step of the second logical problem under the reference model, determine the divergence of the reference model and the second largest language model corresponding to the current model parameters at the reasoning step, so as to obtain the divergence of the reference model and the second largest language model corresponding to the current model parameters at each reasoning step in the adjacent reasoning steps; A fourth loss value is determined based on the score of each reasoning step in the adjacent reasoning steps and the divergence between the reference model and the second largest language model corresponding to the current model parameters at each reasoning step in the adjacent reasoning steps.

[0027] The large language model training method provided in this embodiment determines a fourth loss value by the score of each reasoning step in adjacent reasoning steps and the divergence of the reference model and the second largest language model corresponding to the current model parameters at each reasoning step in adjacent reasoning steps, and adjusts the model parameters of the second largest language model based on the fourth loss value to obtain a target large language model with generalization ability, which helps to guide the model to learn a more reasonable and accurate reasoning process, thereby improving the performance of the model in logical reasoning tasks and making the reasoning steps output by it more logical and realistic.

[0028] In an optional implementation, determining a fourth loss value based on a score of each reasoning step in adjacent reasoning steps and a divergence between a reference model and a second largest language model corresponding to the current model parameters at each reasoning step in adjacent reasoning steps includes: determining a first reward value based on a score of a first reasoning step in the adjacent reasoning steps and a divergence between the reference model and a second largest language model corresponding to the current model parameters under the first reasoning step; determining a second reward value for a second reasoning step in the adjacent reasoning steps based on the score of each reasoning step in the adjacent reasoning steps; determining a third reward value based on a score of a second reasoning step in the adjacent reasoning steps, a divergence of a second largest language model corresponding to the reference model and the current model parameters at the second reasoning step, and a second reward value of the second reasoning step in the adjacent reasoning steps; determining a fourth loss value based on the first reward value and the third reward value; The adjacent reasoning steps include the first reasoning step and the second reasoning step.

[0029] The large language model training method provided in this embodiment rewards the self-improvement behavior of the model during the reasoning process, motivating the model to continuously optimize in subsequent reasoning and try to generate better reasoning steps, thereby gradually improving the overall performance and reasoning quality of the model.

[0030] In a second aspect, the present invention provides a method for processing a logic problem, comprising: Get pending logic issues; The logic problem to be processed is input into the target large language model trained by the large language model training method according to the first aspect or any corresponding implementation method thereof, to obtain the thought chain data corresponding to the logic problem to be processed.

[0031] The logic problem processing method provided in this embodiment processes the logic problem to be processed by using the target large language model trained by the large language model training method of the first aspect or any corresponding implementation method mentioned above, thereby achieving efficient and accurate processing of logic problems.

[0032] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the large language model training method of the first aspect or any corresponding embodiment thereof or the logic problem processing method of the second aspect by executing the computer instructions.

[0033] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the large language model training method of the first aspect or any corresponding embodiment thereof, or to execute the logic problem processing method of the second aspect.

[0034] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the large language model training method of the above-mentioned first aspect or any corresponding embodiment thereof, or to execute the logic problem processing method of the above-mentioned second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 is a flowchart of a large language model training method according to an embodiment of the present invention; Figure 2 is a flowchart of another language model training method according to an embodiment of the present invention; Figure 3 is a flowchart of another language model training method according to an embodiment of the present invention; Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0038] The large language model is a complex model system built with the help of deep learning technology and trained on massive data. This type of model has powerful capabilities and can accurately understand natural language while also generating fluent and natural language content.

[0039] The large language model exhibits two thinking modes when processing information, just like fast thinking and slow thinking in human thinking. In the fast thinking mode, the large language model relies on intuition and existing pattern recognition capabilities to quickly give answers. This mode is suitable for scenarios that require quick responses, such as daily conversations, simple questions and answers, and other scenarios that do not require complex logical reasoning.

[0040] Slow thinking involves complex logical reasoning and analysis. In slow thinking mode, the large language model needs to analyze the problem, construct a complex thinking process, and gradually derive the answer. This mode is suitable for complex problems that require in-depth analysis and reasoning, such as scientific problem solving, long-form writing, etc.

[0041] The large language models in related technologies have made significant progress in fast thinking and have been used effectively in scenarios that require quick responses. However, when dealing with complex logical reasoning problems, that is, in terms of slow thinking, the large language models in related technologies often need to rely on technologies such as prompt engineering or multi-solution search, which have poor processing effects when solving complex logical reasoning problems such as those in science and mathematics.

[0042] Specifically, in the related technologies, the large language model improves the logical reasoning ability of the large language model through chain-of-thought technology (COT), tree of thoughts technology (TOT), search reasoning enhancement technology, STaR method, Quiet-STaR method, etc.

[0043] Among them, COT is a technology that enhances the reasoning ability of LLM, which aims to improve the performance of the model in complex tasks by generating intermediate reasoning steps. The characteristics of COT include: Step-by-step reasoning: COT requires the model to gradually deduce a series of intermediate steps before generating the final answer. These steps constitute a chain of thinking to help the model understand the problem more clearly and draw correct conclusions. Explainability: By displaying the results of reasoning, COT improves the transparency of model decisions, enabling users to understand how the model reaches a certain conclusion. Scope of application: COT technology is applicable to a variety of reasoning tasks, including arithmetic reasoning, common sense reasoning, and symbolic reasoning. It helps the model to perform more effective logical reasoning when faced with problems that require combining multiple pieces of information.

[0044] TOT aims to enhance the performance of LLM in complex reasoning tasks. It is an evolution of the COT method. By introducing a tree structure to organize the logical thinking process, the model can consider multiple paths and options when solving problems, thereby improving the model's reasoning ability and output quality.

[0045] Search reasoning enhancement technology can use LLM to generate multiple answers during reasoning, and then determine the final answer by voting or combining it with a reward model.

[0046] The STaR method gradually guides the model to improve its ability to perform complex reasoning by iteratively utilizing a small number of reasoning examples and a large number of non-reasoning data sets. This method relies on a simple cycle: generate reasoning procedures for multiple questions, prompt them with a small number of reasoning examples; if the generated answer is wrong, then try to generate the reasoning procedure again given the correct answer, fine-tune all the reasoning procedures that ultimately produced the correct answer, and repeat the above process.

[0047] The Quiet-STaR method allows the LLM to perform a series of internal thinking steps before generating the final answer, that is, to perform sufficient internal reasoning before the model output. These thinking steps can help the model understand the problem more deeply, thereby generating more accurate and comprehensive answers.

[0048] However, COT, TOT and search reasoning enhancement techniques do not involve updating the weights of LLM, which means that they rely on LLM to have long-term thinking ability and self-correction ability in the pre-training and fine-tuning stages, which is obviously unrealistic, so the model is less effective in dealing with complex logical problems. Long-term thinking ability means the ability to conduct complex, coherent and long-chain thinking and reasoning. Even if the data set for training LLM contains some reasoning chain data, the diversity and generalization of the obtained LLM are still poor, and the effect of dealing with complex logical problems is poor. STaR and Quiet-STaR techniques attempt to enhance the model's complex logical reasoning ability through fine-tuning. Fine-tuning is to further train the model using a specific data set based on the pre-trained model to optimize the model's performance on specific tasks. However, the way these two methods obtain specific data sets is based on direct guidance from COT. The long-term thinking ability and self-correction ability of the data set obtained in this way only remain at the solution level rather than the step level, resulting in poor results for the model in dealing with complex logical problems.

[0049] An embodiment of the present invention provides a large language model training method. For any first logical problem in a first logical problem set, the first logical problem is input into a first large language model corresponding to any set of hyperparameters to obtain thinking chain data of the first logical problem, wherein the thinking chain data includes multiple reasoning steps. The large language model generates a current reasoning step based on the first logical problem, the previous reasoning step and the evaluation information of the previous reasoning step; based on the first logical problem, the thinking chain data of the first logical problem and the evaluation information of the reasoning step in the thinking chain data, the model parameters of the first large language model are adjusted to obtain a second large language model with preliminary logical reasoning ability and a reasoning step scoring model; a second logical problem set is obtained, and based on the second logical problem set and the reasoning step scoring model, the model parameters of the second large language model are adjusted to obtain a target large language model with generalization ability, wherein the amount of logical problem data of the second logical problem set is greater than that of the first logical problem set, thereby achieving the effect of improving the logical reasoning ability and generalization ability of the large language model, so that the target large language model can accurately handle complex logical problems.

[0050] According to an embodiment of the present invention, an embodiment of a large language model training method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0051] In this embodiment, a large language model training method is provided, which can be used in mobile terminals, such as servers, central processing units, etc. Figure 1 is a flow chart of a large language model training method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps: Step S101, obtaining a first language model for output formatting.

[0052] Among them, a pre-trained large language model in the related technology is obtained, and the first large language model with output formatting is obtained by fine-tuning the model parameters in the pre-trained large language model.

[0053] Model parameters include weight parameters.

[0054] Step S102, obtaining a first set of logical questions and multiple sets of hyperparameters of a first language model.

[0055] The first logic problem set includes a plurality of first logic problems. Logic problems are problems involving logical reasoning, which usually require conclusions to be drawn through reasonable reasoning steps based on given premises, rules or information.

[0056] For example, the logic problem may be: "A circle with a radius of 5 is circumscribed to a circle with a radius of 3. Connect the centers of the two circles and draw the perpendicular bisector of the line segment. If the perpendicular bisector intersects the two circles at A and C respectively, find the length of the line segment AC".

[0057] Hyperparameters can be adjusted manually, and hyperparameters can be the temperature and sampling range of the first language model output.

[0058] Exemplarily, the number of groups of hyperparameters may be 32. Of course, the number of groups may be any other number, which is not specifically limited here.

[0059] Step S103, for any first logical problem in the first logical problem set, input the first logical problem into the first large language model corresponding to any set of hyperparameters to obtain the thinking chain data of the first logical problem, the thinking chain data includes multiple reasoning steps, and the large language model generates the current reasoning step based on the first logical problem, the previous reasoning step and the evaluation information of the previous reasoning step.

[0060] It can be understood that, for any first logical problem, multiple thought chain data can be obtained.

[0061] When generating any reasoning step in the thought chain data, the large language model generates the reasoning step based on the first logical problem, the reasoning step before the reasoning step, that is, the previous reasoning step, and the evaluation information of the previous reasoning step.

[0062] It should be noted that the evaluation information of the previous reasoning step includes the evaluation information of the most recent reasoning step in the reasoning steps before the reasoning step.

[0063] That is, when generating any reasoning step in the thought chain data, the large language model generates the reasoning step based on the first logical question, the reasoning step before the reasoning step, i.e., the previous reasoning step, and the evaluation information of the most recent reasoning step in the reasoning step before the reasoning step, i.e., the evaluation information of the most recent reasoning step in the previous reasoning step. The evaluation information includes scores and guidance opinions.

[0064] Step S104, based on the first logical problem, the thinking chain data of the first logical problem and the evaluation information of the reasoning steps in the thinking chain data, the model parameters of the first language model are adjusted to obtain the second language model and the reasoning step scoring model with preliminary logical reasoning ability.

[0065] Among them, the loss function used to obtain the second largest language model with preliminary logical reasoning ability is different from the loss function used to obtain the reasoning step scoring model.

[0066] The loss function used to obtain the second largest language model with preliminary logical reasoning ability calculates the loss value of the reasoning step content output by the first largest language model.

[0067] The loss function used to obtain the scoring model for the inference step calculates the loss value of the score after each inference step.

[0068] Step S105, obtaining a second set of logical questions, and adjusting the model parameters of the second large language model based on the second set of logical questions and the reasoning step scoring model to obtain a target large language model with generalization capability, wherein the amount of logical question data in the second set of logical questions is greater than that in the first set of logical questions.

[0069] The logic problems in the second logic problem set may include some of the logic problems in the first logic problem set, or may include all of the logic problems in the first logic problem set.

[0070] It should be noted that the second largest language model has preliminary logical reasoning ability, that is, it has preliminary long thinking chain ability, but its generalization is not strong enough. Therefore, it is necessary to use reinforcement learning methods to adjust the model parameters of the second largest language model based on the second logical problem set and reasoning step scoring model to obtain the target large language model with generalization ability.

[0071] The large language model training method provided in this embodiment, by inputting the first logical problem into the first large language model corresponding to any set of hyperparameters for any first logical problem in the first logical problem set, obtains the thinking chain data of the first logical problem, the thinking chain data includes multiple reasoning steps, and the large language model generates the current reasoning step based on the first logical problem, the previous reasoning step and the evaluation information of the previous reasoning step, so as to ensure that the generated thinking chain data has the action of making mistakes and correcting mistakes, thereby adjusting the model parameters of the first large language model based on the first logical problem, the first logical chain data and the evaluation information of the reasoning steps in the thinking chain data, and the obtained second large language model has more accurate and reasonable logical reasoning ability. A second logical problem set with more logical problem data is obtained, and the model parameters of the second large language model are adjusted based on the second logical problem set and the reasoning step scoring model, so that the second large language model can be exposed to more abundant and diverse logical problem scenarios, learn a wider range of logical patterns and laws, thereby enhancing the generalization ability of the model when facing new and unseen logical problems. The target large language model obtained by the large language model training method of this embodiment can accurately handle complex logical problems.

[0072] In this embodiment, a large language model training method is provided, which can be used in mobile terminals, such as servers, central processing units, etc. Figure 2is a flow chart of a large language model training method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps: Step S201, obtaining the first language model for output formatting.

[0073] Specifically, the above step S201 includes: Step S2011, obtaining a data set, where the training samples in the data set include logic problems and reasoning steps corresponding to the logic problems.

[0074] The reasoning steps corresponding to the logic problem are that the logic problem is decomposed into a series of logically related steps. The reasoning steps can be generated using COT and few-shot prompt engineering. This dataset is used to fine-tune the pre-trained large language model.

[0075] For example, a training sample can be as follows: Question: A circle with a radius of 5 is circumscribed to a circle with a radius of 3. Connect the centers of the two circles and draw the perpendicular bisector of the line segment. If the perpendicular bisector intersects the two circles at points A and C respectively, find the length of line segment AC.

[0076] Step 1: Analyze the geometric relationship and center distance of the two circles.

[0077] According to the question, the two circles are circumscribed, and their radii are 5 and 3 respectively, so the distance between their centers is equal to the sum of the radii of the two circles, that is, 5+3=8. The distance between the centers is recorded as O1O2=8.

[0078] (Should" " is a symbol in the format) Step 2: Find the geometric characteristics of the length of AC.

[0079] Connect the centers O1 and O2 and draw the perpendicular bisector of the line segment. Then A and C are the intersection points of the two circles on the perpendicular bisector. This means that line segment AC is perpendicular to O1O2 and on the perpendicular bisector of the line connecting the centers of the circles. Since A and C are located on the large circle and the small circle respectively, the distance from A to the center O1 is 5, and the distance from C to the center O2 is 3.

[0080]

[0081] Step 3: Determine the length expression of AC.

[0082] In the right triangle O1AO2, the sum of the distance from A to the straight line O1O2 and the distance from C to the straight line O1O2 is the length of AC, that is, AC=O1A+O2C=5+3.

[0083]

[0084] Step 4: Calculate the length of AC.

[0085] Therefore, the length of AC is 5 + 3 = 8.

[0086]

[0087] Answer: The length of line segment AC is 8.

[0088] It can be understood that the questions in the training samples are logic problems, and the logic steps corresponding to the logic problems in the training samples can be the first step, the second step, the third step, the fourth step, the answer, etc.

[0089] Step S2012: Based on the data set, adjust the model parameters of the pre-trained large language model to obtain the first large language model with output formatting.

[0090] After the data set is obtained, the model parameters of the pre-trained large language model are adjusted using the data set to obtain the first large language model with formatted output. The pre-trained large language model is a large language model in the related art that has the ability to understand and process natural language text.

[0091] The first large language model for formatted output is a large language model that can generate formatted reasoning steps. The formatted reasoning steps can refer to outputting a streamlined solution line by line according to the reasoning steps for solving the problem until the problem is solved.

[0092] That is, the first language model can output data according to the formatted reasoning steps.

[0093] Step S202, obtaining a first set of logical questions and multiple sets of hyper parameters of the first language model. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0094] Step S203, for any first logical problem in the first logical problem set, input the first logical problem into the first large language model corresponding to any set of hyperparameters to obtain the thinking chain data of the first logical problem, the thinking chain data includes multiple reasoning steps, and the large language model generates the current reasoning step based on the first logical problem, the previous reasoning step and the evaluation information of the previous reasoning step.

[0095] Specifically, the above step S203 includes: Step S2031, input the first logical problem into the first language model corresponding to any set of hyperparameters to obtain the first reasoning step.

[0096] It can be understood that since the first language model can output formatted reasoning steps, after the first logical problem is input into the first language model corresponding to any set of hyperparameters, different reasoning steps are sampled using the specified step delimiter as the generation terminator.

[0097] Step S2032, input the first logical problem and multiple first reasoning steps corresponding to the first logical problem into the third largest language model, obtain the target first reasoning step and evaluation information of the target first reasoning step, wherein the logical reasoning ability of the third largest language model is higher than the logical reasoning ability of the pre-trained large language model.

[0098] Among them, the evaluation information includes scores and guidance.

[0099] The first logical problem is input into the first largest language model corresponding to any one of the multiple sets of hyperparameters, and the first reasoning step output by the first largest language model corresponding to each set of hyperparameters is obtained, then multiple first reasoning steps are obtained for the first logical problem.

[0100] The third largest language model is used to screen multiple first reasoning steps to obtain a target first reasoning step. Exemplarily, 32 first reasoning steps are obtained, and the 32 first reasoning steps and the first logical question are input into the third largest language model to obtain eight target first reasoning steps.

[0101] The third language model can also be used to obtain the evaluation information of the first reasoning step of the eight targets.

[0102] Step S2033: Based on the target first reasoning step, determine the target hyperparameters corresponding to the target first reasoning step.

[0103] After determining the target first reasoning step, a hyperparameter corresponding to the first largest language model outputting the target first reasoning step is determined as the target hyperparameter.

[0104] Step S2034, input the first logical problem, the target first reasoning step, and the evaluation information of the target first reasoning step into the first language model corresponding to the target hyperparameter to obtain the second reasoning step.

[0105] Among them, after obtaining the first reasoning step of the target, the long thinking chain can be expanded based on the first reasoning step of the target to obtain step-level thinking chain data.

[0106] Step S2035, using the third language model, based on the target first reasoning step, the second reasoning step and the first logical problem, determine the evaluation information of the second reasoning step.

[0107] After obtaining the second reasoning step corresponding to each target first reasoning step, the target first reasoning step, the second reasoning step and the first logical question are input into the third language model to obtain evaluation information of the second reasoning step. The evaluation information includes guidance, and the guidance includes guidance for the second reasoning step and guidance for the next reasoning step of the second reasoning step.

[0108] When using the third language model to generate guidance for the second reasoning step, the following prompts can be used: "Please provide a detailed analysis of the current step in the following chain of reasoning: 1. First, check whether the step is logically correct and reasonable, whether it conforms to the question, and whether there is any misunderstanding or deviation.

[0109] 2. Confirm whether the step clearly expresses the reasoning process and can effectively lead to the next step of reasoning.

[0110] 3. Provide specific suggestions for possible improvements, including how to optimize expression, supplement missing information, or adjust the order of reasoning.

[0111] 4. Finally, based on the analysis of the current step, give specific guidance for the next step of generation to ensure that the next step of reasoning can be closely connected and promote problem solving.

[0112] Please analyze the current steps in point-by-point form and give corresponding opinions."

[0113] The reasoning chain may include currently acquired reasoning steps arranged in the order of acquisition.

[0114] Step S2036, input the target first reasoning step, the second reasoning step, the first logical problem, and the evaluation information of the second reasoning step into the first language model corresponding to the target hyperparameter to obtain the third reasoning step.

[0115] Step S2037, iteratively execute using the third language model, based on the first logical problem, the reasoning step before the currently obtained reasoning step and the currently obtained reasoning step, determine the evaluation information of the currently obtained reasoning step, input the first logical problem, the reasoning step before the currently obtained reasoning step, the currently obtained reasoning step and the evaluation information of the currently obtained reasoning step into the first language model corresponding to the target hyperparameter, obtain the next reasoning step, until the logical reasoning is completed, and obtain the thinking chain data of the first logical problem.

[0116] It is understandable that according to the thinking of the two agents' debate, the final thought chain data is obtained through continuous iteration and guidance. Among them, the two agents are the first and third largest language models.

[0117] It should be noted that the method of obtaining thought chain data in this embodiment can also be applied to the acquisition of multimodal long thought chain data.

[0118] Step S204: Based on the first logical question, the thinking chain data of the first logical question and the evaluation information of the reasoning steps in the thinking chain data, the model parameters of the first language model are adjusted to obtain the second language model with preliminary logical reasoning ability and the reasoning step scoring model. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0119] Step S205, obtain a second set of logical questions, adjust the model parameters of the second large language model based on the second set of logical questions and the reasoning step scoring model, and obtain a target large language model with generalization ability, wherein the amount of logical question data in the second set of logical questions is greater than the amount of logical question data in the first set of logical questions. For details, please refer to Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0120] It should be noted that step S204 and step S205 can be performed alternately, which can not only improve the generalization ability, but also have better logical reasoning ability in the corresponding technical field.

[0121] It should be further explained that the target large language model of this embodiment can be used orthogonally with the multi-path search technology to further improve the logical reasoning ability of the large language model, so that the large language model has multi-path long-term thinking ability.

[0122] The large language model training method provided in this embodiment adjusts the model parameters of the pre-trained large language model to obtain the first large language model with formatted output, thereby improving the interpretability and transparency of the model output, ensuring the format consistency and standardization of the model output results, and reducing misunderstandings or errors caused by inconsistent formats of output results.

[0123] The large language model training method provided in this embodiment obtains the first reasoning step by inputting the first logical problem into the first large language model corresponding to any set of hyperparameters, and then uses the third large language model with stronger logical reasoning ability to screen the first reasoning step to obtain the target first reasoning step and the evaluation information of the target first reasoning step, thereby ensuring the diversity of the generated thinking chain data and reducing the amount of calculation for generating the thinking chain data.

[0124] During each iteration, the subsequent reasoning steps will be further adjusted and optimized based on the current reasoning steps and their evaluation information, ensuring the logical consistency, coherence and accuracy of the final output thinking chain data, and ensuring that the generated thinking chain data has the actions of making and correcting errors, thereby improving the logical reasoning ability of the second largest language model and improving the processing effect of complex logical problems.

[0125] In some optional implementations, the above step S2012 includes: Step a1: input the training sample into the pre-trained large language model to obtain a first output result, where the first output result represents the probability value of the label output by the pre-trained large language model being the label corresponding to the training sample.

[0126] Step a2: determining a first loss value based on the first output result, wherein the first loss value includes a loss value of a format symbol.

[0127] Step a3: Based on the first loss value, adjust the model parameters of the pre-trained large language model until the adjustment termination condition is reached, and obtain the first large language model with output formatting.

[0128] Among them, the model parameters of the pre-trained large language model are updated using the gradient descent algorithm to minimize the first loss value.

[0129] Based on the first loss value, the model parameters of the pre-trained large language model are adjusted until the adjustment termination condition is reached, and the first large language model with output formatting is obtained, including: Based on the first loss value, the model parameters of the pre-trained large language model are adjusted until the first loss value is no greater than the first loss value threshold, thereby obtaining the first large language model with output formatting.

[0130] It is understandable that the adjustment termination condition may be that the first loss value is not greater than a first loss value threshold, wherein the first loss value threshold is set by a technician and is not specifically limited here.

[0131] The large language model training method provided in this embodiment can significantly improve the accuracy and robustness of the model by continuously optimizing the model parameters so that the first loss value is gradually reduced until it is no greater than the set first loss value threshold.

[0132] The large language model training method provided in this embodiment obtains the loss value of the format symbol, and adjusts the model parameters of the pre-trained large language model according to the loss value of the format symbol to obtain the first large language model with formatted output, thereby improving the interpretability and transparency of the model output, ensuring the format consistency and standardization of the model output results, and reducing misunderstandings or errors caused by inconsistent formats of output results.

[0133] In some optional implementations, the above step a2 includes: Step a21, determining the position of the format symbol mark based on the reasoning steps corresponding to the logical problem in the training sample.

[0134] Token refers to the information of a single word understood by the computer, which can be understood as a standard word or Chinese phrase. The output of a large language model is a token.

[0135] For example, the format symbols may be "First step:", "Second step:", " " and other format symbols. The format symbol mark is the mark corresponding to the format symbol.

[0136] Step a22, based on the position of the format symbol mark, determining from the first output result the probability value of the mark corresponding to the position of the format symbol mark being the mark corresponding to the format symbol.

[0137] Step a23, determining a first loss value based on the probability value that the mark corresponding to the position of the format symbol mark is the mark corresponding to the format symbol.

[0138] Specifically, the first loss value is determined by inputting the probability value of the mark corresponding to the position of the format symbol mark as the mark corresponding to the format symbol into the first loss function.

[0139] Among them, the first loss function is:

[0140] in, is the first loss value, k is the position of the format symbol mark, is the first (i-1) label of the first output result. , and the model parameters of the pre-trained large language are 1, the i-th token of the first output result is the probability value of the token corresponding to the format symbol.

[0141] That is to say, is the probability value that the mark corresponding to the position marked by the format symbol in the first output result is the mark corresponding to the format symbol.

[0142] It should be noted that this step only fine-tunes the output format of the pre-trained large language model, so when calculating the loss value, only the loss value of the format symbol is calculated.

[0143] The first loss function can measure the difference between the token corresponding to the format symbol in the output result of the pre-trained large language model and the token corresponding to the format symbol in the actual reasoning step.

[0144] The large language model training method provided in this embodiment determines a first loss value by determining the probability value of the mark corresponding to the position of the format symbol mark determined in the first output result being the mark corresponding to the format symbol, and adjusts the model parameters of the pre-trained large language model based on the first loss value to obtain the first large language model with formatted output, thereby ensuring the formatting and consistency of the model output.

[0145] In some optional implementations, the above step S2032 includes: Step b1, using the third language model, based on the first logical problem, screen multiple first reasoning steps corresponding to the first logical problem from the dimensions of correctness, uncertainty and repeatability to obtain the target first reasoning step.

[0146] Among them, since the first reasoning step directly affects the analysis of the logical problem and the subsequent thinking chain, in order to ensure the diversity of the thinking chain, the importance difference sampling technology is adopted. Using the third language model, based on the first logical problem, multiple first reasoning steps corresponding to the first logical problem are screened from the correctness dimension, uncertainty dimension and repeatability dimension to obtain the target first reasoning step.

[0147] Specifically, the third language model selects the target first reasoning step by using the prompt word prompt engineering method. For example, the prompt word can be: "Please select 8 items that meet the conditions from the following 32 data items. Please judge according to the following 3 indicators: 1. Correctness: Select data that is error-free, logical, and clear; 2. Uncertainty: Prioritize data that is unique, inspiring, or has additional room for interpretation to avoid selecting content that is too certain; 3. Repeatability: Avoid repetitive or similar content, and give priority to data that is independent and non-repetitive.

[0148] The 8 selected data should be representative in terms of overall content and quality, while avoiding data that does not meet the requirements. Please strictly follow these three criteria to complete the screening and output a list of 8 data".

[0149] It can be understood that the data in the prompt word is the first reasoning step.

[0150] The large language model training method provided in this embodiment utilizes the third large language model to screen multiple first reasoning steps corresponding to the first logical problem from the dimensions of correctness, uncertainty and repeatability based on the first logical problem, obtain the target first reasoning steps, remove the first reasoning steps that do not meet the requirements, and ensure the accuracy of the obtained thinking chain data.

[0151] In some optional implementations, the above step S204 includes: Step c1, input the first logical question and the thinking chain data of the first logical question into the first large language model to obtain a second output result, and the second output result represents the probability value of the mark output by the first large language model corresponding to the first logical question and the thinking chain data of the first logical question.

[0152] Among them, the first logical question and a chain of thinking data corresponding to the first logical question are a training sample.

[0153] Step c2, based on the second output result, determine a second loss value, the second loss value includes the loss value of the reasoning step of the thinking chain data.

[0154] The second output result is input into the second loss function to obtain a second loss value.

[0155] It should be noted that the second loss function is similar to the first loss function, but it calculates the loss value of the reasoning step of the thinking chain data.

[0156] Step c3: Based on the second loss value, adjust the model parameters of the first language model to obtain a second language model with preliminary logical reasoning capabilities.

[0157] Step c4, input the first logical problem, the thinking chain data of the first logical problem and the evaluation information of the reasoning steps in the thinking chain data into the first large language model to obtain a third output result, and the third output result represents the probability value of the mark corresponding to the first logical problem, the thinking chain data of the first logical problem and the evaluation information of the reasoning steps in the thinking chain data output by the first large language model.

[0158] Among them, the evaluation information of the reasoning steps in the thinking chain data can be placed after the corresponding reasoning steps of the thinking chain data of the first logical problem.

[0159] The first logical problem, a piece of thinking chain data corresponding to the first logical problem, and evaluation information of the reasoning steps in the thinking chain data are a training sample.

[0160] Step c5: determining a third loss value based on the third output result, where the third loss value includes a loss value of the score of the reasoning step.

[0161] The evaluation information includes a score. When the score of the reasoning step is placed after each reasoning step, a loss value of the score after each reasoning step, ie, a third loss value, is calculated.

[0162] The third output result is input into the third loss function to determine the third loss value. It should be noted that the third loss function is similar to the first loss function, but the loss value of the score of the reasoning step is obtained.

[0163] Step c6: Based on the third loss value, adjust the model parameters of the first language model to obtain a reasoning step scoring model, wherein the evaluation information includes a score.

[0164] The large language model training method provided in this embodiment calculates the loss value of the reasoning step of the thinking chain data and the loss value of the score of the reasoning step, and adjusts the model parameters of the first large language model based on the loss value of the reasoning step of the thinking chain data and the loss value of the score of the reasoning step, thereby improving the model's logical reasoning ability and the accuracy of the reasoning steps.

[0165] In some optional implementations, the above step S205 includes: Step d1, creating multiple environments, and the multiple environments share the second largest language model.

[0166] Among them, the second largest language model has preliminary logical reasoning ability, but its generalization is poor. In order to improve the generalization ability of the model, the multi-environment asynchronous proximal policy optimization (PPO) method is used to enhance the generalization ability of the second largest language model and obtain the target large language model. The PPO method is a reinforcement learning method.

[0167] Among them, the process of using the PPO method to enhance the generalization ability of the second largest language model includes: Step 1: Multiple environments (Env) are created by multithreading. Each environment (Env) has a common agent. Based on any second logic problem in the second logic problem set, the agent in each environment infers the first reasoning step, records it as action A1, i.e., action (A1), and updates the state (observation) of the environment to the first logic problem and the first reasoning step. In this embodiment, the agent is the second largest language model.

[0168] Step 2: Create a runtime instance of reinforcement learning training, which mainly includes: multiple created environments, agents, buffers, parametric reward models (PRM), and reinforcement learning trainer PPO. PRM is the scoring model for the aforementioned reasoning step.

[0169] Step 3: Reset the environment, give a second logic problem, and start a reinforcement learning process.

[0170] Step 4: Start the inference of the second largest language model and obtain action (A1), which is the decoded tokens or tokens, and the probability value (logits) of action (A1). The state of the environment and action (A1) are input into LLM to obtain the probability value of action (A1).

[0171] Step 5: Send the action and environment status inferred by the second largest language model to PRM to obtain the reward value, that is, the score of the action.

[0172] Step 6: Return to step 1 to get the status of the new environment.

[0173] Step 7: Insert the reward value, probability value and action into the buffer for later use.

[0174] Steps 4 to 7 are repeated until the simulations in all environments are completed. The completion of the simulations in all environments is marked by the environment reaching a done state or reaching a simulation depth (inference step number threshold) reaching a preset depth threshold. For example, the preset depth threshold may be 8.

[0175] Step 8: In the training phase, the data stored in the buffer, i.e., the value of each reasoning step, is used to perform learning and training using the PPO strategy. The data in the buffer includes: (1) obs_batch (state data, i.e., logical problems); (2) action_batch (actions under the state, i.e., reasoning steps); (3) log_prob_batch (confidence value of the action, i.e., the probability value of the reasoning step); (4) reward_batch (reward value of the current state, i.e., the score of the reasoning step); (5) action_tokens_batch (action identifier, the position of the reasoning step in the preset dictionary).

[0176] Step 9: During the training process, update the model parameters of the agent, that is, update the model parameters of the second largest language model to obtain the target large language model.

[0177] Step d2, in each environment, for any second logical problem in the second logical problem set, input the second logical problem into the second largest language model to obtain the first reasoning step of the second logical problem.

[0178] Corresponding to the aforementioned inference of the second largest language model, action (A1) is obtained.

[0179] Step d3, obtaining the probability value of the first reasoning step.

[0180] Step d4, inputting the first reasoning step and the second logical question into the reasoning step scoring model to obtain the score of the first reasoning step.

[0181] Step d5, inputting the second logical problem and the first reasoning step into the second largest language model to obtain the second reasoning step of the second logical problem.

[0182] Step d6, obtaining the probability value of the second reasoning step.

[0183] Step d7, inputting the second reasoning step, the second logical question and the first reasoning step into the reasoning step scoring model to obtain the score of the second reasoning step.

[0184] Step d8, iteratively executes inputting the second logical problem and the reasoning step before the reasoning step to be obtained into the second largest language model, obtaining the reasoning step to be obtained, obtaining the probability value of the reasoning step to be obtained, inputting the second logical problem, the reasoning step to be obtained and the reasoning step before the reasoning step to be obtained into the reasoning step scoring model, obtaining the score of the reasoning step to be obtained, until the environment completes the logical reasoning for the second logical problem.

[0185] Step d9, saving the second logical problem, the reasoning steps of the second logical problem, the probability values ​​of the reasoning steps, and the scores of the reasoning steps in the logical reasoning process.

[0186] Step d10, based on the saved second logical problem, the reasoning steps of the second logical problem, the probability values ​​of the reasoning steps and the scores of the reasoning steps, the model parameters of the second largest language model are adjusted to obtain a target large language model with generalization ability.

[0187] For the description of steps d1 to d10, please refer to steps 1 to 9 in the process of using the PPO method to enhance the generalization ability of the second language model.

[0188] The large language model training method provided in this embodiment creates multiple environments so that the model can be fully trained in different environments, thereby enhancing the generalization ability of the model, enabling the model to provide reliable and accurate answers when faced with unprecedented logical problems, thereby improving its applicability in various scenarios.

[0189] In some optional implementations, the above step d3 includes: Step d31, input the second logical question and the first reasoning step into the second largest language model to obtain the probability value of the first reasoning step.

[0190] It can be understood that the calculation method of the probability values ​​of the remaining reasoning steps is similar to the calculation method of the probability value of the first reasoning step, and will not be repeated here.

[0191] The large language model training method provided in this embodiment obtains the probability value of the first reasoning step, guides the model to optimize based on the probability value of the first reasoning step, and enhances the logical reasoning ability of the model.

[0192] In some optional implementations, the above step d10 includes: Step d101, based on the saved second logical problem, the reasoning steps of the second logical problem and the probability values ​​of the reasoning steps, determine the adjacent reasoning steps of the second logical problem under the reference model and the probability value of each reasoning step in the adjacent reasoning steps, wherein the reference model is the second largest language model without model parameter adjustment.

[0193] Exemplarily, the second logic problem is input into the reference model to obtain the reasoning steps of the second logic problem under the reference model, wherein the first reasoning step and the second reasoning step in the reasoning steps of the second logic problem under the reference model are adjacent reasoning steps, and the second reasoning step and the third reasoning step are adjacent reasoning steps. It should be noted that the first and second representations in the first reasoning step and the second reasoning step obtain the order of the reasoning steps.

[0194] It can be understood that the saved reasoning steps and probability values ​​of the reasoning steps of the second logical problem are obtained by inputting the reference model of the second logical problem, that is, the saved reasoning steps and probability values ​​of the reasoning steps of the second logical problem are the reasoning steps and probability values ​​of the reasoning steps of the second logical problem under the reference model.

[0195] Step d102, based on the saved scores of the reasoning steps, determine the score of each reasoning step in the adjacent reasoning steps.

[0196] The scores of the saved inference steps are obtained through the inference step scoring model.

[0197] Step d103, determining a fourth loss value based on the adjacent reasoning steps of the second logical problem under the reference model, the probability value of each reasoning step in the adjacent reasoning steps, and the score of each reasoning step in the adjacent reasoning steps.

[0198] Step d104: Based on the fourth loss value, adjust the model parameters of the second largest language model to obtain a target large language model with generalization capability.

[0199] The large language model training method provided in this embodiment determines a fourth loss value based on adjacent reasoning steps of the second logical problem under a reference model, the probability value of each reasoning step in the adjacent reasoning steps, and the score of each reasoning step in the adjacent reasoning steps, and adjusts the model parameters of the second large language model based on the fourth loss value to obtain a target large language model with generalization ability, thereby enhancing the model's logical reasoning ability and generalization ability.

[0200] In some optional implementations, the above step d103 includes: Step d1031, obtaining the probability value of each reasoning step in the adjacent reasoning steps of the second logical problem under the second largest language model corresponding to the current model parameters.

[0201] The probability value of each reasoning step is obtained by inputting the second logical question and the corresponding reasoning steps into the second largest language model, and then the probability value of each reasoning step in adjacent reasoning steps is determined.

[0202] Step d1032, for any reasoning step in adjacent reasoning steps, based on the probability value of the second logical problem in the adjacent reasoning steps under the second largest language model corresponding to the current model parameters and the probability value of the second logical problem in the adjacent reasoning steps under the reference model, determine the divergence of the reference model and the second largest language model corresponding to the current model parameters at the reasoning step, so as to obtain the divergence of the reference model and the second largest language model corresponding to the current model parameters at each reasoning step in the adjacent reasoning steps.

[0203] Step d1033, determining a fourth loss value based on the score of each reasoning step in the adjacent reasoning steps and the divergence between the reference model and the second largest language model corresponding to the current model parameters at each reasoning step in the adjacent reasoning steps.

[0204] Specifically, the score of each reasoning step in adjacent reasoning steps and the divergence between the reference model and the second largest language model corresponding to the current model parameters at each reasoning step in adjacent reasoning steps are input into the fourth loss function to obtain a fourth loss value.

[0205] Among them, the fourth loss function is:

[0206] in, is the fourth loss value, The current model parameters are The second largest language model, To obtain the maximum value, is the divergence, E is the expectation, For the second logical problem, is the score of the i-th reasoning step in the adjacent reasoning steps, is the inference step output by the reference model, is a hyperparameter, is the probability value of the i-th reasoning step in the adjacent reasoning steps of the second logical problem under the reference model, For the second logical problem, the current model parameters are The probability value of the i-th reasoning step in the adjacent reasoning steps under the second largest language model, is the reference model, is the model input corresponding to the i-th reasoning step in the adjacent reasoning steps of the second logic problem, 1 is the first reasoning step in the contiguous reasoning steps, 2 is the second reasoning step in the adjacent reasoning steps, and the reference model is the second largest language model without model parameter adjustment.

[0207] It can be understood that when the adjacent reasoning steps are the first reasoning step and the second reasoning step, when i=1, B is the second logical problem, and when i=2, B is the second logical problem and the first reasoning step.

[0208] That is to say, is the divergence of the second largest language model corresponding to the reference model and the current model parameters at the i-th reasoning step in the adjacent reasoning steps. Among them, the value of B changes according to the value of i, that is, Changes according to the value of i.

[0209] The large language model training method provided in this embodiment determines a fourth loss value by the score of each reasoning step in adjacent reasoning steps and the divergence of the reference model and the second largest language model corresponding to the current model parameters at each reasoning step in adjacent reasoning steps, and adjusts the model parameters of the second largest language model based on the fourth loss value to obtain a target large language model with generalization ability, which helps to guide the model to learn a more reasonable and accurate reasoning process, thereby improving the performance of the model in logical reasoning tasks and making the reasoning steps output by it more logical and realistic.

[0210] In some optional implementations, the above step d1033 includes: Step e1, determining a first reward value based on a score of a first reasoning step in adjacent reasoning steps and a divergence between a reference model and a second largest language model corresponding to current model parameters under the first reasoning step.

[0211] Among them, for the fourth loss function, when i=1, by calculating [ - ], determine the first reward value.

[0212] Step e2, determining a second reward value of a second reasoning step in the adjacent reasoning steps based on the score of each reasoning step in the adjacent reasoning steps.

[0213] Specifically, the second reward value is obtained by inputting the score of each reasoning step in the adjacent reasoning steps into the reward loss function.

[0214] Among them, the reward loss function is:

[0215] in, is the second reward value, is a hyperparameter, is the score of the second reasoning step in a sequence of reasoning steps, Score for the first reasoning step in a sequence of reasoning steps.

[0216] Step e3, determining a third reward value based on the score of the second reasoning step in the adjacent reasoning steps, the divergence of the second largest language model corresponding to the reference model and the current model parameters under the second reasoning step, and the second reward value of the second reasoning step in the adjacent reasoning steps.

[0217] Among them, for the fourth loss function, when i=2, the fourth loss function Updated to { + }Calculate the third reward value.

[0218] That is, by calculating [ - ], determine the third reward value.

[0219] Step e4, determining a fourth loss value based on the first reward value and the third reward value.

[0220] The adjacent reasoning steps include a first reasoning step and a second reasoning step. The first reasoning step is a reasoning step with an earlier acquisition order among the adjacent reasoning steps, and the second reasoning step is a reasoning step with a later acquisition order among the adjacent reasoning steps.

[0221] Specifically, by inputting the first reward value and the third reward value into the fourth loss function, the fourth loss value is determined.

[0222] The fourth loss function is an improved PPO strategy loss function, which can reward reflective and corrective behaviors between reasoning steps.

[0223] The large language model training method provided in this embodiment rewards the self-improvement behavior of the model during the reasoning process, motivating the model to continuously optimize in subsequent reasoning and try to generate better reasoning steps, thereby gradually improving the overall performance and reasoning quality of the model.

[0224] In this embodiment, a large language model training method is provided, which can be used in mobile terminals, such as servers, central processing units, etc. Figure 3 is a flow chart of a large language model training method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps: Step S301: Format and fine-tune the large language model.

[0225] Among them, LLM format fine-tuning is to obtain the first language model for output formatting. Please refer to the description of the aforementioned step S201 for details, which will not be repeated here.

[0226] Step S302: data collection for the first reasoning step of the large language model.

[0227] Among them, LLM step1 (first reasoning step) data collection is to input the first logical problem and multiple first reasoning steps corresponding to the first logical problem into the third language model to obtain the target first reasoning step. Please refer to the description of the aforementioned step S2032 for details, which will not be repeated here.

[0228] Step S303: long thinking chain data collection based on the second reasoning step and subsequent reasoning steps of multiple agents.

[0229] The data collection of the long thinking chain based on step2-stepN (the second reasoning step and the subsequent reasoning steps) of the multi-agent is described in the aforementioned steps S2034 to S2037, which will not be repeated here. The multi-agent includes the first language model and the third language model.

[0230] Step S304: fine-tune the content of the large language model (long thought chain).

[0231] Among them, the LLM content (long thought chain) fine-tuning is to adjust the model parameters of the first language model based on the first logical problem, the thought chain data of the first logical problem and the evaluation information of the reasoning steps in the thought chain data, so as to obtain the second language model with preliminary logical reasoning ability and the reasoning step scoring model. For details, please refer to the description of the aforementioned step S204, which will not be repeated here.

[0232] Step S305: reinforcement learning generalization training.

[0233] The reinforcement learning generalization training is to adjust the model parameters of the second language model based on the second logic problem set and the reasoning step scoring model to obtain a target language model with generalization ability. For details, please refer to the description of the aforementioned step S205, which will not be repeated here.

[0234] It is understandable that when cultivating the logical reasoning ability of LLM, it is very important to obtain data containing wrong steps and corresponding modifications and correct long-term thinking chain data. The large language model training method provided in this embodiment obtains thinking chain data by using the importance difference sampling method and the long-term thinking chain data of the multi-agent expansion step level, and fine-tunes the first large language model based on the thinking chain data to obtain the second large language model with preliminary logical reasoning ability, that is, the long-term thinking ability of logical consistency, step-by-step thinking, and self-reflection modification. The second large language model has long-term thinking ability in a certain knowledge field, but due to data limitations, generalization and step reflection ability are poor. Using the PPO method, the step-level error correction is added to the loss function constraint, and the enhancement and generalization of long-term thinking ability are completed in the environment of reinforcement learning, and the target large language model is obtained. The logical reasoning ability of the target large language model is strong, and the processing effect of complex logical problems is improved.

[0235] In this embodiment, a logic problem processing method is provided, which can be used in a mobile terminal, such as a server, a central processing unit, etc. The logic problem processing method includes: Get pending logic issues.

[0236] The logic problem to be processed is input into the target large language model trained by the large language model training method shown in any of the above embodiments to obtain the thinking chain data corresponding to the logic problem to be processed.

[0237] The logic problem processing method provided in this embodiment processes the logic problem to be processed by using the target large language model obtained by training with the large language model training method shown in any of the above embodiments, thereby achieving efficient and accurate processing of logic problems.

[0238] The embodiment of the present invention also provides a computer device, see Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4As shown, the computer device includes: one or more processors 401, memory 402, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Each component utilizes different buses to communicate with each other, and can be installed on a common mainboard or installed in other ways as required. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 401 is taken as an example.

[0239] The processor 401 may be a central processing unit, a network processor or a combination thereof. The processor 401 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable logic gate array, a general purpose array logic or any combination thereof.

[0240] The memory 402 stores instructions executable by at least one processor 401 , so that the at least one processor 401 executes the method shown in the above embodiment.

[0241] The memory 402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 402 may optionally include a memory remotely arranged relative to the processor 401, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0242] The memory 402 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 402 may also include a combination of the above types of memory.

[0243] The computer device also includes a communication interface 403, which is used for the computer device to communicate with other devices or a communication network.

[0244] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0245] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0246] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A large language model training method, characterized in that: The method comprises: Get the first language model for output formatting; Obtain multiple sets of hyperparameters for a first logic problem set and a first language model; For any first logical problem in the first logical problem set, the first logical problem is input into a first large language model corresponding to any set of hyperparameters to obtain thinking chain data of the first logical problem, wherein the thinking chain data includes a plurality of reasoning steps, and the large language model generates a current reasoning step based on the first logical problem, a previous reasoning step, and evaluation information of the previous reasoning step; Based on the first logical question, the thinking chain data of the first logical question and the evaluation information of the reasoning steps in the thinking chain data, adjusting the model parameters of the first language model to obtain a second language model with preliminary logical reasoning ability and a reasoning step scoring model; A second logical question set is obtained, and based on the second logical question set and the reasoning step scoring model, model parameters of the second large language model are adjusted to obtain a target large language model with generalization capability, wherein the amount of logical question data in the second logical question set is greater than the amount of logical question data in the first logical question set.

2. The method according to claim 1, characterized in that The first language model for obtaining output formatting includes: Acquire a data set, wherein training samples in the data set include logic problems and reasoning steps corresponding to the logic problems; Based on the data set, model parameters of the pre-trained large language model are adjusted to obtain a first large language model with formatted output.

3. The method according to claim 2, characterized in that The step of adjusting the model parameters of the pre-trained large language model based on the data set to obtain the first large language model with output formatting includes: Inputting the training sample into the pre-trained large language model to obtain a first output result, wherein the first output result represents a probability value of a label output by the pre-trained large language model being a label corresponding to the training sample; Based on the first output result, determining a first loss value, the first loss value comprising a loss value of a format symbol; Based on the first loss value, the model parameters of the pre-trained large language model are adjusted until an adjustment termination condition is reached, thereby obtaining a first large language model with output formatting.

4. The method according to claim 3, characterized in that The determining a first loss value based on the first output result includes: Determining the position of the format symbol mark based on the reasoning steps corresponding to the logic problem in the training sample; Based on the position of the format symbol mark, determining from the first output result a probability value that the mark corresponding to the position of the format symbol mark is the mark corresponding to the format symbol; A first loss value is determined based on a probability value that the mark corresponding to the position of the format symbol mark is the mark corresponding to the format symbol.

5. The method according to claim 2, characterized in that: The step of inputting the first logical question into a first language model corresponding to any set of hyperparameters to obtain the thought chain data of the first logical question includes: Inputting the first logical question into a first language model corresponding to any set of hyperparameters to obtain a first reasoning step; Inputting the first logic question and a plurality of first reasoning steps corresponding to the first logic question into a third large language model to obtain a target first reasoning step and evaluation information of the target first reasoning step, wherein the logic reasoning ability of the third large language model is higher than the logic reasoning ability of the pre-trained large language model; Based on the target first reasoning step, determining a target hyperparameter corresponding to the target first reasoning step; Inputting the first logical question, the target first reasoning step, and evaluation information of the target first reasoning step into a first large language model corresponding to the target hyperparameter to obtain a second reasoning step; Determining evaluation information of the second reasoning step based on the target first reasoning step, the second reasoning step and the first logical question using the third language model; Inputting the target first reasoning step, the second reasoning step, the first logical question, and the evaluation information of the second reasoning step into the first large language model corresponding to the target hyperparameter to obtain a third reasoning step; Iterative execution utilizes the third language model to determine evaluation information of the currently obtained reasoning step based on the first logical problem, the reasoning step before the currently obtained reasoning step, and the currently obtained reasoning step, inputs the first logical problem, the reasoning step before the currently obtained reasoning step, the currently obtained reasoning step, and the evaluation information of the currently obtained reasoning step into the first language model corresponding to the target hyperparameter, obtains the next reasoning step, until the logical reasoning is completed, and obtains the thinking chain data of the first logical problem; The evaluation information includes scores and guidance.

6. The method according to claim 5, characterized in that The step of inputting the first logical question and a plurality of first reasoning steps corresponding to the first logical question into a third language model to obtain a target first reasoning step includes: By using the third language model and based on the first logical problem, multiple first reasoning steps corresponding to the first logical problem are screened from the dimensions of correctness, uncertainty and repeatability to obtain the target first reasoning step.

7. The method according to claim 1, characterized in that The method of adjusting the model parameters of the first language model based on the first logical question, the thinking chain data of the first logical question and the evaluation information of the reasoning steps in the thinking chain data to obtain the second language model and the reasoning step scoring model with preliminary logical reasoning ability includes: Inputting the first logical question and the thinking chain data of the first logical question into the first large language model to obtain a second output result, wherein the second output result represents a probability value of a mark output by the first large language model that is marked as corresponding to the first logical question and the thinking chain data of the first logical question; Based on the second output result, determining a second loss value, wherein the second loss value includes a loss value of a reasoning step of the thought chain data; Based on the second loss value, adjusting the model parameters of the first language model to obtain a second language model with preliminary logical reasoning ability; Input the first logical question, the thinking chain data of the first logical question, and the evaluation information of the reasoning steps in the thinking chain data into the first large language model to obtain a third output result, wherein the third output result represents the probability value of the mark output by the first large language model corresponding to the first logical question, the thinking chain data of the first logical question, and the evaluation information of the reasoning steps in the thinking chain data; Based on the third output result, determine a third loss value, wherein the third loss value includes a loss value of the score of the reasoning step; Based on the third loss value, adjusting the model parameters of the first language model to obtain an inference step scoring model; The evaluation information includes a score.

8. The method according to claim 1, characterized in that The adjusting the model parameters of the second large language model based on the second logical question set and the reasoning step scoring model to obtain a target large language model with generalization capability includes: Creating multiple environments, where the multiple environments share the second largest language model; In each environment, for any second logical problem in the second logical problem set, input the second logical problem into the second largest language model to obtain a first reasoning step of the second logical problem; Obtaining a probability value of the first reasoning step; Inputting the first reasoning step and the second logic question into the reasoning step scoring model to obtain a score for the first reasoning step; Inputting the second logical problem and the first reasoning step into the second largest language model to obtain a second reasoning step for the second logical problem; Obtaining a probability value of the second reasoning step; Inputting the second reasoning step, the second logical question and the first reasoning step into the reasoning step scoring model to obtain a score for the second reasoning step; Iteratively execute inputting the second logical question and the reasoning step before the reasoning step to be obtained into the second largest language model, obtaining the reasoning step to be obtained, acquiring the probability value of the reasoning step to be obtained, inputting the second logical question, the reasoning step to be obtained, and the reasoning step before the reasoning step to be obtained into the reasoning step scoring model, obtaining the score of the reasoning step to be obtained, until the logical reasoning of the environment for the second logical question is completed; The second logical question, the reasoning steps of the second logical question, the probability values ​​of the reasoning steps, and the scores of the reasoning steps in the logical reasoning process are saved; Based on the saved second logical problem, the reasoning steps of the second logical problem, the probability values ​​of the reasoning steps and the scores of the reasoning steps, the model parameters of the second large language model are adjusted to obtain a target large language model with generalization ability.

9. The method according to claim 8, characterized in that The obtaining the probability value of the first reasoning step includes: The second logical question and the first reasoning step are input into the second large language model to obtain a probability value of the first reasoning step.

10. The method according to claim 8, characterized in that The method of adjusting the model parameters of the second large language model based on the saved second logical problem, the reasoning steps of the second logical problem, the probability values ​​of the reasoning steps, and the scores of the reasoning steps to obtain a target large language model with generalization ability includes: Determine adjacent reasoning steps of the second logical problem under a reference model and the probability value of each of the adjacent reasoning steps based on the saved second logical problem, the reasoning steps of the second logical problem, and the probability value of the reasoning steps, wherein the reference model is the second largest language model without model parameter adjustment; Determining a score for each of the adjacent reasoning steps based on the saved scores of the reasoning steps; Determine a fourth loss value based on adjacent reasoning steps of the second logical problem under the reference model, a probability value of each reasoning step in the adjacent reasoning steps, and a score of each reasoning step in the adjacent reasoning steps; Based on the fourth loss value, the model parameters of the second large language model are adjusted to obtain a target large language model with generalization capability.

11. The method according to claim 10, characterized in that The determining of the fourth loss value based on the adjacent reasoning steps of the second logical problem under the reference model, the probability value of each reasoning step in the adjacent reasoning steps, and the score of each reasoning step in the adjacent reasoning steps comprises: Obtaining a probability value of each reasoning step in adjacent reasoning steps of the second logical problem under the second largest language model corresponding to the current model parameters; For any reasoning step in adjacent reasoning steps, based on the probability value of the reasoning step in the adjacent reasoning step of the second logical problem under the second largest language model corresponding to the current model parameters and the probability value of the reasoning step in the adjacent reasoning step of the second logical problem under the reference model, determine the divergence of the reference model and the second largest language model corresponding to the current model parameters under the reasoning step, so as to obtain the divergence of the reference model and the second largest language model corresponding to the current model parameters under each reasoning step in the adjacent reasoning steps; A fourth loss value is determined based on the score of each reasoning step in the adjacent reasoning steps and the divergence of the reference model and the second largest language model corresponding to the current model parameters at each reasoning step in the adjacent reasoning steps.

12. The method according to claim 11, characterized in that The determining of a fourth loss value based on a score of each reasoning step in the adjacent reasoning steps and a divergence between the reference model and the second largest language model corresponding to the current model parameters at each reasoning step in the adjacent reasoning steps comprises: Determining a first reward value based on a score of a first reasoning step in the adjacent reasoning steps and a divergence between the reference model and a second largest language model corresponding to the current model parameters under the first reasoning step; determining a second reward value for a second reasoning step in the adjacent reasoning steps based on the score of each reasoning step in the adjacent reasoning steps; determining a third reward value based on a score of a second reasoning step in the adjacent reasoning steps, a divergence of a second largest language model corresponding to the reference model and the current model parameters under the second reasoning step, and a second reward value of the second reasoning step in the adjacent reasoning steps; determining the fourth loss value based on the first reward value and the third reward value; The adjacent reasoning steps include a first reasoning step and a second reasoning step.

13. A method for processing a logic problem, characterized in that: include: Get pending logic issues; The logic problem to be processed is input into a target large language model trained by the large language model training method according to any one of claims 1 to 12 to obtain the thought chain data corresponding to the logic problem to be processed.

14. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the large language model training method described in any one of claims 1 to 12 or the logic problem processing method described in claim 13 by executing the computer instructions.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the large language model training method described in any one of claims 1 to 12 or the logic problem processing method described in claim 13.

16. A computer program product, characterized in that It includes computer instructions, which are used to enable a computer to execute the large language model training method described in any one of claims 1 to 12 or the logic problem processing method described in claim 13.

Images