Large language model reasoning methods, devices, electronic devices, and readable storage media

Abstract

This invention provides a large language model reasoning method, apparatus, electronic device, and readable storage medium. The method includes: based on the currently generated reasoning content of the problem to be solved using a large language model, the large language model performs self-evaluation of the current reasoning content through prompts, generating a state vector; inputting the state vector into a pre-trained navigator model, outputting the current optimal action; if the current optimal action is not a termination action, using the prompt text corresponding to the current optimal action to drive the large language model to generate new reasoning content, adding the new reasoning content to the current reasoning content, and returning to execute the self-evaluation step; if the current optimal action is a termination action, driving the large language model to generate the final answer based on the current reasoning content and ending the reasoning. This method achieves dynamic and adaptive enhancement of the reasoning logic without modifying the parameters of the large language model, significantly improving the accuracy and structure of the reasoning while maintaining high efficiency.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making technology, and in particular to a large language model reasoning method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] In recent years, Large Language Models (LLMs) have made significant progress in the field of natural language processing. However, when faced with tasks that require complex logical structures, multi-step reasoning, or long-term dependencies (such as mathematical proofs, complex question answering, or planning problems), their word-based autoregressive generation mechanism reveals obvious limitations: the model tends to generate words one by one, making it difficult to explicitly model the global structure and dynamic adjustment of the reasoning path.

[0003] To improve the inference capabilities of LLM (Limited Model Learning), existing technologies mainly fall into two categories. The first category is fine-tuning methods, which involve retraining the model parameters on a specific task dataset. While these methods can effectively improve task performance, they rely on large amounts of labeled data and expensive computational resources, exhibit poor generalization, and have high deployment costs. The second category is inference-time augmentation techniques, typically represented by Chain-of-Thought, Tree-of-Thoughts, and Graph-of-Thoughts. These methods guide the model's step-by-step inference by introducing pre-defined external logical structures, without requiring modification of model parameters, and offer better cost-effectiveness.

[0004] However, the aforementioned reasoning enhancement methods generally employ manually designed, static, and task-independent logical frameworks, forcibly applying the same reasoning structure regardless of changes in task type, difficulty, or context. This strategy lacks the ability to perceive and adapt to task characteristics, making it difficult to flexibly adjust the depth, width, or topological form of the reasoning path in dynamic, heterogeneous, and complex reasoning scenarios, thus limiting its reasoning efficiency and accuracy.

[0005] Therefore, there is an urgent need for a mechanism that can dynamically construct or adjust the reasoning structure according to specific task requirements, in order to overcome the bottlenecks of existing methods in terms of flexibility, adaptability and versatility. Summary of the Invention

[0006] This invention provides a large language model reasoning method, apparatus, electronic device, and readable storage medium to overcome the shortcomings of existing reasoning enhancement methods that lack adaptability due to the use of fixed and static logical frameworks. It can dynamically generate adaptive logical structures for specific tasks during reasoning, thereby enhancing the complex reasoning capabilities of large language models.

[0007] On one hand, the present invention provides a large language model reasoning method, comprising: based on the current reasoning content generated for the problem to be solved by the large language model, prompting the large language model to perform self-evaluation of the current reasoning content, generating a state vector representing the current reasoning state; inputting the state vector into a pre-trained navigator model, outputting the current optimal action; the navigator model is obtained by reinforcement learning training based on a problem sample set; if the current optimal action is not a termination action, using the prompt text corresponding to the current optimal action to drive the large language model to generate new reasoning content, adding the new reasoning content to the current reasoning content, and returning to execute the self-evaluation step; if the current optimal action is a termination action, driving the large language model to generate the final answer based on the current reasoning content and ending the reasoning.

[0008] Furthermore, the step of prompting the large language model to perform self-evaluation of the current reasoning content and generate a state vector representing the current reasoning state includes: prompting the large language model to score the current reasoning content from three dimensions: correctness, complexity, and completeness; and combining the scores of the three dimensions to obtain the state vector.

[0009] Furthermore, the current optimal action can be any one of a single-step reasoning action, a decomposition action, a debate action, an optimization action, and a termination action. Accordingly, if the current optimal action is a decomposition action, the large language model is driven to break down the current problem into multiple sub-tasks and process them sequentially; if the current optimal action is a debate action, the large language model is driven to generate multiple solution schemes and compare them before selecting the optimal scheme to continue reasoning; if the current optimal action is an optimization action, the large language model is driven to correct or improve the current reasoning content.

[0010] Further, the navigator model is trained using reinforcement learning, including: constructing a question sample set based on questions that the large language model directly answers incorrectly; freezing the parameters of the large language model and using the large language model to reason about question samples in the question sample set; prompting the large language model to self-evaluate the existing reasoning content of the question samples, and inputting the self-evaluation result into the navigator model to output a decision action; using the prompt words corresponding to the decision action to drive the large language model to generate the next reasoning content; calling a pre-trained process reward model to perform a quality evaluation of the next reasoning content, and using the score result as an immediate reward signal for reinforcement learning training to update the parameters of the navigator model.

[0011] Furthermore, the navigator model is constructed based on a lightweight neural network.

[0012] Furthermore, throughout the entire reasoning process, no parameters of the large language model are modified; the navigator model dynamically selects the current optimal action to regulate the reasoning flow.

[0013] Secondly, the present invention also provides a large language model reasoning device, comprising: a reasoning state self-evaluation module, used to, based on the current reasoning content generated by the large language model for the problem to be solved, prompt the large language model to self-evaluate the current reasoning content, and generate a state vector representing the current reasoning state; a logical action decision module, used to input the state vector into a pre-trained navigator model, and output the current optimal action; the navigator model is obtained by reinforcement learning training based on a problem sample set; a reasoning loop control module, used to, when the current optimal action is not a terminating action, use the prompt text corresponding to the current optimal action to drive the large language model to generate new reasoning content, add the new reasoning content to the current reasoning content, and return to execute the self-evaluation step; and an answer generation and termination module, used to, when the current optimal action is a terminating action, drive the large language model to generate a final answer based on the current reasoning content and end the reasoning.

[0014] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the large language model reasoning methods described above.

[0015] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the large language model reasoning method as described above.

[0016] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the large language model reasoning method as described above.

[0017] The large language model reasoning method provided by this invention, based on the current reasoning content generated by the large language model for the problem to be solved, prompts the large language model to self-evaluate the current reasoning content, generating a state vector representing the current reasoning state; the state vector is input into a pre-trained navigator model, which outputs the current optimal action; the navigator model is trained through reinforcement learning based on a problem sample set; if the current optimal action is not a termination action, the prompt text corresponding to the current optimal action drives the large language model to generate new reasoning content, which is then added to the current reasoning content, and the self-evaluation step is returned; if the current optimal action is a termination action, the large language model is driven to generate the final answer based on the current reasoning content and the reasoning ends. This method achieves dynamic and adaptive enhancement of the reasoning logic without modifying the parameters of the large language model, significantly improving the accuracy and structure of the reasoning while maintaining high efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the large language model reasoning method provided in this embodiment of the invention.

[0020] Figure 2 This is a schematic diagram illustrating the self-evaluation of the current inference content by the large language model provided in this embodiment of the invention.

[0021] Figure 3 This is a schematic diagram of the overall framework for Markov decision process modeling provided in the embodiments of the present invention.

[0022] Figure 4 This is a schematic diagram of the basic logic module provided in the embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram of the overall process of the large language model reasoning method provided in the embodiments of the present invention.

[0024] Figure 6 This is a schematic diagram of the structure of the large language model reasoning device provided in the embodiment of the present invention.

[0025] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided in the embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] It's important to note that existing technologies for enhancing the reasoning capabilities of large language models mainly fall into two categories. The first category is fine-tuning methods, which achieve significant performance improvements by training model parameters on task-specific datasets. However, these methods require substantial computational resources and data, resulting in high costs. The second category is inference-time augmentation techniques, such as thought chains, thought trees, and mind maps. These methods guide the model's reasoning process through predefined external logical structures without modifying model parameters, thus offering higher cost-effectiveness. However, these methods rely on manually designed, task-independent, fixed logical frameworks, applying uniformly to all different task types. They lack adaptability to task diversity and dynamism, limiting their effectiveness in complex reasoning scenarios.

[0028] In view of this, the present invention proposes a novel reasoning method for large language models, specifically, Figure 1 A flowchart illustrating the large language model reasoning method provided in an embodiment of the present invention is shown.

[0029] like Figure 1 As shown, the method includes: S110, based on the current reasoning content generated for the problem to be solved using a large language model, prompting the large language model to perform self-evaluation of the current reasoning content, and generating a state vector representing the current reasoning state; S120, inputting the state vector into a pre-trained navigator model, and outputting the current optimal action; the navigator model is obtained through reinforcement learning training based on a problem sample set; S130, if the current optimal action is not a termination action, using the prompt text corresponding to the current optimal action to drive the large language model to generate new reasoning content, and adding the new reasoning content to the current reasoning content, returning to execute the self-evaluation step; S140, if the current optimal action is a termination action, driving the large language model to generate the final answer based on the current reasoning content and ending the reasoning.

[0030] The following will provide a detailed description of steps S110-S140 and related steps.

[0031] S110: Based on the large language model, the current reasoning content generated for the problem to be solved is prompted to the large language model to perform self-evaluation of the current reasoning content, and a state vector representing the current reasoning state is generated.

[0032] It is easy to understand that when a user submits a problem to be solved (such as an advanced mathematics problem or other difficult natural science problem), the reasoning process is initiated first.

[0033] In the initial stage, the large language model may not have generated any reasoning content yet, or may only output a restatement of the question. As reasoning progresses, at the beginning of each iteration, a specially designed self-evaluation prompt text is constructed based on the current reasoning content generated by the large language model so far. This self-evaluation prompt text guides the large language model to score its own reasoning quality / state from three predefined dimensions.

[0034] The three predefined dimensions are correctness, complexity, and completeness. The correctness dimension is used to determine whether there are logical, factual, or computational errors in the current reasoning; the complexity dimension is used to reflect the level of abstraction or technical difficulty of the concepts or steps involved in the current reasoning; and the completeness dimension is used to measure which key steps are still missing from the current reasoning to fully solve the problem.

[0035] Each predefined dimension has a score range of 1 to 3. The three scores are combined to form a three-dimensional state vector, which is used to represent the current reasoning progress / state.

[0036] In step S110, based on the current reasoning content generated for the problem to be solved by the large language model, and by prompting the large language model to perform self-evaluation of the current reasoning content and generating a state vector representing the current reasoning state, step S120 is further executed.

[0037] S120, the state vector is input into the pre-trained navigator model, and the current optimal action is output; the navigator model is obtained by reinforcement learning training based on the problem sample set.

[0038] Specifically, the state vector generated in step S110 is input into a pre-trained navigator model to output the current optimal action. The navigator model is a small neural network (e.g., a three-layer multilayer perceptron), whose parameters are obtained through reinforcement learning on a set of problem samples consisting of a large number of problem samples.

[0039] In this embodiment, the problem samples are challenging questions, which are questions that the large language model cannot answer correctly using conventional methods (such as direct question-and-answer or standard thought chains). During training, the navigator model learns to select the actions most conducive to problem-solving based on different state vectors. These actions come from a set of predefined basic logical modules, including five categories: single-step reasoning actions, decomposition actions, debate actions, optimization actions, and termination actions.

[0040] The process is divided into several stages: Single-step reasoning, Decomposition, and Termination. Single-step reasoning involves executing a single reasoning step in the current task, contributing a portion of the problem-solving approach to the overall process. Decomposition involves breaking down a complex task into multiple simpler subtasks and executing them sequentially. Debate involves generating multiple different problem-solving solutions or approaches for the current task, comparing them, and selecting the optimal solution to continue. Optimization involves reviewing and revising existing reasoning steps to improve their clarity and accuracy. Termination involves generating the final answer and ending the reasoning process based on all previous reasoning steps.

[0041] During actual reasoning, the navigator model generates the current optimal action as any one of the following: single-step reasoning action, decomposition action, debate action, optimization action, and termination action.

[0042] Based on the state vector input into the pre-trained navigator model in step S120 and the output of the current optimal action, step S130 is further executed.

[0043] S130, if the current optimal action is not a termination action, use the prompt text corresponding to the current optimal action to drive the large language model to generate new inference content, add the new inference content to the current inference content, and return to the execution self-evaluation step.

[0044] Specifically, if the current optimal action is not a termination action (i.e., the current optimal action is one of the following: single-step reasoning action, decomposition action, debate action, and optimization action), then the corresponding prompt text template is invoked based on the action type. The large language model generates a new inference content based on the corresponding prompt text template and appends this new inference content to the existing current inference content to form an updated complete inference sequence.

[0045] The prompt text template is a natural language instruction that explicitly tells the large language model how to respond. For example, if the current optimal action is decomposition, the prompt is "Decompose the current problem into several independently solvable subproblems and list them in order"; if it is optimization, the prompt is "Review your previous reasoning, check for logical flaws or calculation errors, and provide the corrected version".

[0046] Subsequently, the process automatically returns to step S110, where the updated reasoning content is re-evaluated, a new state vector is generated, and the next decision loop begins. This process continues, gradually building a clear and logically coherent reasoning path.

[0047] S140, if the current optimal action is a termination action, drive the large language model to generate the final answer based on the current reasoning content and end the reasoning.

[0048] Specifically, if the navigator model determines that the current reasoning content is sufficient, it will output a termination action as the current optimal action. At this point, intermediate reasoning steps are no longer requested; instead, a final answer generation prompt is constructed, instructing the large language model to "provide a clear and concise final answer to the original problem based on all the above reasoning content." The large language model integrates the existing reasoning content according to the final answer generation prompt, outputs the final conclusion, and then ends the entire reasoning process.

[0049] It should be noted that the large language model reasoning method provided in this embodiment of the invention is applicable to all reasoning task scenarios, especially to high-difficulty task scenarios that require multi-step deduction and are easily affected by initial thought bias, such as mathematical problem solving, natural science question answering, multi-hop reading comprehension and complex question answering, logic puzzles and algorithm design, etc., and is not specifically limited here.

[0050] In this embodiment, based on the large language model, the current reasoning content generated for the problem to be solved is self-evaluated by the large language model, generating a state vector representing the current reasoning state. This state vector is then input into a pre-trained navigator model, which outputs the current optimal action. The navigator model is trained using reinforcement learning based on a problem sample set. If the current optimal action is not a termination action, the prompt text corresponding to the current optimal action drives the large language model to generate new reasoning content, which is then added to the current reasoning content before returning to the self-evaluation step. If the current optimal action is a termination action, the large language model generates the final answer based on the current reasoning content and the reasoning process ends. This method achieves dynamic and adaptive enhancement of the reasoning logic without modifying the parameters of the large language model, significantly improving the accuracy and structure of the reasoning while maintaining high efficiency.

[0051] Based on the above embodiments, the process of self-evaluation of the large language model in step S110 will be described in detail below.

[0052] The current reasoning content is self-evaluated by the prompting large language model, generating a state vector representing the current reasoning state. This includes: scoring the current reasoning content from three dimensions—correctness, complexity, and completeness—by the prompting large language model; and combining the scores from the three dimensions to obtain the state vector.

[0053] It is easy to understand that once a large language model has generated some reasoning content (i.e., the current reasoning content) for the problem being solved, it needs to quantitatively represent the current reasoning progress / state to support dynamic decision-making for subsequent logical actions. To this end, this embodiment introduces a self-evaluation mechanism, which guides the large language model to perform structured reflection and scoring on its own generated current reasoning content by constructing specific natural language prompts.

[0054] Specifically, Figure 2 This diagram illustrates the self-evaluation of the current inference content by the large language model provided in an embodiment of the present invention. Figure 2 As shown, a self-evaluation prompt text is first constructed for the current reasoning content (existing reasoning steps). The self-evaluation prompt text explicitly requires the large language model to independently score the current reasoning content from three predefined dimensions: correctness, complexity, and completeness.

[0055] Regarding the correctness dimension, it is used to assess whether the current reasoning is accurate in terms of logical deduction, factual citation, or numerical calculation. If there are obvious errors or contradictions, the score is lower; if the reasoning steps are reasonable and the conclusion is reliable, the score is higher.

[0056] The complexity dimension is used to assess the conceptual depth, number of steps, or cognitive load involved in the current reasoning. For example, stating only common-sense conclusions is considered low complexity, while reasoning involving multiple layers of nested arguments or domain-specific knowledge is considered high complexity.

[0057] The completeness dimension is used to assess the extent to which the current reasoning covers the original problem in relation to a complete solution. High completeness indicates that key intermediate conclusions and necessary premises are covered; low completeness indicates that core elements are missing or the essence of the problem is not addressed.

[0058] For the three predefined dimensions above, the large language model will output an integer score, with the score range limited to 1 to 3 points, where 1 point indicates poor performance in the predefined dimension, 2 points indicates average performance, and 3 points indicates good performance.

[0059] After receiving the three scores returned by the large language model, the scores are combined in a fixed order (e.g., correctness, complexity, completeness) into a three-dimensional vector, namely the state vector. As a structured representation of the current reasoning state, the state vector can comprehensively reflect the accuracy, difficulty level, and progress of the current reasoning process, thus providing a reliable basis for decision-making for the navigator model.

[0060] It's important to note that the self-evaluation prompts are applied to the updated inference content in each iteration, thus the generated state vector dynamically tracks changes in the inference process. For example, in the initial stage, the current inference content may only contain a restatement of the problem, resulting in a completeness score of 1 and low correctness and complexity. As the inference unfolds, if the model successfully identifies key sub-problems and provides reasonable derivations, the completeness and correctness scores will improve accordingly. If logical deviations occur later, the correctness score may decrease, triggering optimization actions to correct the errors.

[0061] In this embodiment, the large language model is prompted to score the current reasoning content from three dimensions: correctness, complexity, and completeness. The scores from the three dimensions are combined to obtain a state vector, thereby realizing the external observability and quantitative expression of the internal reasoning state of the large language model.

[0062] Based on the above embodiments, the reinforcement learning training process of the navigator model will be described in detail below.

[0063] The navigator model is trained using reinforcement learning, including: constructing a question sample set based on incorrect answers from the large language model; freezing the parameters of the large language model and using it to reason about the question samples in the question sample set; prompting the large language model to self-evaluate the existing reasoning content of the question samples, and inputting the self-evaluation result into the navigator model to output a decision action; using the prompt words corresponding to the decision action to drive the large language model to generate the next reasoning content; calling the pre-trained process reward model to evaluate the quality of the next reasoning content, and using the quality evaluation score as an immediate reward signal for reinforcement learning training to update the parameters of the navigator model.

[0064] It is easy to understand that the navigator model must be trained before actual deployment. The navigator model is a lightweight neural network (such as a three-layer multilayer perceptron) whose function is to dynamically select the most appropriate logical action based on the current reasoning state during the reasoning process. To achieve this goal, this embodiment models the complex reasoning process of the large language model as a Markov Decision Process (MDP) and regards the navigator model as an agent in the MDP, performing end-to-end training through reinforcement learning.

[0065] Figure 3 A schematic diagram of the overall framework for Markov decision process modeling provided in an embodiment of the present invention is shown. Figure 3As shown, the left side represents the reasoning process. First, the problem to be solved is read. Then, in each step of the reasoning, the large language model generates reasoning content (e.g., step 1: debate, step N: optimization, etc.). After each step, the large language model self-evaluates the existing reasoning content with prompts, extracting scores including dimensions such as correctness and complexity, forming a self-evaluation result (a three-dimensional state vector) to represent the current reasoning state. The middle part represents the agent (i.e., the navigator model). After receiving the self-evaluation result, the agent selects an action (e.g., decomposes the action) according to a reinforcement learning strategy and outputs it to the next reasoning instruction. The right side represents the action space and reward mechanism. The selected action corresponds to a predefined basic logic module, driving the large language model to generate the next reasoning content. Simultaneously, the process reward model is called to score the newly generated next reasoning content as an immediate reward signal (e.g., R(s,a)=80.5), used to train the navigator model.

[0066] To train the navigator model, the first step is to construct a question sample set. Specifically, from the training data of the target task domain (such as mathematics, natural sciences, or complex question answering), questions that the large language model would answer incorrectly using conventional methods (such as directly asking questions or generating answers without guidance) are selected as question samples. These question samples have high cognitive difficulty or logical complexity, and are usually difficult problems. The set of these difficult problems constitutes the question sample set, which is specifically used to train the navigator model.

[0067] Subsequently, the parameters of the large language model are frozen, and an inference simulation is executed. Specifically, during training, the parameters of the large language model remain frozen, meaning they are not fine-tuned or updated. For each question sample in the question sample set, a complete inference simulation process is initiated. Initially, the existing inference content is empty; subsequently, in each step, the large language model is prompted to self-evaluate its existing inference content, obtaining a self-evaluation result consisting of three dimensions: correctness, complexity, and completeness (each dimension's score ranges from 1 to 3). The self-evaluation result is input into the navigator model, which outputs a decision action. The decision action originates from a set of predefined basic logic modules. Figure 4 A schematic diagram of the basic logic module provided in an embodiment of the present invention is shown.

[0068] like Figure 4As shown, from left to right, the actions include single-step reasoning, decomposition, debate, optimization, and termination. Single-step reasoning represents executing an independent reasoning step in the current task, generating information such as "In this step, we calculated..." to advance the reasoning process. Decomposition breaks down complex tasks into multiple subtasks (e.g., subtask 1, subtask 2, etc.) and executes them sequentially, suitable for multi-stage problems. Debate proposes multiple solutions to the current task (e.g., solution 1, solution 2, etc.), comparing and selecting the optimal solution to continue execution, enhancing the diversity and robustness of reasoning. Optimization reviews existing reasoning content, checking and correcting errors or ambiguities, such as "We corrected the error..." to improve reasoning quality. Termination outputs the final answer based on all reasoning results, ending the entire reasoning process.

[0069] Next, the large language model is driven to generate the next inference. Specifically, based on the selected decision action, the corresponding prompt word (i.e., a preset natural language instruction template) is invoked to construct new prompt text, which is then input into the large language model to drive it to generate the next inference. For example, if the decision action is a decomposition action, the prompt word will instruct the large language model to "decompose the current problem into several sub-problems and list them one by one"; if it is an optimization action, the prompt word will require "please check if there are any errors in your previous reasoning and provide a corrected version".

[0070] Then, an immediate reward signal is obtained to update the navigator model. Specifically, a pre-trained process reward model is invoked to evaluate the quality of the generated next inference content, and the quality evaluation score is directly used as the immediate reward signal in reinforcement learning. Here, the process reward model is an independent model pre-trained on a large amount of data labeled with inference quality, which can output a continuous quality evaluation score for any piece of inference text, reflecting its logical coherence, factual accuracy, and degree of progress towards the problem.

[0071] Within the reinforcement learning framework, the goal of the navigator model is to maximize cumulative reward. This embodiment employs the Double-Dueling-DQN algorithm as the training strategy, utilizing the empirical tuples composed of the self-evaluation results, decision actions, and immediate reward signals to continuously update the network parameters of the navigator model through an experience replay mechanism. Notably, throughout the entire training process, the parameters of the large language model and the process reward model remain frozen, while only the parameters of the navigator model are updated, thereby significantly reducing computational overhead and improving training efficiency.

[0072] Finally, training is complete, and the inference deployment phase begins. Once the navigator model has undergone sufficient training iterations (e.g., thousands of inference simulations on a set of question samples), its parameters converge, and training ends. At this point, the process reward model no longer needs to participate in subsequent inference processes.

[0073] In practical applications, when faced with new problems, it is only necessary to call the pre-trained navigator model and the large language model to work together to dynamically generate a task-adaptive reasoning logic structure without the need for additional supervision signals or external scoring mechanisms.

[0074] In this embodiment, a question sample set is constructed by directly answering incorrect questions based on a large language model; the parameters of the large language model are frozen, and the large language model is used to reason about the question samples in the question sample set; the large language model is prompted to self-evaluate the existing reasoning content of the question samples, and the self-evaluation result is input into the navigator model to output a decision action; the prompt words corresponding to the decision action drive the large language model to generate the next reasoning content; a pre-trained process reward model is called to evaluate the quality of the next reasoning content, and the score result is used as an immediate reward signal for reinforcement learning training to update the parameters of the navigator model. This method trains a navigator model capable of intelligently scheduling logic modules and effectively guiding complex reasoning at extremely low computational cost without modifying the parameters of the large language model.

[0075] In other embodiments, Figure 5 This diagram illustrates the overall flow of the large language model reasoning method provided in an embodiment of the present invention. Figure 5 As shown, the overall process is divided into two stages: the navigator model training stage and the navigator model deployment stage.

[0076] During the navigator model training phase, data preparation and model initialization are performed first. Then, a cyclical training process begins, with each round including the following steps: 1) Generating a state evaluation vector: Based on the existing reasoning content of the large language model, a state evaluation vector containing scores for correctness, complexity, and completeness is generated through prompts for self-evaluation (i.e., the self-evaluation results mentioned in the training process above); 2) Selecting a logic module: The state evaluation vector is input into the navigator model, and a decision action is output; 3) Generating reasoning steps: Based on the prompt words corresponding to the selected decision action, the large language model is driven to generate the next reasoning content; 4) Calculating process rewards: A pre-trained process reward model is called to evaluate the quality of the newly generated next reasoning content, serving as an immediate reward signal; 5) Reinforcement learning to update model parameters: Using the aforementioned state evaluation vector, decision action, and reward information, the parameters of the navigator model are updated through a reinforcement learning algorithm.

[0077] After training, the navigator model deployment phase begins. This phase no longer uses the process reward model, retaining only the core reasoning flow: 1) Generate state vector: Obtain the state vector based on the current reasoning content; 2) Select logic module: The trained navigator model selects the current optimal action; 3) Generate reasoning steps: Execute the corresponding reasoning operation based on the current optimal action until a termination action is selected and the final answer is output.

[0078] In one specific embodiment, the user wishes to apply the large language model reasoning method provided by the embodiments of the present invention to improve the ability of the large language model to solve MATH mathematical problems.

[0079] Step 1: Data Preparation and Navigator Model Initialization. From the MATH training set of 7500 questions, 1736 challenging questions that large language models cannot correctly answer using conventional methods (such as direct questioning) were selected and used to train the navigator model. A three-layer Multilayer Perceptron (MLP) was initialized as the navigator model.

[0080] Step 2: Train the Navigator Model. At the start of each training round, a challenging problem from the MATH training set is randomly selected. At each step of reasoning, the current problem-solving progress is first self-evaluated using a prompting language model, generating a state evaluation vector. This state evaluation vector is then input into the navigator model, which outputs a decision action. Next, based on the prompt word template corresponding to the selected decision action, the prompting language model is driven to generate the next reasoning step. For example, if the decomposition action is chosen, the model will first break the problem down into sub-tasks and then solve them one by one. Then, a pre-trained mathematical process reward model is used to score the newly generated next reasoning step, serving as an immediate reward signal. Using this immediate reward signal, the parameters of the navigator model are updated using a reinforcement learning algorithm. This process is repeated until the problem is solved or the maximum number of steps is reached. The entire training process loops for 3000 rounds.

[0081] Step 3: Deploy the Navigator Model. For a new MATH problem, the trained navigator model guides the large language model in reasoning. The navigator model generates a logical chain, such as "decomposition-optimization-debate-termination," based on the problem's characteristics and the evaluation status of intermediate steps. For example, when solving a complex geometric area problem, the navigator first selects a single-step reasoning action to calculate the key intersection points, then selects an optimization action to correct incorrect judgments about the shape of the figure (e.g., correcting a quadrilateral to a rhombus), and then uses a debate action to compare various area calculation schemes and select the optimal rhombus formula method, ultimately arriving at the correct answer.

[0082] In another specific embodiment, the user wishes to apply the large language model reasoning method provided in the embodiments of the present invention to improve the ability of the large language model to solve GPQA natural science problems.

[0083] Step 1: Data Preparation and Navigator Model Initialization. From the 448 questions in the GPQA training set, 255 challenging questions that the large language model cannot answer correctly using conventional methods (such as direct questioning) were selected and used to train the navigator model. A three-layer Multilayer Perceptron (MLP) was initialized as the navigator model.

[0084] Step 2: Train the Navigator Model. At the start of each training round, a challenging problem from the GPQA training set is randomly selected. At each step of reasoning, the current problem-solving progress is first self-evaluated using a prompting language model, generating a state evaluation vector. This state evaluation vector is then input into the navigator model, which outputs a decision action. Next, based on the prompt word template corresponding to the selected decision action, the prompting language model is driven to generate the next reasoning step. For example, if the decomposition action is chosen, the model will first break the problem down into sub-tasks and then solve them one by one. Then, a pre-trained mathematical process reward model is used to score the newly generated next reasoning step, serving as an immediate reward signal. Using this immediate reward signal, the parameters of the navigator model are updated using a reinforcement learning algorithm. This process is repeated until the problem is solved or the maximum number of steps is reached. The entire training process loops for 3000 rounds.

[0085] Step 3: Deploy the navigator model. For a new GPQA problem, use the trained navigator model to guide the large language model in reasoning. The navigator model will generate a logical chain such as "decomposition-optimization-debate-termination" based on the characteristics of the problem and the evaluation state of intermediate steps.

[0086] Corresponding to the large language model reasoning methods described in the above embodiments, this invention also proposes a large language model reasoning device. Specifically, Figure 6 A schematic diagram of the structure of the large language model inference device provided in an embodiment of the present invention is shown.

[0087] like Figure 6 As shown, the device includes: a reasoning state self-evaluation module 610, used to self-evaluate the current reasoning content generated for the problem to be solved based on the large language model, and generate a state vector representing the current reasoning state by prompting the large language model; a logical action decision module 620, used to input the state vector into a pre-trained navigator model and output the current optimal action; the navigator model is obtained by reinforcement learning training based on a problem sample set; a reasoning loop control module 630, used to drive the large language model to generate new reasoning content by using the prompt text corresponding to the current optimal action when the current optimal action is not a termination action, and add the new reasoning content to the current reasoning content, and return to execute the self-evaluation step; and an answer generation and termination module 640, used to drive the large language model to generate the final answer and end the reasoning when the current optimal action is a termination action.

[0088] In this embodiment, the reasoning state self-evaluation module 610, based on the large language model, evaluates the current reasoning content generated for the problem to be solved, and generates a state vector representing the current reasoning state. The logic action decision module 620 inputs the state vector into a pre-trained navigator model and outputs the current optimal action. The navigator model is trained through reinforcement learning based on a problem sample set. When the current optimal action is not a termination action, the reasoning loop control module 630 uses the prompt text corresponding to the current optimal action to drive the large language model to generate new reasoning content, adds the new reasoning content to the current reasoning content, and returns to execute the self-evaluation step. When the current optimal action is a termination action, the answer generation and termination module 640 drives the large language model to generate the final answer based on the current reasoning content and ends the reasoning. This device achieves dynamic and adaptive enhancement of the reasoning logic without modifying the parameters of the large language model, significantly improving the accuracy and structure of the reasoning while maintaining high efficiency.

[0089] It should be noted that the large language model reasoning device provided in the embodiments of the present invention can be referred to in correspondence with the large language model reasoning methods described in the above embodiments, and will not be repeated here.

[0090] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a large language model reasoning method, which includes: based on the current reasoning content generated by the large language model for the problem to be solved, prompting the large language model to perform self-evaluation of the current reasoning content, generating a state vector representing the current reasoning state; inputting the state vector into a pre-trained navigator model, outputting the current optimal action; the navigator model is obtained by reinforcement learning training based on a problem sample set; if the current optimal action is not a termination action, using the prompt text corresponding to the current optimal action to drive the large language model to generate new reasoning content, adding the new reasoning content to the current reasoning content, and returning to execute the self-evaluation step; if the current optimal action is a termination action, driving the large language model to generate the final answer based on the current reasoning content and ending the reasoning.

[0091] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the large language model reasoning method provided by the above methods. The method includes: based on the current reasoning content generated for the problem to be solved by the large language model, prompting the large language model to perform self-evaluation of the current reasoning content, generating a state vector representing the current reasoning state; inputting the state vector into a pre-trained navigator model, and outputting the current optimal action; the navigator model is obtained by reinforcement learning training based on a problem sample set; if the current optimal action is not a termination action, using the prompt text corresponding to the current optimal action to drive the large language model to generate new reasoning content, adding the new reasoning content to the current reasoning content, and returning to execute the self-evaluation step; if the current optimal action is a termination action, driving the large language model to generate the final answer based on the current reasoning content and ending the reasoning.

[0093] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the large language model reasoning method provided by the above methods. The method includes: based on the current reasoning content generated for the problem to be solved by the large language model, prompting the large language model to perform self-evaluation of the current reasoning content, generating a state vector representing the current reasoning state; inputting the state vector into a pre-trained navigator model, and outputting the current optimal action; the navigator model is obtained by reinforcement learning training based on a problem sample set; if the current optimal action is not a termination action, using the prompt text corresponding to the current optimal action to drive the large language model to generate new reasoning content, adding the new reasoning content to the current reasoning content, and returning to the self-evaluation step; if the current optimal action is a termination action, driving the large language model to generate a final answer based on the current reasoning content and ending the reasoning.

[0094] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0095] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reasoning method for large language models, characterized in that, include: Based on the current reasoning content generated for the problem to be solved by the large language model, the large language model is prompted to self-evaluate the current reasoning content and generate a state vector representing the current reasoning state. The state vector is input into a pre-trained navigator model, which outputs the current optimal action. The navigator model is obtained by reinforcement learning training based on a set of problem samples; If the current optimal action is not a termination action, the prompt text corresponding to the current optimal action is used to drive the large language model to generate new inference content, and the new inference content is added to the current inference content, and the self-evaluation step is returned. If the current optimal action is the termination action, the large language model is driven to generate the final answer based on the current inference content and end the inference.

2. The large language model reasoning method according to claim 1, characterized in that, The step of prompting the large language model to perform self-evaluation of the current reasoning content and generating a state vector representing the current reasoning state includes: The large language model is prompted to score the current reasoning content from three dimensions: correctness, complexity, and completeness. The state vector is obtained by combining the scores from the three dimensions.

3. The large language model reasoning method according to claim 1, characterized in that, The current optimal action is any one of the following: single-step reasoning action, decomposition action, debate action, optimization action, and termination action; Accordingly, if the current optimal action is a decomposition action, the large language model is driven to decompose the current problem into multiple sub-tasks and process them sequentially. When the current optimal action is the debate action, the large language model is driven to generate multiple problem-solving solutions, and the optimal solution is selected after comparison to continue reasoning; If the current optimal action is an optimized action, the large language model is driven to correct or improve the current inference content.

4. The large language model reasoning method according to claim 1, characterized in that, The navigator model is trained using reinforcement learning, including: Based on the large language model, directly answer the incorrect questions and construct the question sample set; Freeze the parameters of the large language model and use the large language model to reason about the problem samples in the problem sample set; The large language model is prompted to self-evaluate the existing reasoning content of the problem sample, and the self-evaluation result is input into the navigator model to output a decision action; The prompt words corresponding to the decision action are used to drive the large language model to generate the next reasoning content; The pre-trained process reward model is invoked to evaluate the quality of the next inference content, and the score is used as an immediate reward signal for reinforcement learning training to update the parameters of the navigator model.

5. The large language model reasoning method according to claim 1, characterized in that, The navigator model is constructed based on a lightweight neural network.

6. The large language model reasoning method according to claim 1, characterized in that, Throughout the reasoning process, no parameters of the large language model are modified; the navigator model dynamically selects the current optimal action to regulate the reasoning flow.

7. A large language model reasoning device, characterized in that, include: The reasoning state self-evaluation module is used to generate a state vector representing the current reasoning state by prompting the large language model to self-evaluate the current reasoning content based on the generated current reasoning content of the problem to be solved. The logical action decision module is used to input the state vector into a pre-trained navigator model and output the current optimal action; the navigator model is obtained by reinforcement learning training based on a set of problem samples. The inference loop control module is used to drive the large language model to generate new inference content when the current optimal action is not a termination action, using the prompt text corresponding to the current optimal action, and adding the new inference content to the current inference content, and then returning to the execution self-evaluation step. The answer generation and termination module is used to drive the large language model to generate the final answer and end the reasoning when the current optimal action is the termination action.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the large language model reasoning method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the large language model reasoning method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the large language model reasoning method as described in any one of claims 1 to 6.

Images