Large language model reasoning method based on time sequence difference learning and rule enhancement

Abstract

The invention relates to the field of natural language processing and decision intelligence, in particular to a large language model reasoning method based on time sequence differential learning and rule enhancement, and the method is widely applied to scenes such as automatic planning, intelligent question answering and intelligent. Comprising the steps of task trajectory sampling, domain knowledge induction, domain rule extraction and rule enhancement-based large language model reasoning: for test task data, matching a most relevant historical task based on vector retrieval, and obtaining a domain rule set of a corresponding task; regular natural language rewriting is carried out through the large language model, so that the large language model has higher interpretability and adaptability; and finally, integrating the optimized rule into a big language model reasoning prompt text so as to optimize reasoning quality and stability.

Description

Technical Field

[0001] This application relates to the fields of natural language processing and decision-making intelligence, etc., and particularly relates to a large language model inference method based on Temporal Difference Learning (TD Learning) and Rule Augmentation. This method can be widely applied to scenarios such as automatic planning, intelligent question answering, and embodied intelligence. Background Art

[0002] In the field of intelligent agent inference and decision-making, the multi-step inference ability is the key to improving the task processing efficiency and accuracy. Existing intelligent agent decision-making methods usually adopt reinforcement learning to optimize the decision-making strategy through interactive trial-and-error learning. However, reinforcement learning methods highly rely on a large amount of environmental interaction data, with high training costs and limited generalization ability when dealing with new tasks. In recent years, the research direction of applying large language models (LLMs) as intelligent agents to multi-step inference and decision-making tasks has attracted wide attention. These models are pre-trained on large-scale text data and possess rich world knowledge and reasoning abilities, enabling them to perform complex multi-step decision-making tasks in the actual environment. Their decision-making paradigm can be summarized as the language interaction between the intelligent agent and the environment: the LLM serves as the intelligent agent, based on the text description provided by the environment, combines the existing knowledge to reason about the appropriate action, and adjusts the strategy based on the environmental feedback. In this process, the LLM relies on internal knowledge and memory mechanisms for long-term reasoning and complex planning, and can combine methods such as In-context Learning (ICL) and Retrieval-Augmented Generation (RAG) to introduce external knowledge and optimize the inference through knowledge enhancement.

[0003] However, the existing knowledge enhancement methods still have limitations in terms of knowledge utilization efficiency, generalization ability, and inference optimization. For example, the context learning-based method guides the large language model to reason by injecting examples into the prompt. However, this method is limited by the context window size and is difficult to store and reuse knowledge in the long term. In addition, some studies inject domain knowledge into the large language model through model finetuning, such as optimizing the inference strategy using a process reward model. However, these methods usually require a large amount of high-quality labeled data, with high training costs and are difficult to adapt to closed-source large language models, thus being restricted in practical applications.

[0004] Based on this, the present invention applies a knowledge enhancement method based on temporal difference learning and rule induction to optimize the multi-step reasoning of large language models, so as to improve the reasoning decision-making level of LLMs. The present invention can effectively solve problems such as the existing knowledge enhancement methods relying on high-quality positive sample data, the difficulty in balancing the external knowledge injection density and the computational cost, and can be widely applied to scenarios such as automatic planning, intelligent question answering, and embodied intelligence. Summary of the Invention

[0005] To overcome the deficiencies of the prior art, the present invention aims to optimize the multi-step reasoning of large language models based on a knowledge enhancement method of temporal difference learning and rule induction. This method is applicable to situations where tasks can be decomposed into several sub-goals. In such tasks, the agent can measure the task progress based on the achievement of the sub-goals of the decision-making task, and combine rule enhancement to optimize the reasoning process.

[0006] The specific technical solution is as follows: A large language model reasoning method based on Temporal Difference Learning (TDLearning) and Rule Augmentation includes the following steps;

[0007] (1) Task trajectory sampling: Based on the base large language model, perform multiple rounds of random sampling on the training task dataset to generate diverse task reasoning trajectories (Trajectory); further, screen the sampled reasoning trajectories and only retain the valid trajectories.

[0008] (2) Domain knowledge induction: Extract the action semantics from the action steps of all valid reasoning trajectories and map them to a unified action space; classify the reasoning trajectories according to the domain to which the task belongs (Domain), aggregate all trajectories in the same task domain, and extract the general execution patterns within the domain; construct a domain knowledge graph (Domain Knowledge Graph) based on the aggregated trajectory set, with the parameter-free action semantics as nodes and the action semantic dependency relationships in the trajectories as directed edges to form the knowledge structure within the task domain.

[0009] (3) Domain rule extraction: Use the temporal difference method to learn the contribution of different actions to task progress until it converges, and combine PageRank to evaluate the global influence of actions in the domain knowledge graph. By fusing the local contribution degree and the global importance, extract the key task execution rules and form the action dependency relationships, thereby establishing a domain rule set that can be used to optimize the reasoning of LLMs.

[0010] (4) Rule-Enhanced Large Language Model Inference: For test task data, retrieve the most relevant historical tasks based on vector retrieval and obtain the domain rule sets corresponding to the tasks; rewrite the rules in natural language through the large language model itself to make them more interpretable and adaptable; finally, integrate the optimized rules into the large language model inference prompt text to optimize the inference quality and stability.

[0011] Specifically, for step (1), the training task dataset Each task instance (m, o) consists of a task domain m and the specific information o of the task. The task domain m represents the category to which the task belongs; the task specific information o contains the input information of the task. The LLM needs to reason about these task instances to generate possible task inference trajectories. These trajectories contain the states at multiple time steps, the actions performed, and the corresponding task progress, describing how the large language model advances the task progress during multi-step reasoning. The sampled trajectory set can be formally represented as:

[0012]

[0013] τ = {(s1, a1, r1), (s2, a2, r2), …, (s T , a T , r T )},

[0014] where τ represents the complete execution trajectory inferred by the large language model on a specific task instance (m, o), containing all decision-making steps from the initial state to the final result. To ensure the diversity and rationality of the sampled trajectories, temperature scaling and Nucleus sampling (Top-p sampling) are used for trajectory sampling.

[0015] Furthermore, in the multi-step decision-making process, for each decision-making step, the agent outputs the current action based on the current environmental information. Since the actions output based on the large language model are uncertain, the quality of the sampled inference trajectories is improved through data screening. The screening process includes the following key steps: eliminating illegal actions: ensuring that each action in the sampled trajectory conforms to the basic logical rules of the task environment; removing invalid actions: avoiding the large language model generating redundant inference paths. Specifically, for step (2), since the large language model may express the same inference step in different ways, it is necessary to map the actions in the sampled trajectory to a unified semantic space. Each decision-making action a t consists of an action semantics and an action parameter θ t . Among them, the action semantics It represents the general operations across different task scenarios, while the action parameters refer to specific task instances. To aggregate the task trajectories in the same domain, the action parameters are removed so that all actions of the same category are mapped to a unified action space, and only the action semantics are retained. Then, according to the task domain m, all the trajectories in the current task domain are aggregated to obtain T m , and then a directed acyclic graph G m =(V, E) for the task domain m is constructed. Among them, the node set represents the de-parameterized action semantics in the task. The edge set represents the action dependency relationships in the trajectories. If an action usually precedes another action during the task execution, then there will be a directed edge in the graph from to reflects the execution logic of the actions during the reasoning process.

[0016] Specifically in step (3), temporal difference learning is used to calculate the local contribution degree of each action, and a calculation method based on link analysis is combined to evaluate the global influence of the action in the task graph. Through this process, task execution rules are constructed and a structured domain knowledge set is formed. During the task execution, the contributions of different actions to the final goal are not equal. Some key actions may play a decisive role in advancing the task, while the influence of other actions may be smaller. Therefore, it is necessary to calculate the TD contribution degree Q of the action, that is, the degree of influence of an action on the future task advancement in a specific state. For this purpose, the temporal difference error method is adopted to update the TD contribution degree Q to ensure that the Q value can correctly reflect the contribution of the action. Further, the eligibility trace is adopted to improve the stability of learning. Its iterative update process is as follows:

[0017]

[0018] Among them, δ t represents the temporal difference error (TD Error), which is used to measure whether the contribution of the current action meets the expectation; p t is the immediate contribution degree, that is, the task advancement increment directly brought by this action; γ is the discount factor, which is used to control the influence weight of future states on the current decision. The value of γ is between [0, 1]. A larger γ makes the model pay more attention to the long-term benefits, while a smaller γ makes the model more inclined to short-term benefits; is the TD contribution degree of the next state, representing the possible benefits in the future; is the TD contribution degree of the current action, which is continuously updated during the training process; ​is the eligibility trace value, which is used to record the contribution degree of an action to the task progress; α is the learning rate, which controls the step size of Q-value update; λ is the trace decay factor.

[0019] In addition to calculating the local contribution degree, it is necessary to evaluate the global influence of each action in the task graph. During the task execution, some actions may be the core steps on the critical path of the task, while others may be just auxiliary operations. Therefore, it is necessary to measure whether an action plays a key role throughout the task. For this purpose, the PageRank calculation method is introduced to evaluate the global influence of actions in the domain knowledge graph. Combine the TD contribution degree and the PageRank influence to calculate the comprehensive action contribution degree, form the task execution rules, and store them in the domain knowledge set:

[0020]

[0021] Among them, represents the final contribution degree of the action, which synthesizes its local benefit and global influence β is the weight parameter, which is used to adjust the relative weights of the TD contribution degree and the PageRank influence.

[0022] Specifically in step (4), when a test task arrives, it is necessary to retrieve the most relevant training tasks in order to obtain the rule knowledge applicable to the current task. For this purpose, a vector retrieval method is adopted. By calculating the cosine similarity between the current task description h i and the training task description h j to find the most similar training task and obtain its rule set:

[0023]

[0024] Among them, Embed(·) represents the text embedding model, which is used to convert the text task description into a high-dimensional vector representation so that the similarity of different tasks can be measured through mathematical calculations; cos(·,·) is the cosine similarity calculation function, which is used to measure the similarity between two task description vectors. Then, select the top K rules with the highest contribution degree from the retrieved task rule set and use them to optimize the inference ability of the large language model. The key to this process is to ensure that the large language model only refers to the optimal rules for inference optimization and avoid introducing irrelevant or inefficient rules to interfere with the inference results. The calculation method for screening rules is as follows:

[0025]

[0026] Among them, Π mis the complete rule set corresponding to task domain m; represents the action contribution degree, that is, the influence degree of a certain action on task promotion; the contribution degree calculation combines temporal difference learning and PageRank influence; TopK(·) represents sorting according to the contribution degree and selects the top K rules that are most helpful for task completion. The retrieved most relevant rule set represents the rules that are most suitable for reference in the current test task. Since these rules are derived from the execution data of training tasks, they not only have high reliability, but also can be directly applied to test tasks, improving the inference ability of the large language model. Adopt few-shot learning to optimize the selected rules through the internal knowledge of the LLM in natural language, making them more interpretable and applicable to different task scenarios.

[0027] Finally, integrate the optimized rules into the inference prompt of the LLM so that it can make more accurate decisions in test tasks.

[0028] The features and beneficial effects of the present invention are:

[0029] (1) The present invention performs LLM knowledge-enhanced reasoning through temporal difference learning and rule induction, making full use of a large amount of non-positive sample information generated by training tasks. Through multiple samplings of training tasks and temporal difference learning, calculate the value contribution of state-actions to the completion of the decision-making goal from the sampled samples, and construct a generalizable external knowledge base.

[0030] (2) The present invention is based on rule-enhanced context learning for LLM self-improvement, and uses the LLM itself to extract and induce rules based on state-action contributions. This rule-enhanced context learning effectively improves the information entropy of context learning, achieving a balance between instance-based prompt learning and model fine-tuning, and balancing the external knowledge injection density and computational cost.

[0031] (3) The present invention can adapt to various types of task scenarios and supports automatic induction and optimization of knowledge in different task domains, thereby enhancing the generalization ability and adaptability of the inference process, enabling it to be widely applied to multiple application scenarios such as complex task planning, intelligent reasoning, and automatic decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a framework diagram of a large language model inference method based on temporal difference learning and rule enhancement.

[0033] Figure 2 is a flow schematic diagram of a large language model inference method based on temporal difference learning and rule enhancement.

[0034] Figure 3This is a comparison chart of model inference performance. Detailed implementation

[0035] The present invention proposes a multi-step decision-making optimization method for large language models based on rule enhancement. First, sample and screen effective inference trajectories, induce action semantics, and construct a task domain knowledge graph. Then, evaluate the contribution of actions based on temporal difference learning and extract key task rules. Finally, retrieve and match relevant tasks during inference, and self-optimize the rules through the LLM and integrate them into text prompts to improve the inference quality and stability.

[0036] The specific implementation steps of the present invention are as Figure 1 shown, mainly including the following four steps:

[0037] (1) Task trajectory sampling: Based on the base large language model, perform multiple rounds of random sampling on the training task dataset to generate diverse task inference trajectories (Trajectory); further, screen the sampled inference trajectories and only retain the valid trajectories.

[0038] Specifically, each task instance (m, o) of the training task dataset consists of the task domain m and the specific information o of the task. The task domain m represents the category to which the task belongs; the task specific information o contains the input information of the task. For each training data sample, perform N samplings with T time steps through the LLM. The inference trajectory structure obtained by each sampling contains the states, actions performed, and corresponding task progress at multiple time steps:

[0039] τ = {(s1, a1, r1), (s2, a2, r2), …, (s T , a T , r T )},

[0040] The set of sampled trajectories for all training data can be formally represented as:

[0041]

[0042] To improve the diversity of sampling and enable the LLM to explore multiple task execution methods, this method adopts a stochastic sampling (Stochastic Decoding) strategy to avoid the LLM generating single-mode trajectories. Specifically, a method combining temperature scaling and Nucleus sampling (Top-p sampling) is used to control the exploration range during the inference process, so as to generate sufficiently diverse trajectories and avoid low-probability and unreasonable inference paths.

[0043] Temperature regulation adjusts the steepness of the action probability distribution output by the model, enabling the sampling process to explore new paths while maintaining reasonable decision-making stability. Its generation process is as follows:

[0044]

[0045] Among them, z a represents the score of action a in the model output, and δ is the temperature parameter that determines the randomness of sampling. Intuitively, the temperature parameter δ controls the caution of the model when choosing the next action: when δ is large, the probabilities of all possible actions become closer, and the model has strong exploration ability and can try different reasoning paths; when δ is small, the large language model is more inclined to choose actions with high probabilities, thereby reducing the uncertainty of reasoning and improving the stability of reasoning results.

[0046] To further optimize the decoding strategy, Nucleus sampling (Top-p sampling) is used to ensure that the large language model only selects from the most representative candidate action sets and does not sample extremely unreasonable actions. Its calculation method is as follows:

[0047]

[0048] Among them, represents the set of candidate actions of the top p probability mass selected by the large language model, ensuring that the large language model only selects among the most likely actions and does not randomly select extremely low-probability options; is the indicator function, indicating that only actions within will be sampled. The above strategy ensures that the actions sampled by the large language model only contain the most representative candidate set while still retaining a certain exploration ability.

[0049] Furthermore, since the trajectories sampled by the LLM may contain illegal or invalid actions, after sampling, the data needs to be filtered to retain only valid paths. All actions can be divided into three categories: (1) Illegal actions: actions that do not belong to the set of actions allowed by the environment (2) Legal but invalid actions: executable but not contributing to the task progress, that is, r t = r t-1 ; (3) Legal and valid actions: executable and contributing to task completion, that is, r t > r t-1 . Since illegal actions themselves do not have any impact on the state transition of the current task, they can be removed. Specifically, for each trajectory sample τ with a final progress r T > 0, all illegal actions are removed, and only the trajectory composed of legal actions is retained:

[0050] τ′ = {(s′1, a′1, r′1), (s′2, a′2, r′2), …, (s′ T′ , a′ T′ , r′ T′ )}, where r′ T′ > 0,

[0051] where T′ ≤ T, and T is the number of time steps of the original sampled trajectory τ.

[0052] (2) Domain knowledge induction: Extract the action semantics from the action steps of all valid inference trajectories and map them to a unified action space; classify the inference trajectories according to the domain to which the task belongs, aggregate all trajectories in the same task domain, and extract the general execution patterns within the domain; construct a domain knowledge graph (Domain Knowledge Graph) based on the aggregated trajectory set, with the parameter-free action semantics as nodes and the action semantic dependency relationships in the trajectories as directed edges to form the knowledge structure within the task domain.

[0053] Specifically, in a multi-step decision-making task, each decision-making action a t consists of an action semantics and an action parameter θ t . Among them, the action semantics represents a general operation that spans different task scenarios, and the action parameter refers to a specific task instance. To aggregate the trajectories under the same category of tasks, the action parameters are removed so that all actions of the same category are mapped to a unified action space, and only the action semantics are retained. Further, define the progress increment p t of the action a t in the trajectory τ as p t = r t-1 - r t , where r t-1 represents the progress metric at the current time t, and r filtered is the progress metric at the previous time. As a part of the trajectory, obtain the filtered sampled trajectory set where each trajectory τ filtered is:

[0054]

[0055] Because the action semantics have logical commonality in the same domain, this representation method can effectively aggregate the decision-making trajectories under different task instances and be used for knowledge induction or policy optimization. According to the task domain m, aggregate all the trajectories under the current task domain to obtain T m , and then construct a directed acyclic graph G = (V, E) for the task domain m, where the node set and edge set are defined as:

[0056]

[0057] Among them, is the action semantics Average progress contribution value in different trajectories:

[0058]

[0059] Among them, represents the set of trajectories containing the action semantics That is, Domain knowledge graph G m can accurately represent the execution process of task domain m and provide domain knowledge support for subsequent action contribution calculation and rule extraction.

[0060] (3) Domain rule extraction: Use the temporal difference method to learn the contribution of different actions to task progression until it converges, and combine PageRank to evaluate the global influence of actions in the domain knowledge graph. By fusing local contribution and global importance, extract key task execution rules and form action dependencies, thus establishing a domain rule set that can be used to optimize LLM reasoning.

[0061] Specifically, after constructing the domain knowledge graph G corresponding to task domain m m , induct the domain rule set (Domain Rule Set) Φ from it, aiming to capture the rule structure across specific tasks within task domain m:

[0062] Φ = {(φ i , ψ i , θ) ∣ i = 1, …, |Φ|},

[0063] Among them, φ i is the causal antecedent action (Antecedent); ψ i is the causal consequent action (Consequence), representing the action preference to be taken under the condition of satisfying φ i ; θ is the contribution degree of the causal consequence to the task completion degree. Specifically, it is necessary to calculate the contribution degree of each action semantics to the task progress. Use the method based on temporal difference (TD) to estimate the local contribution of different actions to task progression. At the same time, introduce the PageRank method to calculate the global influence of actions in the domain knowledge graph, and finally obtain a comprehensive estimate of the contribution to task progress.

[0064] In the task trajectory, represents the contribution of the action semantics to task progression, that is, the TD contribution degree. Use temporal difference for estimation, and assume that the state transition and immediate reward Δp t in the trajectory are:

[0065]

[0066] Δp t = p t+1 - p t ,

[0067] Based on this, the TD error is calculated as follows:

[0068]

[0069] Among them, γ ∈ [0, 1] is the discount factor, which is used to balance the influence of the current reward and future rewards. To improve the stability and convergence speed of learning, the eligibility trace is introduced, and TD(λ) is used for Q-value update. The update rule is as follows:

[0070]

[0071] Among them, λ ∈ [0, 1] is the eligibility trace decay factor, which is used to control the influence degree of historical information; α is the learning rate, which determines the step size of Q-value update. By accumulating the past TD errors, this method balances the short-term reward and long-term contribution while reducing the estimation variance and improving the adaptability to long-sequence tasks. To effectively control the computational complexity, a random trajectory sampling strategy is adopted, based on N total complete paths of the entire graph, and TD(λ) calculation is only performed on N samples sampling paths in each round, and multiple iterations are performed for update until convergence. Specifically, after the k-th round of iteration, the Q-values of each action semantics are If in several consecutive rounds of iteration, the change amplitude of the Q-values of all actions is lower than a certain threshold ∈, it is considered that the Q-values converge:

[0072]

[0073] Among them, the value range of ∈ is 10e2 or 10e3 to ensure the stability of the estimated value. To prevent misjudging local oscillations as convergence, this condition needs to hold in multiple consecutive rounds of iteration.

[0074] In addition to calculating the direct contribution degree of the TD action to the task progress, it is also necessary to evaluate the global influence of the action in the entire domain knowledge graph. Some actions in the domain knowledge graph G m may play a core role in task completion due to their high frequency of occurrence or connection to multiple key paths. Therefore, the PageRank calculation method is introduced to evaluate the importance of actions in the domain knowledge graph:

[0075]

[0076] Among them, is the PageRank value of the action , indicating the importance of this action in the domain knowledge graph; d is the damping factor, which controls the degree of dependence on other actions during the random walk process. The core idea of the PageRank calculation method is that if an action is depended on by multiple important previous actions, then it itself is also important. Therefore, PageRank assigns higher scores to those actions that play a bridging role during task execution.

[0077] Combine the TD contribution and the PageRank influence to calculate the comprehensive action contribution, form the task execution rules, and store them in the domain knowledge base:

[0078]

[0079] Among them, represents the final contribution of the action, which combines its local benefit (TD contribution ) and global influence β is the weight parameter, used to adjust the relative weights of the TD contribution and the PageRank influence. Sort all actions according to the contribution score to obtain the domain rule set:

[0080]

[0081] Among them, represents the set of parent action nodes of the semantic action in the domain knowledge graph, that is:

[0082]

[0083] Among them, E m is the directed edge set of the domain knowledge graph G=(V, E), representing the dependency relationship in the task process. In particular, the domain rule set Π m is the set of actions arranged in descending order of contribution under the task domain m, and each rule contains a semantic action, a contribution estimate, and a pre-dependency action.

[0084] (4) Large language model reasoning based on rule enhancement: For the test task data, retrieve the most relevant historical tasks based on vector matching and obtain the corresponding domain rule set; rewrite the rules naturally through the large language model itself to make them more interpretable and adaptable; finally, integrate the optimized rules into the large language model reasoning prompt text to optimize the reasoning quality and stability.

[0085] Specifically, based on the rule set Π m, further introduce a context learning mechanism based on rule reinforcement, and adopt a rule retrieval enhancement method in the inference stage to provide reference rule knowledge for the LLM, thereby optimizing its inference process and improving interpretability and generalization ability.

[0086] To construct rule knowledge that can be used for test tasks, we first use the Vector Retrieval mechanism to retrieve the tasks most relevant to the current test task from the training task description set and determine its corresponding task domain m. Calculate the embedding similarity (Cosine i between the initial description h of the current task and the training task description to find the most similar training task and its corresponding task domain m:

[0087] Similarity), and find the most similar training task and its corresponding task domain m:

[0088]

[0089] After retrieving the similar training task description h m , obtain its corresponding task domain m, and select the top K rules with the highest contribution from the rule set Π m :

[0090]

[0091] Among them, TopK(·) represents sorting according to the contribution degree and selecting the top K rules. Further, the selected rules are inductively refined. Each rule is formalized as an Antecedent-Consequence rule:

[0092] Antecedent[Conditions]→Consequence[Actions],

[0093] where the precondition is used as the Antecedent part; is used as the Consequence part. To enhance the interpretability of the rules, we are based on few-shot learning and use the LLM to perform Domain Rule Self-Refinement on the Antecedent-Consequence rules to make them more in line with natural language expressions. Let the initial rule be:

[0094] R=(Antecedent A,Consequence C).

[0095] Define the LLM decoding generation as Optimized under the few-shot learning paradigm:

[0096]

[0097] Among them, is a sample set of few-shot learning, including high-quality rule rewriting samples, so that the optimized rule R′ has stronger interpretability and can be integrated into the inference prompt structure of the LLM as interpretable task experience to guide the inference process:

[0098]

[0099] Among them, s t represents the state information of the current test task input to the agent, and a t represents the next action output by the agent in the domain rule set R′.

[0100] Through the above technical solutions, the present invention significantly improves the adaptability and stability of large language models in multi-step reasoning decision-making tasks by introducing a knowledge enhancement mechanism based on rule learning. Through task trajectory sampling and screening, this method can generate diverse task reasoning paths, and use trajectory aggregation and knowledge induction methods to construct a structured domain knowledge graph, thereby extracting general task knowledge. Combining temporal difference learning and link analysis calculation, this method can accurately evaluate the local contribution and global influence of each action, and automatically induce the rule knowledge for optimizing the inference process of large language models. Compared with the traditional direct reasoning method of large language models, this application can not only use vector retrieval to match the most relevant rules in the test task, but also perform rule induction through few-shot learning, making the inference results more interpretable and generalizable. In addition, by integrating the optimized rules into the large language model inference prompt, this method effectively reduces the occurrence rate of inference errors and hallucinations, and improves the credibility of task decision-making. Generally speaking, the present invention introduces a new technical method in the multi-step reasoning task of large language models, promotes the development of knowledge-enhanced intelligent decision-making optimization solutions, and can be widely applied to task scenarios that require complex reasoning capabilities such as automatic planning, intelligent question answering, and embodied intelligence.

[0101] Example 1: Reasoning and execution of household tasks based on rule enhancement

[0102] In this embodiment, the reasoning method is applied to the household task environment of an embodied agent, and the agent needs to complete complex multi-step operation tasks through text interaction. Taking "put two soap packages into the trash can" as a representative task, the specific application process of the method is shown:

[0103] 1. Task trajectory sampling stage:

[0104] (1) During the training phase, the base large language model (LLM) samples task trajectories related to "item classification and processing" in home scenarios such as the kitchen and bathroom, such as "put dirty dishes in the sink", "throw food into the trash can", etc.

[0106] (2) Each trajectory contains state-action pairs at multiple time steps, including text state descriptions (such as "you are standing at the bathroom door") and actions generated by the model (such as "open the door", "enter the room", "pick up the soap").

[0107] (3) The temperature control and Top-p sampling mechanisms are adopted to ensure trajectory diversity, and illegal and redundant actions are eliminated through the trajectory screening algorithm, and valid trajectories are retained.

[0108] 2. Domain knowledge induction and rule extraction:

[0109] (1) All trajectory actions are mapped to a unified semantic space. For example, actions such as "open the refrigerator" and "open the trash can" are both normalized to the "OPEN_CONTAINER" semantics.

[0110] (2) Build a domain knowledge graph and extract action dependencies, such as "OPEN_CONTAINER" → "PICK_OBJECT"

[0111] → "PUT_OBJECT".

[0112] (3) Calculate the TD contribution of each action based on temporal difference learning, and combine PageRank to calculate the global influence, forming rules:

[0113] Rule 1: IF the target container is closed THEN execute OPEN_CONTAINER (TD contribution = 0.35, PageRank = 0.42) Rule 2: IF the item is in a reachable position THEN execute PICK_OBJECT (TD contribution = 0.31, PageRank = 0.30)

[0114] 3. Rule enhanced reasoning phase:

[0115] (1) In the test task of "put two soap packages into the trash can", the system retrieves and matches the "home scenario - garbage disposal" task domain through vector retrieval, and recalls the high contribution rule set.

[0116] (2) Rewrite the rules into natural language prompts through the LLM:

[0117] "Please first confirm whether the trash can is open. If necessary, the opening operation should be performed first."

[0118] "Before picking up the target item, please first approach the target position."

[0119] (3) The optimized rule prompts are embedded into the model’s reasoning context, significantly improving its action rationality and completion efficiency.

[0120] Example 2: Reasoning optimization of scientific experiment process based on rule enhancement

[0121] In this example, the reasoning method is applied to task reasoning and operation planning in a scientific experiment scenario. The agent is required to complete a causal reasoning task based on observation results in an environment equipped with various experimental equipment. Using the task of "determining whether an iron block has thermal expansion characteristics" as a representative example, the specific application process of the method is demonstrated:

[0122] 1. Mission trajectory sampling phase:

[0123] (1) During the training phase, the base large language model (LLM) samples trajectories on tasks related to “metal physical property measurement”, such as “measuring the length of an aluminum rod before and after heating”, “monitoring the heating process of metallic copper using a thermometer”, and “recording the trend of copper block size changes with temperature”.

[0124] (2) Each trajectory contains multiple time-step state-action pairs, including textual state descriptions (such as “you stand in front of the lab bench”, “the thermometer shows 30 degrees Celsius”) and model-generated actions (such as “measure the length”,

[0125] "Heating Metal", "Recording Data").

[0126] (3) Generate diverse experimental strategy paths through temperature control and Top-p sampling, and screen the trajectories to eliminate redundant or logically incorrect operations and retain only valid reasoning paths.

[0127] 2. Domain knowledge induction and rule extraction:

[0128] (1) All trajectory actions are mapped into a unified semantic space. For example, actions such as “measure metal size” and “read temperature” are normalized into semantic actions such as “MEASURE_OBJECT” and “READ_TEMPERATURE”.

[0129] (2) Construct a knowledge graph in the field of “metal thermal properties experiment” and mine the dependency sequence between typical experimental steps, such as “MEASURE_OBJECT” → “HEAT_OBJECT” → “MEASURE_OBJECT”.

[0130] (3) Temporal difference learning is used to calculate the action TD contribution, and combined with PageRank to evaluate the global influence,

[0131] Extract key experimental rules:

[0132] Rule 1: IF the initial size has been recorded THEN the size change can be judged (TD contribution = 0.45, PageRank = 0.48); Rule 2: IF the metal has been heated THEN the high-temperature measurement operation can be performed (TD contribution = 0.39, PageRank = 0.41)

[0133] 3. Rule enhancement and reasoning stage:

[0134] (1) In the test task of "judging whether an iron block has thermal expansion characteristics", the system retrieves and matches through vector search to

[0135] the task domain of "material thermal property analysis" and recalls the most relevant rule set.

[0136] (2) The LLM rewrites the key rules in natural language to generate inference-guiding prompt information:

[0137] "Please first measure the size of the metal at room temperature and record the data."

[0138] "After the metal is heated, please re-measure and compare the size change to judge whether there is an expansion phenomenon."

[0139] (3) The optimized rule prompts are embedded as inputs into the model's inference context to guide the agent to reasonably plan the experimental process and improve the logic of operations and the reliability of reasoning.

[0140] Performance of model inference:

[0141] Inference task dataset

[0142] ALFWorld [1] Focuses on embodied tasks in a home environment and evaluates the agent's exploration, manipulation, and commonsense reasoning abilities in a text-interactive home environment. This dataset includes 134 different tasks covering home scenarios such as kitchens, bedrooms, and bathrooms. The tasks involve item recognition, manipulation, and environmental interaction, such as "put two soaps in the trash can" or "put a clean egg in the microwave". The interaction in this environment is in text form, and the agent needs to parse the environmental feedback and select appropriate operations, such as "open the fridge", "take out the egg", etc. The average number of interaction rounds for ALFWorld tasks is 6, the action space size is 13, and its context length is approximately 900 tokens.

[0143] Evaluation metrics

[0144] We use the following three evaluation metrics to evaluate the effectiveness of each method on the test set: Grounding Rate, Progress Rate, and Success Rate:

[0145] The GroundingRate measures whether the text generation of the agent is a legal action in the current environment, reflecting the agent's cognitive ability and text alignment ability in the task environment. Its definition is as follows:

[0146]

[0147] Where \(T\) represents the total number of time steps (turns); \(a_t\) represents the action generated by the agent at the \(t\)-th step; \(\text{Env}_t\) represents the environmental state at the \(t\)-th step; (1(IsValid(a t ,Env t ))) is the indicator function, which takes 1 if \(a_t\) belongs to the legal actions of the current environment, and 0 otherwise. The value range of GroundingRate is \([0,1]\), and the higher the value, the higher the matching degree between the agent's text output and the environment, indicating that the model has stronger observation understanding and description capabilities.

[0148] The ProgressRate measures the degree to which the agent achieves the goal during the task execution, reflecting the progress of task completion. In this study, the progress rate is mainly calculated through the completion of sub-goals. The calculation of the progress rate of the task is as follows:

[0149]

[0150] Where \(s_t\) represents the state of the agent at time step \(t\); \(G\) represents the definition of the task goal; \(|G|\) represents the total number of sub-goals in the task goal; \(|C|\) represents the total number of constraint conditions in the task goal; \(achieved(s t ,G)\) represents whether the agent achieves the sub-goal at time step \(t\); \(\match(s t ,G)\) represents whether the agent meets the constraint conditions of the target state at time step \(t\). The value range of ProgressRate is \([0,1]\), and the higher the value, the greater the progress made by the agent during the task execution.

[0151] The SuccessRate evaluates the final success rate of the agent in the task, that is, whether the agent successfully completes the entire task goal. Different from ProgressRate (which focuses on the step-by-step progress), SuccessRate is only counted as successful when the agent completely achieves the task goal. Therefore, its calculation method is usually a binary indicator. The Success Rate is calculated by whether the final state \(s_T\) meets the task goal \(G\), and its definition is as follows:

[0152]

[0153] Among them, \(N\) represents the total number of tasks; \(s_T^i\) represents the final state of the \(i\)-th task; \(G^i\) represents the target state of the \(i\)-th task; is an indicator function that takes 1 when the agent's final state \(s_T^i\) meets the task goal, and 0 otherwise.

[0154] Experimental results

[0155] To compare the inference performance, we adopted Zero-shot (0-shot) and One-shot (1-shot) as comparison methods, and compared them with the method of this application (TD-RuAG) to evaluate the impact of different learning paradigms on the LLM agent:

[0156] Zero-shot directly generates answers without any examples, relying on the world knowledge and reasoning ability obtained during the LLM pre-training process.

[0157] One-shot provides an example for the LLM to help it understand the task format and solution strategy. Compared with Zero-shot, One-shot provides limited task context, enabling the LLM to better align with the task requirements.

[0158] The comparison methods are all based on the Act mechanism [2] . Currently, there are many module-based agent frameworks that can improve the performance of agents. However, these frameworks usually involve complex designs and are not applicable to all LLMs. Therefore, we choose Act as a unified prompting framework, which only requires minimal design and is applicable to most instruction-following LLMs. To avoid biases caused by a single base model, we selected three open-source 7B models, and all the base comparison methods were compared on these three base models: Mistral-7B-Instruct-v0.2 (Mistral-7B), Meta-Llama-3-8B-Instruct (Llama3-8B), and Qwen2.5-7B-Instruct (Qwen-7B).

[0159] In the ALFWorld task, the agent needs to interact with objects in a partially observable home environment. The task involves multiple sub-goals, and the environmental state may change during task execution. Therefore, the agent must possess good reasoning and planning abilities. As shown in Table 1, the Success Rate of the Zero-shot and One-shot methods on this task is only 0.000 to 0.008, indicating that relying solely on the pre-trained knowledge of the LLM or the prompt of a single example is not sufficient for the agent to effectively complete the task. In contrast, the TD-RuAG method combines TD(λ) contribution calculation and rule-enhanced learning. Except for the overall low Success Rate of Mistral-7B, the Success Rates of the other two base models reach 0.135 and 0.155 respectively, and the Progress Rate reaches a maximum of 0.387, achieving a significant improvement compared to the comparison methods. At the same time, TD-RuAG enables the agent to effectively avoid invalid exploration by automatically screening high-contribution state-action pairs, and combines rule-enhanced strategies to dynamically retrieve the experience of similar tasks during the reasoning process, thereby optimizing the decision-making quality. Therefore, in the ALFWorld task, TD-RuAG has significant task planning capabilities, can improve the success rate of multi-step reasoning tasks, and effectively cope with the high-complexity challenges of home environment tasks.

[0160]

[0161]

[0162] Table 1: ALFWorld Inference Results

[0163] Furthermore, as Figure 3 shown, compare the domain rules of TD-RuAG and context learning (Demo-AG) that only uses high-quality reasoning trajectories during the sampling process as examples. TD-RuAG retrieves the rules that contribute the most to the task progress in the historical tasks through the rule extraction strategy, and combines context learning to optimize the decision-making, while Demo-AG only uses the trajectory with the highest Progress Rate found during the sampling process of the large language model directly as an example without additional rule enhancement. As Figure 3As shown, although Demo-AG can improve the task progress to a certain extent, the Success Rate is still low. For example, the Success Rate of Llama3-8B under example learning (Demo-AG) is only 0.008, indicating that it is difficult for the agent to generalize to different task instances through a single example. In contrast, TD-RuAG combines rule retrieval and in-context learning, enabling the agent to dynamically adjust its strategy. For example, the Progress Rate of Qwen-7B under TD-RuAG is increased to 0.387, and the Success Rate is increased to 0.135. Demo-AG can only provide static examples, while TD-RuAG enhances the generalization ability of the large language model in the reasoning process by combining multiple task experiences through rules, improving the generalization ability of decision-making.

[0164] [1]Wang R,Jansen P, M-A,et al.ScienceWorld:Is your Agent Smarterthan a 5th Grader?[C] / / Goldberg Y,Kozareva Z,Zhang Y.In Proceedings of the2022 Conference on Empirical Methods in Natural Language Processing,AbuDhabi,United Arab Emirates,December 2022:11279–11298.

[0165] [2]Yao S,Zhao J,Yu D,et al.React:Synergizing reasoning and acting inlanguage models[C].In International Conference on Learning Representations(ICLR),2023.

Claims

1. A large language model inference method based on temporal difference learning and rule enhancement, characterized in that The steps include: Task trajectory sampling: Based on the base large language model, multiple rounds of random sampling are performed on the training task dataset to generate diverse task reasoning trajectories. The sampled reasoning trajectories are then filtered to retain only valid trajectories. Domain knowledge induction: Extract action semantics from the action steps of valid reasoning trajectories and map them into a unified action space. Classify reasoning trajectories according to the domain to which the task belongs, and aggregate all trajectories in the same task domain to extract common execution patterns within the domain. Based on the aggregated trajectory set, a domain knowledge graph is constructed, with deparameterized action semantics as nodes and action semantic dependencies in the trajectory as directed edges, forming a knowledge structure within the task domain. Domain rule extraction: A temporal difference method is used to learn the contribution of different actions to task advancement until convergence. PageRank is then combined to evaluate the global influence of actions in the domain knowledge graph. By fusing local contributions with global importance, key task execution rules are extracted and action dependencies are formed, thereby establishing a domain rule set that can be used to optimize LLM reasoning. Rule-enhanced large language model reasoning: For test task data, the most relevant historical tasks are matched based on vector retrieval, and the domain rule set for the corresponding task is obtained. The rules are rewritten in natural language using the large language model itself, making them more interpretable and adaptable. Finally, the optimized rules are integrated into the large language model inference prompt text to optimize the inference quality and stability.

2. The method according to claim 1, characterized in that, Specifically, in step (1), the training task dataset Each task instance (m, o) consists of a task domain m and task-specific information o. The task domain m represents the category to which the task belongs; the task-specific information o includes the input information of the task; the LLM needs to reason about these task instances to generate possible task inference trajectories; these trajectories contain the states, actions performed, and corresponding task progress at multiple time steps, describing how the large language model advances the task progress during multi-step reasoning; The set of sampled trajectories can be formally expressed as: τ = {(s1, a1, r1), (s2, a2, r2), …, (s T , a T , r T )}, Here, τ represents the complete execution trajectory inferred by the large language model on a specific task instance (m, o), including all decision steps from the initial state to the final result; To ensure the diversity and rationality of sampling trajectories, temperature control and Nucleus sampling were used for trajectory sampling; Furthermore, in the multi-step decision-making process, for each decision step, the agent outputs the current action based on the current environmental information; the screening process includes the following key steps: eliminating illegal actions: ensuring that every action in the sampled trajectory complies with the basic logical rules of the task environment; removing invalid actions: avoiding the large language model from generating redundant reasoning paths.

3. The method according to claim 1, characterized in that Step (2) Specifically, map the actions in the sampling trajectory to a unified semantic space; each decision-making action a t is composed of action semantics and action parameters θ t ; among them, action semantics represent general operations that span different task scenarios, while action parameters refer to specific task instances; in order to aggregate task trajectories in the same domain, remove the action parameters so that all actions of the same category are mapped to a unified action space, only retaining the action semantics; then, according to the task domain m, aggregate all trajectories under the current task domain to obtain T m , and then construct a directed acyclic graph G m =(V, E) for the task domain m; where the node set represents the de-parameterized action semantics in the task; the edge set represents the action dependency relationship in the trajectory; if an action usually occurs before another action during task execution, then there will be a directed edge in the graph from to The directed edge reflects the execution logic of actions during the reasoning process.

4. The method according to claim 1, wherein Step (3) Specifically, temporal difference learning is used to calculate the local contribution of each action, and the global influence of the action in the task graph is evaluated in combination with a calculation method based on link analysis; through this process, task execution rules are constructed and a structured domain knowledge set is formed; during task execution, the action TD contribution Q needs to be calculated, that is, the degree of influence of a certain action on the future task progress under a specific state; the TD contribution Q is updated using the temporal difference error method to ensure that the Q value can correctly reflect the contribution of the action; further, the eligibility trace is used to improve the stability of learning; The iterative update process is as follows: Among them, δ t represents the temporal difference error, which is used to measure whether the contribution of the current action meets the expectation; p t is the immediate contribution degree, that is, the increment of task progress directly brought by this action; γ is the discount factor, which is used to control the influence weight of future states on the current decision-making; the value of γ is between [0, 1]. A larger γ makes the model pay more attention to long-term benefits, while a smaller γ makes the model more inclined to short-term benefits; is the TD contribution degree of the next state, representing the possible benefits in the future; is the TD contribution degree of the current action, which is continuously updated during the training process; is the eligibility trace value, which is used to record the contribution degree of a certain action to task progress; α is the learning rate, which controls the step size of Q-value update; λ is the eligibility trace decay factor; The PageRank calculation method is introduced to evaluate the global influence of actions in the domain knowledge graph; the TD contribution and PageRank influence are combined to calculate the comprehensive action contribution, form task execution rules, and store them in the domain knowledge set: Among them, represents the final contribution of the action, which synthesizes its local benefits and global influence β is a weight parameter used to adjust the relative weights of the TD contribution and the PageRank influence.

5. The method according to claim 1, wherein Step (4) Specifically, when a test task arrives, the most relevant training tasks need to be retrieved in order to obtain the rule knowledge applicable to the current task; for this purpose, a vector retrieval method is adopted, and by calculating the cosine similarity between the current task description h i and the training task description h j the closest training task is found and its rule set is obtained: Among them, Embed(·) represents the text embedding model, which is used to convert the text task description into a high-dimensional vector representation, so that the similarity of different tasks can be measured by mathematical calculation; cos(·,·) is the cosine similarity calculation function, which is used to measure the similarity between two task description vectors; Then, screen the top K rules with the highest contribution from the retrieved task rule set and use them to optimize the inference ability of the large language model; the key to this process is to ensure that the large language model only refers to the optimal rules for inference optimization and avoid introducing irrelevant or inefficient rules to interfere with the inference results; The calculation method for screening rules is as follows: Among them, Π m is the complete rule set corresponding to the task domain m; represents the action contribution degree, that is, the influence degree of a certain action on task promotion; the contribution degree calculation combines temporal difference learning and PageRank influence; TopK(·) represents sorting according to the contribution degree and selecting the top K rules that are most helpful for task completion; the most relevant rule set recalled represents the rules most suitable for reference in the current test task; Finally, integrate the optimized rules into the inference prompt of the LLM so that it can make more accurate decisions in test tasks.

6. The method according to claim 1, wherein Applied to automatic planning, intelligent question answering, and embodied intelligence scenarios.

Images