Large language model driven offline target conditioned reinforcement learning method and system
By constructing a state interpretation dictionary and a key state discriminator, combined with a self-checking mechanism, the problem of target selection not distinguishing between importance and noise interference in offline target-conditional reinforcement learning is solved, thus achieving efficient and stable offline target-conditional reinforcement learning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INST OF SOFTWARE - CHINESE ACAD OF SCI
- Filing Date
- 2026-02-12
- Publication Date
- 2026-06-05
AI Technical Summary
Existing offline target-conditional reinforcement learning methods do not distinguish the importance of targets, lack relabeling quality evaluation standards, directly use large language models with high costs and illusion risks, have insufficient support for high-dimensional state noise, and lack unified theoretical evaluation standards, resulting in low training efficiency, poor stability and low sample efficiency.
By constructing a state interpretation dictionary, a key state discriminator is generated. A two-stage mechanism of compile-time self-checking and re-marking-time self-checking is introduced. A large language model is used to generate semantic rules and an executable discriminator. Combined with the state subspace index, target re-marking and policy generation are performed. TD error is used as an indicator for adaptive optimization.
It significantly reduces the cost of calling large language models during long-term training, improves the interpretability and stability of key state recognition, enhances sample efficiency and training robustness, reduces noise interference, and achieves adaptive calibration and interpretability in the training process.
Smart Images
Figure CN122154973A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, specifically to an offline target-conditional reinforcement learning method and system driven by a large language model. Background Technology
[0002] In existing technologies, offline target-conditional reinforcement learning (offline GCRL) typically treats the target as an element in the state space, constructing a formalized Markov decision process. And, given an offline trajectory dataset, by each transfer Configure a target Recalculate the reward This allows for the learning of target-conditional policies without requiring online interaction with the environment. These methods generally assume that a sufficient number of training samples useful for learning can be constructed from existing trajectories through a suitable "target relabeling" strategy to alleviate the difficulties caused by sparse rewards.
[0003] The most typical approach to goal relabeling is hindsight experience replay (Andrychowicz, M., Crow, D. et al. (2017). Hindsight Experience Replay. Neural Information Processing Systems.). Within a trajectory, HER uniformly and randomly selects a state from future states after the current moment as the "hindsight goal," and then recalculates the reward based on whether the goal was achieved, turning a previously failed trajectory segment into a "successful experience" after reinterpretation. Subsequent work has proposed some improvements in areas such as intra-trajectory sampling and cross-trajectory sampling, but most still treat all states in the trajectory as homogeneous candidate goals, lacking a distinction between "critical states that truly drive task progress" and "ordinary states with low information content," resulting in limited information content in the relabeled samples.
[0004] On the other hand, mainstream offline objective-conditional reinforcement learning methods mainly focus on how to better estimate the value function or policy given the objective. For example, some works are based on weighted behavioral cloning or imitation learning using demonstration data (Yang, R. et al. (2022). Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL. ArXiv, abs / 2202.04478.), some works focus on the estimation of the goal-conditional Q function or value function (Yang, R. et al. (2023). What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL? ArXiv, abs / 2305.18882). Other methods introduce hierarchical goal decomposition, breaking down long-range goals into a series of intermediate sub-goals (Park, S. et al. (2023). HIQL: Offline Goal-Conditioned RL with Latent Statesas Actions. ArXiv, abs / 2307.11949). Still other methods utilize metric learning (Tongzhou Wang et al. (2023). Optimal Goal-Reaching Reinforcement). Learning via Quasimetric Learning (PMLR). Contrastive learning (Benjamin Eysenbach et al. (2022). Contrastive Learning as Goal-Conditioned Reinforcement Learning. In Advances in Neural Information Processing Systems, Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (Eds.)). and other methods construct representation spaces more suitable for expressing "state-goal relationships." These methods have made progress in algorithmic modeling, but overall they focus more on "how to learn under a given goal" and less on systematically designing from the perspective of "how to select or construct a more suitable target distribution in offline data."
[0005] In recent years, large language models (LLMs) have been increasingly introduced into the field of reinforcement learning for applications such as high-level planning, action code generation, and mapping of natural language to reward functions. For example, some studies have used LLMs to translate natural language objectives into dense, executable reward functions (Guanwen Xie et al. (2024). Large Language Models as Efficient Reward Function Searchers for Custom Environment Multi-Objective Reinforcement Learning. ArXiv:2409.02428). Alternatively, LLMs can be used as high-level planners to output abstract action sequences, which are then implemented by lower-level RL policies (Shaoteng Liu et al. (2024). RL-GPT: Integrating Reinforcement Learning and Code-as-policy. In The Thirty-eighth Annual Conference on Neural Information Processing). Such works demonstrate the potential of LLM in injecting prior knowledge and improving sample efficiency, but most are geared towards online interactive scenarios, often requiring frequent calls to LLM for inference, and the output is mostly natural language plans or pseudocode. System modeling and theoretical analysis of "target relabeling quality" in offline GCRL scenarios are still insufficient.
[0006] In addition, some existing technologies attempt to incorporate LLM into the target selection or sub-target planning stages. For example, some methods use LLM to generate a set of candidate targets based on the task description, and then map them into the state space to guide policy learning; there are also online GCRL solutions that request LLM to select potentially useful targets or sub-targets for the current state during interaction with the environment (Jianhai Su and Qi Zhang. 2023. Subgoal Proposition Using aVision-Language Model. In 2nd Workshop on Language and Robot Learning: Language as Grounding). While these methods utilize the semantic reasoning capabilities of LLM to some extent, they generally share several commonalities: first, they rely on online interaction and multiple rounds of LLM calls, resulting in high overall operating costs; second, they lack large-scale target relabeling mechanisms for offline data, and have not yet formed a reusable, structured key state discrimination module; and third, they do not adequately consider the problems of LLM illusion and noise and uneven distribution in offline data, and lack relabeling quality evaluation and adaptive optimization mechanisms based on unified metrics. Therefore, in offline GCRL scenarios, how to systematically utilize LLM to mine key states from offline data, construct high-quality target distributions, and theoretically characterize their impact on sample complexity and learning performance remains an unresolved issue.
[0007] In summary, the existing technology has the following main technical defects.
[0008] 1. Target selection does not distinguish importance: Existing target relabeling methods (such as HER and its variants) generally adopt a uniform sampling mechanism based on the future state of the trajectory, treating all states as equal candidate targets without distinguishing their semantic importance in the task. This inherent limitation of the technical mechanism means that a large number of sampled targets are actually weakly related to task progress, providing limited information. This leads to sparse reward signals, low temporal difference errors, and suppressed value function learning, resulting in technical problems such as low training efficiency and slow convergence speed.
[0009] 2. Lack of a systematic evaluation standard for "relabeling quality": Because existing methods lack a unified evaluation standard for "relabeling target quality," the target selection process often relies solely on heuristic experience rather than data-driven quantitative indicators. Therefore, it is impossible to determine whether the relabeled target truly contributes to the approximation of the value function. Without a quality assessment mechanism, the system struggles to automatically optimize the relabeling strategy and cannot correct for anomalies, resulting in poor training stability, weak controllability, and a lack of adaptive adjustment capabilities.
[0010] 3. Direct use of LLM as an aid carries high costs and the risk of illusion: While some solutions incorporating LLM into reinforcement learning are used in high-level planning and natural language reward generation, they often require frequent calls to LLM for inference, leading to high operating costs and poor real-time performance. Furthermore, the lack of structured constraints in LLM outputs makes them prone to illusions, posing a significant technical risk to their direct application in target selection or relabeling. Existing technologies do not provide reliable mechanisms to verify and correct the rules output by LLMs, lacking guarantees regarding their executability, matching rate, or reasonableness within data distribution. Therefore, they are difficult to use safely in large-scale offline relabeling scenarios.
[0011] 4. Insufficient support for noisy data and high-dimensional states: For high-dimensional, noisy state spaces, existing relabeling methods often rely solely on geometric distance across the entire space as the basis for judgment. This fails to identify key factors in the task's semantics and is easily affected by noise, leading to a decrease in target selection accuracy. Especially in situations with sensor noise or unobservable states, strategies based on full-dimensional automatic measurement struggle to reliably extract effective targets, further reducing the sample efficiency of policy learning.
[0012] 5. Lack of unified theoretical evaluation standards: Existing technologies lack a mechanism to quantitatively determine "which target distribution is conducive to learning" from a theoretical perspective. They are unable to evaluate the effectiveness of targets based on indicators such as value function approximation error and temporal difference error, nor can they use these quantitative indicators to guide target generation strategies. Therefore, traditional solutions cannot dynamically adjust target relabeling strategies according to the training process, and cannot cope with changes in model requirements at different training stages, resulting in poor performance in complex or long-view tasks. Summary of the Invention
[0013] The purpose of this invention is to address the problems in existing technologies, such as the lack of semantic distinction in target relabeling, uncontrollable relabeling quality, difficulty in directly using LLM output for large-scale training, severe noise interference from high-dimensional states, and the lack of unified theoretical evaluation standards. This invention proposes an offline target conditional reinforcement learning method and system driven by a large language model. By providing high-quality training signals for offline target conditional reinforcement learning, improving sample efficiency, enhancing system stability and interpretability, and generating accurate control actions.
[0014] To achieve the above objectives, the technical solution of the present invention includes the following:
[0015] An offline target-conditional reinforcement learning method driven by a large language model, the method comprising: Obtain the sample task and sample task environment, and construct a state interpretation dictionary based on the offline trajectory dataset in the sample task environment; Based on the sample task, sample task environment and state interpretation dictionary, the large language model is called multiple times to generate a key state discriminator; The offline trajectory dataset is relabeled using a key state discriminator to generate training samples. On these training samples, target condition policies corresponding to different task types are generated based on an offline target conditional reinforcement learning algorithm. Based on the task type of the target task, the corresponding target condition strategy is invoked to obtain the control action corresponding to the current state.
[0016] Furthermore, the construction of the state interpretation dictionary based on the offline trajectory dataset in the sample task environment includes: The offline trajectory data is cleaned and aligned; the cleaning and alignment includes: removing missing fields, removing outliers, aligning timestamps, unifying units of measurement, normalizing / standardizing states, and de-jittering discrete event flags. For each state dimension or field in the cleaned and aligned offline trajectory data, the physical meaning, dimensions, value range, noise type, available metrics, and derivation calculation method are given to generate a state interpretation dictionary.
[0017] Furthermore, based on the sample task, sample task environment, and state interpretation dictionary, the large language model is called multiple times to generate a key state discriminator, including: Obtain the task objective and environmental description of the sample task environment; Based on the task objective, environment description, and state interpretation dictionary, the large language model is invoked to obtain key state categories and key state category descriptions; wherein, the key state category description includes the category name, the significance of occurrence, and the observable cues corresponding to the key state category; Based on the key state category description, task objective, environment description, and state interpretation dictionary, the large language model is invoked to generate key state semantic rules; Based on the key state semantic rules and the state interpretation dictionary, the large language model is called to determine the state subspace index corresponding to the key state semantic rule; Based on the key state semantic rules and state subspace index, the large language model is invoked to generate the key state discrimination function code corresponding to the key state semantic rule, and an initial key state discriminator is constructed based on the key state discrimination function code; The initial critical state discriminator code is iteratively updated based on the sample task dataset of the corresponding task type until the generated critical state discriminator passes the compile-time self-check and meets the matching rate constraint.
[0018] Furthermore, the key state semantic rules include criteria through combinations of AND, OR, and NOT. These criteria include numerical criteria, relational criteria, and / or event-based criteria. The numerical criteria include threshold intervals satisfied by continuous quantities, the relational criteria include constraints satisfied by object relationships, and the event-based criteria include triggering conditions satisfied by events. The key state semantic rules also include noise resistance conditions.
[0019] Furthermore, the process by which the large language model determines the state subspace index corresponding to the key state semantic rule includes: Obtain semantic features from key state semantic rules; By using a state interpretation dictionary, keywords in semantic features are mapped to corresponding state fields; Based on this state field, the state subspace index corresponding to the key state semantic rule is obtained.
[0020] Furthermore, the initial key state discriminator code is iteratively updated based on the sample task dataset of the corresponding task type until the generated key state discriminator passes the compile-time self-check and satisfies the matching rate constraint, including: Run the current critical state discriminator on the sample task dataset of the corresponding task type to check the sample task data; the check includes: correct syntax, complete variable definition, array out of bounds / type error, and non-existent array, and the running time meets the preset upper limit; The system statistically analyzes the trigger frequency and matching rate of the current key status discriminator on the sample task dataset of the corresponding task type during the inspection process, and generates error logs. For the current key state discriminator whose matching rate is not in the set range, the error log, trigger frequency, key state category, key state semantic rules and state subspace index are fed back to the large language model. After adjusting the key state semantic rules or state subspace index and updating the current key state discriminator, the current key state discriminator is re-executed on the sample task dataset of the corresponding task type. For a current critical state discriminator whose matching rate is within a set range, it is determined that the current critical state discriminator passes the compile-time self-check and satisfies the matching rate constraint.
[0021] Furthermore, a key state discriminator is used to relabel targets in the offline trajectory dataset to generate training samples, including: For any moment in the offline trajectory data, run the key state discriminator for all key state categories within the subsequent time window of that moment to collect a set of candidate moments for each key state category; For a candidate time set that is not empty, randomly sample state data from the candidate time set and generate a relabeled target based on the state data; For a candidate time set that is empty, a state data point is randomly sampled from a subsequent time window, and a relabeled target is generated based on that state data. Generate training samples; the training samples include: current state data, current action command, relabeling target corresponding to the current state data, relabeling reward calculated based on the relabeling target, and next state data.
[0022] Furthermore, when generating target-conditional policies for different task types based on the offline target-conditional reinforcement learning algorithm on this training sample, it also includes: The system statistically learns a signal and characterizes the relabeled target based on the learned signal; wherein the learned signal includes: the mean and variance of the time difference error or the quantile and moving average of the time difference error, and the learned signal also includes: success rate, Q-value scaling drift and policy entropy change; If the characterization result meets the preset triggering conditions, a self-check during the remarking period is performed, and the window length of the key state semantic rules, subspace index and / or subsequent time window is adjusted based on the diagnostic results of the self-check. A compile-time self-check is performed based on the tuning results to update the training sample.
[0023] An offline objective-conditional reinforcement learning system driven by a large language model, the system comprising: The dictionary building module is used to obtain the sample task and sample task environment, and build a state interpretation dictionary based on the offline trajectory dataset in the sample task environment; The discriminator generation module is used to call the large language model multiple times based on the sample task, sample task environment and state interpretation dictionary to generate a key state discriminator. The training module is used to relabel targets in the offline trajectory dataset using a key state discriminator, generate training samples, and generate target condition policies for different task types based on the offline target conditional reinforcement learning algorithm on the training samples. The testing module is used to invoke the corresponding target condition policy based on the task type of the target task in order to obtain the control action corresponding to the current state.
[0024] A computer device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the large language model-driven offline target conditional reinforcement learning method described above.
[0025] Compared with the prior art, the present invention has at least the following beneficial effects.
[0026] 1) This invention, based on LLM semantic rules and a coded key state discriminator, provides offline relabeling support from "one-time generation" to "calibrable and reusable". Specifically, this invention generates semantic rules corresponding to key states through a large language model and further automatically generates an executable key state discriminator function. This solidifies the high-level semantic knowledge of the large language model into a discriminator module that can be repeatedly called during training in a "rule-based and coded" form, thereby significantly reducing the cost of frequently calling the large language model during long-term training and improving the interpretability and consistency of key state recognition. CN118333183A emphasizes "generating a key state discriminator function in a one-round dialogue based on a preset prompt template" and using this discriminator function for key state recognition in the sampling trajectory to "promote efficient exploration by multiple agents". In addition to "generating a discriminator", this invention focuses more on using the discriminator for offline target relabeling and combining it with subsequent two-stage self-checking closed-loop continuous calibration, making the discriminator more usable and stable in offline data and training processes.
[0027] 2) This invention provides robust discrimination and improved relabeling stability for high-dimensional noisy states through state subspace indexing. Specifically, this invention synchronously generates a state subspace index for each type of key state, enabling the discriminator to make judgments based on only a small number of state dimensions that are highly relevant to the task semantics. This significantly suppresses interference from irrelevant and noisy dimensions in high-dimensional, noisy states, improving the robustness and stability of key state recognition, and further enhancing the quality of relabeled target selection based on key states.
[0028] 3) This invention achieves closed-loop stability gains from "executability assurance" to "training-phase adaptive calibration" based on a two-stage self-checking mechanism. Specifically, this invention introduces a two-stage mechanism combining compile-time self-checking and re-marking-time self-checking. The compile-time self-check verifies the discriminator code is executable and constrains the trigger frequency / matching rate to avoid extreme situations caused by overly strict or lenient rules. The re-marking-time self-check dynamically monitors the learning signals (such as TD error statistics) during offline GCRL training and triggers adaptive fine-tuning of key state semantic rules and the discriminator, ensuring the re-marking strategy remains effective as value learning progresses and reducing the instability caused by LLM generation bias and data noise. Although CN118333183A also mentions a self-checking feedback mechanism to improve the quality of discriminator function generation (including "self-reflection on whether it is reasonable" and "code executability testing"). However, its self-check mainly revolves around "generation stage / executability"; this invention further extends the self-check to a two-stage closed-loop calibration for the entire offline relabeling and training process: ensuring both "operability and reasonable triggering" and "continuous usefulness during training".
[0029] 4) Through GOAL-BE dimension analysis, it is demonstrated that under certain conditions, using a relabeling distribution that produces a larger TD error can significantly compress the GOAL-BE dimension, thereby reducing the sample complexity of offline GCRL. This invention utilizes this property, using TD error as an indicator to guide discriminator optimization, to achieve a theoretically interpretable adaptive relabeling strategy.
[0030] 5) This method can directly replace the relabeling module in existing Offline GCRL algorithms, keeping the rest unchanged. Experiments show that integrating this invention into various baselines such as GCIVL, GCIQL, QRL, CRL, and HIQL can achieve continuous improvements in various environments such as navigation, control, and puzzle-solving, especially on long-view and noisy datasets, and can still maintain a high success rate under half-data conditions, fully demonstrating the advantage of sample efficiency. Attached Figure Description
[0031] Figure 1 A flowchart of an offline objective-conditional reinforcement learning method driven by a large language model. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0033] The large language model-driven offline target-conditional reinforcement learning method of this invention, such as Figure 1 As shown, it includes the following steps.
[0034] Step 1: Obtain the sample task and sample task environment, and construct a state interpretation dictionary based on the offline trajectory dataset in the sample task environment.
[0035] Step 1.1: Collect or import offline trajectory datasets in the target task environment. The trajectory format is as follows: Each trajectory consists of a state, an action, a next state (and an optional termination flag).
[0036] Step 1.2: Clean and align the trajectory: This includes removing missing fields, eliminating outliers, aligning timestamps, unifying units, normalizing / standardizing states, and de-jittering discrete event flags (contact / grab / switch), to form a data format usable for training.
[0037] Step 1.3: Construct a “State Interpretation Dictionary”: For each state dimension (or field), provide its physical meaning, dimensions, reasonable range, noise type, available metrics (distance / angle / contact / logical event), and necessary derivation calculation methods (such as Euclidean distance, angle difference, velocity norm, contact count, etc.).
[0038] Step 2: Based on the sample task, sample task environment, and state interpretation dictionary, call the large language model multiple times to generate the key state discriminator.
[0039] Step 2.1: Organize the explanation dictionary, task objective, environment description, and success conditions (if any) into structured text or JSON format, and use this as input prompts for the large language model, so that the large language model outputs several key state categories. Each category It includes: category name, significance (impact on task progress / sub-goals), and observable cues corresponding to that category. For example, "approaching the corridor entrance" or "completing the grabbing action." Optionally, the prompt information may also include a small number of representative segments (several state segments or statistical summaries) extracted from the dataset as supplementary examples.
[0040] Step 2.2: For each key state category It invokes a large language model to generate key state semantic rules based on the interpretation dictionary, task objectives, and environment descriptions. The semantic rules consist of a set of decidable semantic features / criteria, including at least one or more of the following forms: Numerical criteria: Certain continuous quantities satisfy a threshold interval (e.g., distance). (Height within the range) Relational criteria: The relative relationship between two objects satisfies the constraints (relative position, relative angle, relative speed); Event-based criteria: Discrete events / contact / grab / arrival flags meet the triggering conditions; Combinatorial logic: The above criteria are combined using AND / OR / NOT, and allow setting "persistence". Frames satisfy "to reduce noise".
[0041] Step 2.3: The large language model is based on each semantic rule. The semantic features involved automatically determine the state subspace indexes to which they depend. Specifically, it is constructed as follows: Based on semantic-to-field mapping: using the interpretation dictionary from step 1.3, keywords in the semantic features (position, orientation, joint angle, gripper opening and closing, contact markers, etc.) are mapped to the corresponding state fields, thereby obtaining... Finally, for each key state category, output a triplet. And store it as a structured file for use in subsequent steps.
[0042] Step 2.4: Extract the key state semantic rules from the LLM output. With state subspace index As input, the corresponding key state discrimination function code is generated by calling the large language model again. This code is used to determine critical states in any sequence of states. Its input must include at least the states. (Live state sequence fragment), output as a boolean value: This indicates that the critical state is met; and the discriminator only accesses [the relevant data]. This is to reduce interference from irrelevant dimensions.
[0043] Step 2.5: Batch run all on the subset of data Check: syntax is correct, variable definitions are complete, arrays are out of bounds / type errors do not exist, and runtime meets the preset upper limit.
[0044] Step 2.6: Calculate the trigger frequency / match rate of each discriminator on the dataset. and within the preset range Compare: like Rules that are too strict or do not match like The rules are too broad or too generalized.
[0045] Step 2.7: Compile error logs, trigger statistics, and corresponding... Feedback is sent to the large language model, requiring it to process the semantic rules. Thresholds, logical combinations, or subspace indexes Make minimal adjustments and update the discriminator code synchronously; repeat steps 2.5–2.7 until all discriminators pass the compile-time self-check and meet the matching rate constraint.
[0046] Step 3: Use the key state discriminator to relabel the targets in the offline trajectory dataset to generate training samples. On these training samples, generate target condition policies corresponding to different task types based on the offline target condition reinforcement learning algorithm.
[0047] Step 3.1: The trajectory used for relabeling is the offline trajectory dataset from Step 1.1, therefore the trajectory contains by default... Remarking only changes the "target field" "and the "remark reward" calculated accordingly ".
[0048] Step 3.2: For any time in the trajectory During its subsequent time window Run all discriminators internally and collect a set of candidate moments that satisfy any key state: .
[0049] Step 3.3: Target re-marking, as follows: like Not empty, from the candidate set Random sampling and set goals ; When no critical state is identified within the time window, i.e. If the set is empty, it degenerates into the traditional uniform sampling strategy for future states, i.e., from... Select one randomly without preference and set goals ; in It can be an identity mapping (using the state directly) or it can be extracting only the target-related fields (such as position / object pose). The specific choice depends on the task type.
[0050] Step 3.4: Regardless of whether critical state sampling or degenerate sampling is used, standard offline GCRL training samples are generated: ,in To relabel the target The calculated relabeled reward, for example, based on a predefined distance function. With threshold : Used for subsequent offline GCRL training.
[0051] Step 3.5: Train the objective conditional value function and policy (such as objective conditional...) on the samples generated in Step 3.4 using any compatible offline objective conditional reinforcement learning algorithm (such as GCIQL, HIQL, etc.). The network parameters are iteratively updated until convergence or the number of training rounds is reached.
[0052] Step 3.6: Periodically statistically analyze the learning signal during training, including at least the mean and variance (or quantiles, moving average) of the TD (Time Difference) error, and optionally statistically analyze: success rate, Q-value scale drift, policy entropy change, etc., to characterize whether the current relabeled target is "too easy / too difficult / noisy".
[0053] Step 3.7: When the monitoring indicators meet the preset trigger conditions (e.g., a consistently small TD error indicates that the target is too easy, and a large variance indicates that the target quality is unstable or the distribution is too complex), then the "remarking period self-check" is initiated, and the diagnostic information is fed back for rule adjustment: The objects to be adjusted include: key state semantic rules (Threshold, Logical, Continuous Frames), Subspace Index (Adding / Removing Dimensions), Window Length ; Adjustment method: Input the above diagnostic information, along with the task description, existing rules, and discriminator code, into the large language model and request it to generate "fine-tuned" rules and code.
[0054] Step 3.8: The discriminator after the large language model is updated must undergo a self-check during compilation again. Only after passing the check can it be used for the next round of relabeling and training, thus forming a closed loop of "monitoring-adjustment-re-verification-retraining".
[0055] Step 4: Invoke the corresponding target condition policy based on the task type of the target task to obtain the control action corresponding to the current state.
[0056] After offline target-conditional reinforcement learning training is complete, a target-conditional policy is output and deployed in the execution system. This policy enables the system to generate corresponding actions based on the current state and the given target without relying on a large language model or relabeling process. Different target-conditional policies correspond to different task types. That is, for the first Task-based output strategy Its characteristic is that it includes the following steps.
[0057] Step 4.1: Targeting the task type Output the target-conditional policy corresponding to this task, which is based on the state and the target. The strategy described above is used to characterize the mapping relationship from state to action under different objective constraints, and can be expressed as: ,in Indicates the current state. Indicates the goal.
[0058] Step 4.2: During the strategy execution phase, the current state information and target information are used as strategy inputs, wherein the target information is the target state, target location, target object attributes, or their equivalent representations, and is combined with the task type. The underlying goal is to maintain consistency.
[0059] Step 4.3: Utilize the strategy corresponding to the task. Output Action This action will be used for online execution of robot control, path navigation, manipulation, or other decision-making tasks.
[0060] Step 4.4: During the strategy deployment and execution process, the large language model is no longer called, and key state discrimination or target re-marking operations are no longer performed, thereby reducing online computational complexity and improving the real-time performance and stability of the system.
[0061] In summary, this invention provides prompts to the LLM through task text descriptions and state variable descriptions, enabling the LLM to output semantic rules oriented towards "critical state categories" (such as turning events, grab events, termination stability, etc.), and automatically translates these rules into executable discriminant function code, forming a reusable critical state discriminator.
[0062] The LLM of this invention not only provides natural language rules, but also provides a set of state dimension indices related to the rules, so that the discriminator can make judgments only in this subspace, thereby reducing high-dimensional noise interference, improving rule interpretability and accuracy, and is particularly suitable for complex manipulation tasks with noise.
[0063] During the compilation phase of this invention, a discriminator is executed on a subset of offline data to check whether it is executable and whether its matching rate is within a preset range. If the conditions are not met, the LLM will make minimal modifications to the rules or code based on the error log and statistical information until the check is passed. During the relabeling period, the TD error distribution of the relabeled samples is statistically analyzed. When the mean and variance are abnormal, the LLM is triggered to reflect on and fine-tune the rules to adaptively improve the "information content" and stability of the relabeling.
[0064] This invention draws on the GOAL-BE dimension theory to prove that in offline GCRL, under certain conditions, a larger TD error corresponds to a smaller GOAL-BE dimension, thereby resulting in lower sample complexity. Therefore, the TD error is used as an evaluation index for the quality of target relabeling, thus achieving a unity of theory and practice.
[0065] The LLM-driven remarking module proposed in this invention only modifies the data remarking process without changing the main structure of the original Offline GCRL algorithm. It can be directly combined with various algorithms such as GCIVL, GCIQL, and HIQL, and has good versatility and scalability.
[0066] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.
Claims
1. A large language model-driven offline target-conditional reinforcement learning method, characterized in that, The method includes: Obtain the sample task and sample task environment, and construct a state interpretation dictionary based on the offline trajectory dataset in the sample task environment; Based on the sample task, sample task environment and state interpretation dictionary, the large language model is called multiple times to generate a key state discriminator; The offline trajectory dataset is relabeled using a key state discriminator to generate training samples. On these training samples, target condition policies corresponding to different task types are generated based on an offline target conditional reinforcement learning algorithm. Based on the task type of the target task, the corresponding target condition strategy is invoked to obtain the control action corresponding to the current state.
2. The method according to claim 1, characterized in that, The offline trajectory dataset based on the sample task environment is used to construct a state interpretation dictionary, including: The offline trajectory data is cleaned and aligned; the cleaning and alignment includes: removing missing fields, removing outliers, aligning timestamps, unifying units of measurement, normalizing / standardizing states, and de-jittering discrete event flags. For each state dimension or field in the cleaned and aligned offline trajectory data, the physical meaning, dimensions, value range, noise type, available metrics, and derivation calculation method are given to generate a state interpretation dictionary.
3. The method according to claim 1, characterized in that, Based on the sample task, sample task environment, and state interpretation dictionary, the large language model is called multiple times to generate a key state discriminator, including: Obtain the task objective and environmental description of the sample task environment; Based on the task objective, environment description, and state interpretation dictionary, the large language model is invoked to obtain key state categories and key state category descriptions; wherein, the key state category description includes the category name, the significance of occurrence, and the observable cues corresponding to the key state category; Based on the key state category description, task objective, environment description, and state interpretation dictionary, the large language model is invoked to generate key state semantic rules; Based on the key state semantic rules and the state interpretation dictionary, the large language model is called to determine the state subspace index corresponding to the key state semantic rule; Based on the key state semantic rules and state subspace index, the large language model is invoked to generate the key state discrimination function code corresponding to the key state semantic rule, and an initial key state discriminator is constructed based on the key state discrimination function code; The initial critical state discriminator code is iteratively updated based on the sample task dataset of the corresponding task type until the generated critical state discriminator passes the compile-time self-check and meets the matching rate constraint.
4. The method according to claim 3, characterized in that, The key state semantic rules include criteria that are combined with, or, and not. These criteria include numerical criteria, relational criteria, and / or event-based criteria. The numerical criteria include threshold intervals satisfied by continuous quantities, the relational criteria include constraints satisfied by object relationships, and the event-based criteria include triggering conditions satisfied by events. The key state semantic rules also include noise resistance conditions.
5. The method according to claim 3, characterized in that, The process by which a large language model determines the state subspace index corresponding to a key state semantic rule includes: Obtain semantic features from key state semantic rules; By using a state interpretation dictionary, keywords in semantic features are mapped to corresponding state fields; Based on this state field, the state subspace index corresponding to the key state semantic rule is obtained.
6. The method according to claim 3, characterized in that, The initial key state discriminator code is iteratively updated based on the sample task dataset of the corresponding task type until the generated key state discriminator passes the compile-time self-check and meets the matching rate constraint, including: Run the current critical state discriminator on the sample task dataset of the corresponding task type to check the sample task data; the check includes: correct syntax, complete variable definition, array out of bounds / type error, and non-existent array, and the running time meets the preset upper limit; The system statistically analyzes the trigger frequency and matching rate of the current key status discriminator on the sample task dataset of the corresponding task type during the inspection process, and generates error logs. For the current key state discriminator whose matching rate is not in the set range, the error log, trigger frequency, key state category, key state semantic rules and state subspace index are fed back to the large language model. After adjusting the key state semantic rules or state subspace index and updating the current key state discriminator, the current key state discriminator is re-executed on the sample task dataset of the corresponding task type. For a current critical state discriminator whose matching rate is within a set range, it is determined that the current critical state discriminator passes the compile-time self-check and satisfies the matching rate constraint.
7. The method according to claim 3, characterized in that, The offline trajectory dataset is relabeled using a key state discriminator to generate training samples, including: For any moment in the offline trajectory data, run the key state discriminator for all key state categories within the subsequent time window of that moment to collect a set of candidate moments for each key state category; For a candidate time set that is not empty, randomly sample state data from the candidate time set and generate a relabeled target based on the state data; For a candidate time set that is empty, a state data point is randomly sampled from a subsequent time window, and a relabeled target is generated based on that state data. Generate training samples; the training samples include: current state data, current action command, relabeling target corresponding to the current state data, relabeling reward calculated based on the relabeling target, and next state data.
8. The method according to claim 7, characterized in that, On this training sample, when generating target-conditional policies for different task types based on the offline target-conditional reinforcement learning algorithm, the following also applies: The system statistically learns a signal and characterizes the relabeled target based on the learned signal; wherein the learned signal includes: the mean and variance of the time difference error or the quantile and moving average of the time difference error, and the learned signal also includes: success rate, Q-value scaling drift and policy entropy change; If the characterization result meets the preset triggering conditions, a self-check during the remarking period is performed, and the window length of the key state semantic rules, subspace index and / or subsequent time window is adjusted based on the diagnostic results of the self-check. A compile-time self-check is performed based on the tuning results to update the training sample.
9. A large language model-driven offline objective-conditional reinforcement learning system, characterized in that, The system includes: The dictionary building module is used to obtain the sample task and sample task environment, and build a state interpretation dictionary based on the offline trajectory dataset in the sample task environment; The discriminator generation module is used to call the large language model multiple times based on the sample task, sample task environment and state interpretation dictionary to generate a key state discriminator. The training module is used to relabel targets in the offline trajectory dataset using a key state discriminator, generate training samples, and generate target condition policies for different task types based on the offline target conditional reinforcement learning algorithm on the training samples. The testing module is used to invoke the corresponding target condition policy based on the task type of the target task in order to obtain the control action corresponding to the current state.
10. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the offline target conditional reinforcement learning method driven by a large language model as described in any one of claims 1-8.