Agent learning training method and device, computer equipment and storage medium

Abstract

The application relates to an agent learning training method and device, computer equipment and a storage medium, the method comprising: converting a high-level intention into executable code by using an LLM model to obtain a cloned data set, constructing an intelligent driving scene as a Markov decision process and constructing an instruction space; constructing an agent policy model; decoupling a training target into the targets of behavior cloning and reinforcement learning and performing weighted summation, training the agent according to the final training target, updating the agent policy model parameters by sampling from the cloned data set, then sampling from the environment interaction track, and updating the agent policy model parameters in a PPO manner. Since the link of high-level intention-instruction-policy is closer to the decision level that can be understood by humans, the agent behavior is convenient to debug, audit and backtrack, the maintainability is improved from the engineering landing angle, and a clearer interface is provided for subsequent access to safety fences, formal constraints or rule checking.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of agent reinforcement learning, and relates to an agent learning training method and device, a computer device and a storage medium. BACKGROUND

[0002] In reinforcement learning, the goal of an agent is to maximize a pre-defined reward function. But a well-trained agent needs to not only "win the game", but also "complete the task according to human instructions", which is more suitable for the real environment with ambiguous tasks and sparse feedback. This novel and important problem is called human intention alignment reinforcement learning (IARL). Language is the most natural interface for human interaction with agents, and it is worth in-depth study for current researchers to make it participate in the policy learning process.

[0003] Existing language-conditioned reinforcement learning methods often treat language as auxiliary information, which is concatenated with the state and fed into the policy network. This simple design ignores two characteristics of language: diversity of expression and hierarchical difference, resulting in the policy's reaction to language often staying at the surface level, making it difficult to achieve consistent and stable behavior guidance. Although large models improve language understanding, they are usually used to convert natural language into templated commands and require a large amount of labeled data. The adjacent research field of VLA or VLN has a direct and surface-level task semantics, and the latest research mostly uses the Transformer structure to handle multi-modal problems, the difficulty of which lies in perception and action execution, not real intention.

[0004] In the field of intelligent driving, facing complex road conditions, multi-agent interaction and uncertainty, this method improves the long-term decision-making quality and safety robustness of the agent in multi-round games by modeling and reasoning the strategies of environmental participants. The core is to treat traffic flow as a partially observable, multi-agent, non-cooperative or semi-cooperative stochastic game process, to identify the type and intention of the opponent, risk-sensitive planning and control, and cross-scene generalization and robustness under the constraints of limited resources (computing, sensing, time budget). The existing methods often have the problems of insufficient strategy generalization ability, difficulty in handling human high-level intentions or environmental uncertainty, etc. SUMMARY

[0005] In view of the problems existing in the above-mentioned traditional method, the application provides an agent learning training method, device, computer device and storage medium, which can enable the agent to understand ambiguous language and handle the consistency of behavior and human intention and the maximization of reward.

[0006] To achieve the above purpose, the embodiments of the application adopt the following technical solutions:

[0007] In one aspect, an agent learning training method is provided, the agent being an automatic driving system, real-time perceived environment data and interaction information being input into the trained agent to realize automatic driving of the vehicle; the agent learning training method comprises:

[0008] The LLM model based on the DIFY platform is used to convert the intention of the driver into executable code to obtain a human demonstration trajectory; the human demonstration trajectory is used as a cloning data set and also as an output policy of reinforcement learning;

[0009] The intelligent driving scene is constructed as a Markov decision process, the Markov decision process comprising a state space, an action space, a state transition probability, an initial state distribution, a reward function, a discount factor and a maximum time step;

[0010] An agent policy model is constructed according to the state space, the instruction space and the action space; each language instruction in the instruction space corresponds to a high-level intention or a low-level operation instruction;

[0011] The target is decoupled into a behavior cloning target and a reinforcement learning target, and a weighted sum is performed to obtain a final training target;

[0012] The agent policy model is trained according to the final training target for multiple rounds to obtain a trained agent; wherein in each round of training, learning in a BC phase is first performed according to the cloning data set, and then PPO proximal policy optimization is performed according to the environment interaction trajectory.

[0013] In another aspect, an agent learning training device is also provided, the agent being an automatic driving system, real-time perceived environment data and interaction information being input into the trained agent to realize automatic driving of the vehicle; the agent learning training device comprises:

[0014] A cloning data set and output policy determination module is configured to convert the intention of the driver into executable code based on the LLM model based on the DIFY platform to obtain a human demonstration trajectory; the human demonstration trajectory is used as a cloning data set and also as an output policy of reinforcement learning;

[0015] A Markov decision process construction module is configured to construct the intelligent driving scene as a Markov decision process, the Markov decision process comprising a state space, an action space, a state transition probability, an initial state distribution, a reward function, a discount factor and a maximum time step;

[0016] An agent policy model construction module is configured to construct an agent policy model according to the state space, the instruction space and the action space; each language instruction in the instruction space corresponds to a high-level intention or a low-level operation instruction;

[0017] a target decoupling module configured to decouple the target into a behavior cloning target and a reinforcement learning target, and to perform weighted summation to obtain a final training target;

[0018] an agent model training module configured to perform multi-round training on the agent policy model according to the final training target to obtain a trained agent; wherein in each round of training, learning in a BC phase is performed according to the cloning dataset, and then PPO proximal policy optimization is performed according to the environment interaction trajectory.

[0019] In yet another aspect, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps of any of the above agent learning and training methods when executing the computer program.

[0020] In still another aspect, a computer readable storage medium is also provided, storing a computer program, and the computer program implementing the steps of any of the above agent learning and training methods when executed by a processor.

[0021] One of the above technical solutions has the following advantages and beneficial effects:

[0022] The above agent learning and training method, device, computer device and storage medium, the method including: converting a high-level intention into executable code using an LLM model to obtain a cloning dataset, which can significantly reduce the dependence of intelligent driving policy learning on artificial rule writing and expert labeling, convert experience into trainable samples, improve data construction efficiency and coverage, and facilitate sample filling for long-tail scenarios and complex interaction scenarios; constructing an intelligent driving scenario as a Markov decision process and constructing an instruction space, so that the decision-making process of the agent can be modeled and optimized under a unified state-action-return framework; the instruction space provides higher-level and more controllable action expression, which can reduce the search difficulty caused by continuous low-level control; constructing an agent policy model; decoupling the target into a behavior cloning target and a reinforcement learning target and performing weighted summation to obtain a final training target, which on the one hand uses behavior cloning to provide effective prior and stable gradient in the early stage of training to alleviate the cold start, sparse reward and exploration risk of pure reinforcement learning, and on the other hand uses reinforcement learning to optimize long-term return in environment interaction to overcome the distribution shift and error accumulation problem of pure imitation learning. The agent model is trained according to the final training target, the agent policy model parameters are updated by sampling from the cloning dataset, and the agent policy model parameters are updated by sampling from the environment interaction trajectory using the proximal policy optimization target of PPO.

[0023] Since the high-level intent-instruction-strategy link is closer to the human-understandable decision level, it is convenient to debug, audit and trace the agent behavior (e.g. locate the risky behavior triggered by a certain type of instruction), improve the maintainability from the engineering landing perspective, and provide a clearer interface for subsequent access to safety guardrails, formal constraints or rule checking. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 The flowchart of the agent learning training method in an embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not intended to limit the present application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0028] It should be noted that the reference herein to "embodiments" means that the particular features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase is exhibited at various places in the specification does not necessarily mean the same embodiment, nor is it independent or alternative to other embodiments. Those skilled in the art can understand that the embodiments described herein can be combined with other embodiments. The term "and / or" used herein refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0029] The embodiments of the present application will be described in detail below in combination with the drawings in the embodiments of the present application.

[0030] In one embodiment, as Figure 1As shown, an agent learning training method is provided, wherein the agent is an automatic driving system, real-time perceived environment data and interaction information are input into the trained agent to realize automatic driving of the vehicle; the agent learning training method can include the following processing steps 100 to step 104:

[0031] Step 100: Based on the DIFY platform, the intention of the driver is converted into executable code by using the LLM model to obtain the human demonstration trajectory; the human demonstration trajectory is used as the cloning data set and also as the output strategy of reinforcement learning.

[0032] Specifically, the user proposes a vague and high-level intention.

[0033] In the data construction link, the proposed tactics are run in the form of hard-coded scripts and generate behavior cloning (BC) data sets. It should be pointed out that this step can be completely extended by automatic means such as LLM, so the construction cost is relatively low. Although script generation is used, since the tactics come from expert artificial design, the data set can cover diversified task modes, ensuring the representativeness and challenge of evaluation.

[0034] The automatic driving system has the capabilities of environment perception, interaction understanding, long-term planning, continuous learning and collaborative cooperation.

[0035] Step 101: The intelligent driving scene is constructed as a Markov decision process, which includes state space, action space, state transition probability, initial state distribution, reward function, discount factor and maximum time step.

[0036] Specifically, the intelligent driving scene is modeled as a Markov decision process (MDP) , wherein represents the state space, which covers all features of the map; is the action space; is the state transition probability; is the initial state distribution; is the reward function; is the discount factor; is the maximum time step.

[0037] Intelligent driving scenarios are a lightweight real-time traffic simulation environment, which, despite the simplification of perception and rendering, retains the core challenges, including complex action space, real-time decision constraints, and sparse win-lose rewards. Six different scenario types are designed. Then, diverse driving tactics are collected on these scenarios, including both successful strategies and failed or near-failed strategies. This reflects the necessity of verifying feasibility in a virtual environment: strategies in real-world roads are not always successful, and the interaction of different road users and changes in road conditions will affect the strategy's performance. How to remain faithful to the instructions under different driving tactics while meeting safety and compliance constraints is the core test objective of the method.

[0038] Step 102: According to the state space, instruction space, and action space, construct an agent strategy model; each language instruction in the instruction space corresponds to a high-level intent or a low-level operation instruction.

[0039] Specifically, to introduce language constraints, the application defines the instruction space where each instruction corresponds to a high-level tactical intent or a low-level operation instruction.

[0040] In real interactions, humans cannot express all tactical intentions at once or continuously give instructions throughout the process when facing complex environments, otherwise the burden will be too heavy. Before the demonstration data is constructed, the LLM is used to complete this step. By analyzing the map and combining the feedback corrections provided by humans, more reasonable script data is obtained. Therefore, the agent's strategy can be formalized as: that is, under the state , given the high-level tactical instruction l , the action selection is performed. By default, when there is no input instruction, the model automatically degenerates to pure RL learning, relying on environmental reward signals to optimize local behavior.

[0041] The agent strategy model is used to output actions s t under the corresponding state and instruction .

[0042] Step 103: Decouple the target into behavior cloning targets and reinforcement learning targets, and perform weighted summation to obtain the final training target.

[0043] Specifically, the target is decoupled, and the training target is divided into behavior cloning and reinforcement learning. According to the behavior cloning and reinforcement learning targets, the final training target is obtained by weighted summation; the behavior cloning focuses on maintaining the consistency of the trajectory and the instruction; the reinforcement learning optimizes the local details using environmental interaction to improve the strategy performance.

[0044] The present application is more concerned about whether the agent has faithfully executed the intention of the human commander, although the human commander's idea may lead to failure. On this basis, the agent is allowed to explore better results again. To this end, the present application decouples the goals: the behavior specification follows the implementation through behavior cloning (BC), while the reward optimization is handled by reinforcement learning (RL).

[0045] Step 104: training the agent policy model according to the final training target for multiple rounds to obtain a trained agent; wherein in each round of training, first learn in the BC phase according to the cloning data set, and then use PPO proximal policy optimization according to the environment interaction trajectory.

[0046] Specifically, in the training process, the policy learning is divided into two parts: BC focuses on keeping the consistency of the trajectory and the instruction, and RL optimizes the local details using environment interaction to improve the policy performance.

[0047] The data set consists of two parts: Demonstration data generated from expert-designed tactics and rule scripts Autonomous interaction of the agent in the environment. The language instruction is parsed and converted into a templated constraint by the large model workflow (Dify), so as to enter the policy learning process in the form of data.

[0048] According to the final training target, the agent model is trained, and in the training process, the agent policy model parameters are updated by sampling from the cloning data set and minimizing the cross-loss. On this basis, the agent policy model parameters are then updated by sampling from the environment interaction trajectory and using the proximal policy optimization target of PPO for multiple rounds of training to obtain a trained agent.

[0049] The intelligent agent learning training method, the method comprises: converting the high-level intention into executable code by using an LLM model to obtain a cloned data set, which can significantly reduce the dependence of intelligent driving strategy learning on artificial rule writing and expert annotation, convert experience into trainable samples, improve data construction efficiency and coverage, and is beneficial to sample filling of long-tail scenes and complex interaction scenes; the intelligent driving scene is constructed as a Markov decision process and an instruction space, so that the decision process of the intelligent agent can be modeled and optimized under a unified state-action-return framework; at the same time, the instruction space provides higher-level and more controllable action expression, which can reduce the search difficulty brought by continuous low-level control; an intelligent agent strategy model is constructed; the target is decoupled into the target of behavior cloning and reinforcement learning and weighted summation to obtain a final training target, which on the one hand utilizes behavior cloning to provide effective prior and stable gradient in the early stage of training, and on the other hand utilizes reinforcement learning to optimize long-term return in environmental interaction, overcoming the distribution deviation and error accumulation problem of pure imitation learning. According to the final training target, the intelligent agent model is trained, the intelligent agent strategy model parameters are updated by sampling from the cloned data set, and the intelligent agent strategy model parameters are updated by sampling from the environmental interaction trajectory and using the proximal policy optimization target of PPO. Since the link of high-level intention-instruction-strategy is closer to the decision level that can be understood by humans, it is convenient to debug, audit and backtrack the behavior of the intelligent agent (for example, positioning the risk behavior triggered by a certain type of instruction), improve the maintainability from the engineering landing angle, and provide a clearer interface for subsequent access to safety barriers, formal constraints or rule checking.

[0050] In one embodiment, the final training target in step 103 is:

[0051] (1)

[0052] wherein, is the final training target, is a dynamic weight factor, is a behavior cloning loss modeled according to the compliance of the language instruction, is a loss of a reinforcement learning process, is a model parameter.

[0053] Specifically, in a multi-agent environment, the RL target is defined as follows:

[0054] (2)

[0055] In the training process, the policy learning is divided into two parts: BC focuses on keeping the consistency of the trajectory and the instruction, and RL optimizes the local details by using environmental interaction to improve the policy performance.

[0056] In the context of intelligent driving, the agent needs to remain loyal to the commands issued by the human driver even if the driver's idea can lead to suboptimal results or even failure. The solution to this problem lies in how to allocate the roles of BC and RL. From the perspective of maximizing the expected discount reward, this problem can be formalized as the following objective :

[0057] (3)

[0058] where the trajectory , the state transition and the reward are given by the environment.

[0059] Now solve the objective formula. The general solution is to optimize the parameters by maximizing the gradient estimate of . But in the technical problem solved by the present application, it is not feasible to optimize the objective of formula (2) in practice for the following two reasons: first, the language instruction mentioned in the foregoing has ambiguity and cannot be converted into a stable constraint function. Second, there is a local conflict between the instruction and the reward signal, and direct joint optimization will lead to unstable gradients. Therefore, the present application adopts a solvable approximation: it converts the problem into a division of labor.

[0060] The compliance of the language instruction is modeled as a pure behavior cloning loss; the local exploration is left to the reinforcement learning to optimize separately, allowing the strategy to improve the reward performance within the constraint range. This division of labor avoids forced balancing of "loyalty" and "optimality" on the same optimization path, alleviating the conflict. The training objective is divided into two parts, which can be combined as shown in formula (1).

[0061] In one embodiment, the loss of the reinforcement learning process is:

[0062] (4)

[0063] (5)

[0064] where is the value function loss coefficient, is the entropy regularization coefficient, is the policy entropy, used to enhance exploration, is the value function loss, is the target value, estimated by GAE accumulation, is the truncation operation, which ensures that the value function update does not exceed range, is the value function estimate under the current parameters, is the truncation parameter, is the policy function loss.

[0065] In one embodiment, the dynamic weight factor is:

[0066] (6)

[0067] wherein, is a Sigmoid function, controls the sensitivity, is the state of the policy taken by the RL phase, is the state of the policy taken by the BC phase, and KL( ) represents the KL regularization.

[0068] Specifically, finally, the dynamic weight factor is explained. In the past combination of RL and BC, the dynamic weight factor is always simply set to 0 / 1; or changes with time steps, the greater, the greater the proportion of RL training; or the use of motion prior constraints. But in this application, we hope that the agent group always acts according to the natural language instructions of the human commander, and RL only acts on the individual agent's own exploration path and does not interfere with the general direction. Therefore, the above three settings cannot be satisfactory. Initially, explicit gating was considered, that is, artificially specifying when to use the expert strategy and when to use the RL strategy, but the result is still not ideal. The reason why the above method is unsuccessful lies in the real-time and diversity of combat simulation. Therefore, finally, the KL regularization method is used to adjust the policy similarity. There may be more ideal ways, such as a dynamic weight factor based on task progress, but finally it is not considered again due to the problem of universality. The dynamic weight factor is finally set as shown in formula (6). In this way, when the policy deviates from the BC trajectory, the dynamic weight factor automatically becomes small, and RL can explore; when the policy is near the BC trajectory, becomes large, maintaining the consistency of the general direction.

[0069] In one embodiment, the formula for updating the agent policy model parameters based on behavior cloning in step 104 is:

[0070] (7)

[0071] wherein, is the expected cumulative return function in the BC phase. It is difficult to solve directly, so an easy-to-handle approximation method is used to approximate the target , which is the gradient of the cloning loss, is the behavior cloning model, is the action at the current time, is the state at the current time, is the high-level instruction, is the model parameter.

[0072] In one embodiment, the agent policy model parameter update formula based on reinforcement learning in step 104 is:

[0073] (8)

[0074] wherein, is the expected cumulative return function of the RL stage, is the gradient of the reinforcement learning loss, , is the current policy, is the old policy, is the advantage function, is the action at the current time, is the state at the current time, is the high-level instruction, is the model parameter, is the mathematical expectation, is the truncation parameter.

[0075] Specifically, the present application proposes a learning method of large model assisted intent alignment learning, which aims to realize the division of responsibilities between "faithful execution of human intent" and "environment reward driven local optimization". Overall, the behavior of the policy is constrained by the language instruction . In order to balance the optimization strength of BC and RL, the embodiment also designs a dynamic weight factor . The expression of the final training target is shown in formula (1). Based on this target, two types of update rules are derived respectively:

[0076] BC update: by sampling from the demonstration data set, the cross-entropy loss is minimized, and formula (7) is used for update.

[0077] RL update: sampling from the environment interaction trajectory, following the proximal policy optimization goal of PPO. Given the old policy , the new policy update target is obtained by using formula (8) for update.

[0078] At the same time, value function clipping is used to stabilize the value network update, as shown in formula (5).

[0079] The final RL loss definition is shown in formula (4).

[0080] Both are updated in parallel during the training process, and the dynamic weight factor is used to dynamically adjust the contribution of the two parts.

[0081] In one embodiment, the LLM model in step 102 adopts the native code version provided by Qwen.

[0082] Specifically, in the instruction parsing link, a specially fine-tuned LLM is not used, but the native code version provided by Qwen is directly adopted. The reason for this choice is that its value is more in the rich prior knowledge, similar to introducing a "highly knowledgeable person" with general knowledge. In complex tasks, although a fine-tuned LLM can further improve the accuracy of instruction parsing, it requires additional large-scale data and engineering costs. Considering that the focus of the present application is to verify the "instruction compliance and reward optimization job division" framework, it is believed that the use of an un-tuned LLM is sufficient to support the required instruction parsing and template filling. The experimental results also show that even without task customization, the prior ability of the native LLM can still guarantee the feasibility of the framework.

[0083] It should be understood that although the above Figure 1 Each step is displayed in sequence according to the arrow, but these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least part of the above Figure 1 The steps can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or at least part of the sub-steps or stages of other steps.

[0084] In one embodiment, an agent learning and training device is also provided, wherein the agent is an automatic driving system, real-time perceived environmental data and interaction information are input into the trained agent to realize automatic driving of the vehicle; the agent learning and training device comprises:

[0085] The cloned data set and output policy determination module is configured to convert the driver's intention into executable code based on the DIFY platform using the LLM model to obtain a human demonstration trajectory; the human demonstration trajectory is used as a cloned data set and also as an output policy of reinforcement learning.

[0086] The Markov decision process construction module is configured to construct the intelligent driving scene as a Markov decision process, the Markov decision process including a state space, an action space, a state transition probability, an initial state distribution, a reward function, a discount factor, and a maximum time step.

[0087] The agent policy model construction module is configured to construct an agent policy model according to a state space, an instruction space, and an action space, wherein each language instruction in the instruction space corresponds to a high-level intention or a low-level operation instruction.

[0088] The target decoupling module is configured to decouple the target into a behavior cloning target and a reinforcement learning target, and perform weighted summation to obtain a final training target.

[0089] The agent model training module is configured to perform multi-round training on the agent policy model according to the final training target to obtain a trained agent, wherein in each round of training, learning in a BC phase is performed according to a cloning data set, and then PPO proximal policy optimization is performed according to an environment interaction trajectory.

[0090] In one embodiment, the final training target in the target decoupling module is shown in formula (1).

[0091] In one embodiment, the loss of the reinforcement learning process in the target decoupling module is shown in formula (4) and formula (5).

[0092] In one embodiment, the dynamic weight factor in the target decoupling module is shown in formula (6).

[0093] In one embodiment, the agent policy model parameter update formula based on behavior cloning in the agent model training module is shown in formula (7).

[0094] In one embodiment, the agent policy model parameter update formula based on reinforcement learning in the agent model training module is shown in formula (8).

[0095] In one embodiment, the LLM model in the cloning data set and output policy determination module adopts a native code version provided by Qwen.

[0096] It can be understood that the specific explanations and descriptions of the agent learning and training apparatus can refer to the corresponding explanations and descriptions of the embodiments of the agent learning and training method described above, and will not be repeated here. The various modules in the above agent learning and training apparatus can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of a device with data processing function in hardware form, or can be stored in the memory of the aforementioned device in software form, so that the processor can call and execute the operations corresponding to the above modules. The aforementioned device can be, but is not limited to, various types of data processing computer devices in the prior art.

[0097] In one embodiment, a computer device is also provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0098] It can be understood that the above computer device includes other software and hardware components not listed in the specification in addition to the above-mentioned memory and processor, which can be determined according to the specific image processing computer model in different application scenarios, and the specification will not be described one by one.

[0099] In one embodiment, the processor can also implement the steps or sub-steps added in the above embodiments of the intelligent agent learning training method when executing the computer program.

[0100] It can be understood by those skilled in the art that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus dynamic random access memory (Rambus DRAM, RDRAM) and interface dynamic random access memory (DRDRAM).

[0101] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0102] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the protection scope of the present application. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are also within the scope of protection of the present application.

Claims

1. A method for learning and training an intelligent agent, characterized in that, The intelligent agent is an autonomous driving system that inputs real-time perceived environmental data and interaction information into a trained intelligent agent to achieve autonomous driving of the vehicle; the intelligent agent learning and training method includes: Based on the DIFY platform, an LLM model is used to convert the driver's intention into executable code, resulting in a human demonstration trajectory. This human demonstration trajectory is used as a clone dataset and also as the output strategy for reinforcement learning. The intention is expressed as natural language commands and actions. The intelligent driving scenario is constructed as a Markov decision process, which includes a state space, action space, state transition probability, initial state distribution, reward function, discount factor, and maximum time step. Based on the state space, instruction space, and action space, an intelligent agent policy model is constructed; each language instruction in the instruction space corresponds to a high-level intent or a low-level operation instruction. The objective is decoupled into a behavior cloning objective and a reinforcement learning objective, and then weighted and summed to obtain the final training objective; wherein, the final training objective is: in, For the ultimate training goal, As a dynamic weighting factor, The behavioral cloning loss is modeled based on compliance with language instructions. To mitigate the loss during the learning process, These are model parameters; The dynamic weighting factor is: in, It is the Sigmoid function. Controlling sensitivity, For state The strategy adopted in the RL phase under certain circumstances For state In this case, the strategy adopted in the BC phase is KL(), which represents KL regularization. The agent policy model is trained in multiple rounds according to the final training objective to obtain a trained agent. In each round of training, the BC phase is first performed based on the cloned dataset, and then the PPO proximal policy is used for optimization based on the environmental interaction trajectory.

2. The intelligent agent learning and training method according to claim 1, characterized in that, The formula for updating the agent policy model parameters based on behavior cloning is: in, The expected cumulative reward function for the BC stage is difficult to solve directly. Therefore, an easily tractable approximation method is adopted, using a gradient-based loss function to approximate the objective, i.e. , which is the gradient of the cloning loss. For behavioral cloning models, For the action at the current moment, To represent the current state, This is a high-level instruction. These are the model parameters.

3. The intelligent agent learning and training method according to claim 1, characterized in that, The formula for updating the parameters of the agent's policy model based on reinforcement learning is: in, Let be the expected cumulative return function for the RL stage. To reinforce the gradient of the learning loss, , The strategy for the current moment, This is the old strategy. For the dominant function, For the action at the current moment, To represent the current state, This is a high-level instruction. For model parameters, For mathematical expectation, This is for truncating parameters.

4. The intelligent agent learning and training method according to claim 1, characterized in that, The loss in the reinforcement learning process is: in, The loss coefficient of the value function. Here is the entropy regularization coefficient. For policy entropy, For the value function loss, For target value, To ensure that the value function update does not exceed the truncation operation, scope, Estimate the value function under the current parameters. To truncate parameters, The loss is the policy function loss.

5. The intelligent agent learning and training method according to claim 1, characterized in that, The LLM model uses the native code version provided by Qwen.

6. An intelligent agent learning and training device, characterized in that, The intelligent agent is an autonomous driving system that inputs real-time perceived environmental data and interactive information into a trained intelligent agent to achieve autonomous driving of the vehicle. The intelligent agent learning and training device includes: The clone dataset and output policy determination module is used to convert the driver's intention into executable code using an LLM model based on the DIFY platform to obtain the human demonstration trajectory; the human demonstration trajectory is used as the clone dataset and also as the output policy of reinforcement learning; the intention is expressed as natural language command actions; The Markov Decision Process Construction Module is used to construct an intelligent driving scenario as a Markov Decision Process, which includes a state space, action space, state transition probability, initial state distribution, reward function, discount factor, and maximum time step. The agent policy model construction module is used to construct an agent policy model based on the state space, instruction space, and action space; each language instruction in the instruction space corresponds to a high-level intent or a low-level operation instruction. The target decoupling module is used to decouple the target into a behavior cloning target and a reinforcement learning target, and then perform a weighted summation to obtain the final training target; wherein, the final training target is: in, For the ultimate training goal, As a dynamic weighting factor, The behavioral cloning loss is modeled based on compliance with language instructions. To mitigate the loss during the learning process, These are model parameters; The dynamic weighting factor is: in, It is the Sigmoid function. Controlling sensitivity, For state The strategy adopted in the RL phase under certain circumstances For state In this case, the strategy adopted in the BC phase is KL(), which represents KL regularization. The agent model training module is used to train the agent policy model in multiple rounds according to the final training objective to obtain a trained agent. In each round of training, the BC phase is first performed based on the cloned dataset, and then the PPO proximal policy is used for optimization based on the environmental interaction trajectory.

7. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the intelligent agent learning and training method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent agent learning and training method according to any one of claims 1 to 5.

Images