Intelligent agent learning training method and device, computer equipment and storage medium

Abstract

The invention relates to an agent learning training method and device, computer equipment and a storage medium, and the method comprises the steps: converting a high-level intention into an executable code through employing an LLM model, obtaining a clone data set, constructing an intelligent driving scene into a Markov decision process, and constructing an instruction space; constructing an intelligent agent strategy model; a training target is decoupled into a behavior cloning and reinforcement learning target and weighted summation is carried out, the intelligent agent is trained according to the final training target, sampling is carried out from a cloning data set, intelligent agent strategy model parameters are updated, then sampling is carried out from an environment interaction track, and the intelligent agent strategy model parameters are updated in a PPO mode. Due to the fact that a high-level intention-instruction-strategy link is closer to a decision-making level which can be understood by human beings, intelligent agent behaviors can be debugged, audited and backtracked conveniently, maintainability is improved from the perspective of engineering landing, and a clearer interface is provided for follow-up safety fence access, formalized constraint or rule verification.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of agent reinforcement learning, and relates to an agent learning training method and device, a computer device and a storage medium. BACKGROUND

[0002] In reinforcement learning, the goal of an agent is to maximize a pre-defined reward function. But a well-trained agent needs to not only "win the game", but also "complete the task according to human instructions", which is more suitable for the real environment with ambiguous tasks and sparse feedback. This novel and important problem is called human intention alignment reinforcement learning (IARL). Language is the most natural interface for human interaction with agents, and it is worth in-depth study for current researchers to make it participate in the policy learning process.

[0003] Existing language-conditioned reinforcement learning methods often treat language as auxiliary information, which is concatenated with the state and fed into the policy network. This simple design ignores two characteristics of language: expression diversity and hierarchical difference, resulting in the policy's reaction to language often staying at the surface level, making it difficult to achieve consistent and stable behavior guidance. Although large models improve language understanding, they are usually used to convert natural language into templated commands and require a large amount of labeled data. The adjacent research field of VLA or VLN has a direct and surface-level task semantics, and the latest research mostly uses the Transformer structure to handle multi-modal problems, the difficulty of which lies in perception and action execution, not real intention.

[0004] In the field of intelligent driving, facing complex road conditions, multi-agent interaction and uncertainty, this method improves the long-term decision-making quality and safety robustness of the agent in multi-round games by modeling and reasoning the strategies of environmental participants. The core is to treat traffic flow as a partially observable, multi-agent, non-cooperative or semi-cooperative stochastic game process, to identify the type and intention of the opponent, risk-sensitive planning and control, and cross-scene generalization and robustness under the constraints of limited resources (computing, sensing, time budget). The existing methods often have the problems of insufficient strategy generalization ability, difficulty in processing human high-level intention or environmental uncertainty, etc. SUMMARY

[0005] In view of the problems in the above-mentioned traditional method, the application provides an agent learning training method, device, computer device and storage medium, which can enable the agent to understand ambiguous language and process the consistency of behavior and human intention and the maximization of reward.

[0006] To achieve the above purpose, the embodiments of the application adopt the following technical solutions: In one aspect, an agent learning training method is provided, the agent being an automatic driving system, real-time perceived environment data and interaction information being input into the trained agent to realize automatic driving of the vehicle; the agent learning training method comprises: The LLM model based on the DIFY platform is used to convert the intention of the driver into executable code to obtain a human demonstration trajectory; the human demonstration trajectory is used as a cloning data set and also as an output policy of reinforcement learning; The intelligent driving scene is constructed as a Markov decision process, the Markov decision process comprising a state space, an action space, a state transition probability, an initial state distribution, a reward function, a discount factor and a maximum time step; An agent policy model is constructed according to the state space, the instruction space and the action space; each language instruction in the instruction space corresponds to a high-level intention or a low-level operation instruction; The target is decoupled into a behavior cloning target and a reinforcement learning target, and a weighted sum is performed to obtain a final training target; The agent policy model is trained for multiple rounds according to the final training target to obtain a trained agent; wherein in each round of training, learning in a BC phase is first performed according to the cloning data set, and then PPO proximal policy optimization is performed according to the environment interaction trajectory.

[0007] In another aspect, an agent learning training device is also provided, the agent being an automatic driving system, real-time perceived environment data and interaction information being input into the trained agent to realize automatic driving of the vehicle; the agent learning training device comprises: A cloning data set and output policy determination module is configured to convert the intention of the driver into executable code based on the LLM model based on the DIFY platform to obtain a human demonstration trajectory; the human demonstration trajectory is used as a cloning data set and also as an output policy of reinforcement learning; A Markov decision process construction module is configured to construct the intelligent driving scene as a Markov decision process, the Markov decision process comprising a state space, an action space, a state transition probability, an initial state distribution, a reward function, a discount factor and a maximum time step; An agent policy model construction module is configured to construct an agent policy model according to the state space, the instruction space and the action space; each language instruction in the instruction space corresponds to a high-level intention or a low-level operation instruction; A target decoupling module is configured to decouple the target into a behavior cloning target and a reinforcement learning target, and perform a weighted sum to obtain a final training target; The agent model training module is configured to perform multi-round training on the agent policy model according to the final training target, to obtain a trained agent; wherein in each round of training, first, learning in the BC phase is performed according to the cloned dataset, and then PPO proximal policy optimization is performed according to the environment interaction trajectory.

[0008] In another aspect, a computer device is also provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of any of the above-mentioned agent learning and training methods when executing the computer program.

[0009] In yet another aspect, a computer readable storage medium is also provided, which stores a computer program, and the computer program implements the steps of any of the above-mentioned agent learning and training methods when executed by a processor.

[0010] One of the above technical solutions has the following advantages and beneficial effects: The above-mentioned agent learning and training method, device, computer device and storage medium, the method includes: converting a high-level intention into executable code using an LLM model to obtain a cloned dataset, which can significantly reduce the dependence of intelligent driving policy learning on manual rule writing and expert labeling, convert experience into trainable samples, improve data construction efficiency and coverage, and help to fill in samples for long-tail scenarios and complex interaction scenarios; constructing an intelligent driving scenario as a Markov decision process and constructing an instruction space, so that the decision-making process of the agent can be modeled and optimized under a unified state-action-return framework; at the same time, the instruction space provides a higher-level and more controllable action expression, which can reduce the search difficulty caused by continuous low-level control; constructing an agent policy model; decoupling the target into the targets of behavior cloning and reinforcement learning and performing weighted summation to obtain a final training target, which on the one hand uses behavior cloning to provide effective prior and stable gradient in the early stage of training, and on the other hand uses reinforcement learning to optimize long-term return in environment interaction, overcoming the distribution shift and error accumulation problems of pure imitation learning. According to the final training target, the agent model is trained, the agent policy model parameters are updated by sampling from the cloned dataset, and the agent policy model parameters are updated by sampling from the environment interaction trajectory using the proximal policy optimization target of PPO.

[0011] Since the link of high-level intention-instruction-policy is closer to the decision level that humans can understand, it is convenient to debug, audit and backtrack the behavior of the agent (for example, to locate the risk behavior triggered by a certain type of instruction), to improve maintainability from the perspective of engineering landing, and to provide a clearer interface for subsequent access to safety barriers, formal constraints or rule checking. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only represent some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative effort.

[0013] Figure 1 A flowchart of an embodiment of an agent learning training method. DETAILED DESCRIPTION

[0014] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the specification only for the purpose of describing specific embodiments and is not intended to limit the present application.

[0016] It should be noted that the term "embodiment" mentioned herein means that the specific features, structures or characteristics described in conjunction with the embodiments can be included in at least one embodiment of the present application. The phrase is shown at various places in the specification does not necessarily refer to the same embodiment, nor is it independent or alternative to other embodiments. Those skilled in the art can understand that the embodiments described herein can be combined with other embodiments. The term "and / or" used herein refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0017] The embodiments of the present application will be described in detail below with reference to the accompanying drawings of the embodiments of the present application.

[0018] In one embodiment, as shown in Figure 1 An agent learning training method is provided, wherein the agent is an automatic driving system, real-time perceived environmental data and interaction information are input into the trained agent to realize automatic driving of the vehicle; the agent learning training method can include the following processing steps 100 to 104: Step 100: converting the driver's intention into executable code based on the DIFY platform using the LLM model to obtain human demonstration trajectory; taking the human demonstration trajectory as a cloning data set and also as an output policy of reinforcement learning.

[0019] Specifically, the user proposes a vague and high-level intention.

[0020] In the data construction phase, the proposed tactics are run in the form of hard-coded scripts and generate behavior cloning (BC) datasets. It should be noted that this step can be extended by automatic means such as LLM, so the construction cost is relatively low. Although script generation is used, since the tactics come from expert manual design, the dataset can cover diverse task patterns, ensuring the representativeness and challenge of the evaluation.

[0021] The autonomous driving system has the capabilities of environment perception, interaction understanding, long-term planning, continuous learning, and cooperative ability.

[0022] Step 101: Construct the intelligent driving scene as a Markov decision process, which includes state space, action space, state transition probability, initial state distribution, reward function, discount factor, and maximum time step.

[0023] Specifically, the intelligent driving scene is modeled as a Markov decision process (MDP) , wherein represents the state space, covering all features of the map; is the action space; is the state transition probability; is the initial state distribution; is the reward function; is the discount factor; is the maximum time step.

[0024] The intelligent driving scene is a lightweight real-time traffic simulation environment. Although the perception and rendering are simplified, the core challenges are retained, including a complex action space, real-time decision constraints, and sparse win-lose rewards. This application designs 6 different scene types. Then, diverse driving tactics are collected on these scenes, including both successful strategies and failed or near-failure strategies. This reflects the necessity of verifying feasibility in a virtual environment: strategies in real-world roads are not always successful, and the interaction of different traffic participants and road conditions will affect the strategy's benefits. How to remain faithful to the instructions under different driving tactics while meeting safety and compliance constraints is the core test goal of this method.

[0025] Step 102: According to the state space, instruction space, and action space, construct the agent policy model; each language instruction in the instruction space corresponds to a high-level intent or a low-level operation instruction.

[0026] Specifically, to introduce language constraints, this application defines the instruction space , wherein each instruction corresponds to a high-level tactical intent or a low-level operation instruction.

[0027] In real interaction, humans cannot express all tactical intentions at once or give continuous instructions throughout the process, otherwise the burden will be too heavy. Before the demonstration data is constructed, the LLM is used to complete this step. By analyzing the map and combining the initial instructions provided by humans to generate feedback corrections, more reasonable script data is obtained. Therefore, the strategy of the agent can be formalized as: , that is, in the state , conditional on the high-level tactical instruction l , the action selection is performed. By default, when there is no input instruction, the model automatically degenerates into pure RL learning, relying on the environment reward signal to optimize local behavior.

[0028] The agent strategy model is used to output action s t under the corresponding state and instruction .

[0029] Step 103: decouple the target into behavior cloning targets and reinforcement learning targets, and perform weighted summation to obtain the final training target.

[0030] Specifically, the target is decoupled, the training target is divided into behavior cloning and reinforcement learning, and the weighted summation is performed according to the behavior cloning and reinforcement learning targets to obtain the final training target; wherein the behavior cloning focuses on maintaining the consistency of the trajectory and the instruction; the reinforcement learning optimizes the local details using environment interaction to improve the strategy performance.

[0031] The present application is more concerned about whether the agent has faithfully executed the intentions of the human commander, although the human commander's ideas may lead to failure. On this basis, the agent is allowed to explore better results. To this end, the present application decouples the target: behavior specification follows the behavior cloning (BC), while reward optimization is handled by reinforcement learning (RL).

[0032] Step 104: train the agent strategy model according to the final training target for multiple rounds to obtain a trained agent; wherein in each round of training, first learn in the BC phase according to the cloning dataset, and then use PPO proximal policy optimization according to the environment interaction trajectory.

[0033] Specifically, in the training process, the strategy learning is divided into two parts: BC focuses on maintaining the consistency of the trajectory and the instruction, and RL optimizes the local details using environment interaction to improve the strategy performance.

[0034] The dataset consists of two parts: demonstration data generated from expert-designed tactical and rule scripts Autonomous interactions of the agent in the environment. The language instructions are parsed and translated into templated constraints by a large model workflow (Dify) and thus enter the policy learning process in data form.

[0035] The agent model is trained according to the final training target, and in the training process, the agent policy model parameters are updated by sampling from the cloned dataset and minimizing the cross-loss. On this basis, the agent policy model parameters are then updated by sampling from the environment interaction trajectory and using the proximal policy optimization (PPO) objective, and multiple rounds of training are performed to obtain the trained agent.

[0036] The above agent learning and training method comprises: converting high-level intentions into executable code using an LLM model to obtain a cloned dataset, which can significantly reduce the dependence of intelligent driving policy learning on manual rule writing and expert labeling, convert experience into trainable samples, improve data construction efficiency and coverage, and help to fill in samples for long-tail scenarios and complex interaction scenarios; constructing an intelligent driving scenario as a Markov decision process and constructing an instruction space, so that the decision-making process of the agent can be modeled and optimized under a unified state-action-return framework; at the same time, the instruction space provides higher-level and more controllable action expressions, which can reduce the search difficulty caused by continuous low-level control; constructing an agent policy model; decoupling the target into the goals of behavior cloning and reinforcement learning and performing weighted summation to obtain the final training target, which on the one hand utilizes behavior cloning to provide effective prior and stable gradient in the early stage of training, and on the other hand utilizes reinforcement learning to optimize long-term return in environment interaction, overcoming the distribution shift and error accumulation problems of pure imitation learning. According to the final training target, the agent model is trained, the agent policy model parameters are updated by sampling from the cloned dataset, and the agent policy model parameters are updated by sampling from the environment interaction trajectory and using the proximal policy optimization (PPO) objective. Since the link between high-level intentions, instructions, and strategies is closer to the decision-making level that humans can understand, it is convenient to debug, audit, and backtrack the agent's behavior (for example, to locate risky behavior triggered by a certain type of instruction), which improves maintainability from an engineering landing perspective and provides a clearer interface for subsequent access to safety barriers, formal constraints, or rule checking.

[0037] In one embodiment, the final training target in step 103 is: (1) wherein, is the final training target, is a dynamic weight factor, is a behavior cloning loss modeled according to the compliance of the language instructions, is a loss of the reinforcement learning process, Model parameters are denoted as

[0038] Specifically, in the multi-agent environment, the RL objective is defined as follows: (2) During training, policy learning is divided into two parts: BC focuses on keeping the trajectory consistent with the instruction, and RL optimizes local details using environment interaction to improve policy performance.

[0039] In the intelligent driving context, the agent needs to remain loyal when executing the commands issued by the human driver, even if the driver's ideas may lead to suboptimal results or even failure. The solution to this problem lies in how to allocate the roles of BC and RL. From the perspective of maximizing the expected discount reward, this problem can be formalized as the following objective : (3) where the trajectory , the state transition and the reward are given by the environment.

[0040] Now solve the objective equation. The general solution is to optimize the parameters by maximizing the gradient estimate of But in the technical problem solved by the present application, it is not feasible to optimize the objective of formula (2) in practice, for the following two reasons: first, the language instruction mentioned in the foregoing has ambiguity and cannot be converted into a stable constraint function. Second, there is a local conflict between the instruction and the reward signal, and direct joint optimization will lead to unstable gradients. Therefore, the present application adopts a solvable approximation: it converts the problem into a division of labor.

[0041] The compliance of the language instruction is modeled as a pure behavior cloning loss; the local exploration is left to the reinforcement learning to optimize separately, allowing the policy to improve the reward performance within the constraint range. This division of labor avoids forced balancing of "loyalty" and "optimality" on the same optimization path, alleviating the conflict. The training objective is divided into two parts, which can be combined as shown in formula (1).

[0042] In one embodiment, the loss of the reinforcement learning process is: (4) (5) where is the value function loss coefficient, is the entropy regularization coefficient, is the policy entropy, used to enhance exploration, is the value function loss, is the target value, estimated by GAE accumulation, For truncation operation, to ensure that the value function update does not exceed the range, for the value function estimate under the current parameters, is the truncation parameter, is the policy function loss.

[0043] In one embodiment, the dynamic weight factor is: (6) where, is a Sigmoid function, controls the sensitivity, is the state under which the policy is taken by the RL phase, is the state under which the policy is taken by the BC phase, and KL( ) represents the KL regularization.

[0044] Specifically, finally, the dynamic weight factor is explained. In the past combination of RL and BC, the dynamic weight factor is always simply set to 0 / 1; or changes with time steps, the larger, the greater the proportion of RL training; or use motion prior constraints. But in this application, we hope that the agent group always acts according to the natural language instructions of the human commander, and RL only acts on the individual agent's own exploration path and does not interfere with the general direction. Therefore, the above three settings are not satisfactory. Initially, explicit gating was considered, that is, artificially specifying when to use the expert strategy and when to use the RL strategy, but the result was still not ideal. The reason why the above method is unsuccessful lies in the real-time and diversity of combat simulation. Therefore, finally, only the KL regularization method can be used to adjust the policy similarity. There may be more ideal ways, such as a dynamic weight factor based on task progress, but finally, due to the problem of universality, it is not considered again. The dynamic weight factor is finally set as shown in formula (6). In this way, when the policy deviates from the BC trajectory, the dynamic weight factor automatically becomes smaller, and RL can explore; when the policy is near the BC trajectory, becomes larger, maintaining the consistency of the general direction.

[0045] In one embodiment, the formula for updating the agent policy model parameters based on behavior cloning in step 104 is: (7) where, is the BC phase expected cumulative return function. It is difficult to solve directly, so an easy-to-handle approximation method is used to approximate the target is the gradient of the cloning loss, is the behavior cloning model, is the action at the current time, is the state at the current time, is the high-level instruction, is the model parameter.

[0046] In one embodiment, the agent policy model parameter update formula based on reinforcement learning in step 104 is: (8) wherein, is the expected cumulative return function of the RL stage, is the gradient of the reinforcement learning loss, , is the policy at the current time, is the old policy, is the advantage function, is the action at the current time, is the state at the current time, is the high-level instruction, is the model parameter, is the mathematical expectation, is the truncation parameter.

[0047] Specifically, the present application proposes a large model assisted intent alignment learning method, which aims to achieve the division of responsibilities between "faithful execution of human intent" and "environment reward driven local optimization". Overall, the behavior of the policy is constrained by the language instruction . In order to balance the optimization strength of BC and RL, the embodiment also designs a dynamic weight factor . The expression of the final training target is shown in formula (1). Based on this target, two types of update rules are derived respectively: BC update: by sampling from the demonstration data set, the cross-entropy loss is minimized, and formula (7) is used for update.

[0048] RL update: sampling from the environment interaction trajectory, following the proximal policy optimization goal of PPO. Given the old policy , the new policy update target is obtained by using formula (8) for update.

[0049] At the same time, value function clipping is used to stabilize the value network update, as shown in formula (5).

[0050] The final RL loss definition is shown in formula (4).

[0051] Both are updated in parallel during the training process, and the dynamic weight factor for dynamically adjusting the contribution of the two parts.

[0052] In one embodiment, the LLM model in step 102 adopts the native code version provided by Qwen.

[0053] Specifically, in the instruction parsing link, a specially fine-tuned LLM is not used, but the native code version provided by Qwen is directly adopted. The reason for this choice is that its value is more in the rich prior knowledge, similar to introducing a "highly knowledgeable person" with general knowledge. In complex tasks, although a fine-tuned LLM can further improve the accuracy of instruction parsing, it requires additional large-scale data and engineering costs. Considering that the focus of the present application is to verify the "instruction compliance and reward optimization duty division" framework, it is believed that the use of an un-tuned LLM is sufficient to support the required instruction parsing and template filling. The experimental results also show that even without task customization, the prior ability of the native LLM can still guarantee the feasibility of the framework.

[0054] It should be understood that although the above Figure 1 Each step is displayed in sequence according to the arrow, but these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the above Figure 1 The steps can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be alternately executed with other steps or at least part of the sub-steps or stages of other steps.

[0055] In one embodiment, an agent learning and training device is also provided, wherein the agent is an automatic driving system, real-time perceived environmental data and interaction information are input into the trained agent to realize automatic driving of the vehicle; the agent learning and training device comprises: The cloned data set and output policy determination module is configured to convert the driver's intention into executable code based on the DIFY platform using the LLM model to obtain a human demonstration trajectory; the human demonstration trajectory is used as a cloned data set and also as an output policy of reinforcement learning.

[0056] The Markov decision process construction module is configured to construct the intelligent driving scene as a Markov decision process, the Markov decision process including a state space, an action space, a state transition probability, an initial state distribution, a reward function, a discount factor, and a maximum time step.

[0057] The agent policy model construction module is configured to construct an agent policy model according to a state space, an instruction space, and an action space, wherein each language instruction in the instruction space corresponds to a high-level intention or a low-level operation instruction.

[0058] The target decoupling module is configured to decouple the target into a behavior cloning target and a reinforcement learning target, and perform weighted summation to obtain a final training target.

[0059] The agent model training module is configured to perform multi-round training on the agent policy model according to the final training target to obtain a trained agent, wherein in each round of training, learning in a BC phase is performed according to a cloning data set, and then PPO proximal policy optimization is performed according to an environment interaction trajectory.

[0060] In one embodiment, the final training target in the target decoupling module is shown in formula (1).

[0061] In one embodiment, the loss of the reinforcement learning process in the target decoupling module is shown in formula (4) and formula (5).

[0062] In one embodiment, the dynamic weight factor in the target decoupling module is shown in formula (6).

[0063] In one embodiment, the agent policy model parameter update formula based on behavior cloning in the agent model training module is shown in formula (7).

[0064] In one embodiment, the agent policy model parameter update formula based on reinforcement learning in the agent model training module is shown in formula (8).

[0065] In one embodiment, the LLM model in the cloning data set and output policy determination module adopts a native code version provided by Qwen.

[0066] It can be understood that the specific explanations and descriptions of the agent learning and training apparatus can refer to the corresponding explanations and descriptions of the embodiments of the agent learning and training method described above, and will not be repeated here. The various modules in the above agent learning and training apparatus can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of a device with data processing function in hardware form, or can be stored in the memory of the aforementioned device in software form, so that the processor can call and execute the operations corresponding to the above modules. The aforementioned device can be, but is not limited to, various types of data processing computer devices in the prior art.

[0067] In one embodiment, a computer device is also provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0068] It can be understood that the above computer device includes other software and hardware components not listed in the specification in addition to the above-mentioned memory and processor, and the specific image processing computer model can be determined according to different application scenarios, and the specification will not be described one by one.

[0069] In one embodiment, the processor can also implement the steps or sub-steps added in the above embodiments of the intelligent agent learning training method when executing the computer program.

[0070] It can be understood by those skilled in the art that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus dynamic random access memory (Rambus DRAM, RDRAM) and interface dynamic random access memory (DRDRAM).

[0071] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0072] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the protection scope of the present application. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are also within the scope of protection of the present application.

Claims

1. A method for learning training of an agent, the method comprising: The intelligent agent is an automatic driving system, real-time perceived environment data and interaction information are input into the trained intelligent agent to realize automatic driving of the vehicle; the intelligent agent learning training method comprises: The LLM model is used to convert the driver's intention into executable code based on the DIFY platform to obtain a human demonstration trajectory; the human demonstration trajectory is used as a cloning data set and also as an output strategy of reinforcement learning; An intelligent driving scene is constructed as a Markov decision process, which comprises a state space, an action space, a state transition probability, an initial state distribution, a reward function, a discount factor and a maximum time step; An intelligent agent strategy model is constructed according to the state space, an instruction space and the action space; each language instruction in the instruction space corresponds to a high-level intention or a low-level operation instruction; A target is decoupled into a behavior cloning target and a reinforcement learning target, and a weighted sum is performed to obtain a final training target; The intelligent agent strategy model is trained according to the final training target for multiple rounds to obtain a trained intelligent agent; wherein, in each round of training, learning in a BC phase is performed according to the cloning data set, and then PPO proximal policy optimization is used according to an environment interaction trajectory. 2.The intelligent agent learning training method of claim 1, wherein, The final training target is: wherein, is a final training target, is a dynamic weight factor, is a behavior cloning loss modeling compliance to linguistic instructions, is a loss of the reinforcement learning process, is a model parameter. 3.The intelligent agent learning training method of claim 2, wherein, The dynamic weight factor is: wherein, is a Sigmoid function, controls the sensitivity, is a state the policy taken by the RL phase in the case, is a state the policy taken by the BC phase in the case, KL() represents the KL regularization process. 4.The intelligent agent learning training method of claim 1, wherein, The formula for updating parameters of the intelligent agent strategy model based on behavior cloning is: where, is the BC phase expected cumulative return function, which is difficult to solve directly, so an easy-to-handle approximation method is adopted to approximate the target , which is the gradient of the cloning loss, is the behavior cloning model, is the action at the current time, is the state at the current time, is the high-level instruction, is the model parameter. 5.The intelligent agent learning training method of claim 1, wherein, The formula for updating parameters of the intelligent agent strategy model based on reinforcement learning is: wherein, is the expected cumulative return function for the RL phase, is the gradient of the reinforcement learning loss, , is the policy at the current time, is the old policy, is the advantage function, is the action at the current time, is the state at the current time, is the high-level instruction, is the model parameter, is the mathematical expectation, is the truncation parameter. 6.The intelligent agent learning training method of claim 2, wherein, The loss of the reinforcement learning process is: where, is a value function loss coefficient, is an entropy regularization coefficient, is a policy entropy, is a value function loss, is a target value, is a clipping operation, ensuring that the value function update does not exceed a range, is a value function estimate under current parameters, is a clipping parameter, is a policy function loss. 7.The intelligent agent learning training method of claim 1, wherein, The LLM model uses a native code version provided by Qwen.

8. An agent learning training apparatus, comprising: The intelligent agent is an automatic driving system, real-time perceived environment data and interaction information are input into the trained intelligent agent to realize automatic driving of the vehicle; The intelligent agent learning training device comprises: A cloning data set and output strategy determination module is configured to convert the driver's intention into executable code based on the DIFY platform using the LLM model to obtain a human demonstration trajectory; the human demonstration trajectory is used as a cloning data set and also as an output strategy of reinforcement learning; A Markov decision process construction module is configured to construct an intelligent driving scene as a Markov decision process, which comprises a state space, an action space, a state transition probability, an initial state distribution, a reward function, a discount factor and a maximum time step; An intelligent agent strategy model construction module is configured to construct an intelligent agent strategy model according to the state space, an instruction space and the action space; each language instruction in the instruction space corresponds to a high-level intention or a low-level operation instruction; A target decoupling module is configured to decouple a target into a behavior cloning target and a reinforcement learning target, and perform a weighted sum to obtain a final training target; An intelligent agent model training module is configured to train the intelligent agent strategy model according to the final training target for multiple rounds to obtain a trained intelligent agent; wherein, in each round of training, learning in a BC phase is performed according to the cloning data set, and then PPO proximal policy optimization is used according to an environment interaction trajectory. 9.A computer device, comprising a memory and a processor, and characterized in that, The memory stores a computer program, and the processor implements the steps of the agent learning training method in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the agent learning training method in any one of claims 1 to 7.

Images