Robot generative adversarial self-imitation learning method based on large language model feedback

Abstract

The invention discloses a robot generative adversarial self-imitation learning method based on large language model feedback, and belongs to the technical field of computer systems based on specific calculation models, and the method comprises the steps: (1) initializing a strategy network, a discriminator network and a demonstration pool; (2) constructing and training a large language model feedback network: obtaining a training sample of the large language model feedback network, and training the large language model feedback network according to the training sample; and (3) an iterative learning step (33): updating the strategy network according to the reward calculated by the new cost function. According to the robot generative adversarial self-imitation learning method based on large language model feedback, the advantages of the large language model in the aspects of semantic comprehension, reasoning and knowledge organization are used as a potential award priori source, and a stable award signal can be provided when the demonstration data quality is poor, so that a strategy exceeding original demonstration is learned.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and robot control, and in particular to a robot generative adversarial self-imitation learning method based on feedback from a large language model. Background Technology

[0002] In recent years, with the development of artificial intelligence and robotics, deep reinforcement learning (DRL) has achieved remarkable results in many simulation and real-world tasks, with applications covering video games, dual-arm and mobile manipulation, autonomous driving, quadcopter control, and even full-body control of humanoid robots. However, designing efficient and dense reward functions for various tasks (especially complex high-dimensional tasks) is both difficult and often impractical, which poses a significant challenge to applying DRL to solve complex, long-cycle, and open-world tasks.

[0003] Similar to human learning, for robot learning, learning from expert trajectories is simpler than carefully constructing reward functions. The simplest method for robots to learn from expert demonstration data is Behavior Cloning (BC), which aims to learn the mapping from states to optimal actions. Research shows that BC based on human demonstration data allows real dual-arm robots to successfully perform banana peeling tasks. However, BC suffers from error accumulation, requires a large amount of data for learning, and cannot generalize to unseen states. In contrast, Inverse Reinforcement Learning (IRL) allows robots to learn policies using a cost function extracted from expert trajectories. Because it assumes expert optimality as a priori assumption for the policy space, IRL can more effectively generalize expert behaviors to unseen states. However, most IRL algorithms require solving a series of planning or reinforcement learning problems in an inner loop, which hinders their application in large-scale complex tasks. Furthermore, if the planning problem does not yield an optimal solution, the performance of the IRL robot may significantly degrade.

[0004] Generative Adversarial Imitation Learning (GAIL) learns policies directly from expert demonstration data by establishing an analogy between imitation learning and Generative Adversarial Networks (GANs), extending IRL to large-scale tasks. However, in many real-world tasks, optimal demonstration data is difficult to obtain, and GAIL often struggles to surpass expert trajectory performance if expert behavior is suboptimal or far from optimal. If the optimal trajectory is unavailable, the robot should imitate its own generated, relatively "better" trajectory. Based on this, Generative Adversarial Self-Imitation Learning (GASIL) was proposed, learning by imitating historically good robot trajectories evaluated based on a pre-defined reward function. However, GASIL contradicts the original intention of the GAIL framework: learning solely from expert trajectories and avoiding pre-defined reward functions. Recent research has extended GAIL by introducing human evaluation feedback on the quality of its behavior, and by allowing the robot to progressively replace provided suboptimal expert demonstration data with self-generated good trajectories selected by human trainers. Both methods enable the robot to achieve near-optimal performance from suboptimal trajectories. However, human experts selecting trajectories or providing evaluation feedback for each robot behavior is costly and time-consuming, especially in complex tasks. Furthermore, experts struggle to consistently provide relatively high-quality trajectories and evaluation feedback for diverse tasks.

[0005] Based on the above analysis, the problems and shortcomings of the existing technology are as follows: (1) Traditional DRL faces significant challenges when applied to complex, long-cycle, and open-world tasks due to the difficulty in designing efficient and dense reward functions.

[0006] (2) BC has the problem of error accumulation, requires a large amount of data to learn, and cannot generalize to unseen states.

[0007] (3) Most IRL algorithms require solving a series of planning or reinforcement learning problems through the model in the inner loop, which hinders their application in large-scale complex tasks. In addition, if the planning problem does not obtain an optimal solution, the performance of the IRL robot may decrease significantly.

[0008] (4) GAIL can learn policies directly from expert trajectories and demonstrates good generalization ability in complex large-scale tasks, but its performance is limited by high-quality expert trajectories and it is difficult to surpass the level of expert trajectories.

[0009] The GASIL method, based on human feedback, allows robots to progressively replace suboptimal trajectories with autonomously generated superior trajectories selected by human trainers, enabling the robot to achieve near-optimal performance from these suboptimal trajectories. However, selecting trajectories or providing evaluation feedback for each robot behavior by human experts is costly and time-consuming, especially in complex tasks. Furthermore, obtaining high-quality demonstration data and evaluation feedback from human experts for diverse tasks is costly and difficult to implement. Summary of the Invention

[0010] To address the challenges of existing generative adversarial self-imitation learning methods, which require human experts to select trajectories or provide evaluations for each robot action, resulting in high feedback costs and significant time consumption, especially in complex tasks, and the difficulty in consistently providing relatively high-quality trajectory and evaluation feedback for diverse tasks, this invention proposes a robot generative adversarial self-imitation learning method based on feedback from a large language model. This method effectively solves these problems.

[0011] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A robot generative adversarial self-imitation learning method based on feedback from a large language model includes: (1) Sample expert trajectories from expert strategies and store them in the demonstration pool. Initialize the network parameters of the policy network, discriminator network, and large language model feedback network. The policy network is used to output actions based on the input state. The discriminator network is used to distinguish whether a given state-action pair comes from the expert trajectory in the demonstration pool or is generated by the policy network, and outputs a reward. The demonstration pool stores several state-action pairs of expert trajectories. (2) The step of constructing and training a large language model feedback network includes: matching the prompt words with the state-action pairs (s) in the demonstration pool. E a E The input is fed into a large language model, which evaluates the input state-action pair based on the cue word and outputs a reward. The training samples (s) of the large language model feedback network are obtained. E a E , ), and train a large language model feedback network based on the training samples; (3) Iterative learning steps, including: (31) The policy network outputs action a based on the input state s; (32) Using the new cost function Calculate the reward for the state-action pair (s, a) using the new cost function. for: ; in, Let $\mathbf{ ... The reward for the large language model feedback network to evaluate the output of the input state-action pair (s, a) is α, where α is the weight vector. (33) According to the new cost function The calculated rewards update the policy network.

[0012] In some embodiments, step (3) further includes an update demonstration pool step, which includes: Obtain robot trajectories for the same task. and expert trajectory The robot trajectory consists of several state-action pairs generated by the policy network according to the task, and the expert trajectory consists of several state-action pairs generated by the expert policy according to the task. The cumulative reward of the robot trajectory and the cumulative reward of the expert trajectory are calculated according to the large language model feedback network. The cumulative reward is the sum of the rewards of the evaluation output of the large language model feedback network for all state-action pairs in any trajectory. The cumulative reward of the robot trajectory is compared with the cumulative reward of the expert trajectory. If the cumulative reward of the robot trajectory exceeds that of the expert trajectory, the robot trajectory is stored in the demonstration pool for subsequent iterative learning steps. When the demonstration pool is full, the expert trajectory with the lowest cumulative reward is removed. By using the feedback network evaluation results of the large language model, the inferior trajectory is gradually replaced with the high-quality trajectory generated by the robot itself.

[0013] In some embodiments, each state-action pair of the robot trajectory is input into the large language model feedback network to obtain the predicted reward of each state-action pair. The predicted rewards of all state-action pairs of the robot trajectory are summed to obtain the cumulative reward of the robot trajectory. Each state-action pair of the expert trajectory is input into the large language model feedback network to obtain the predicted reward for each state-action pair. The predicted rewards of all state-action pairs of the expert trajectory are summed to obtain the cumulative reward of the expert trajectory.

[0014] In some embodiments, the system further includes designing prompt word templates for each task, and the large language model matches each state-action pair (s) in the demonstration pool with the prompt words. E a E The following assessment will be conducted: ; The parameter N is set to a different constant in different tasks; The evaluation results output by the large language model are used as reward labels, and these labels, along with the corresponding state-action pairs, constitute the training samples (s). E a E , ), used for training the large language model feedback network.

[0015] In some embodiments, the training method for the large language model feedback network in step (2) includes: The training samples (s) E aE , The large language model feedback network is trained by minimizing a loss function, wherein the loss function is: ; in, This indicates that the large language model feedback network responds to the input (s) E a E The predicted output of ) For the number of samples, this large language model feedback network is used to replace the large language model in subsequent iterative learning steps to evaluate and assign rewards to the state-action pairs generated by the robot's policy and its interaction with the environment.

[0016] In some embodiments, the large language model feedback network is constructed using a neural network, containing two hidden layers, each with 100 units, using the ReLU non-linear activation function, and the output layer using the tanh non-linear function.

[0017] In some embodiments, the expert trajectories in the demonstration pool mentioned in step (1) are generated using the DQN / TD3 algorithm based on the expert policies obtained from training the corresponding task.

[0018] In some embodiments, the large language model is any one of GPT, Gemini, or Claude.

[0019] Compared with existing technologies, the advantages and positive effects of this invention are as follows: The robot generative adversarial self-imitation learning method based on large language model feedback of this invention can provide external evaluation feedback to the robot's state-action pairs by setting up a large language model feedback network, and calculate the reward by combining the cost function initialized by the discriminator network. This reward is used to update the policy network in the iterative learning step, which helps to guide the policy to explore in a meaningful direction in the early stage of adversarial training. It utilizes the advantages of large language models in semantic understanding, reasoning and knowledge organization as a potential prior source of reward, and can provide stable reward signals when the quality of demonstration data is poor, thereby learning a policy that surpasses the original demonstration.

[0020] Other features and advantages of the present invention will become clearer after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. Attached Figure Description

[0021] Figure 1 This is the overall flowchart of the robot generative adversarial self-imitation learning method based on large language model feedback proposed in this invention; Figure 2 This is a schematic diagram of four physics-based control tasks and simulated mobile robot navigation tasks provided in the embodiments of the present invention; Figure 3This is a schematic diagram illustrating the impact of the weight vector α on the final performance in the new cost function provided by this embodiment of the invention; Figure 4 This is a schematic diagram of the learning curves of three robots in four tasks under the condition of poor demonstration data provided by an embodiment of the present invention; Figure 5 This is a schematic diagram of the learning curves of three robots in a simulated mobile robot navigation task under the condition of poor demonstration data provided by an embodiment of the present invention. Figure 6a This is a schematic diagram illustrating the significance analysis of the performance differences of three robots in Task 1, provided by an embodiment of the present invention. Figure 6b This is a schematic diagram illustrating the significance analysis of the performance differences of three robots in Task 2, provided by an embodiment of the present invention. Figure 6c This is a schematic diagram illustrating the significance analysis of the performance differences of three robots in Task 3, provided by an embodiment of the present invention. Figure 6d This is a schematic diagram illustrating the significance analysis of the performance differences of three robots in Task 4, provided by an embodiment of the present invention. Figure 7 This is a schematic diagram provided by an embodiment of the present invention, showing the score distribution of three robots in four new simulated navigation tasks using a violin diagram; Figure 8 This is a schematic diagram of the test trajectories based on three robot final strategies in four new simulated navigation tasks provided by embodiments of the present invention; Figure 9 This is a schematic diagram illustrating the migration and adaptation of LiDAR data from simulation to reality, provided in an embodiment of the present invention. Figure 10 This is a schematic diagram of a mobile robot test trajectory implemented by migrating to the real environment through a simulation strategy, as provided in an embodiment of the present invention. Detailed Implementation

[0022] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Example 1, see Figure 1As shown, this embodiment proposes a robot generative adversarial self-imitation learning method based on feedback from a large language model, including: (1) Sample expert trajectories from expert policies and store them in the demonstration pool. Initialize the network parameters of the policy network, discriminator network, and large language model feedback network. The policy network is used to output actions based on the input state. The discriminator network is used to distinguish whether the input state-action pair comes from the expert trajectory in the demonstration pool or from the policy network and outputs the reward. The demonstration pool stores several state-action pairs of expert trajectories.

[0025] (2) The steps of constructing and training the large language model feedback network include: combining the prompt words with the state-action pairs (s) in the demonstration pool. E a E The input is fed into a large language model, which evaluates the input state-action pair based on the prompt words and outputs a reward. The training samples (s) of the large language model feedback network are obtained. E a E , The large language model feedback network is trained based on the training samples.

[0026] (3) Iterative learning steps, including: (31) The policy network outputs action a based on the input state s.

[0027] Specifically, the state s of the robot's interaction with the environment is input into the policy network, and the policy network outputs action a based on the state s, thus obtaining a state-action pair (s, a).

[0028] (32) Using the new cost function Calculate the reward for the state-action pair (s, a) using the new cost function. for: .

[0029] in, The cost function for initializing the discriminator network. α is the reward for the large language model feedback network to evaluate the output of the input state-action pair (s, a), where α is the weight vector.

[0030] (33) According to the new cost function The calculated reward update policy network.

[0031] This embodiment of the robot generative adversarial self-imitation learning method based on large language model feedback provides external evaluation feedback to the robot's state-action pairs by setting up a large language model feedback network. It also calculates rewards by combining the cost function initialized by the discriminator network. These rewards are used to update the policy network in the iterative learning steps, which helps guide the policy to explore in a meaningful direction in the early stages of adversarial training. By leveraging the advantages of the large language model in semantic understanding, reasoning, and knowledge organization, it serves as a potential prior source of rewards and can provide stable reward signals even when the quality of the demonstration data is poor, thereby learning a policy that surpasses the original demonstration.

[0032] This method is particularly designed to address the challenges of defining reward functions for complex tasks and setting and verifying them in the real world. By learning strategies from expert trajectories, it avoids the complexity of manually designing reward functions.

[0033] In some embodiments, step (3) further includes an update demonstration pool step, which includes: Obtain robot trajectories for the same task. and expert trajectory The robot trajectory is composed of several state-action pairs in the policy network, and the expert trajectory is composed of several state-action pairs in the demonstration pool. The cumulative reward of the robot trajectory and the cumulative reward of the expert trajectory are calculated according to the large language model feedback network. The cumulative reward is the sum of the rewards of the evaluation output of the large language model feedback network for all state-action pairs in any trajectory.

[0034] The cumulative reward of the robot trajectory is compared with the cumulative reward of the expert trajectory. If the cumulative reward of the robot trajectory exceeds that of the expert trajectory, the robot trajectory is stored in the demonstration pool for subsequent iterative learning steps. When the demonstration pool is full, the expert trajectory with the lowest cumulative reward is removed. By using the feedback network evaluation results of the large language model, the inferior trajectory is gradually replaced with the high-quality trajectory generated by the robot itself.

[0035] This solution leverages a large language model feedback network to enable the robot to learn from low-quality demonstration data and gradually replace the original demonstration trajectories with its own high-quality trajectories evaluated by the large language model feedback. This aims to address the problem that the performance of generative adversarial learning is limited by high-quality expert demonstration data and struggles to surpass its performance. This solution eliminates the need for human experts to select trajectories or provide evaluation feedback for each robot action, thus avoiding the high costs and significant time consumption associated with such methods.

[0036] Large Language Model Feedback Networks (GASL) and self-imitation learning work synergistically, enabling robots to learn from poor or failed demonstration data, even when using poor-performing or incomplete demonstration data. 3MF can still accelerate robot learning and approach optimal performance, demonstrate good generalization ability in different tasks, and achieve excellent performance in real-world scenarios through simulation-to-real-world adaptation, opening up new avenues for its application in a wide range of real-world tasks.

[0037] This scheme does not have high requirements for the expert trajectories in the initial demonstration pool, and allows for inferior trajectories. During the iteration process, this scheme can leverage the advantages of large language models in semantic understanding, reasoning, and knowledge organization as potential reward priors. Based on the powerful information search capabilities of the large language model itself, and with the evaluation of the large language model feedback network, the high-quality trajectories learned by the robot are gradually stored in the demonstration pool, while the initial inferior trajectories are eliminated.

[0038] In some embodiments, each state-action pair of the robot trajectory is input into a large language model feedback network to obtain the predicted reward for each state-action pair. The predicted rewards of all state-action pairs of the robot trajectory are summed to obtain the cumulative reward of the robot trajectory.

[0039] Each state-action pair of the expert trajectory is input into the large language model feedback network to obtain the predicted reward for that state-action pair. The predicted rewards of all state-action pairs of the expert trajectory are summed to obtain the cumulative reward of the robot trajectory.

[0040] In some embodiments, when the demonstration pool is full, the expert trajectory with the lowest cumulative reward is removed. This gradually eliminates inferior expert trajectories, causing them to gradually become higher quality, forming a virtuous cycle. As a result, the performance of adversarial imitation learning is no longer limited by high-quality expert demonstration data, but can even surpass the level of expert demonstration data.

[0041] In some embodiments, step (3) further includes an update discriminator network step, including: updating the expert trajectory and robot trajectory The discriminator network is updated using the Adam gradient estimation method. The core objective of this method is to learn the optimal policy, making the distribution of robot state-action pairs as close as possible to the expert trajectory distribution. This method also makes discriminator training more stable and converges faster, and the output reward signal more accurate and less noisy, thereby improving policy update efficiency and promoting the robot trajectory distribution to approximate the expert trajectory distribution, thus enhancing imitation learning performance.

[0042] In some embodiments, the training method for the large language model feedback network in step (2) includes: (21) Design prompt word templates for each task, and the large language model uses prompt words to match state-action pairs (s) in the demonstration pool. E a E ) Evaluate and output rewards .

[0043] (22) Constructing the loss function Minimize the loss function to train the large language model feedback network: Where n is the number of samples. This indicates that the large language model feedback network responds to the input (s) E a E The large language model feedback network is used to replace the large language model in subsequent iterative learning steps to evaluate and assign rewards to the state-action pairs generated by the robot's policy and its interaction with the environment.

[0044] In some embodiments, the large language model feedback network is constructed using a neural network, containing two hidden layers, each with 100 units, using the ReLU non-linear activation function, and the output layer using the tanh non-linear function.

[0045] In some embodiments, the system further includes designing prompt word templates for each task, and the large language model matches each state-action pair (s) in the demonstration pool with the prompt words. E a E The following assessment will be conducted: .

[0046] The parameter N is set to a different constant in different tasks.

[0047] The evaluation results output by the large language model are used as reward labels, and these labels, along with the corresponding state-action pairs, constitute the training samples (s). E a E , ), used for training the large language model feedback network.

[0048] Existing large language models possess powerful capabilities for deep understanding and generation of natural language, efficiently handling various language-related tasks and adapting to diverse scenarios. However, many tasks and scenarios are not applicable to robots with specific needs. Therefore, this solution utilizes only a subset of tasks and scenarios from the large language model, avoiding the need for its powerful and complex computational resources. Instead, a feedback network is trained to replace the large language model, evaluating and allocating rewards to the state-action pairs generated by the robot's strategy and environment interaction. The tasks and scenarios within the feedback network only need to be adapted to the robot trained using this solution, reducing computational complexity and workload.

[0049] In some embodiments, the prompt consists of the following parts: guidelines, task description and objectives, state space and action instructions, evaluation criteria, inputs, and outputs. The guidelines strictly adhere to the evaluation criteria, preventing the large language model from modifying or changing them based on input, prior dialogue, or internal reasoning. The task, state, and action descriptions provide a clear and structured definition of the environment, enabling the large language model to evaluate the quality of actions within the task context. The evaluation criteria are expressed in natural language from a human observer's perspective, which reduces reliance on numerical thresholds and produces general, expert-free evaluation rules. In the input section, the agent provides its current state and current action, while the output section restricts the large language model to generate a single scalar score within a specified range without additional explanation. This design requires the large language model to independently evaluate each... Tuples are used to ensure the consistency, controllability, and repeatability of feedback signals.

[0050] In some embodiments, the policy network is updated using the TRPO algorithm in step (33).

[0051] Unlike previous studies that learned from best or second-best examples, in some embodiments, the expert trajectories in the demonstration pool in step (1) are generated using expert policies trained on the corresponding task (discrete or continuous action space) using the DQN / TD3 algorithm.

[0052] The initial expert trajectory acquisition method in the demonstration pool is as follows: the policy network acquires the trajectory based on the current state. Output Action The robot performs actions The environmental state becomes , will data ( , The samples are stored in the demonstration pool. In this embodiment, only the poorly performing trajectories are used as demonstrations, and all tasks use trajectories whose performance accounts for only 50% of the overall performance as demonstrations.

[0053] In some simulated robot navigation task experiments, all demonstrations failed to complete the task. Each of the four physical simulation tasks used 10 demonstration trajectories, while the mobile robot navigation task used 20. Although the performance of generative adversarial learning is usually limited by the quality of the demonstration data, this invention, with the help of a self-imitation learning module, can achieve this through a large language model feedback network. The system gradually replaces poor-quality examples with high-quality ones from the agent itself, thereby learning strategies that surpass the original examples. Even when faced with poorly performed or uncompleted examples, the GASL of this invention... 3 The MF robot can still achieve near-optimal performance with a learning speed far exceeding that of generative adversarial learning. When trajectories with better performance than those in this embodiment are initially stored in the demonstration pool, the GASL of this invention is employed. 3The generative adversarial self-imitation learning method for MF robots can achieve better technical results.

[0054] The agent reinforcement learning method in this embodiment is insensitive to the choice of large language model. Experimental results show that using GPT-3.5-turbo, Gemini-1.5, or Claude-3.5 can improve sample efficiency and success rate. Different models only affect the convergence speed, but all can ensure the stability and applicability of the method.

[0055] Example 2: In this example, the present invention employs a real-world mobile robot navigation task. The company An experiment was conducted using an XL mobile robot. This invention defines a reward function to train and evaluate the performance of the method in a simulated mobile robot navigation task. This function comprises four components: .

[0056] in Depends on the robot's excess angular velocity : .

[0057] like Exceeding the threshold The robot will be punished. After numerous tests and experiments, Set the value to 1. Based on distance to the nearest obstacle : .

[0058] If the mobile robot's lidar detects that the nearest obstacle is less than the safety value D, the robot is highly likely to collide, in which case a penalty of -200 will be imposed. In the experiments of this invention, the safety distance D was set to 0.45 meters. The following calculations were made based on the change in distance between the robot and the target point: .

[0059] in and These represent the distances from the robot to the target point at the previous and current moments, respectively. This is a weighting coefficient used to balance rewards; after multiple tests and experiments, this invention sets it to 4.8 in the research. Furthermore, if the robot fails to complete the task within the time limit, it will receive a penalty reward of -200; if it successfully reaches the target point, it will receive a reward of +200, as detailed below: .

[0060] In the simulated mobile robot navigation task, the positions of five obstacles are randomly generated within the shaded circular area between the start and end points at the beginning of each training cycle. It should be noted that the reward functions for the four physics control tasks in OpenAL Gym and MuJoCo, as well as the reward functions defined for the simulated and real-world mobile robot navigation tasks, are only used to evaluate the performance of the three robots and have never been used for training.

[0061] The trajectory sample data is collected by the policy network based on the current state. Output Action The robot performs actions The environmental state becomes , will data ( , Store in the demonstration pool.

[0062] Unlike previous studies that learned from optimal or suboptimal examples, this invention used only poorly performing trajectories as examples in this experiment. These example data were generated by policies trained using the DQN / TD3 algorithm in the corresponding tasks (discrete or continuous action spaces). All tasks used trajectories with performance only 50% of the optimal level as examples. In the simulated robot navigation task, none of the examples successfully completed the task. Ten example trajectories were used for each of the four physical simulation tasks, and 20 were used for the mobile robot navigation task. For GAL... 3 MF and the GASL of the present invention 3 The MF robot's provided state-action pairs are first fed into a large language model feedback network for quality evaluation. The evaluation feedback from the large language model feedback network and the corresponding state-action pairs are then used as samples to train the large language model feedback network, providing rewards during the robot's learning process.

[0063] In this demonstration, the state-action pairs are first input into a large language model for quality evaluation. The obtained evaluation feedback from the large language model and the corresponding state-action pairs are then used as samples to train the large language model feedback network, which provides rewards during the robot's learning process.

[0064] The large language model feedback network is implemented using a neural network and pre-trained using state-action pairs from expert trajectories. For each task, a prompt word template is designed, and the large language model feedback network will process each state-action pair from the expert trajectories. The following assessment will be conducted: .

[0065] The parameter N is set as follows for different tasks: 2 for pole balance and mountain bike, 4 for inverted double pendulum, 5 for single-leg hopping robot, and 200 for mobile robot navigation task.

[0066] The prompt consists of the following parts: guidelines, task description and objectives, state space and action instructions, evaluation criteria, inputs, and outputs. The guidelines strictly adhere to the evaluation criteria, preventing the Large Language Model Feedback Network (LLMRF) from modifying or altering them based on inputs, prior dialogue, or internal reasoning. The task, state, and action descriptions provide a clear and structured definition of the environment, enabling the LLMRF to evaluate the quality of actions within the task context. The evaluation criteria are expressed in natural language from a human observer's perspective, reducing reliance on numerical thresholds and generating general, expert-free evaluation rules. In the input section, the agent provides its current state and current action, while the output section restricts the LLMRF to generate a single scalar score within a specified range without additional explanation. This design requires the LLMRF to independently evaluate each (s, a) tuple to ensure the consistency, controllability, and repeatability of the feedback signal.

[0067] The evaluation feedback provided by the large language model feedback network will serve as a reward label, which, along with the corresponding state-action pair, will constitute the sample. These samples were used to train the LLM feedback network by minimizing the following loss function: .

[0068] This feedback model will replace the large language model in the subsequent learning process, evaluating and assigning rewards to the state-action pairs generated by the robot's strategy and its interaction with the environment.

[0069] Among them, the large language model feedback network Will provide robot trajectory Each state-action pair Provide prediction rewards Then the robot trajectory is calculated. Cumulative rewards At the same time, the present invention will be through Computational demonstration pool Cumulative rewards for all tracks If the robot trajectory Cumulative rewards Cumulative rewards exceeding all demonstration tracks The trajectory will be stored in the demonstration pool. For use in subsequent similar cases The strategy learning phase. When the demonstration pool When the system is full, it will remove the expert track with the lowest accumulated reward. By utilizing... Based on the evaluation results, inferior trajectories are gradually replaced with high-quality trajectories generated by the robot itself, ultimately achieving a complete closed loop of self-imitation learning.

[0070] In the GASL of the present invention 3 In the MF method, the policy training process is similar to the original GAIL framework. However, in the original GAIL, the cost function... Obtained through the discriminator: .

[0071] in This is the output of the discriminator. In contrast, GASL... 3 MF will learn a large language model feedback mechanism With cost function The fusion forms a new cost function: .

[0072] in Representation of large language model feedback network For input The predicted output, For the weight vector, Figure 3 for A diagram illustrating the impact of the value on final performance, used to balance the discriminator output. Feedback network for large language models Based on previous research, this invention sets different α values ​​for different tasks in the new cost function: 1 for pole balancing, mountain bike, and mobile robot navigation tasks; and 0.5 for monopodial hopping robot tasks and inverted double pendulum tasks.

[0073] Specifically: First, similar to GAIL, GASL 3 Discriminator in MF Trained to distinguish state-action pairs in expert trajectories State-action pair in robot trajectory Simultaneously measure the network (i.e., policy) The robot is then trained to "confuse" the discriminator, causing it to misinterpret the state-action pairs in the robot's trajectory. Mistaken for expert trajectory Samples from the database. Robot trajectory samples. Through its current strategy Obtained through interaction with the environment.

[0074] In each iteration, GASL 3 MF first started from the demonstration pool Sampling demonstration trajectory At the same time, the current strategy is obtained through environmental interaction. The robot trajectory below The latest sampled trajectory data and Will be used to pass Gradient update discriminator network . The core objective is to learn the optimal policy so that the distribution of the robot's state-action pairs is as close as possible to the demonstrator's distribution.

[0075] Among them, strategy State actions for an unnormalized distribution are defined as its occupancy measure.

[0076] .

[0077] Expert Strategy With robot strategy The matching of occupancy metrics between them can be expressed as: ; in It is a strategy The entropy function is defined as follows: , For the regularization term of the convex cost function, express The convex conjugate can be interpreted as a similarity measure between the occupancy metrics of the expert policy and the robot policy. The regularization term of the convex cost function... It imposes a slight penalty on functions that assign low-cost actions to expert states, while imposing a strong penalty on other cases. Based on the formula... The new cost function, the regularization term of the cost function can Represented as: .

[0078] in Defined as: .

[0079] Based on this, the regularization term of the cost function can be further derived. Convex conjugate form: .

[0080] .

[0081] Among them, according to maximum entropy inverse reinforcement learning Duality theory, convex conjugate function It can be represented as a discriminator The maximization problem: .

[0082] Based on formula The new cost function defined in this invention is achieved through LLM feedback. Incorporating the robot's expectations, the formula is extended to: .

[0083] Finally, by substituting the above expression back into the occupancy metric to match the target and adding an entropy regularization term, the target is matched. The present invention yields the final saddle point optimization problem: .

[0084] New cost function It will be used for robot strategy through the TRPO algorithm. Updates provide rewards. This adversarial training process will be repeated until the desired strategy is learned.

[0085] The results of this invention demonstrate that while GAIL's performance is limited by the quality of the demonstration data, the self-imitation learning module (GAL³MF) can gradually replace inferior demonstrations with the robot's own high-quality trajectories using an LLM feedback network, thereby learning strategies that surpass the original demonstrations. Furthermore, LLM feedback produces a facilitating effect similar to human evaluation feedback during the robot's learning process, and forms a synergistic effect with self-imitation learning. Even when faced with poorly performed or incomplete demonstrations, the GASL³MF robot of this invention can still achieve near-optimal performance with a learning speed far exceeding that of GAIL and GAL³MF.

[0086] The technical effects of the present invention will be described in detail below with reference to experiments.

[0087] To demonstrate the potential effectiveness of the proposed method, experiments were conducted on four physics-based control tasks.

[0088] (1) Experimental tasks, screenshots of the four tasks are shown below. Figure 2 .

[0089] This invention first evaluated the method in four physics-based control tasks: task (a) pole balancing, task (b) mountain bike, task (c) inverted double pendulum, and task (d) monopodial hopping robot. The inverted double pendulum task was derived from MuJoCo, while the other three tasks were derived from OpenAL Gym. Furthermore, the proposed method was tested in both the Gazebo simulation environment and real-world mobile robot navigation tasks. The illustrations show these four tasks and the simulated mobile robot navigation scenarios.

[0090] Pole Balancing: A pole is connected to a cart via a non-powered joint, and the cart moves on a frictionless track. The goal is to maintain the pole's balance by applying forces to the left or right. The observation space includes four dimensions: cart position, cart velocity, pole tilt angle, and pole angular velocity. The action space is discrete, containing two actions: pushing left or pushing right. Each step is rewarded with +1, and the goal is to keep the pole upright for as long as possible.

[0091] Mountain Bike: A bike is located at the bottom of a valley, and the goal is to accelerate to reach the top of the mountain on the right. The observation space contains two dimensions: the bike's position and speed. The action space is discrete, containing three actions: accelerate to the left, do not accelerate, and accelerate to the right. The reward for each step is set to -1, and the goal is to reach the top of the mountain on the right in the shortest time.

[0092] Inverted Double Pendulum: A double pendulum is connected to a cart via a non-powered joint, and the cart moves on a frictionless track. The goal is to keep the double pendulums upright by applying forces. The observation space contains six dimensions: cart position, cart velocity, and the angles and angular velocities of the two pendulums. The action space is continuous, representing the forces applied to the cart. The reward is based on the degree to which the double pendulums remain upright and the stability of the cart's position.

[0093] Monopod Jumping Robot: This task involves a two-dimensional monopod robot consisting of four main body parts: a torso at the top, a thigh in the middle, a lower leg at the bottom, and a single foot supporting the entire body. The goal is to enable the robot to jump forward (to the right) by applying torque at three joints connecting the four body parts. The observation space contains eleven dimensions: the z-coordinate of the torso (robot height), torso angle, thigh joint angle, lower leg joint angle, foot joint angle, velocity at the x-coordinate of the torso, velocity at the z-coordinate of the torso (height), torso angular velocity, and angular velocities of the thigh, lower leg, and foot joints. The motion space is continuous and contains three motions, representing the torques applied to the thigh, lower leg, and foot joints, respectively.

[0094] Task (e) Mobile Robot Navigation Task: This simulation task is built on the Gazebo simulation platform and requires a mobile robot to navigate from a starting point to a target point. In this task, the robot needs to learn to navigate within a 20-meter long and 10-meter wide area, where 1×1×1 meter obstacles that move randomly within a certain range are set up to simulate a dynamic obstacle course. The observation space contains 24 dimensions, including the Euclidean distance between the robot and the target point, the robot's orientation angles along the xyz axes, and 20-dimensional pre-processed LiDAR data. The motion space consists of two continuous variables: representing the robot's linear velocity and angular velocity, respectively.

[0095] (2) Experimental setup In each task, this invention trained three types of robots: a baseline GAIL robot based on demonstration learning, a GAL robot that simultaneously learns from demonstrations and feedback from a large language model through the GAIL framework, and a robot that learns from both demonstrations and feedback from a large language model. 3 MF robots, and GASL, which learns through self-imitation while absorbing feedback from demonstrations and large language models. 3 MF Robot. In the simulated mobile robot navigation task, the positions of five obstacles are randomly generated within a shaded circular area between the start and end points at the beginning of each training cycle. This invention uses the same neural network architecture to train all robot policies across all tasks. This architecture contains two hidden layers, each with 100 units, and employs a tanh nonlinear activation function between layers. The discriminator network for the three robots shares the same architecture: it consists of three fully connected layers. The first two layers are hidden layers, each containing 50 neurons and using the tanh activation function, and the third layer is the output layer, using a linear activation function. The LLM feedback network contains two hidden layers, each with 100 units, using the ReLU nonlinear activation function, and the output layer uses a tanh nonlinear function. At the start of training, all networks are randomly initialized; this invention uses five random seeds to initialize the task. All experimental tasks use the large language model GPT-3.5-turbo. The cue words for the four physical control tasks describe the various dimensions of the state and action space, as well as the criteria for good actions, bad actions, and task success and failure defined from a human perspective. The cue words for the mobile robot navigation task describe the vehicle state representation (such as the distance between the robot and the target point) and obstacle distance data for 20 radar areas. The cue words also provide instructions on safety assessment (such as avoiding collisions with obstacles).

[0096] (3) Experimental results This invention compares GAIL and GAL. 3 MF and GASL 3 The learning curves of the MF robot in four physical simulation tasks and a mobile robot navigation task were analyzed for performance. The shaded area represents the 0.95 confidence interval, and the bold curve represents the average performance. Furthermore, the generalization ability of the three robots was verified through testing on four new simulated mobile robot navigation tasks and real physical mobile robots.

[0097] 1) Learning curve: Figure 4 The demonstration showed the performance of GAIL and GAL in four physical tasks: (a) pole balancing, (b) mountain bike, (c) inverted double pendulum, and (d) monopodial hopping robot, based on low-quality demonstration data (i.e., only reaching 50% of optimal performance). 3 MF and GASL 3The learning curves of the three MF methods are shown in the figure. As can be seen from the figure: when the demonstration data quality is poor, the performance of the GAIL robot consistently fails to surpass or fluctuates around the demonstration data level; however, by introducing additional feedback from a large language model, GAL… 3 The MF robot demonstrates significantly better overall learning performance and speed than the GAIL method. Furthermore, the GASL method proposed in this invention... 3 The MF method—allowing robots to learn from feedback from large language models and gradually replace low-quality teaching with their own high-quality trajectories—enables robots to achieve speeds far exceeding GASL. 3 MF and GAIL achieve near-optimal performance in terms of speed.

[0098] Furthermore, this invention applies GAL to mobile robot navigation simulation tasks. 3 MF, GASL 3 The MF, GAIL, and three other methods were tested and compared. This task is more complex than the previous four tasks, and the provided teaching data only reaches 50% of the optimal performance. Moreover, the robot failed to complete the navigation task in all the teaching trajectories.

[0099] Figure 5 The learning curves of three robots under low-quality teaching conditions are shown. As can be seen from the figure, the performance of the GAIL robot is far below the provided teaching level; using GAL... 3 After MF (Mastery Learning), although the mobile robot can surpass the teaching performance, the learning process is unstable and fluctuates wildly; in contrast, the GASL (Gas-Induced Learning System) proposed in this invention... 3 When using the MF method, the mobile robot can learn a more stable policy faster and achieve near-optimal teaching performance. This result is consistent with the experimental findings of the first four tasks.

[0100] 2) Ability to generalize to new simulation tasks: This invention also tested GASL 3 MF's ability to generalize to new tasks and situations. Specifically, through GAIL, GAL... 3 MF and GASL 3 The final strategy for the mobile robot trained by MF in simulated navigation tasks was tested and compared in four new simulated mobile robot navigation tasks, with 50 trials in each task. Task 1 was the same as the previously described strategy training task, but the positions of the five obstacles between the start and end points were fixed, representing a typical scenario in the training process; Tasks 2 and 3 had start and end points opposite to the training task and different obstacle layouts; Task 4 used a completely different start and end point and obstacle spatial configuration.

[0101] Figures 6a-6dThis paper presents the significance analysis results of the differences in scores, distance to obstacles, time steps, and path length among three robots performing four new simulated navigation tasks (Tasks 1-4). Statistical comparisons were performed using Student's t-test, with significance levels indicated by asterisks: "*" indicates p-value < 0.05, "**" indicates p-value < 0.01, "***" indicates p-value < 0.001, and "ns" indicates no significant difference. Regarding collision rate and timeout rate, the mobile robots failed to complete the tasks. Figure 7 The distribution of scores (i.e., single-round rewards) obtained by the three robots in four test tasks is presented. Figure 8 Demonstrated through GAIL, GAL 3 MF and GASL 3 A schematic diagram of the test trajectories of a mobile robot trained using the MF method in four new simulated navigation tasks.

[0102] Table 1 shows that mobile robots trained with GAIL are basically unable to complete new tasks; those trained with GAL... 3 The robot trained by MF has a task success rate of approximately 0.5 (with a lower success rate of only 0.38 for task 3); while the robot trained by GASL proposed in this invention has a success rate of approximately 0.5. 3 The robot trained by the MF method performed exceptionally well, achieving a success rate of 0.98 in Task 1, 0.78 in Task 3, and over 0.6 in both Tasks 2 and 4. Figure 8 The typical trajectory shown also confirms this conclusion. Regarding the reasons for incomplete tasks: the GAIL robot was mainly due to timeouts (timeout rates exceeded 0.8 for tasks 1, 2, and 4, and 0.76 for task 3); GAL 3 The MF robot mainly failed in tasks 1 and 3 due to timeouts, and in tasks 2 and 4 mainly due to obstacle collisions; while the GASL of this invention... 3 The MF robot failed to complete all four tasks solely due to obstacle collisions.

[0103] Table 1. Average test performance of the three robots in four new simulated navigation tasks. Task method Success rate Fraction Minimum obstacle distance Collision rate Timeout rate Time Steps Path length 1 GAIL 0 -52.2(±82.28) 0.677(±0.200) 0.16 0.84 395.50(±151.40) 25.90(±18.98) <![CDATA[GAL 3 MF]]> 0.64 150.07(±156.03) 0.651(±0.138) 0.08 0.28 328.89(±171.68) 23.10(±7.02) <![CDATA[GASL 3 MF]]> 0.98 -249.47(±61.39) 0.686(±0.094) 0.02 0 146.38(±52.84) 16.83(±1.78) 2 GAIL 0 -17.03(±45.17) 0.679(±0.095) 0.06 0.94 484.45(±52.25) 25.50(±4.83) <![CDATA[GAL 3 MF]]> 0.48 57.31(±203.80) 0.637(±0.218) 0.32 0.2 149.49(±133.51) 21.17(±3.72) <![CDATA[GASL 3 MF]]> 0.62 108.64(±203.26) 0.500(±0.072) 0.38 0 170.03(±25.42) 16.52(±2.29) 3 GAIL 0 -64.11(±70.92) 0.97(±0.628) 0.24 0.76 463.28(±92.15) 39.27(±15.21) <![CDATA[GAL 3 MF]]> 0.38 87.43(±142.84) 0.794(±0.232) 0.12 0.5 459.88(±79.39) 25.77(±4.50) <![CDATA[GASL 3 MF]]> 0.78 163.01(±173.77) 0.736(±0.209) 0.22 0 185.66(±71.15) 18.74(±3.62) 4 GAIL 0 -60.23(±80.94) 0.527(±0.090) 0.18 0.82 464.59(±90.38) 34.95(±11.86) <![CDATA[GAL 3 MF]]> 0.56 60.106(±222.78) 0.481(±0.091) 0.44 0 95.46(±52.56) 15.23(±8.30) <![CDATA[GASL 3 MF]]> 0.64 107.4(±209.91) 0.537(±0.093) 0.36 0 72.33(±84.45) 9.76(±8.86)

[0104] In addition, combined Figures 6a-6d and Figure 7 It can be seen that GAL 3 The MF robot scored significantly higher than GAIL (except for Task 1, p < 0.01), indicating that self-imitation learning enables mobile robots to generalize their strategies better. However, the GASL of this invention... 3 The MF robot scored significantly higher than the GAL robot only in tasks 2 and 4. 3The MF robot (p<0.05 and p<0.01, respectively) demonstrates that the additional large language model feedback network can further improve the generalization ability of the robot's policy in certain tasks. In contrast, the GASL of this invention, which combines self-imitation learning from poor demonstration data and a large language model feedback network, achieves better results. 3 The MF robot scored significantly higher than GAIL in all tasks (p<0.01 in tasks 1 and 3, p<0.001 in tasks 2 and 4). This indicates a synergistic effect between self-imitation learning and the large language model feedback network, consistent with previous findings on learning from suboptimal demonstration data and human evaluation feedback. This suggests that even in the presence of poor demonstration data, the large language model feedback network can produce effects similar to human evaluation feedback for robot learning.

[0105] Furthermore, in terms of both time and distance required to complete the tasks, the GAL³MF robot required significantly fewer time steps than the GAIL robot in all tasks except Task 3 (p<0.01), demonstrating the advantage of self-imitation learning even based on poor demonstration data. Additional LLM feedback further reduced the number of time steps required to complete tasks 1 and 3 (p<0.01 and p<0.001, respectively), but had no significant effect in tasks 2 and 4. Combining self-imitation learning from poor demonstration data with LLM feedback, the GASL³MF robot of this invention significantly reduced the number of time steps required to complete the tasks compared to the original GAIL robot (p<0.001 in Tasks 1, 2, and 4, p<0.01 in Task 3). This suggests a possible synergistic effect between self-imitation learning and LLM feedback, which helps the GASL³MF robot of this invention learn strategies that enable faster task completion.

[0106] Finally, regarding the distance traveled to complete the tasks, self-imitation learning also enabled the GAL³MF robot to complete the tasks with significantly shorter travel distances in all tasks except Task 1 (p<0.01). The additional large language model feedback network further shortened the GASL distance in all four tasks. 3 MF robots compared to GAL 3 The task completion distance of the MF robot (p<0.01). Simultaneously, by combining self-imitation learning from poor demonstration data and a large language model feedback network, the GASL of this invention... 3 The MF robot traveled significantly shorter distances than the GAIL robot in all four tasks (p < 0.001 in tasks 2, 3, and 4, and p < 0.01 in task 1). However, in terms of minimum obstacle distance (Min-Lidar) metric, there was no significant difference among the three robots in tasks 3 and 4; only in tasks 1 and 2 did the GAIL robot of this invention show a significant difference. 3The MF robot allows the mobile robot to approach obstacles at distances slightly closer than the GAIL robot (p<0.01 and p<0.05, respectively), and in Task 1, it is slightly closer than the GAL robot. 3 MF robot (p<0.01). These results demonstrate that the GASL of the present invention... 3 The MF method enables mobile robots to learn policies to complete tasks with shorter travel distances while maintaining a slightly closer or similar distance from obstacles.

[0107] 3) Physical robot experiments: This invention will be implemented via GASL 3 The final strategy trained using the MF method was directly deployed on a Robotnik Summit XL physical mobile robot to perform the first simulated mobile robot navigation task in a real environment. The Summit XL mobile robot is 1 meter long and 0.6 meters wide, and the experimental area is 10 meters long and 7 meters wide. Five cubic obstacles, each measuring 0.5 meters × 0.5 meters × 1 meter (length × width × height), were placed within the area. The starting point was marked with a green box, and the ending point with a red box. The mobile robot was equipped with sensors including a 2D LiDAR, an inertial navigation unit (INS), and an odometry. The 2D LiDAR provided obstacle position information, the INS calculated the robot's orientation angle and angular velocity, and the odometry measured the distance, velocity, and acceleration relative to the target point. Consistent with the simulation environment, these data collectively constituted the robot's observation state during the task. The motion space was also kept identical to the simulation settings. This invention sets the linear velocity range of the mobile robot to 0 to 2 m / s and the angular velocity range to -2 to -2 rad / s.

[0108] However, there is a gap between the observation data of the physical mobile robot and the simulation environment: in the real environment, the mobile robot's LiDAR collects 811 sampling points to form an observation, while in the simulation environment, the LiDAR collects 1081 points. To achieve effective transfer of simulation strategies to reality, this invention employs a domain adaptation method to narrow the observation gap. Specifically, the data collected by the physical LiDAR is adapted to align with the simulation environment data, such as... Figure 9 As shown in the figure, the entire scanning area of ​​the simulated and physical lidar is uniformly divided into 20 sectors, each sector constituting one dimension of the observation space. In each sector of the simulated and real environments, the point closest to the obstacle is selected as the observation value for that dimension.

[0109] This invention underwent eight sets of tests in physical robot experiments, achieving an overall success rate of 75%. (Figure) Demonstrated via GASL 3 The MF method directly transfers the policy trained in the simulation environment to the test trajectory in the real environment. Figure 10It is evident that physical mobile robots equipped with simulation training strategies can successfully complete obstacle avoidance and navigation tasks in real-world environments with exceptionally smooth trajectories. Figure 10 The trajectory plots at time t, t+1, ... t+5 can be seen, which fully demonstrates the GASL of this invention. 3 The MF method has excellent generalization ability and practical applicability in real-world tasks.

[0110] The GAIL algorithm relies on high-quality demonstration data and struggles to surpass the performance ceiling of demonstration data. Existing research has accelerated GAIL learning and surpassed demonstration levels by introducing human evaluation feedback, but this method still requires suboptimal demonstration data, and human feedback is costly—especially in complex tasks. To address these issues, this invention proposes a generative adversarial self-imitation learning framework (GASL) based on demonstration data and a large language model feedback network. 3 MF enables the robot to learn from low-quality demonstrations and progressively replace them with its own high-quality trajectories evaluated by a large language model feedback network. Experimental results in four physical control tasks and mobile robot navigation simulation tasks demonstrate that even when faced with poorly performing or incomplete demonstrations, the GASL of this invention... 3 MF robots can still outperform GAIL and GAL. 3 MF achieves near-optimal learning speed. Further analysis shows that through GASL... 3 The mobile robot simulation strategy trained by MF exhibits excellent environmental generalization ability and maintains superior performance even after adaptive transfer from simulation to reality. Future work will enhance the algorithm's generalization and adaptability in novel and complex tasks through methods such as world model planning, and will conduct testing using low-quality human demonstrations in real-world environments.

[0111] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A robot generative adversarial self-imitation learning method based on feedback from a large language model, characterized in that, include: (1) Sample expert trajectories from expert strategies and store them in the demonstration pool. Initialize the network parameters of the policy network, discriminator network, and large language model feedback network. The policy network is used to output actions based on the input state. The discriminator network is used to distinguish whether a given state-action pair comes from the expert trajectory in the demonstration pool or is generated by the policy network, and outputs a reward. The demonstration pool stores several state-action pairs of expert trajectories. (2) The step of constructing and training a large language model feedback network includes: matching the prompt words with the state-action pairs (s) in the demonstration pool. E a E The input is fed into a large language model, which evaluates the input state-action pair based on the cue word and outputs a reward. The training samples (s) of the large language model feedback network are obtained. E a E , ), and train a large language model feedback network based on the training samples; (3) Iterative learning steps, including: (31) The policy network outputs action a based on the input state s; (32) Using the new cost function Calculate the reward for the state-action pair (s, a) using the new cost function. for: ； in, Let $\mathbf{ ... The reward for the large language model feedback network to evaluate the output of the input state-action pair (s, a) is α, where α is the weight vector. (33) According to the new cost function The calculated rewards update the policy network.

2. The robot generative adversarial self-imitation learning method according to claim 1, characterized in that, Step (3) also includes an update demonstration pool step, which includes: Obtain robot trajectories for the same task. and expert trajectory The robot trajectory consists of several state-action pairs generated by the policy network according to the task, and the expert trajectory consists of several state-action pairs generated by the expert policy according to the task. The cumulative reward of the robot trajectory and the cumulative reward of the expert trajectory are calculated according to the large language model feedback network. The cumulative reward is the sum of the rewards of the evaluation output of the large language model feedback network for all state-action pairs in any trajectory. The cumulative reward of the robot trajectory is compared with the cumulative reward of the expert trajectory. If the cumulative reward of the robot trajectory exceeds that of the expert trajectory, the robot trajectory is stored in the demonstration pool for subsequent iterative learning steps. When the demonstration pool is full, the expert trajectory with the lowest cumulative reward is removed. By using the feedback network evaluation results of the large language model, the inferior trajectory is gradually replaced with the high-quality trajectory generated by the robot itself.

3. The robot generative adversarial self-imitation learning method according to claim 2, characterized in that, Each state-action pair of the robot trajectory is input into the large language model feedback network to obtain the predicted reward of each state-action pair. The predicted rewards of all state-action pairs of the robot trajectory are summed to obtain the cumulative reward of the robot trajectory. Each state-action pair of the expert trajectory is input into the large language model feedback network to obtain the predicted reward for each state-action pair. The predicted rewards of all state-action pairs of the expert trajectory are summed to obtain the cumulative reward of the expert trajectory.

4. The robot generative adversarial self-imitation learning method according to claim 2, characterized in that, It also includes designing prompt word templates for each task, and the large language model matching each state-action pair (s) in the demonstration pool with the prompt words. E a E The following assessment will be conducted: ； The parameter N is set to a different constant in different tasks; The evaluation results output by the large language model are used as reward labels, and these labels, along with the corresponding state-action pairs, constitute the training samples (s). E a E , ), used for training the large language model feedback network.

5. The robot generative adversarial self-imitation learning method according to claim 1, characterized in that, The training method for the large language model feedback network mentioned in step (2) includes: The training samples (s) E a E , The large language model feedback network is trained by minimizing a loss function, wherein the loss function is: ； in, This indicates that the large language model feedback network responds to the input (s) E a E The predicted output of ) For the number of samples, this large language model feedback network is used to replace the large language model in subsequent iterative learning steps to evaluate and assign rewards to the state-action pairs generated by the robot's policy and its interaction with the environment.

6. The robot generative adversarial self-imitation learning method according to claim 5, characterized in that, The large language model feedback network is constructed using a neural network, containing two hidden layers, each with 100 units, using the ReLU non-linear activation function, and the output layer uses the tanh non-linear function.

7. The robot generative adversarial self-imitation learning method according to any one of claims 1-6, characterized in that, The expert trajectories in the demonstration pool mentioned in step (1) are generated using the DQN / TD3 algorithm based on the expert policies obtained from training on the corresponding tasks.

8. The robot generative adversarial self-imitation learning method according to any one of claims 1-6, characterized in that, The large language model can be any one of GPT, Gemini, or Claude.

Images