Robot walking control method for multi-agent reinforcement learning assisted by multi-objective evolutionary algorithm
By introducing multi-objective evolution algorithms in multi-agent reinforcement learning, using strategy novelty and cumulative reward value to evolve strategy populations, the instability, instability, fragile convergence and lack of exploration in multi-agent reinforcement learning are solved. The problem of robot walking stability and strategy diversity have been improved.
Patent Information
- Application Number
- CN202411625348.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-02-18
AI Technical Summary
The existing multi-agent reinforcement learning technology has instability, instability, fragile convergence and lack of exploration in reward signals, resulting in unstable walking of robots, long training time, and reduced strategy diversity.
Multi-objective evolution algorithm assists multi-agent reinforcement learning. By using the robot's arms, legs and trunk, the robot's arm, legs and trunk, the strategy novelty and cumulative reward value are used as the objective function, the strategy population is evolved to improve the strategy novelty. and diversity.
It improves the stability of robot walking and the diversity of strategies, reduces the possibility of local optimum, and ensures that the robot walks smoothly and has a short training time.
Smart Images
Figure CN119472681A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of multi-agent reinforcement learning and evolutionary algorithms, and in particular to a robot walking control method of multi-agent reinforcement learning assisted by a multi-objective evolutionary algorithm. Background Art
[0002] Multi-agent reinforcement learning (MARL) has become a very popular research direction in recent years, performing well in multi-task games such as StarCraft II, and also showing good results in a series of robot control tasks. In MARL, agents interact with the environment while communicating and collaborating with each other. Each agent has its own observations and actions, and updates its strategy based on reward signals. However, unlike single-agent reinforcement learning, each agent cannot focus only on maximizing its own reward. Instead, it needs to ensure that the collective reward of the entire system is maximized, with the goal of global optimization.
[0003] Despite the current development of MARL, there are still some problems with existing technologies: 1. The reward signal in reinforcement learning may be unstable, making the reward function deceptive and inaccurate. Deceptive strategies may mislead the agent to pursue overestimated rewards without substantial improvement, resulting in poor feedback and getting stuck when interacting with the environment. 2. MARL may exhibit instability and fragile convergence. During training, multiple agents can interact and influence each other, resulting in competition or cooperation. When the reward signal is sparse, some agents may need to complete specific tasks to obtain rewards, but if there is no timely and effective feedback, it will greatly prolong the training time and affect the strategies and behaviors of other agents, resulting in poor convergence. 3. Reinforcement learning itself lacks exploration. In traditional reinforcement learning methods, policy updates usually adopt gradient-based methods, which often fall into local optimality and lack exploration mechanisms, resulting in reduced diversity of agent strategies over time, and ultimately leading to decreased stability and hindering the search for better strategies. Summary of the invention
[0004] In order to address the deficiencies in the prior art, the present invention proposes a robot walking control method of multi-agent reinforcement learning assisted by a multi-objective evolutionary algorithm, in order to solve the problem of unstable robot walking. Within a certain training time, more walking strategies are explored to adjust the speed and movement of the robot, so that the joints of the robot can cooperate with each other and walk smoothly.
[0005] The present invention adopts the following technical solutions to solve the above technical problems:
[0006] The present invention provides a robot walking control method using multi-agent reinforcement learning assisted by a multi-objective evolutionary algorithm, which comprises the following steps:
[0007] S1. In a multi-task multi-agent collaborative environment, the robot's arms, legs, and torso correspond to three agents respectively, and are recorded as arm agents accordingly. , legged agent , Trunk Agent ;
[0008] Define the robot's state set ,in, Represents the state of the robot at the initial moment, represents the state of the robot at time t;
[0009] Define the robot's action set ,..., ,..., ,in, represents the action of the robot at the initial moment, represents the action of the robot at time t, and ,in, represents the strategy group at time t function, and } and save it in the strategy collection , Representing the arm agent The strategy at time t is Representing the leg agent The strategy at time t is Representing the torso agent The strategy at time t;
[0010] Initialize the actor network actor, critic network critic, and buffer D;
[0011] S2, initialize t=0;
[0012] S3, the state of the robot at time t and the three strategies of the agent at time t Input actor network actor, and output the action at time t , execute the action at time t The reward value obtained , and the robot's state at time t+1 , thus obtaining an experience tuple at time t , , } and store it in buffer D;
[0013] S4, using formula (1), formula (2) and formula (3) respectively Update and get the three strategies of the agent at time t+1 And constitute the strategy group at time t+1 Deposit Strategy Collection middle:
[0014] (1)
[0015] (2)
[0016] (3)
[0017] In formula (1), formula (2) and formula (3), , for three different update rates;
[0018] S5. , }as well as Input into the critic network critic for processing, and use formula (4) to construct the loss function of the critic network critic at time t :
[0019] (4)
[0020] In formula (4), is the discount factor, is the value function of the critic network critic; is the expected function;
[0021] S6. After assigning to t, return to S3 and execute sequentially until t> So as to obtain the final strategy set ,in, Indicates the time threshold;
[0022] S7: The final strategy set Each strategy group in is taken as an individual, thus obtaining the t-th generation population = ,in, represents the tth generation population The jth individual in ;
[0023] S8. Take out the experience tuple from buffer D , , }, and Recorded as , Recorded as , Recorded as , Recorded as , thus forming an evolutionary tuple , , };
[0024] S9, Evolution Tuple , , }Respectively and team strategy Input to the policy expansion value function approximator Process it and output the population of the t+1 generation ,in, is the population of generation t+1 The jth individual of ;
[0025] S10. Use formula (5) to construct the t-th generation population Team strategy in The loss function , so as to construct the t-th generation population The overall value function ;
[0026] (5)
[0027] In formula (5), is the policy extension value function approximator The value function of
[0028] S11. After assigning to t, return to S9 and execute sequentially until t> So far, we get the t-th generation population ,in, Indicates the time threshold;
[0029] S12 Input into the actor network for processing and get the jth cumulative reward value , thus obtaining the t-th generation population The cumulative reward value ;
[0030] S13. According to , , }and , get the j-th action matrix , thus obtaining the t-th generation population Action Matrix ;
[0031] S14, calculated using formula (6) Novelty , thus obtaining the t-th generation population The novelty of the strategy :
[0032] (6)
[0033] In formula (6), for The kth individual in The action matrix;
[0034] S15. According to and , using the non-dominated sorting algorithm in NSGA-2 to sort the t-th generation population After multi-objective sorting, the Pareto front is obtained and graded, so as to generate the t+1 generation population after sorting from high to low. ;
[0035] S16, the t+1 generation population The first individual with the highest level Select as the elite strategy and Individual Select as the worst strategy and make the second individual Selected as a suboptimal strategy;
[0036] S17. The worst strategy Discard it directly and use the elite strategy and suboptimal strategies After uniform crossover, we get a child strategy And put it into the population of generation t+1 In order to replace the worst strategy ;
[0037] S18, will After assigning to t, return to S12 and execute sequentially until t> So far, we get Generation population Each individual corresponding to the is used as the best strategy to control the robot's walking.
[0038] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the robot walking control method, and the processor is configured to execute the program stored in the memory.
[0039] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the robot walking control method when the computer program is executed by a processor.
[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0041] 1. The present invention introduces an evolutionary algorithm combined with multi-agent reinforcement learning to control the robot's walking, collects the walking strategies used by the robot into a population, and the evolutionary algorithm simulates the natural evolution process, does not rely on gradient information, and improves the stability of the robot's exploration walking during training;
[0042] 2. The present invention proposes a strategy novelty metric as an additional objective function besides reward, so as to use a multi-objective evolutionary algorithm to evolve the strategy population, and take the difference between the robot's walking strategy and other strategies in the population as the strategy novelty, aiming to encourage the robot to explore more diverse walking strategies, and then select the best walking strategy from the population for crossover to generate new offspring strategies, thereby reducing the possibility of falling into local optimality, ensuring that the robot makes steady progress during the walking process, and finally controlling the robot to walk smoothly. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION
[0044] In this embodiment, a multi-agent reinforcement learning walking control method assisted by a multi-objective evolutionary algorithm is to establish an actor network and a critic network in the framework of multi-agent reinforcement learning, use strategy novelty and cumulative reward value as objective functions, and collect the walking strategies used by the robot as a population. During the training process, the multi-objective evolutionary algorithm is used to evolve the population, and then crossover is performed to obtain new offspring to improve the novelty and diversity of the strategy, reduce local optimal situations, and thus obtain the optimal population. The overall process is as follows Figure 1 As shown, the method is performed in the following steps:
[0045] S1. In a multi-task multi-agent collaborative environment, the robot's walking process is set as a partially observable Markov decision process, and the robot's walking task is decomposed into a multi-agent reinforcement learning collaborative task. The robot's arms, legs, and torso correspond to three agents respectively, and are recorded as arm agents accordingly. , legged agent , Trunk Agent ;
[0046] Define the robot's state set ,in, Represents the state of the robot at the initial moment, represents the state of the robot at time t;
[0047] Define the robot's action set ,..., ,..., ,in, represents the action of the robot at the initial moment, represents the action of the robot at time t, and ,in, represents the strategy group at time t function, and } and save it in the strategy collection , Representing the arm agent The strategy at time t is Representing the leg agent The strategy at time t is Representing the torso agent The strategy at time t is Corresponding to the speed of the robot agent at time t and angle , by adjusting and To control the robot's actions;
[0048] Initialize the actor network actor, the critic network critic, and the buffer D.
[0049] S2, initialize t=0;
[0050] S3, the state of the robot at time t and the three strategies of the agent at time t Input the actor network actor, the actor network is the training of the reinforcement learning strategy, the strategy is the weight of the neural network, and outputs the action at time t , execute the action at time t The reward value obtained , and the robot's state at time t+1 , thus obtaining an experience tuple at time t , , } and store it in buffer D;
[0051] S4, using formula (1), formula (2) and formula (3) respectively Update and get the three strategies of the agent at time t+1 And constitute the strategy group at time t+1 Deposit Strategy Collection middle:
[0052] (1)
[0053] (2)
[0054] (3)
[0055] In formula (1), formula (2) and formula (3), , For three different update rates.
[0056] S5. , }as well as Input into the critic network critic for processing, and use formula (4) to construct the loss function of the critic network critic at time t , the critic network evaluates the strategy and guides the training of the strategy:
[0057] (4)
[0058] In formula (4), is the discount factor, is the value function of the critic network critic; is the expected function.
[0059] S6. After assigning to t, return to S3 and execute sequentially until t> So as to obtain the final strategy set ,in, Indicates the time threshold;
[0060] S7: The final strategy set Each strategy group in is taken as an individual, thus obtaining the t-th generation population = In single-agent reinforcement learning, each agent only needs to consider its own state and action, but in multi-agents, other agents must be considered, so groups are used as population individuals, where represents the tth generation population The jth individual in .
[0061] S8. Take out the experience tuple from buffer D , , }, and Recorded as , Recorded as , Recorded as , Recorded as , thus forming an evolutionary tuple , ,};
[0062] S9, Evolution Tuple , , }Respectively and team strategy Input to the policy expansion value function approximator Process it and output the population of the t+1 generation , because in partially observable Markov space, the agent can only see its own state, but not the state of other agents, so we use To unify the value of the estimation strategy, where is the population of generation t+1 The jth individual of .
[0063] S10. Use formula (5) to construct the t-th generation population Team strategy in The loss function , so as to construct the t-th generation population The overall value function ;
[0064] (5)
[0065] In formula (5), is the policy extension value function approximator The value function of .
[0066] S11. After assigning to t, return to S9 and execute sequentially until t> So far, we get the t-th generation population ,in, Indicates the time threshold;
[0067] The S12 multi-objective optimization problem can be expressed as , by optimizing the two objectives to evolve the population, Input into the actor network for processing and get the jth cumulative reward value , thus obtaining the t-th generation population The cumulative reward value .
[0068] S13. According to , , }and , get the j-th action matrix , thus obtaining the t-th generation population Action Matrix ;
[0069] S14, calculated using formula (6) Novelty , thus obtaining the tth generation population The novelty of the strategy ,Novelty is to hope that the robot can obtain more different walking strategies. The more differentiated the strategy, the more novel it is, and new walking strategies can be explored:
[0070] (6)
[0071] In formula (6), for The kth individual in The action matrix.
[0072] S15. According to and , using the non-dominated sorting algorithm in NSGA-2 to sort the t-th generation population After multi-objective sorting, the Pareto front is obtained and graded, so as to generate the t+1 generation population after sorting from high to low. , individuals of different levels can be sorted directly, but individuals of the same level need to be sorted according to the crowding distance;
[0073] S16, the t+1 generation population The first individual with the highest level Select as the elite strategy and Individual Select as the worst strategy and make the second individual Selected as a suboptimal strategy;
[0074] S17. The worst strategy Discard it directly and use the elite strategy and suboptimal strategies After uniform crossover, we get a child strategy And put it into the population of generation t+1 In order to replace the worst strategy ;
[0075] S18, will After assigning to t, return to S12 and execute sequentially until t> So far, we get Generation population Each individual corresponding to the is used as the best strategy to control the robot's walking.
[0076] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.
[0077] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.
Claims
1. A robot walking control method based on multi-agent reinforcement learning assisted by a multi-objective evolutionary algorithm, characterized in that: The steps include: S1. In a multi-task multi-agent collaborative environment, the robot's arms, legs, and torso correspond to three agents respectively, and are recorded as arm agents accordingly. , legged agent , Trunk Agent ; Define the robot's state set ,in, Represents the state of the robot at the initial moment, represents the state of the robot at time t; Define the robot's action set ,..., ,..., ,in, represents the action of the robot at the initial moment, represents the action of the robot at time t, and ,in, represents the strategy group at time t function, and } and save it in the strategy collection , Representing the arm agent The strategy at time t is Representing the Leg Agent The strategy at time t is Representing the torso agent The strategy at time t; Initialize the actor network actor, critic network critic, and buffer D; S2, initialize t=0; S3, the state of the robot at time t and the three strategies of the agent at time t Input actor network actor, and output the action at time t , execute the action at time t The reward value obtained , and the robot's state at time t+1 , thus obtaining an experience tuple at time t , , } and store it in buffer D; S4, using formula (1), formula (2) and formula (3) respectively Update and get the three strategies of the agent at time t+1 And constitute the strategy group at time t+1 Deposit Strategy Collection middle: (1) (2) (3) In formula (1), formula (2) and formula (3), , for three different update rates; S5. , }as well as Input into the critic network critic for processing, and use formula (4) to construct the loss function of the critic network critic at time t : (4) In formula (4), is the discount factor, is the value function of the critic network critic; is the expected function; S6. After assigning to t, return to S3 and execute sequentially until t> So as to obtain the final strategy set ,in, Indicates the time threshold; S7: The final strategy set Each strategy group in is taken as an individual, thus obtaining the t-th generation population = ,in, represents the tth generation population The jth individual in ; S8. Take out the experience tuple from buffer D , , }, and Recorded as , Recorded as , Recorded as , Recorded as , thus forming an evolutionary tuple , , }; S9, Evolution Tuple , , }Respectively and team strategy Input to the policy expansion value function approximator Process it and output the population of the t+1 generation ,in, is the population of generation t+1 The jth individual of ; S10. Use formula (5) to construct the t-th generation population Team strategy in The loss function , so as to construct the t-th generation population The overall value function ; (5) In formula (5), is the policy extension value function approximator The value function of S11. After assigning to t, return to S9 and execute sequentially until t> So far, we get the t-th generation population ,in, Indicates the time threshold; S12 Input into the actor network for processing and get the jth cumulative reward value , thus obtaining the t-th generation population The cumulative reward value ; S13. According to , , }and , get the j-th action matrix , thus obtaining the t-th generation population Action Matrix ; S14, calculated using formula (6) Novelty , thus obtaining the t-th generation population The novelty of the strategy : (6) In formula (6), for The kth individual in The action matrix; S15. According to and , using the non-dominated sorting algorithm in NSGA-2 to sort the t-th generation population After multi-objective sorting, the Pareto front is obtained and graded, so as to generate the t+1 generation population after sorting from high to low. ; S16, the t+1 generation population The first individual with the highest level Select as the elite strategy and Individual Select as the worst strategy and make the second individual Selected as a suboptimal strategy; S17. The worst strategy Directly discard and use the elite strategy and suboptimal strategies After uniform crossover, we get a child strategy And put it into the population of generation t+1 In order to replace the worst strategy ; S18, will After assigning to t, return to S12 and execute sequentially until t> So far, we get Generation population Each individual corresponding to the is used as the best strategy to control the robot's walking.
2. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the robot walking control method according to claim 1, and the processor is configured to execute the program stored in the memory.
3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the robot walking control method according to claim 1 are executed.