A reinforcement learning method and system for swarm robots to avoid capture

Abstract

The present application discloses a reinforcement learning method and system for cluster robots to avoid capture. First, the individuals in the cluster are initialized; the current action data of each individual in the target cluster is determined through a greedy strategy; a dynamic model is established for learning; specifically, the task of learning the number of surrounding individuals is learned through low-dimensional input, and the observation dimension is expanded to learn the tasks of aggregation and anti-collision, and the observation dimension is further expanded to learn the task of avoiding capture. Continuously acquire environmental information and perform actions, use the reward function to update the policy network, and finally complete the learning of specific tasks. The present invention draws on the movement pattern and learning method of biological clusters and is highly biomimetic. At the same time, the learning of this framework does not rely on any prior behavioral rules to constrain the behavior of the robot, but allows all individuals to start learning from the most initial random state, reducing manual dependence and having good robustness and scalability.

Description

Technical Field

[0001] The present invention relates to the technical field of robot swarm motion, and in particular to a reinforcement learning method and system for swarm robots to avoid capture. Background Art

[0002] Spectacular clustering phenomena can often be observed in animals such as fish and birds. In particular, when a large group of animals encounters a predator, they can show amazing consistency to avoid the predator.

[0003] The ability of swarms to collectively avoid predators improves the survival of each individual. However, the origins of this ability remain unclear, and its biological function remains largely unexplored: how do such a large number of organisms perceive information from their environment and use and share this information within the swarm to coordinate their actions? Learning from biological wisdom to feed back into human progress has long been a crucial strategy that has enabled humanity to stand out from the crowd. Learning this ability from swarms of animals like fish and birds and applying it to robotics holds immense practical significance. In recent years, advances in deep learning have led to the continuous development of swarm robotics algorithms, with reinforcement learning in particular finding widespread application. However, existing research on learning policy networks still relies on prior behavioral rules, which presents certain limitations. Summary of the Invention

[0004] Based on this, an embodiment of the present application provides a reinforcement learning method and system for cluster robots to avoid capture. This method does not rely on any prior behavioral rules to constrain the behavior of the robots, but instead allows all individuals to start learning from the most initial random state, reducing manual dependence and having good robustness and scalability.

[0005] In a first aspect, a reinforcement learning method for swarm robots to avoid capture is provided, the method comprising:

[0006] S1, obtaining a target cluster and initializing each individual in the target cluster; the target cluster includes multiple captured robots and one hunting robot;

[0007] S2, determining the current moment action data of each individual in the target cluster through a greedy strategy; wherein the action includes the linear acceleration and steering angle of the individual;

[0008] S3, establishing a dynamic model of the captured robot and the hunting robot; the dynamic model includes three sub-network structures, wherein the first sub-network structure is used to obtain the number of individuals around the current individual, the second sub-network structure is used to obtain the aggregation and anti-collision structure, and the third sub-network structure is used to obtain the relative position result of avoiding capture;

[0009] S4, traversing all individuals in the target cluster as current individuals to train the first sub-network structure, specifically using q-learning to update the policy network of each individual;

[0010] S5, determining network data of individuals surrounding the current individual, and matching the network data of the surrounding individuals with the current individual for collaborative learning;

[0011] S6, updating the probability parameters in the greedy strategy and starting to execute the next cycle of training until the preset training cycle is reached;

[0012] S7, when the first sub-network converges after reaching the preset training cycle, the second sub-network structure and the third sub-network structure are trained in sequence through steps S2 to S5 to obtain the target dynamics model, and the target dynamics model is used to obtain the cluster robot avoidance and capture results.

[0013] Optionally, the first sub-network structure and the second sub-network structure specifically include a 4-layer neural network, the middle layer has 32 neurons, and the activation function is ReLU.

[0014] Optionally, the third sub-network structure specifically includes three intermediate layer structures, wherein the first intermediate layer includes two parts, each with 16 neurons, and the second intermediate layer and the third intermediate layer each contain 32 neurons.

[0015] Optionally, training the first sub-network structure specifically includes:

[0016] Input the speed direction of the current individual and the difference between the surrounding individuals, and use the formula

[0017]

[0018] Get the number of individuals around the current individual, where Indicates the number of individuals around the current individual. Where R is the current individual perception radius, is the number of surrounding individuals within the current individual’s perception range, is the current individual speed, is the current individual position, t is the current moment, and through Get reward value.

[0019] Optionally, training the second sub-network structure specifically includes:

[0020] exist Add eight-dimensional input based on The sensing range is divided into eight directions. If an individual in that direction reaches the warning distance, the corresponding The value is 1, otherwise it is 0.

[0021] And through the reward function

[0022]

[0023] Get the reward value of training, where c g ,c c is a constant, d collision is the warning distance, R is the current individual perception radius, is the number of surrounding individuals within the current individual’s perception range, is the current individual position, and t is the current moment.

[0024] Optionally, training the third sub-network structure specifically includes:

[0025] pass Determine the speed and direction difference between the hunting robot and the current individual; Determine the position angle difference between the hunting robot and the current individual, where pass Determine the distance of the hunting robot;

[0026] The speed direction difference, position angle difference and distance between the hunting robot and the current individual are used as three-dimensional inputs.

[0027]

[0028] Get the reward value, where c0, c1, c2 are constants, and R is the current individual perception radius. is the number of surrounding individuals within the current individual’s perception range, is the current individual position, is the position of the hunting robot, and t is the current moment.

[0029] Optionally, the probability parameter in the greedy strategy is updated according to the formula:

[0030] ε t+1 ←max(0.01,ε t -ε d )

[0031] Determine the update probability parameter, where ε d is 1e-6.

[0032] Optionally, use q-learning to update the policy network of each individual according to the formula:

[0033]

[0034] The updated policy network is obtained, where α is the learning rate, γ is the discount coefficient, and (s, a, r, s′, a′) are the state value, action value, reward value, state value at the next moment, and action value at the next moment in reinforcement learning, respectively.

[0035] In a second aspect, a reinforcement learning method system for swarm robots to avoid capture is provided, the system comprising:

[0036] An acquisition module is used to acquire a target cluster and initialize each individual in the target cluster; the target cluster includes a plurality of captured robots and a hunting robot;

[0037] A determination module, configured to determine the current moment action data of each individual in the target cluster using a greedy strategy; wherein the action includes the linear acceleration and steering angle of the individual;

[0038] Establishing a module for establishing a dynamic model of the captured robot and the hunting robot; the dynamic model includes three sub-network structures, wherein the first sub-network structure is used to obtain the number of individuals around the current individual, the second sub-network structure is used to obtain the aggregation and anti-collision structure, and the third sub-network structure is used to obtain the relative position result of avoiding capture;

[0039] The training module is used to traverse all individuals in the target cluster as current individuals to train the first sub-network structure, specifically using q-learning to update the policy network of each individual; determine the network data of the surrounding individuals of the current individual, and match the network data of the surrounding individuals with the current individual for cooperative learning; update the probability parameters in the greedy strategy, and start the training for the next cycle until the preset training cycle is reached; when the first sub-network converges after the preset training cycle is reached, the second sub-network structure and the third sub-network structure are trained in turn to obtain the target dynamics model, which is used to obtain the cluster robot's avoidance and capture results.

[0040] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0041] (1) Compared with traditional robot swarm algorithms, the present invention does not require human constraints on the robot's behavior, but achieves specific tasks through thorough reinforcement learning and has good autonomy.

[0042] (2) Compared with traditional learning frameworks, the present invention can learn multiple complex tasks simultaneously, has good scalability, and is a fully distributed system, which improves computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0044] Figure 1 A diagram of a progressive learning framework based on multi-agent reinforcement learning provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of the experimental environment structure provided for the embodiments of the present application;

[0046] Figure 3 A schematic diagram of the network structure provided in the embodiment of the present application;

[0047] Figure 4 A schematic diagram of the segmentation of each individual observation space provided in an embodiment of the present application;

[0048] Figure 5 The overall direction evolution process in the optional embodiment of this application;

[0049] Figure 6 This is a schematic diagram of the robot status at the end of training in an optional embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] In the description of the present invention, unless otherwise specified, "plurality" means two or more. The terms "first," "second," "third," "fourth," and so on, used in the specification and claims of the present invention and in the accompanying drawings, are intended to distinguish the objects referred to. For schemes with sequential flows, this terminology does not necessarily describe a specific order or precedence. For schemes with device structures, this terminology does not distinguish between levels of importance or positional relationships.

[0052] In addition, the terms "comprise", "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that have been explicitly listed, but may also include other steps or units that are not explicitly listed but are inherent to these processes, methods, products or apparatuses, or steps or units that are added based on further optimization solutions conceived by the present invention.

[0053] The present invention discloses a hybrid progressive reinforcement learning framework, which includes a basic biodynamic model, multi-agent reinforcement learning, and cooperative learning. In this framework, the robot will carry out three-stage learning. First, it will learn the simple task of maintaining the number of surrounding individuals through low-dimensional input, then learn the tasks of aggregation and anti-collision by expanding the observation dimension, and finally learn the task of avoiding capture by further expanding the observation dimension. Each learning individual moves in a simulated two-dimensional environment, continuously obtains environmental information and performs actions, and uses a pre-set reward function to update its own strategy network, and finally completes the learning of a specific task. The learning framework of the present invention fully draws on the movement pattern and learning method of biological clusters and has a high degree of biomimetic properties. At the same time, the learning of this framework does not rely on any prior behavioral rules to constrain the behavior of the robot, but allows all individuals to start learning from the most initial random state, reducing manual dependence and having good robustness and scalability.

[0054] Specifically, please refer to Figure 1 , a progressive learning framework diagram of this application is given, and the method may include the following steps:

[0055] S1, obtain the target cluster and initialize each individual in the target cluster; Figure 2 ,The target cluster includes multiple captured robots and one hunting robot, where the black solid arrows can be represented as captured robots and the white arrows can be represented as hunting robots.

[0056] Specifically, for each individual in the cluster, initialize its sampling heap D i (i=1,2,...,M), strategy network Q i , target network speed Location The maximum length of the sampling pile is 10,000, and it follows the first-in-first-out principle. For the tasks of "maintaining the number of surrounding individuals" and "aggregation and collision prevention", the policy network is a 4-layer neural network with 32 neurons in the middle layer and the activation function is ReLU; for the task of "avoiding capture", the policy network structure is as follows: Figure 3 ,The middle layer 1 contains two parts, each with 16 neurons, and the middle layers 2 and 3 contain 32 neurons each. The neighbor information and the captor information are input separately for processing.

[0057] S2, determines the current action data of each individual in the target cluster through a greedy strategy; the action includes the individual's linear acceleration and steering angle.

[0058] Specifically, for each individual, a greedy strategy is used to select the action taken at the current time t, for example:

[0059]

[0060] The action space consists of linear acceleration and steering angle.

[0061] S3, establish the dynamic model of the captured robot and the hunting robot; the dynamic model includes three sub-network structures, among which the first sub-network structure is used to obtain the number of individuals around the current individual, the second sub-network structure is used to obtain the aggregation and anti-collision structure, and the third sub-network structure is used to obtain the relative position result for avoiding capture.

[0062] In the embodiment of the present application, the dynamic model is used to update the environment and obtain the state and reward value s t+1 ,r t ←Environment update function (s t ,a t ), and the data Store into the sample stack.

[0063] For the first sub-network structure, i.e. the task of “maintaining the number of surrounding individuals”, its input state is the difference between its own speed direction and the surrounding individuals. Specifically:

[0064]

[0065] in R is its perception radius, is the number of individuals within the perception range; the reward function is

[0066] For the second sub-network structure, namely the "aggregation and anti-collision" task, an eight-dimensional input is added (Refer to Figure 4 ), the perception range is divided into eight directions. If an individual in the direction reaches the warning distance, its value is 1, otherwise it is 0; the reward function is where c g ,c c are constants, and in this embodiment, they are taken as 0.5, 1, and d collision For warning distance.

[0067] For the third sub-network structure, the "avoid capture" task, three-dimensional input is added again, which is the speed difference between the capturer and the self:

[0068]

[0069] The difference in position angle between the captor and the captor

[0070] The distance of the captor

[0071] Then the reward function is

[0072]

[0073] The constants c0, c1, c2 can be 1, 2, -3.

[0074] S4, traverse all individuals in the target cluster as current individuals to train the first sub-network structure, specifically using q-learning to update the policy network of each individual.

[0075] Specifically, for each individual, if its sampling pile capacity is larger than the unit batch (depending on computing power), random sampling is performed from it for training. The policy network of each individual is updated using q-learning. Where α is the learning rate, which can be 1e-3, and γ is the discount factor, which can be 0.99. They are the state, action, reward, state at the next moment, and action at the next moment in reinforcement learning, respectively.

[0076] S5, determining the network data of the surrounding individuals of the current individual, and matching the network data of the surrounding individuals with the current individual to perform cooperative learning.

[0077] Specifically, using network data of surrounding individuals for collaborative learning

[0078] λ is the trust coefficient of the surrounding individuals, which can be

[0079] S6, updates the probability parameters in the greedy strategy and starts training for the next cycle until the preset training cycle is reached.

[0080] Specifically, update the greedy strategy probability ε t+1 ←max(0.01,ε t -ε d ), in this embodiment, ε d 1e-6. Every 200 training cycles, the target network is reset using the parameters of the policy network.

[0081] S7, when the first sub-network converges after reaching the preset training cycle, the second sub-network structure and the third sub-network structure are trained in sequence through steps S2 to S5 to obtain the target dynamics model, which is used to obtain the cluster robot's avoidance and capture results.

[0082] In the embodiment of the present application, the task of "maintaining the number of surrounding individuals" is learned first, and after the model converges, the task of "aggregation and collision avoidance" is learned, and finally the task of "avoiding capture" is learned.

[0083] The following gives a size L x ×L y =7m×7m environment to verify the biomimetic, scalable and adaptive optional embodiment of the present invention. To simplify the experiment, the robot autonomous exploration rules are set, including (1) adopting periodic boundary conditions, that is, the environment is connected left and right, and up and down, to ensure that the robot can continuously learn in this fixed environment. (2) The robot has a maximum and minimum movement speed of 0.1m / s and 0.3m / s respectively, a maximum linear acceleration of 0.05m / s2, and a maximum rotation angle of π / 6rad / s.

[0084] In this task, the robots need to learn to avoid being caught while remaining clustered and avoiding collisions. Figure 5 , the figure shows the evolution of the overall orientation index as the training progresses under this task. It can be found that under this complex task, the robots still learned to keep the speed consistent, that is, they move in the same direction at the same speed. Figure 6 In the later stages of training, all robot individuals learned to maintain an appropriate distance from the predator while staying together. At the same time, when the robots sensed the predator, they would immediately change their orientation and avoid collision at the same time, thus achieving good learning of the given task.

[0085] The experimental results show that: with robot information as input, the hybrid system proposed in the present invention can achieve effective learning of avoiding capture. Under the learning goal of avoiding capture, the learned policy network can keep all individuals in the cluster in a clustered state without colliding with each other. When some individuals perceive the capturer, they will quickly adjust their movement direction, while those individuals who have not yet perceived the danger will follow their neighbors to adjust their own direction. This adjustment will quickly spread throughout the cluster, forming a phenomenon of synchronous movement of the entire cluster. In addition, cooperative learning can significantly accelerate the convergence speed of the model and further improve the program execution efficiency by sharing and absorbing the parameters of other policy networks. At the same time, thorough reinforcement learning is feasible and effective, and has achieved certain breakthroughs in the robot's autonomous capabilities. Therefore, the present invention implements a learning method for robot clusters to avoid capture in a biomimetic, robust and autonomous manner.

[0086] The present application also provides a reinforcement learning method system for swarm robots to avoid capture. The system includes:

[0087] The acquisition module is used to acquire the target cluster and initialize each individual in the target cluster; the target cluster includes multiple captured robots and one hunting robot;

[0088] The determination module is used to determine the current action data of each individual in the target cluster through a greedy strategy; the action includes the individual's linear acceleration and steering angle;

[0089] Establish a module for establishing the dynamic model of the captured robot and the hunting robot; the dynamic model includes three sub-network structures, of which the first sub-network structure is used to obtain the number of individuals around the current individual, the second sub-network structure is used to obtain the aggregation and anti-collision structure, and the third sub-network structure is used to obtain the relative position result of avoiding capture;

[0090] The training module is used to traverse all individuals in the target cluster as the current individuals to train the first sub-network structure, specifically using q-learning to update the policy network of each individual; determine the network data of the individuals surrounding the current individual, and match the network data of the surrounding individuals with the current individual for cooperative learning; update the probability parameters in the greedy strategy, and start the next cycle of training until the preset training cycle is reached; when the first sub-network converges after the preset training cycle, the second sub-network structure and the third sub-network structure are trained in turn to obtain the target dynamics model, which is used to obtain the cluster robot's avoidance and capture results.

[0091] The reinforcement learning method system for cluster robots to avoid capture provided in the embodiment of the present application is used to implement the above-mentioned reinforcement learning method for cluster robots to avoid capture. For the specific definition of the reinforcement learning method system for cluster robots to avoid capture, please refer to the definition of the reinforcement learning method for cluster robots to avoid capture in the above text, which will not be repeated here. The various parts of the above-mentioned reinforcement learning method system for cluster robots to avoid capture can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the device in the form of hardware, or can be stored in the memory in the device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0092] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A reinforcement learning method for swarm robots to avoid capture, characterized by: The method comprises: S1, obtaining a target cluster and initializing each individual in the target cluster; the target cluster includes multiple captured robots and one hunting robot; S2, determining the current moment action data of each individual in the target cluster through a greedy strategy; wherein the action includes the linear acceleration and steering angle of the individual; S3, establishing a dynamic model of the captured robot and the hunting robot; the dynamic model includes three sub-network structures, wherein the first sub-network structure is used to obtain the number of individuals around the current individual, the second sub-network structure is used to obtain the aggregation and anti-collision structure, and the third sub-network structure is used to obtain the relative position result of avoiding capture; S4, traversing all individuals in the target cluster as current individuals to train the first sub-network structure, specifically using q-learning to update the policy network of each individual; S5, determining network data of individuals surrounding the current individual, and matching the network data of the surrounding individuals with the current individual for collaborative learning; S6, updating the probability parameters in the greedy strategy and starting to execute the next cycle of training until the preset training cycle is reached; S7, when the first sub-network converges after reaching the preset training cycle, the second sub-network structure and the third sub-network structure are trained in sequence through steps S2 to S5 to obtain the target dynamics model, and the target dynamics model is used to obtain the cluster robot avoidance and capture results.

2. The method according to claim 1, characterized in that The first sub-network structure and the second sub-network structure specifically include a 4-layer neural network, the middle layer has 32 neurons, and the activation function is ReLU.

3. The method according to claim 1, characterized in that The third sub-network structure specifically includes three intermediate layer structures, wherein the first intermediate layer includes two parts, each with 16 neurons, and the second intermediate layer and the third intermediate layer each contain 32 neurons.

4. The method according to claim 1, wherein Training the first sub-network structure specifically includes: Input the current individual's own speed direction and the difference between the surrounding individuals, through the formula Get the number of individuals around the current individual, where Indicates the number of individuals around the current individual. Where R is the current individual perception radius, is the number of surrounding individuals within the current individual’s perception range, is the current individual speed, is the current individual position, t is the current moment, and through Get reward value.

5. The method according to claim 1, wherein Training the second sub-network structure specifically includes: exist Add eight-dimensional input based on The sensing range is divided into eight directions. If an individual in that direction reaches the warning distance, the corresponding The value is 1, otherwise it is 0; And through the reward function Get the reward value of training, where c g ,c c is a constant, d collision is the warning distance, R is the current individual perception radius, is the number of surrounding individuals within the current individual’s perception range, is the current individual position, and t is the current moment.

6. The method according to claim 1, characterized in that Training the third sub-network structure specifically includes: pass Determine the speed and direction difference between the hunting robot and the current individual; Determine the position angle difference between the hunting robot and the current individual, where pass Determine the distance of the hunting robot; The speed direction difference, position angle difference and distance between the hunting robot and the current individual are used as three-dimensional inputs. Get the reward value, where c0, c1, c2 are constants, and R is the current individual perception radius. is the number of surrounding individuals within the current individual’s perception range, is the current individual position, is the position of the hunting robot, and t is the current moment.

7. The method according to claim 1, characterized in that Update the probability parameters in the greedy strategy according to the formula: e t+1 ←max(0.01,e t -e d ) Determine the update probability parameter, where ε d is 1e-6.

8. The method according to claim 1, characterized in that The strategy network of each individual is updated using q-learning, specifically according to the formula: The updated policy network is obtained, where α is the learning rate, γ is the discount coefficient, and the sequence (s, a, r, s′, a′) is the state, action, reward, state at the next moment, and action at the next moment in reinforcement learning, respectively.

9. A reinforcement learning method system for swarm robots to avoid capture, characterized by: The system comprises: An acquisition module is used to acquire a target cluster and initialize each individual in the target cluster; the target cluster includes a plurality of captured robots and a hunting robot; A determination module, configured to determine the current moment action data of each individual in the target cluster using a greedy strategy; wherein the action includes the linear acceleration and steering angle of the individual; Establishing a module for establishing a dynamic model of the captured robot and the hunting robot; the dynamic model includes three sub-network structures, wherein the first sub-network structure is used to obtain the number of individuals around the current individual, the second sub-network structure is used to obtain the aggregation and anti-collision structure, and the third sub-network structure is used to obtain the relative position result of avoiding capture; The training module is used to traverse all individuals in the target cluster as current individuals to train the first sub-network structure, specifically using q-learning to update the policy network of each individual; determine the network data of the surrounding individuals of the current individual, and match the network data of the surrounding individuals with the current individual for cooperative learning; update the probability parameters in the greedy strategy, and start the training for the next cycle until the preset training cycle is reached; when the first sub-network converges after the preset training cycle is reached, the second sub-network structure and the third sub-network structure are trained in turn to obtain the target dynamics model, which is used to obtain the cluster robot's avoidance and capture results.

Images