A robot motion control method based on swarm intelligence learning

Abstract

This invention relates to a robot motion control method based on swarm intelligence learning. First, a robot simulation environment and reward rules are built according to requirements. Then, interactive training is conducted by imitating the behavioral strategies of mutual learning among individuals in a social group. Finally, through multiple iterations of learning, the robot achieves optimal motion control, outputting the optimal joint angles according to different environmental states, enabling the robot to obtain a smooth and optimal motion trajectory throughout the entire movement process. The three-mode strategy in this invention effectively balances the relationships between exploration and utilization, and global and local aspects during the iterative solution process. It has few adjustable hyperparameters, is easy to converge quickly, and can be computed in parallel using multi-threaded methods, significantly reducing computation time. Furthermore, it effectively avoids the problem of sparse rewards over long time spans, and can be widely applied to intelligent control in the field of robot motion control.

Description

Technical Field

[0001] This invention belongs to the field of robot motion control technology, specifically relating to a robot motion control method based on swarm intelligence learning. Background Technology

[0002] Traditional robot motion control methods require initial analysis and modeling of the robot and its environment, followed by posture estimation and trajectory planning using forward and inverse kinematics. However, due to the complexity and variability of models and environments, manually designed controllers cannot cover all situations. Therefore, applying deep reinforcement learning to robot motion control is a viable approach. Deep reinforcement learning, as a black-box optimization method, does not require robot modeling or kinematic analysis. It trains an optimal control strategy adapted to the current environment through interaction, enabling the robot to achieve optimal joint control outputs under the current environmental conditions. However, most reinforcement learning models consist of multiple neural networks, and backward gradient updates are required during computation, leading to time-consuming simulation and training. Furthermore, it is difficult to train excellent models over long time spans and with sparse rewards, causing the method to easily get trapped in local optima and fail to achieve optimal robot motion control. Therefore, to save training time and improve control accuracy, the current technical challenge is to design a computationally inexpensive, globally intelligent robot control method with few adjustable hyperparameters. Summary of the Invention

[0003] Technical problems to be solved

[0004] To avoid the shortcomings of existing technologies, this invention provides a robot motion control method based on swarm intelligence learning.

[0005] Technical solution

[0006] A robot control method based on swarm intelligence learning, characterized by comprising:

[0007] S1: Build a robot simulation environment and design learning reward rules;

[0008] S2: Based on swarm intelligence learning, interactive training is conducted by imitating the behavioral strategies of mutual learning among individuals in a social group. The swarm intelligence learning method is similar to the decision layer of a robot: First, the robot's joint movements and environmental state at the current moment are acquired and used as input. Then, the robot's joint angles for the next moment are output through learning decisions. At this time, the output joint angles will be executed in the environment to form a new action state for the robot. At the same time, the environment will output the next environmental state and the reward brought by the current robot action based on the current joint movements of the robot. Through the above interactive training, the training continues until the current training ends.

[0009] The swarm intelligence learning described includes a learning mode, an imitation mode, and a self-learning mode. In the learning mode, each individual in the swarm learns from the best member of the swarm. In the imitation mode, each individual in the swarm imitates the learning techniques of the best member of the swarm. In the self-learning mode, each individual in the swarm learns learning methods and problem-solving techniques on their own. Each individual in the swarm is an independent intelligent agent, composed of deep neural networks. Each individual in the swarm uses a parallel mode for multi-threaded computation.

[0010] The learning model described employs the Lévy flight stochastic search strategy and is approximated using the Mantegna method. Its mathematical expression is as follows:

[0011]

[0012]

[0013]

[0014]

[0015]

[0016] in, Both u and ν follow a pattern with expectation of 0 and variance of σ. 2 The normal distribution is given by β, which represents the relationship between flight distance and time, and Γ is the gamma function. i (t) represents the current network parameters, θ i (t+1) represents the updated network parameters, and α is the scaling factor for the step size. The operator represents point-to-point multiplication;

[0017] The aforementioned imitation model employs a random perturbation strategy, that is, in the historical best individual θ best The nearby perturbation, whose perturbation update expression is:

[0018]

[0019] Where rand represents a random function, Pr represents a probability value, and ind1 and ind2 represent random numbers between the total number of network parameters in each layer;

[0020] The self-learning model described above employs a normal distribution sampling strategy, that is, based on the historical best individual θ best The parameters are sampled normally, and the sampling update expression is:

[0021]

[0022] Where, θ i (t+1) follows a pattern with expectation μ and variance σ.2 The normal distribution;

[0023] S3: Through multiple iterative learning, the robot can achieve optimal decision-making, that is, output the optimal joint angle according to different environmental conditions, so that the robot can obtain a smooth and optimal gait throughout the movement process.

[0024] A further technical solution of the present invention: S3 multiple iterations specifically include:

[0025] Let n be the population size in the learning mode, p be the population size in the imitation mode, and q be the population size in the self-learning mode. The total number of individuals in the three modes is equal to the total population size.

[0026] Set the total number of iterations, the initial number of samples, and the average number of tests;

[0027] Under the conditions of satisfying the total number of iterations and the initial number of samplings, the network parameters of the initial population are sampled using a standard normal distribution θ ~ N(0,1);

[0028] Initialize the task environment and obtain the initial state s0, and use a continuously controlled stochastic strategy to obtain each state s. t The following action a t ;

[0029] An individual receives a reward value r in the current state by interacting with the environment. t and new state s t+1 This continues until the end of the round and the fitness value f = f + r is obtained;

[0030] The fitness values ​​of the population are sorted to obtain the network model parameters corresponding to the highest fitness score, which is the optimal value in the current iteration. The optimal value is then verified by averaging the results multiple times.

[0031] By averaging multiple tests, the average fitness value and the corresponding network model parameters are returned, along with the average number of tests num = num + 1 and the total number of iterations iter = iter + (n + p + q).

[0032] Under the conditions of satisfying the total number of iterations and reaching the initial number of samplings, the system enters the learning mode, imitation mode, and self-learning mode stages.

[0033] Find the network model with the highest historical average test fitness value, i.e., the historical optimum;

[0034] The three modes update network parameters according to their respective strategies;

[0035] Repeat the above process of interaction between the individual and the environment to obtain the action and reward value in each state until the end of the round, and obtain the fitness value f for the round.

[0036] A computer system is characterized by comprising: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described above.

[0037] A computer-readable storage medium is characterized by storing computer-executable instructions, which, when executed, are used to implement the above-described method.

[0038] Beneficial effects

[0039] The present invention provides a robot motion control method based on swarm intelligence learning. Compared with existing methods, the advantages of the present invention are as follows:

[0040] 1. This invention has strong search capabilities. By utilizing three strategies—learning mode, imitation mode, and self-learning mode—it can effectively balance the relationship between exploration and utilization, increase population diversity, effectively expand the global search range, and also effectively escape local optima, thus achieving optimal motion control of the robot.

[0041] 2. This invention has a fast solution speed. During the solution process, multi-threaded parallel computing can be used, and there is no need for reverse gradient updates, which saves computing resources and reduces computing time by a factor of two, thereby improving the efficiency of robot simulation training.

[0042] 3. This invention has few adjustable hyperparameters, and very few hyperparameters need to be adjusted during the update and iteration process, making it easy to converge quickly through iteration and reducing the difficulty of robot simulation training.

[0043] 4. This invention has high reward utilization. It adopts a round-based reward update mechanism, which can effectively avoid the problem of sparse rewards when the time span is long, and improve the utilization rate of the robot simulation training process. Attached Figure Description

[0044] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0045] Figure 1 This is a schematic diagram of the present invention based on the swarm intelligence learning method;

[0046] Figure 2 This is a flowchart illustrating the implementation of the robot motion control of the present invention;

[0047] Figure 3 This is an interactive process diagram. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0049] This invention provides a robot motion control method based on swarm intelligence learning. First, a robot simulation environment and reward rules are built according to requirements. Then, interactive training is conducted by imitating the behavioral strategies of mutual learning among individuals in a social group. Finally, through multiple iterative learning processes, the robot achieves optimal motion control, outputting the optimal joint angles based on different environmental states, thus enabling the robot to obtain a smooth and optimal motion trajectory throughout the entire movement process. The swarm intelligence learning-based control method is a robot intelligent learning method with low computational requirements and few adjustable hyperparameters, enabling the robot to achieve optimal motion control.

[0050] The swarm intelligence learning described is a global black-box intelligent control method applicable to various types of robot motion control. It builds different simulation environments and designs different reward rules for different robots, and through interactive training and learning, the robot finally obtains the optimal motion trajectory. In swarm intelligence learning, each individual in the group is an independent intelligent agent, composed of deep neural networks. Each individual uses a parallel mode for multi-threaded computation.

[0051] The aforementioned swarm intelligence learning encompasses three knowledge acquisition pathways: learning mode, imitation mode, and self-learning mode. In the learning mode, each individual in the group learns from the most capable member, similar to most students in a class learning from the most knowledgeable teacher, who imparts existing knowledge. In the imitation mode, each individual imitates the learning techniques of the most capable member, similar to some students in a class imitating the teacher's problem-solving techniques, allowing them to quickly arrive at the correct answer. In the self-learning mode, each individual randomly discovers learning methods and problem-solving techniques, similar to a small number of students in a class not following the teacher's instructions but learning independently using their own developed methods and approaches. The learning mode involves the largest number of individuals, followed by the imitation mode, and then the self-learning mode, aligning with the distribution and learning patterns within human societies.

[0052] The learning model described above employs the Lévy flight random search strategy, the mathematical expression of which is:

[0053] Lévy~u=t -λ ,1<λ≤3 (1)

[0054] The Levi flight described consists of short steps over a longer period and long steps over a shorter period, which can increase the diversity of the population, effectively expand the search range, and make it easier to escape local optima.

[0055] The Lévy flight distribution described is very complex and difficult to realize, so the Mantegna method is usually used for approximate simulation, and its mathematical expression is:

[0056]

[0057]

[0058]

[0059] in, β = 1.5, and Γ is the gamma function.

[0060] To better update the parameters of the neural network, the expression for updating individual parameters in its learning mode is as follows:

[0061]

[0062] Where, θ i (t) represents the current parameter, θ i (t+1) represents the updated parameters, and α is the scaling factor for the step size. The operator represents point-to-point multiplication.

[0063] The α step size scaling factor determines the scale of the search. If the value of α is large, although the global search capability is enhanced, it is difficult to obtain a high-precision solution. If the value of α is small, although the local search capability is enhanced, the method requires more iterations, and the efficiency decreases accordingly.

[0064] Furthermore, a dynamic search Lévy flight mechanism is proposed, where the step size scaling factor α is changed from a fixed value to a dynamic step size factor that changes with the number of iterations. The step size factor α is defined as follows:

[0065]

[0066] Where, α max This represents the maximum value of α. min Let α represent the minimum value, step represent the total number of iterations, and max_step represent the total number of iterations.

[0067] The dynamic step size scaling factor α has a relatively large value and a slow decay rate in the early stages of iteration, resulting in strong global search capability and helping to avoid the population getting trapped in local optima. In the later stages of iteration, α decays rapidly to a smaller value, increasing local search capability and helping to accelerate population convergence.

[0068] The aforementioned imitation model employs a random perturbation strategy, that is, in the historical best individual θ best The nearby perturbation, whose perturbation update expression is:

[0069]

[0070] The perturbation range of the perturbation strategy used in the imitation mode is [-1, 1].

[0071] The perturbation strategy employed in the aforementioned imitation model has a 50% probability of reversing the historical best individual θ. best Perturb all parameters, and there is a 50% probability that the historical best individual θ will be affected. best The random parameters are perturbed.

[0072] The self-learning model described above employs a normal distribution sampling strategy, that is, based on the historical best individual θ best The parameters are sampled normally, and the sampling update expression is:

[0073]

[0074] The self-learning mode adopts a normal strategy independent of each layer of the network. That is, the mean of each layer distribution is the mean of the network parameters of the current layer in the historical best individual, and the variance of each layer distribution is the variance of the network parameters of the current layer in the historical best individual.

[0075] To enable those skilled in the art to better understand the present invention, the present invention will be described in detail below with reference to specific embodiments.

[0076] See Figures 1 to 2 A robot motion control method based on swarm intelligence learning mainly consists of a learning mode, an imitation mode, and a self-learning mode, all of which are related to the current best performer. The best performer is the one with the highest current knowledge, i.e., the historical optimal solution. In the learning mode, individuals learn from the best performer, resulting in the closest relationship and the largest number of individuals. In the imitation mode, individuals imitate the best performer, resulting in the next closest relationship and the next smallest number of individuals. In the self-learning mode, individuals learn from the best performer while also learning from each other, resulting in the least close relationship and the smallest number of individuals. Each individual in the swarm is an independent intelligent agent, composed of a deep neural network. Various network types are available; this embodiment uses a fully connected deep neural network. After fixing the network structure, the parameters that need to be updated include the weights and biases of each layer.

[0077] The specific technical process is as follows:

[0078] Build a robot simulation environment and design learning reward rules.

[0079] Initialize the structure and activation functions of the neural network.

[0080] Let n be the population size in learning mode, p be the population size in imitation mode, and q be the population size in self-learning mode. The total number of individuals in the three modes is equal to the total population size.

[0081] Set the total number of iterations, the initial number of samples, and the average number of tests.

[0082] Under the conditions of satisfying the total number of iterations and the number of initial samplings, the network parameters of the initial population are sampled using a standard normal distribution θ ~ N(0,1). The purpose of the initial sampling is to obtain more initial population samples.

[0083] Initialize the task environment and obtain the initial state s0, and use a continuously controlled stochastic strategy to obtain each state s. t The following action a t .

[0084] An individual receives a reward value r in the current state by interacting with the environment. t and new state s t+1 And obtain the fitness value f = f + r, until the end of the round.

[0085] The fitness values ​​of the population are sorted to obtain the network model parameters corresponding to the highest fitness score, which is the optimal value in the current iteration. These parameters are then subjected to multiple averaging tests for validation. The purpose of these multiple averaging tests is to verify the robustness of the model parameters, ensuring that they are not easily affected by the randomness of the testing environment.

[0086] By averaging multiple tests, the system returns the average fitness value and the corresponding network model parameters, as well as the average number of tests num = num + 1 and the total number of iterations iter = iter + (n + p + q).

[0087] Once the total number of iterations and the initial number of samples are met, the method enters the swarm intelligence learning phase.

[0088] Find the network model with the highest historical average test fitness value, i.e., the historical optimum.

[0089] In learning mode, network parameters are updated as shown in Equations 2-6, and the dynamic step size scaling factor α... max Set to 0.1, α min Set to 0.01.

[0090] In imitation learning, the network parameters are updated as shown in Equation 7.

[0091] In self-learning mode, update network parameters as shown in Equation 8.

[0092] Repeat the above process of interaction between individuals in the population and the environment to obtain the actions and reward values ​​in each state until the end of the round, and obtain the fitness value f for the round.

[0093] The calculation is repeated until the total number of iterations is met, and the model corresponding to the historical best value is used as the final controller for this task.

[0094] Using the final controller as the robot's output control strategy means that the robot can output the optimal joint angles based on the current environment and its own state, making the robot more stable and efficient throughout the entire movement process.

[0095] like Figure 3 The diagram illustrates the interactive training process. During learning, the individual continuously interacts with the environment. After acquiring a state within the environment, the individual uses that state to output an action. This action is then executed within the environment, which, based on the individual's action, outputs the next state and the reward derived from that action. The purpose of repeated training is to enable the individual to obtain as many rewards as possible from the environment.

[0096] To verify the effectiveness of the proposed method, the swarm intelligence learning method was compared and evaluated with current reinforcement learning methods such as DDPG (Deep Deterministic Policy Gradient), SAC (Soft Actor-Critic), and PPO (Proximal Policy Optimization). The test tasks were six consecutive robot motion control tasks in the Mujoco simulation environment. The comparison results are shown in Table 1. The reward value in the table is the steady-state mean within the simulation steps; a larger reward value indicates better robot motion control and smoother trajectory output. The comparison results show that the swarm intelligence learning method proposed in this invention performs best on Swimmer and Reacher robots, exhibiting the best control effect, and its performance on other robot motion control tasks is not significantly worse. Meanwhile, the swarm intelligence learning method proposed in this invention can save computational resources and significantly reduce computation time during simulation calculations. For example, under the same hardware conditions, the swarm intelligence learning method takes approximately 25 minutes to reach two million steps in the Swimmer robot simulation example, while DDPG takes nearly 6 hours, PPO takes nearly 1 hour, and SAC takes nearly 12 hours. Through the above discussion and analysis, it can be demonstrated that the swarm intelligence learning method proposed in this invention has its own characteristics and advantages, and possesses certain research value and promotional significance.

[0097] Table 1. Calculation results of the test task under different methods

[0098]

[0099] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in the present invention, and such modifications or substitutions should all be covered within the scope of protection of the present invention.

Claims

1. A robot control method based on swarm intelligence learning, characterized in that... include: S1: Build a robot simulation environment and design learning reward rules; S2: Based on swarm intelligence learning, interactive training is conducted by imitating the behavioral strategies of mutual learning among individuals in a social group. The swarm intelligence learning method is similar to the decision layer of a robot: First, the robot's joint movements and environmental state at the current moment are acquired and used as input. Then, the robot's joint angles for the next moment are output through learning decisions. At this time, the output joint angles will be executed in the environment to form a new action state for the robot. At the same time, the environment will output the next environmental state and the reward brought by the current robot action based on the current joint movements of the robot. Through the above interactive training, the training continues until the current training ends. The swarm intelligence learning described includes a learning mode, an imitation mode, and a self-learning mode. In the learning mode, each individual in the swarm learns from the best member of the swarm. In the imitation mode, each individual in the swarm imitates the learning techniques of the best member of the swarm. In the self-learning mode, each individual in the swarm learns learning methods and problem-solving techniques on their own. Each individual in the swarm is an independent intelligent agent, composed of deep neural networks. Each individual in the swarm uses a parallel mode for multi-threaded computation. The learning model described employs the Lévy flight stochastic search strategy and is approximated using the Mantegna method. Its mathematical expression is as follows: s v =1.1≤β≤3 in, Both u and ν follow a function with expectation 0 and variance σ. 2 The normal distribution is given by β, which represents the relationship between flight distance and time, and Γ is the gamma function. i (t) represents the current network parameters, θ i (t+1) represents the updated network parameters, and α is the scaling factor for the step size. The operator represents point-to-point multiplication; The aforementioned imitation model employs a random perturbation strategy, that is, in the historical best individual θ best The perturbation near the object is updated by the following expression: Where rand represents a random function, Pr represents a probability value, and ind1 and ind2 represent random numbers between the total number of network parameters in each layer; The self-learning model described above employs a normal distribution sampling strategy, that is, based on the historical best individual θ best The parameters are sampled normally, and the sampling update expression is: Where, θ i (t+1) follows a pattern with expectation μ and variance σ. 2 The normal distribution; S3: Through multiple iterative learning, the robot can achieve optimal decision-making, that is, output the optimal joint angle according to different environmental conditions, so that the robot can obtain a smooth and optimal gait throughout the movement process.

2. The robot control method based on swarm intelligence learning according to claim 1, characterized in that... The S3 iterations are as follows: Let n be the population size in the learning mode, p be the population size in the imitation mode, and q be the population size in the self-learning mode. The total number of individuals in the three modes is equal to the total population size. Set the total number of iterations, the initial number of samples, and the average number of tests; Under the conditions of satisfying the total number of iterations and the initial number of samplings, the network parameters of the initial population are sampled using a standard normal distribution θ ~ N(0,1); Initialize the task environment and obtain the initial state s0, and use a continuously controlled stochastic strategy to obtain each state s. t The following action a t ; An individual receives a reward value r in the current state by interacting with the environment. t and new state s t+1 This continues until the end of the round and the fitness value f = f + r is obtained; The fitness values ​​of the population are sorted to obtain the network model parameters corresponding to the highest fitness score, which is the optimal value in the current iteration. The optimal value is then verified by averaging the results multiple times. By averaging multiple tests, the average fitness value and the corresponding network model parameters are returned, along with the average number of tests num = num + 1 and the total number of iterations iter = iter + (n + p + q). Under the conditions of satisfying the total number of iterations and reaching the initial number of samples, it enters the learning mode, imitation mode, and self-learning mode stages; Find the network model with the highest historical average test fitness value, i.e., the historical optimum; The three modes update network parameters according to their respective strategies; Repeat the above process of interaction between the individual and the environment to obtain the action and reward value in each state until the end of the round, and obtain the fitness value f for the round.

3. A computer system, characterized in that... include: One or more processors, a computer-readable storage medium for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of claim 1.

4. A computer-readable storage medium, characterized in that... The device stores computer-executable instructions, which, when executed, are used to implement the method of claim 1.

Images