VEX robot path planning method based on transfer reinforcement learning

Abstract

The robot needs to have the ability of movement control and path planning when performing tasks, and reasonable path planning can reduce time cost and improve work efficiency. In view of the problem that the experience learned by traditional path planning is difficult to transfer between different tasks, the application provides a VEX robot path planning method based on transfer reinforcement learning. Firstly, the current competition field is divided by using a grid method, and the agent uses reinforcement learning to design the optimal path of a single competition to obtain learning experience. Then, the agent uses transfer reinforcement learning to reuse the prior experience of the source task and migrate it to the next competition, thereby shortening the path search time of the new field. The simulation results under the "alliance" and "one touch" competitions show that, compared with the traditional A* algorithm and Dijkstra algorithm, the method can reduce the model training time while improving the accuracy of path planning, and effectively solves the experience sharing problem between high correlation path planning tasks.

Description

TECHNICAL FIELD

[0001] The application is an application method of AI technology in VEX robot competition, in particular to a VEX robot path planning method based on transfer reinforcement learning. BACKGROUND

[0002] The era trend of informatization and intelligentization promotes the rapid development of robot technology. Robots are widely used in various industries, agriculture, medical and home services, and important fields such as military and scientific research. At present, robot technology is listed as a key high-tech by many countries, and the related industry is an important symbol to measure the strength of the comprehensive national power. In order to actively adapt to the situation of international new technology development and create a new intelligent industrial mode, the Ministry of Education actively promotes the construction of new engineering since February 2017. The goal is to cultivate a batch of innovative engineering and technological talents and enhance the international competitiveness of high-tech. [2] Especially with the continuous popularization and application of robots, a large number of robot technology engineers engaged in research and practice of complex and diverse application scenarios are in urgent need. Robot competition has become a main means and important breakthrough for cultivating innovative talents due to its advantages of frontier, comprehensiveness and practicality.

[0003] VEX is a robot project jointly supported by the United States Space Agency, American EY insurance company, Asian robot alliance Chevron, Texas Instruments, Northrop Grumman Corporation and other American companies. The competition consists of fifteen seconds of autonomous time and one minute and forty-five seconds of operator control time. The goal of the competition is to move the scoring objects to the designated area by the robot to achieve a higher score than the opponent team. In order to obtain the highest possible score in the automatic program stage, it is particularly important to design a reasonable optimal path planning of the robot.

[0004] The goal of robot path planning is to quickly plan a collision-free optimal path in a space with obstacles, so that the robot can smoothly and stably reach the target from the starting point. Path planning algorithms include traditional A* algorithm [4], Dijkstra algorithm [5], artificial potential field method [6] and evolutionary algorithms such as genetic algorithm [7], particle swarm optimization [8], ant colony algorithm [9] and so on. The traditional algorithms need to accurately model the complex planning space and obstacles, and the more complex the environment, the lower the planning efficiency. Evolutionary algorithms have strong learning ability, but poor real-time performance, large amount of calculation and easy to fall into local optimal solution. In view of the problems existing in traditional algorithms, researchers use reinforcement learning (RL) to guide the robot to interact with the environment in unknown environment state, and find a more accurate moving direction through trial and error. Reinforcement learning does not require prior knowledge, selects actions according to the perceived environment information, and judges the value of actions combined with rewards to guide the robot to correctly perform the designed task, so it is very suitable for autonomous learning of robot path planning. Literature

[10] proposes a global guidance reinforcement learning method (G2RL), which does not need to call reinforcement learning to find an alternative path when the robot encounters obstacles, but uses the learned global experience to solve the robot path planning problem in a distributed response manner. Experimental results show that G2RL has good path optimization performance under different map types and obstacle densities. Literature

[11] proposes an energy-aware coverage path planning (CTPP) framework, which designs the action space and reward function of reinforcement learning according to the movement of the robot (including transformation, translation and orientation action) and the required energy, and obtains the optimal path from the source location to the target location with the least energy and time through model training. Considering the optimal path containing static and dynamic obstacles, literature

[12] proposes an action selection strategy combining the ε-greedy strategy with simulated annealing cooling scheduling control, and combines the heuristic reward function to solve the exploration-exploitation problem in unknown environment. At the same time, the global search performance and learning efficiency of collision-free path planning are improved in the actual robot operating system (ROS).

[0005] The above literatures design various path planning methods based on reinforcement learning in a single scenario. It is noted that the competition rules and venue settings in the same season of VEX robot competition are consistent, and the competition rules and venue settings in different seasons are similar. In order to improve the path planning efficiency of the automatic program stage in various competition systems, programmers in the team need to reuse learning experience and transfer it between different competitions. Transfer learning (TL) uses prior learned knowledge and skills to train the model to achieve fast solution of new tasks in the case of small sample, while improving the robustness and generalization of the model. Literature

[13] designs a dynamic near-optimal policy optimization method based on covariance matrix adaptive evolution strategy. In the training phase, the robot learns a strategy that can avoid obstacles; in the test phase, transfer learning is used to transfer the training parameters to the new environment as the initial parameters to explore the optimal path, greatly reducing the calculation time required to train the deep network. By introducing transfer learning into reinforcement learning, researchers have obtained a new deep learning model-transfer reinforcement learning (TRL)

[14]

[15] , which can reduce the training time of reinforcement learning in multiple related scenarios. In the actual scenario of Open AI gym, literature

[16] tries to apply reinforcement learning to generate the optimal evacuation path in the case of fire spreading. First, the trained deep Q-learning (DeepQ-Learning, DQL) model is tested on a 91-room building body, and then the learning experience is transferred to other large and complex building bodies using transfer learning, so that DQL can speed up the exploration of the optimal evacuation path in a larger operation space. Literature

[17] uses transfer learning to design a robot fast path planning method that adapts to dynamic environments. First, pre-train the Q network in a static environment; then use the trained parameters as the initial parameters of the Q network in the new environment, and through parameter fine-tuning, the training convergence time is shortened from 1033 rounds to 450 rounds, while ensuring the accuracy of path planning and speeding up the training speed. Literature

[18] proposes a transfer learning algorithm based on improved Q-learning to solve the robot path planning problem. The Kantorovich metric based on the action set, state transition function and reward function is used to effectively estimate the similarity between two application scenarios Markov decision process (MDP), thereby realizing the transfer of improved Q-learning between different scenarios, so that the agent can explore the obstacle-free path faster than traditional Q-learning. SUMMARY

[0006] In view of the problems of low path planning efficiency of a traditional path planning method and incapability of sharing path planning experience in similar scenarios, the application provides a VEX robot path planning method based on transfer reinforcement learning.

[0007] Firstly, in a grid map environment, an agent explores an optimal path of a robot in a single game of "alliance connection and horizontal connection" and "one touch" through continuous interaction with an environment by using reinforcement learning.

[0008] Then, the agent transfers learning experience of a previous game to a next game of the same game system or a next game of a different game system by using transfer learning, and then completes an action selection strategy according to a current environment to efficiently find the optimal path of the robot.

[0009] Finally, experimental results show that the path planning time of the transfer reinforcement learning in the games of "alliance connection and horizontal connection" and "one touch" is reduced by 66.5%, 16.5% and 52%, 19.4% respectively compared with A* algorithm and Dijkstra algorithm. The application reduces invalid exploration of the agent in highly related games, reduces model training time, improves path planning accuracy, and effectively solves the experience transfer problem between highly related path planning tasks. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a top view of a "alliance connection and horizontal connection" game layout;

[0011] Figure 2 is a top view of a "one touch" game layout;

[0012] Figure 3 is a grid diagram of the "alliance connection and horizontal connection" game;

[0013] Figure 4 is a grid diagram of the "one touch" game;

[0014] Figure 5 is an illustration of eight moving directions of a robot;

[0015] Figure 6 is an illustration of experience transfer between two games of "alliance connection and horizontal connection";

[0016] Figure 7 is an illustration of experience transfer between two games of "alliance connection and horizontal connection" and "one touch";

[0017] Figure 8 is a convergence curve diagram of Q-learning in "alliance connection and horizontal connection";

[0018] Figures 9(a) to 9(c)are the optimal path maps of Q-learning under "alliances" respectively corresponding to Experiment 1-Experiment 3;

[0019] Figure 10 are the convergence curves of transfer reinforcement learning under "alliances";

[0020] Figures 11(a) to 11(c) are the optimal path maps of transfer reinforcement learning under "alliances" respectively corresponding to Experiment 4-Experiment 6;

[0021] Figure 12 is the convergence curve of transfer reinforcement learning from "alliances" to "touch";

[0022] Figure 13 is the optimal path map of transfer reinforcement learning from "alliances" to "touch";

[0023] Figures 14(a) to 14(c) are the optimal path maps of three methods respectively corresponding to the starting grid coordinate (0, 2) under "alliances";

[0024] Figures 15(a) to 15(c) are the optimal path maps of three methods respectively corresponding to the starting grid coordinate (0, 3) under "alliances";

[0025] Figures 16(a) to 16(c) are the optimal path maps of three methods respectively corresponding to the starting grid coordinate (0, 1) under "touch". DETAILED DESCRIPTION

[0026] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. 1 SUMMARY

[0028] The present application uses transfer reinforcement learning to explore the optimal path of a robot in the VEX automatic competition stage. First, the competition field is divided by using a grid method, and a target optimization model for robot path planning is established. Then, reinforcement learning is used to solve the optimal path of a robot in a single competition. Then, according to the learning experience of the last competition, transfer reinforcement learning is used to solve the optimal path of a robot in the next competition. Finally, simulation experiments are conducted on "alliances" and "touch" competitions to evaluate the performance of path planning based on transfer reinforcement learning.

[0029] This method shows the process of quickly and efficiently exploring the optimal path of a robot in VEX competition. Compared with traditional A* algorithm and Dijkstra algorithm, this method reduces the model training time while improving the accuracy of path planning, effectively solving the experience sharing problem between high correlation path planning tasks.

[0030] 2 Modeling of robot path planning

[0031] 2.1 Description of competition task

[0032] VEX robot competition is held on a 12ft x 12ft square field, and each team of two robots of red and blue alliance compete with each other in a match including a 15-second autonomous phase and a 45-second teleoperated phase. In the autonomous phase, the programmers of each team need to design a path for the robot to avoid obstacles and reach the goal as fast as possible to complete the task. VEX competition is a robot competition including various game modes, such as “Cross the Line” and “Skystone”.

[0033] As shown in Figure 1 , the field of “Cross the Line” has 9 vertical and horizontal pegs and 16 balls of red and blue alliance respectively. Before the start of the game, each team of red and blue alliance uses 1 red ball and 1 blue ball as pre-loaded. During the game, each team puts the corresponding color ball into the vertical and horizontal pegs to score. For example, the robot of the red team carries the pre-loaded ball from any coordinate of the starting area to the southeast corner of the vertical and horizontal peg A, and scores after putting the ball into the vertical and horizontal peg. At the end of the autonomous phase, any team that achieves a cross in its vertical and horizontal can obtain an automatic winning point, and the team with the highest score will obtain an automatic phase bonus point.

[0034] As shown in Figure 2 , the field of “Skystone” has 3 neutral ring towers and 60 curved rings, and 6 pre-loaded curved rings, 2 team ring towers and 1 balance bridge of each team of red and blue alliance. Any team can put the curved ring into the ring tower, move the ring tower into its own area and climb the balance bridge during the game, and score at the end of the game. For example, the robot of the red team can directly move to the curved ring position of the ring tower A and put the pre-loaded curved ring into the ring tower; or it can first move to coordinate a, and then directly move the ring tower B into its own area after the opponent puts the curved ring into the ring tower B. At the end of the game, any team that achieves removal of the AWP line and at least one curved ring in the ring tower of the two teams can obtain an automatic winning point, and the team with the highest score will obtain an automatic phase bonus point.

[0035] As can be seen from the above, the path planning of the robot in the VEX competition is the key to scoring in each link of the game, especially in the autonomous phase. The programmers in the team need to design a feasible and efficient path planning according to the field layout of Figure 1 and Figure 2 to find a collision-free path with the shortest length for the robot from the starting point to the target point.

[0036] 2.2 Path planning model

[0037] In order to establish a model environment for robot path planning, the VEX competition field is divided into a two-dimensional grid, and the grid attributes are defined as the movement of the robot and the information of the obstacles. Figure 3 and Figure 4The "coalition" and "one-touch" fields are divided into 9x9 grids, each with a width of l = 4 / 3 feet, where black grids represent vertical and horizontal poles, ring towers, or balance bridges, and white grids represent movable areas without obstacles.

[0038] The robot has several moving directions on each grid, but in order to avoid high complexity models, eight representative moving directions are considered: forward, backward, left, right, right front, right back, left back, and left front, as shown in Figure 5 The robot moves from the initial grid coordinates (x origin ,y origin ) to the target grid coordinates (x target ,y target ). If the moving speed is v, the moving time to reach the current grid coordinates (x, y) can be represented as

[0039]

[0040] where n1 and n2 represent the total number of grids moved by the robot in the forward, backward, left, and right directions, and the right front, right back, left back, and left front directions, respectively, and n = n1 + n2 represents the total number of grids moved by the robot in the eight directions. In addition, corresponding to the eight moving directions, the forward direction is defined as the zero-degree direction, the clockwise direction is positive, and the counterclockwise direction is negative, so the set of robot rotation angles θ can be represented as

[0041] Combining the moving direction and the rotation angle, the path planning of the robot in the automatic race stage can be modeled as the following objective optimization problem

[0042]

[0043] s.t.t n ≤ 15s (2a)

[0044] θ ∈ Θ (2b)

[0045] where constraint (2a) represents that the robot moving time does not exceed 15s, and constraint (2b) represents that the robot has eight moving directions. Next, we first use reinforcement learning to solve problem P1 to find the optimal path of the robot in a single field competition.

[0046] 2 Single Field Competition Path Planning Based on Reinforcement Learning

[0047] 2.1 Q-learning Principle

[0048] Q-learning is a model-free reinforcement learning algorithm proposed by Watkins in 1989, which regards the interaction between agent and environment as a Markov decision process, whose model can be represented as a four-tuple <S, A, P, R>, where S = {s1, s2,...} denotes the set of environment states, A = {a1, a2,...} denotes the set of executable actions, P denotes the state transition probability, and R denotes the immediate reward after executing action A. n} represents the set of environment states, A = {a1, a2,... n} represents the set of executable actions, P represents the state transition probability, and R represents the immediate reward after executing action A.

[0049] In each round of learning, the agent perceives the environment state and selects an action according to the current policy π, which is then applied to the environment. The environment state changes accordingly, and a reward value r = (s, a) is generated and fed back to the agent. The agent updates the policy based on the reward and the current environment state. Through continuous learning, the agent eventually finds the optimal policy π π (s) that maximizes the expected long-term cumulative reward V * (s).

[0050]

[0051] where γ ∈ (0, 1) is the discount factor. According to the Bellman optimality criterion, the maximum value of equation (3) is

[0052]

[0053] where R(s, a) is the mathematical expectation of r(s, a), and P s,s′ (a) is the probability of state s transitioning to state s' under action a.

[0054] Q-learning can find the optimal policy π * that satisfies equation (4) through simple Q-value iteration when R(s, a) and P s,s′ (a) are unknown. Each state-action pair (s, a) corresponding to policy π is associated with a Q-value

[0055]

[0056] The update formula for Q-value in equation (5) is

[0057]

[0058] where the larger the learning rate α (0 < α < 1) is, the faster the algorithm converges. As the number of iterations increases, Q(s, a) will converge to the optimal value Q * (s, a) with probability 1, and the optimal policy π

[0059]

[0060] 2.2 Path planning based on reinforcement learning

[0061] Each match of VEX competition requires the programmers in the team to design the robot's moving path in the autonomous phase. For the path planning of a single match, the programmers use Q-learning to explore the optimal path of the robot. Specifically, the quadruple <S, A, P, R> can be described as follows:

[0062] (1) The external environment state S = {1,..., 81} represents the 81 grid sequences divided in the competition venue.

[0063] (2) The action set A = {1,..., 8} represents the 8 moving direction sequences of the robot, i.e., forward, backward, left, right, right front, right back, left back, and left front.

[0064] (3) The reward function r = r(s, a) is the maximum benefit of the system in state s e S choosing action a e A, i.e.,

[0065]

[0066] where Dis represents the Euclidean distance between the current coordinates of the robot and the target coordinates. The closer the robot is to the target grid, the smaller the negative reward it gets for advancing one grid. The negative reward obtained when encountering an obstacle will prompt the agent to make better movement decisions and help the robot reach the target grid faster.

[0067] According to the above problem description, the path planning method based on Q-learning is designed as follows:

[0068] Step 1 Initialization: the grid positions of the vertical and horizontal pegs, the ring tower, the curved ring, and the balance bridge, the coordinates of the starting grid and the target grid of the robot, the Q table and the R table;

[0069] Step 2 Action selection: the agent uses the e-greedy strategy to select the moving direction of the robot in the action set A, i.e., with a probability e e [0, 1] to select the action with the maximum Q value in state s, and with a probability of 1-e to randomly select an action;

[0070] Step 3 Update Q value: combine the recorded state-action (s, a) and reward value r, and update Q(s, a) according to formula (6);

[0071] Step 4 Iterative training: repeat steps Step2-Step3 until the Q table converges;

[0072] Step 5 Path solving: the agent looks up the optimal path of the robot in the current match on the trained Q table, and sends moving instructions to the main control board to guide the robot to reach the target grid at the fastest speed.

[0073] 3 Two-game path planning based on transfer reinforcement learning

[0074] 3.1 Principle of transfer reinforcement learning

[0075] Generally, the practical scene using AI technology lacks development experience in the early application stage, while transfer learning can introduce a mature AI model trained in another application scene to help improve the performance of the local AI model

[19] . Specifically, given the source domain D s and learning task T s , the target domain D t and the learning task T t , transfer learning helps improve the learning ability of T s in D s by acquiring the knowledge of D t and T t , where D s ≠ D t , T s ≠ T t . Obviously, transfer learning applies the learning experience of the source task to the target task, making the training of the target task more efficient.

[0076] In the initial learning stage of the new task, the exploration of reinforcement learning is blind. The agent converges to the optimal solution only after continuous training through interaction with the environment. If the new task has strong correlation with the source task, the exploration actions of the agent will be repeated, greatly increasing the computational cost. In order to avoid the blind exploration of the agent in the new environment, transfer learning is introduced into reinforcement learning to form transfer reinforcement learning

[20] . The agent reuses the learning knowledge of the source task as prior knowledge to accelerate the model training of the new task and reduce the difficulty of global optimization.

[0077] 3.2 Path planning based on transfer reinforcement learning

[0078] Each season of the VEX competition is divided into provincial, national, and world championships at different levels. Considering the actual situation of opponent replacement, rule changes, and other factors, programmers need to adjust the robot path strategy for the next game after the last game. The environment and goals of the two games are highly similar, which meets the application conditions of transfer reinforcement learning. Therefore, the path planning of the last game is defined as the source task, and the path planning of the next game is defined as the target task. Without training the Q table from scratch, programmers reuse the pathfinding experience of the last game to find the optimal path for the next game while saving a lot of training cost.

[0079] Figure 6The migration of path planning experience between two games in the "allied horizontal and vertical" is given, that is, the knowledge migration between games of the same category. When the opponent is replaced so that the red team or the blue team needs to replace the starting grid of the robot, the agent first reads the Q table and R table of the last game, and then re-trains the Q table according to the initial state of the next game to obtain a new optimal path.

[0080] The "allied horizontal and vertical" and the "one touch" game environment and robot task also have high similarity, for example, the game time and the field size are consistent, and the task of moving the scoring object to the target grid by avoiding obstacles is consistent. Therefore, programmers can use transfer reinforcement learning to perform knowledge migration between games of different categories, as shown in Figure 7 After modifying the R table of the "one touch" field environment, the agent trains the model on the Q table of the "allied horizontal and vertical", and quickly explores a new optimal path under the guidance of prior knowledge.

[0081] 4 Simulation experiment and result analysis

[0082] Next, the VEX robot path planning method based on transfer reinforcement learning of the present application is simulated. By comparing A* algorithm and Dijkstra algorithm, the performance of robot path planning in VEX competition is analyzed and evaluated to verify the feasibility and effectiveness of the present application.

[0083] 4.1 Performance analysis of reinforcement learning path planning

[0084] The present application takes the "allied horizontal and vertical" as an example to show the robot path exploration process based on Q-learning, wherein the learning rate α is set to 0.01, the discount factor γ is set to 0.9, and the maximum number of iterations is set to 2000 in the model training process.

[0085] The present application designs three groups of experiments: the starting grid coordinates are set to (0, 5), (0, 6), and (0, 7) respectively, and the corresponding target grid coordinates are all (8, 8); all of which are trained for 2000 times. As shown in Figure 8 With the gradual deepening of the agent's understanding of the environment, the learning experience plays a guiding role, making the reward value gradually increase, and finally converges after 862, 778, and 748 iterations respectively. Correspondingly, Fig. 9 shows the optimal path of the robot in experiments 1-3 after iteration convergence. It is easy to see that the optimal path of different starting grids can be successfully solved by using Q-learning, greatly increasing the scoring probability in the automatic competition stage.

[0086] 4.2 Performance analysis of transfer reinforcement learning path planning

[0087] The single-game Q-learning training provides learning experience, which can be transferred to the next game by the agent using transfer reinforcement learning. The path planning of experiment 2 is selected as the source task and is transferred from one game of "alliance horizontal and vertical" to the other three games. The starting grid coordinates are set as (0, 1), (0, 2), (0, 3) respectively, and the corresponding target grid coordinates are all (8, 8). Figure 10 It is shown that the reward values of experiments 4-6 all show a rapid upward trend at the beginning of iteration, and the convergence is completed after 564, 473, and 296 iterations respectively, which is significantly less than experiments 1-3. Obviously, the learning experience of the source task plays a guiding role for the target task, reducing the blindness of the agent exploring the environment. Correspondingly, Fig. 11 shows that the robot moves very smoothly on the three shortest paths and can quickly reach the target grid.

[0088] Then, the learning experience of "alliance horizontal and vertical" is transferred to the "one touch" game, i.e. the knowledge transfer between different types of games. Experiments 7-8 set the starting grid coordinates as (0, 0), (0, 1) respectively, and the corresponding target grid coordinates are all (8, 8). At this time, the learning experience of the source task cannot be completely applied to the target task because the field environment and path starting point of the source and target tasks have changed greatly, so the agent needs to modify the learning experience of the source task according to the new environment state. Compared with Fig. 11 where the target task has high similarity with the source task, Figure 12 However, with the increase of learning times and the accumulation of learning experience, the reward values gradually tend to converge. Finally, experiments 7-8 complete the convergence after 539 and 473 iterations respectively, which is more than the iteration times of experiments 4-6 and less than the iteration times of experiments 1-3. Combined with the two robot movement paths in Fig. 13, the transfer of learning experience between high-similarity games can obtain the optimal path at low computational cost.

[0089] 4.3 Performance comparison of three path planning methods

[0090] The present application compares the path planning performance of A* algorithm, Dijkstra algorithm and transfer reinforcement learning method. The starting grid coordinates of "allied horizontal and vertical" are set as (0, 2) and (0, 3), and the corresponding target grid coordinates are both (8, 8). Figures 14 and 15 respectively show the optimal paths of the three methods, and the lengths are all 8. Similarly, Figure 16 shows the three optimal paths under "one touch" when the starting grid coordinates are (0, 1), and the lengths are all 9. Although the optimal path lengths of the three methods are equal, there is a large difference in training time. Table 1 shows that under "allied horizontal and vertical", the training time of transfer reinforcement learning is reduced by 66.5% and 16.5% compared with A* algorithm and Dijkstra algorithm respectively, and under "one touch", it is reduced by 52% and 19.4% respectively. This shows that whether it is "allied horizontal and vertical" or "one touch", transfer reinforcement learning reuses the knowledge of the last game and improves the learning ability for new tasks. Therefore, it can search the optimal path at the fastest speed, thereby effectively solving the problem of low efficiency of traditional path planning, and greatly reducing the model training time.

[0091] Table 1 Comparison of training time of three path planning methods

[0092]

[0093] 5CONCLUSION

[0094] In view of the problem of low efficiency of traditional path planning method, the present application proposes a VEX robot path planning method based on transfer reinforcement learning. First, the game field is divided by grid method, and the reward function is defined through the interaction between agent and environment. Then, the transfer of path planning experience is used to guide the action selection of the agent, and the new optimal path of the robot is obtained. Reducing the blind exploration of the agent to the environment during path search further improves the model convergence speed. Compared with A* algorithm and Dijkstra algorithm, the simulation results under "allied horizontal and vertical" and "one touch" show that this method can more efficiently solve the robot path planning problem.

[0095] 6REFERENCES

[0096] [1] Tao Y, Liu H T, Wang T M, Han D M. Research progress and industrialization development trend of service robots in China[J]. Journal of Mechanical Engineering, 2022: 1-19.

[0097] [2] Zuo G Y, Lei F, Qiao J F. Innovation talent training mode based on robot competition under the background of new engineering[J]. Journal of Higher Education, 2021, 6: 44-47.

[0098] [3] Dam T, Chalvatzaki G, Peters J, Pajarinen J. Monte-Carlo robot path planning[J]. IEEE Robotics and Automation Letters, 2022, 7(4): 11213-11220.

[0099] [4] Li X H, Miao M, Ran B J, et al. UAV obstacle avoidance path planning based on improved A* algorithm[J]. Computer Systems Applications, 2021, 30(02): 255-259.

[0100] [5] Che J T, Gao F Y, Xie Y W, et al. Path planning of underwater robot based on Dijkstra algorithm[J]. Mechanical Design and Research, 2020, 36(1): 44-48.

[0101] [6] Wang S, Duan R K, Liao Y H. Improved research on rapid expansion random tree algorithm in robot path planning[J]. Journal of Xi'an Jiaotong University, 2022, 56(7): 1-8.

[0102] [7] Alireza M, Vincent D, Tony W. Experimental study of path planning problem using EMCOA for a holonomic mobile robot[J]. Journal of Systems Engineering and Electronics, 2021, 32(6): 1450-1462.

[0103] [8] Wahab M N A, Lee C M, Akbar M F, Hassan F H. Path planning for mobile robot navigation in unknown indoor environments using hybrid PSOFS algorithm[J]. IEEE Access, 2020, 8: 161805-161815.

[0104] [9] Duan C X, Zhang P. Path planning for welding robot based on ant colony optimization algorithm [C]. 2021 3rd International Conference on Artificial Intelligence and Advanced Manufacture (AIAM) Manchester, United Kingdom, 23-25 October 2021, pp: 23-25.

[0105]

[10] Wang B, Liu Z, Li Q, Prorok A. Mobile robot path planning in dynamic environments through globally guided reinforcement learning [J]. IEEE Robotics and Automation Letters, 5(4): 6932-6939.

[0106]

[11] Le A V, Parween P T, Kyaw R E, Mohan T H. Reinforcement learning-based energy-aware area coverage for reconfigurable Rombo Tiling robot [J]. IEEE Access, 2020, 8: 209750-209761.

[12] Pei M, An H, Liu B, Wang C. An improved Dyna-Q algorithm for mobile robot path planning in unknown dynamic environment [J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2022, 52(7): 4415-4425.

[0107]

[13] Wen S H, Wen Z T, Zhang D. A multi-robot path-planning algorithm for autonomous navigation using meta-reinforcement learning based on transfer learning[J]. Applied Soft Computing. 2021, 110: 107605-107619.

[0108]

[14] Niu S, Liu Y, Wang J, Song H. A decade survey of transfer learning (2010-2020)[J]. IEEE Transactions on Artificial Intelligence, 2020, 1(2): 151-166.

[0109]

[15] Zeng R, Zhou J, Liu M L, et al. Transfer reinforcement learning algorithm based on double Q network learning[J]. Computer Application Research, 2021, 38(06): 1699-1703.

[0110]

[16] Sharma J, Andersen P A, Granmo O C, Goodwin M. Deep Q-learning with Q-matrix transfer learning for novel fire evacuation environment. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2021, 51(12): 7363-7381.

[0111]

[17] Hu X D, Huang X X, Hu T J, Wang F L, Liang S L. A fast path planning method for space robots in dynamic environment[J]. Space Control Technology and Application, 2018, 44(05): 14-21.

[0112]

[18] Wu S, Hu J, Zhao C, Pan Q. Path planning for autonomous mobile robot using transfer learning-based Q-learning [C]. 2020 3rd International Conference on Unmanned Systems (ICUS), Harbin, China, 2020, pp: 88-93.

[0113]

[19] Cui Y N, Li J, Chen Y, et al. TransPath: a knowledge reasoning method based on deep transfer reinforcement learning [J]. Small and Microcomputer System, 2022, 43(03): 536-543.

[0114]

[20] Zeng R, Zhou J, Liu M L, et al. Transfer reinforcement learning algorithm based on double Q network learning [J]. Computer Application Research, 2021, 38(06): 1699-1703.

Claims

1. A VEX robot path planning method based on transfer reinforcement learning, the application scenario is: in the VEX competition, the red team and the blue team are composed of two teams, and the two parties compete with each other in the competition including the first fifteen seconds of the automatic competition stage and the next forty-five seconds of the manual control stage; in the automatic competition stage, the programmer of each team designs the moving path of the robot to avoid obstacles according to the requirements of the season, so that the robot reaches the target at the fastest speed and completes the confrontation task; The application is characterized in that The path planning method comprises: 1) establishing a path planning model of the VEX robot; 1.1) The competition field of VEX robot competition is divided into a 9x9 grid map, each grid width is foot, and the grid attribute is defined as the movement of the robot and the information of the obstacles; 1.2) the moving direction of the robot on each grid is defined as eight moving directions: forward, backward, left, right, right front, right back, left back and left front; In the process that the robot moves from the initial grid coordinate to the target grid coordinate, let the moving speed be The moving time to reach the current grid coordinate is expressed as (1) wherein, and respectively represent the total number of grids in which the robot moves in the forward, backward, left, right directions, and in the right front, right rear, left rear, left front directions, ; defining the forward direction as the zero-degree direction and the clockwise direction as the positive direction, the angle of rotation of the robot is represented as ; 1.3) combined with the moving direction and the rotation angle, the path planning of the robot in the automatic competition stage is modeled as the following target optimization problem P1 (2) (2a) (2b) 2) using reinforcement learning to solve the optimal path of a single competition; Step 2) using Q-learning to explore the optimal path of the robot; 3) using transfer reinforcement learning to solve the optimal path of two competitions; 3.1) defining that the path planning of the last competition in the "horizontal and vertical connection" or "one touch" automatic competition stage is the source task, and the path planning of the next competition is the target task; only the path finding experience of the last competition is reused to obtain the optimal path of the next competition; 3.2) the competition environment and robot task of "horizontal and vertical connection" and "one touch" automatic competition stage have high similarity, so the R table of "horizontal and vertical connection" is modified according to the "one touch" field environment, the Q table of "horizontal and vertical connection" is trained, and the new optimal path is quickly explored under the guidance of prior knowledge; 4) simulation experiments are carried out for "horizontal and vertical connection" and "one touch" competition examples to evaluate the planning performance of the paths obtained in steps 2) and 3); 4.1) performance analysis of reinforcement learning path planning Taking VEX competition "horizontal and vertical connection" as an example, the robot path exploration process based on Q-learning is displayed, and the performance advantage of the path planning is verified; 4.2) performance analysis of transfer reinforcement learning path planning The path planning experience of "horizontal and vertical connection" one competition is transferred to another three competitions to verify the feasibility of knowledge transfer between similar competitions; at the same time, the path planning experience of "horizontal and vertical connection" is transferred to "one touch" competition to verify the feasibility of knowledge transfer between different types of competitions; 4.3) comparison with A* algorithm and Dijkstra algorithm.

2. The VEX robot path planning method based on transfer reinforcement learning according to claim 1, characterized in that In step 2), the specific process of using Q-learning to explore the optimal path of the robot is as follows: Quadruple The following description is made (1) External environment state represents a sequence of 81 grids in which the game field is divided; (2) Action set Indicates the sequence of 8 movement directions of the robot, forward, backward, left, right, right front, right back, left back, left front. (3) reward function To select the action with the maximum reward, i.e. ​ ， Wherein, Dis represents the Euclidean distance between the current coordinates of the robot and the target coordinates; the closer the robot is to the target grid, the smaller the negative reward obtained by advancing one grid; encountering obstacles will obtain a negative reward, which will prompt the agent to make better movement decisions and help the robot reach the target grid faster.

3. The VEX robot path planning method based on transfer reinforcement learning according to claim 1, characterized in that In step 2), the steps of designing the robot to find the optimal path are as follows: Step 1 initialization: the grid positions of vertical and horizontal stakes, ring towers, curved rings and balance bridges, the coordinates of the starting grid and the target grid of the robot, the Q table and the R table; Step 2 Action Selection: The agent utilizes - The greedy policy selects the direction of movement of the robot in the action set A, i.e. with probability The action with the maximum Q-value in state s is selected, while with probability a random action is selected; Step 3 Update Q-values: Combine recorded state-action and reward values to update ; Step 4 Iterative training: repeat Step 2~Step 3 until the Q table converges; Step 5 Path solving: the agent finds the optimal path for the current game on the trained Q table, and the host board issues a moving instruction to guide the robot to reach the target grid at the fastest speed.

4. The VEX robot path planning method based on transfer reinforcement learning according to claim 1, characterized in that In Step 3, the path planning of the last game is defined as the source task, and the path planning of the next game is defined as the target task. The programmer reuses the pathfinding experience of the last game to find the optimal path for the next game.

Images