Adaptive frequency modulation navigation method and storage medium

Abstract

The application discloses an adaptive frequency modulation navigation method and a storage medium, wherein the method comprises the following steps: designing a robot navigation simulator close to a real environment, and mathematically modeling a semi-Markov process for a navigation problem; dimensionally reducing an action space of the semi-Markov process by using a trajectory parameter space; sampling from the dimensionally reduced action space to obtain a large amount of interactive navigation experience; training an optimal frequency modulation navigation strategy according to the collected navigation experience; and migrating the trained navigation strategy to a physical robot. The adaptive frequency modulation navigation method uses adaptive frequency modulation navigation technology, so that a navigation algorithm based on deep reinforcement learning can adapt to more diverse environments and has better generalization. In an environment with low scene complexity, the robot is controlled at a low frequency as much as possible, so as to reduce the calculation overhead of the algorithm, and in an environment with high scene complexity, the robot is controlled at a suitable high frequency, so as to ensure the safety of the robot.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot navigation, and particularly relates to an adaptive frequency modulation navigation method and a storage medium. BACKGROUND

[0002] Although the classic mobile robot navigation technology has been applied in actual scenes, most of them have the problems of large amount of calculation and poor scene generalization. In order to solve these problems, the prior art proposes a method of using deep reinforcement learning, so that the robot learns the strategy of avoiding obstacles and reaching the destination by automatically labeling the success and failure experience through a reward function in the process of interacting with the environment. Most of the existing navigation technologies based on deep reinforcement learning ensure safety by issuing speed instructions to the robot at a high control frequency, so that the robot can quickly respond to changes in the environment. However, this approach has a large overhead on computing resources, increasing the hardware cost and energy consumption of the computing platform. And in the training environment of deep reinforcement learning, a higher control frequency (i.e. a smaller control interval) makes the robot more prone to local optimization problems, making it difficult to complete long-distance navigation tasks in unknown complex environments. SUMMARY

[0003] The adaptive frequency modulation navigation method proposed by the present application can solve the above technical problems.

[0004] To achieve the above object, the present application adopts the following technical solutions:

[0005] An adaptive frequency modulation navigation method, comprising the following steps,

[0006] Designing a robot navigation simulator close to the real environment, and mathematically modeling the navigation problem with a semi-Markov process;

[0007] Using a trajectory parameter space to reduce the dimension of the action space of the semi-Markov process;

[0008] Sampling from the reduced action space to obtain a large amount of interactive navigation experience, and training an optimal frequency modulation navigation strategy according to the collected navigation experience;

[0009] Migrating the trained navigation strategy to a physical robot.

[0010] Further, the robot navigation simulator close to the real environment is designed, and the navigation problem is mathematically modeled with a semi-Markov process, including:

[0011] Let the state S of the robot contain two parts, which are the local map of the nearby 6m x 6m area perceived by the laser sensor of the robot, and the relative position (x, y, theta) of the target point, wherein x and y represent coordinates, and theta represents the direction of the target point;

[0012] The robot's action command A comprises three dimensions: linear velocity, angular velocity, and execution time. In each state S, after executing action A, a corresponding reward function R evaluates the appropriateness of action A; the more appropriate the action, the larger the value of R. In this embodiment, t... i The reward function at time step is defined as:

[0013]

[0014] in, The reward for getting closer to the target point is calculated by subtracting the current distance from the target point from the previous distance, then multiplying by 10. This means that the closer you are to the target point, the greater the reward, and the farther you are from the target point, the greater the penalty. In the above formula, pg represents the robot's target point. Indicates t i-1 The robot's location is constantly changing. Indicates t i The robot's location is constantly changing. express The straight-line distance between the position and pg;

[0015] For t i The reward value is reached at a certain time. When the robot reaches the target point (pt-pg < 0.20), the reward value is 500. Otherwise, the value is 0, that is...

[0016] For t i The collision penalty value at time t, if t i If the robot collides with another robot, it will be penalized with -500. Otherwise, the value is 0, indicating a collision has occurred.

[0017] For t i The action execution time penalty value at any given time is added to a fixed value, set to... The longer the execution time of a single action, the greater the penalty, in order to make the robot reach the target point as quickly as possible and minimize the number of planned actions.

[0018] Furthermore, the dimensionality reduction of the action space of the semi-Markov process using the trajectory parameter space includes designing the neural network structure of the navigation strategy and initializing it as a stochastic strategy, specifically including:

[0019] The neural network is used to represent the navigation strategy, and the structure of the neural network includes the first three layers of 2D convolution and maximum pooling for extracting grid image information, the first convolution layer is 64-dimensional, the convolution kernel is 3x3, and the ReLU activation function is used, the second convolution layer is 128-dimensional, the convolution kernel is 3x3, and the ReLU activation function is used, and the third convolution layer is 256-dimensional, the convolution kernel is 3x3, and the ReLU activation function is used;

[0020] Among them, the kernel of the three maximum pooling layers is 2x2, and the step is 2;

[0021] The output of the convolution layer is converted to 512-dimensional after passing through a fully connected layer, and is spliced with the position of the target point, and then passes through two 512-dimensional fully connected layers to obtain a 2-dimensional data, and then passes through random sampling, ELU activation function, and 2-dimensional to 3-dimensional conversion in the trajectory parameter space, and finally obtains the linear velocity, angular velocity and execution time;

[0022] The parameters in the neural network are initialized by the xavier algorithm, and the corresponding initial strategy is a random strategy.

[0023] Further, the dimension-reduced action space is sampled to obtain a large amount of interactive navigation experience, and the optimal frequency modulation navigation strategy is trained according to the collected navigation experience, including generating a random scene in the simulator, collecting experience according to the current strategy, and updating the navigation strategy with the collected experience, specifically including:

[0024] S2.1, the positions of all obstacles and the starting point and target point of the robot are randomly set, the robot tries to reach the target point according to the current strategy, at each step the robot can know the state S of itself, the current strategy generates a corresponding action A, then the robot obtains a certain value of reward R after executing the action A in the simulator, then reaches the next state S', and then continuously loops until the robot reaches the target point or collision occurs; In this process, the (S, A, R) combination of each step is recorded and stored in the experience database;

[0025] The three-dimensional action space is converted into a two-dimensional trajectory parameter space for experience storage, and the action is converted back to three dimensions for actual execution;

[0026] The trajectory parameter space is a two-dimensional space, in which each point corresponds to a drivable trajectory of the robot, a series of robot poses in the three-dimensional space (x, y, θ) are represented by a two-dimensional manifold in the trajectory family parameter space, a 2D virtual action (v TP , ω TP ) and a fixed time τ TP are used to define a parameterized circular trajectory, which is called a virtual action because v TP and ωTP is the virtual translational and rotational velocity in the TPspace, not limited by the actual robot maximum translational velocity v m and rotational velocity ω m This virtual 2D action is converted into a unique 3D executable action, converting (v TP , ω TP ) into (v, ω, d) as follows:

[0027]

[0028] where

[0029] S2.2, repeat step S2.1, after collecting experience of 100 scenes, update the navigation strategy with the experience, update the parameters of the navigation strategy neural network model based on the distributed proximal policy optimization algorithm.

[0030] In another aspect, the application also discloses a computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the steps of the above method.

[0031] As can be seen from the above technical solution, the adaptive frequency navigation method of the application uses adaptive frequency navigation technology, so that the navigation algorithm based on deep reinforcement learning can adapt to more diverse environments and obtain better generalization. In an environment with low scene complexity, the robot is controlled at a low frequency as much as possible to reduce the computational overhead of the algorithm, and in an environment with high scene complexity, the robot is controlled at a suitable high frequency to ensure its safety.

[0032] Specifically, the application sets a dynamically adjusted control frequency, so that the motion mode of the robot is more diverse, and uses a semi-Markov model to accurately model the problem, supplemented by action dimension reduction in the trajectory parameter space, and successfully trains a navigation strategy that can adapt to more diverse environments using a deep reinforcement learning method, and obtains better strategy generalization. The generalization refers to controlling the robot at a low frequency as much as possible in a large environment with low scene complexity to reduce the computational overhead of the algorithm, and the long-distance navigation task can be completed more easily compared with the conventional reinforcement learning navigation method; in an environment with high scene complexity, the robot is controlled at a suitable high frequency, which is more secure than the conventional reinforcement learning navigation method. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a method flowchart of the application;

[0034] Figure 2a is a training environment of a simulator;

[0035] Figure 2b is a local map schematic diagram;

[0036] Figure 3 is a strategy neural network structure diagram of the present application;

[0037] Figure 4 is a test effect diagram of the present application in a simulator;

[0038] Figure 5 is a physical robot running effect diagram; wherein (a) is a physical robot used when testing the algorithm; (b) and (c) are navigation scenes that are easy to make the robot fall into local optimization, which are a spiral scene and a zigzag scene respectively; (d) is a long-distance complex test scene containing dynamic pedestrians. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments.

[0040] As shown in Figure 1 , the adaptive frequency modulation navigation method described in the present embodiment includes designing a robot navigation simulator close to a real environment, and mathematically modeling a navigation problem by using a semi-Markov process;

[0041] Dimensionality reduction is performed on the action space of the semi-Markov process by using a trajectory parameter space;

[0042] A large amount of interactive navigation experience is obtained by sampling from the dimensionality-reduced action space, and the optimal frequency modulation navigation strategy is trained according to the collected navigation experience;

[0043] The trained navigation strategy is migrated to a physical robot.

[0044] The embodiments of the present application can be understood as realizing frequency modulation by adjusting the execution time of each instruction. First, a large amount of navigation experience with different instruction execution times is collected in a simulator, then the navigation strategy is trained by using the technology of deep reinforcement learning, after the training is completed, the migration to the entity is simulated, and then the strategy is deployed to the physical robot, so as to realize the real application.

[0045] As shown in Figure 2a , the simulator used in the present embodiment contains static and dynamic obstacles, and the line connection is the robot and its target point, as shown in Figure 2b , the simulator simulates the laser sensor, and the local map can be generated by the simulated laser data.

[0046] Mathematical model

[0047] To enable the data collected in the simulator to guide the robot in learning adaptive navigation strategies, we designed a reward function R as a standard to evaluate the quality of the data. We also defined a unified data format for the robot's state and available actions, thus using a semi-Markov decision process (SMDP) to characterize the entire problem. In this embodiment, the robot's state S consists of two parts: a local map of a nearby 6m × 6m area perceived by the robot's laser sensor, and the relative position (x, y, θ) of the target point, where x and y represent coordinates and θ represents the direction of the target point. The robot's action command A has three dimensions: linear velocity, angular velocity, and execution time. Compared to ordinary navigation methods, we added the time dimension, allowing the execution time of each action command to be different, thereby achieving adaptive frequency modulation navigation. After executing action A in each state S, a corresponding reward function R evaluates whether action A is appropriate; the more appropriate the action, the larger the value of R. In this embodiment, t... i The reward function at time step is defined as:

[0048]

[0049] in, The reward for getting closer to the target point is calculated by subtracting the current distance from the target point from the previous distance, then multiplying by 10. This means that the closer you are to the target point, the greater the reward, and the farther you are from the target point, the greater the penalty. In the above formula, pg represents the robot's target point. Indicates t i-1 The robot's location is constantly changing. Indicates t i The robot's location is constantly changing. express The straight-line distance between the position and pg;

[0050] For t i The reward value is reached at a certain time. When the robot reaches the target point (at this point, pt-pg < 0.20), the reward value is 500, i.e. Otherwise, the value is 0, that is...

[0051] For t i The collision penalty value at time t, if t i If the robot collides with another robot, it will be penalized with -500. Otherwise, the value is 0, indicating a collision has occurred.

[0052] For t iThe action execution time penalty value at the moment is added to a fixed value, which is set to The longer the execution time of a single action, the greater the penalty of the item, which is used to make the robot reach the target point as soon as possible and reduce the number of planned actions as much as possible;

[0053] After we model the navigation problem using SMDP, we can use deep reinforcement learning to obtain a policy π(A|S) that maximizes the sum of the reward function (the policy π can choose an action A that it considers most appropriate according to any state S). The optimal policy in this embodiment is a navigation policy that reaches the end point in the least amount of time and without collision.

[0054] The specific steps are as follows:

[0055] Step one, design the neural network structure of the navigation policy and initialize it as a random policy;

[0056] Step two, generate a random scene in the simulator, collect experience according to the current policy, and to improve training efficiency, reduce the three-dimensional action space to two-dimensional by trajectory parameter space;

[0057] Step three, repeat step two, collect experience for 100 scenes, and then update the navigation policy with the experience. If the probability of the robot reaching the end point is greater than 99% and the policy neural network update is very small, it is judged that the training has converged, and step four is performed, otherwise the updated policy is used to repeat steps two and three;

[0058] Step four, design various scenes in the simulator to verify the generalization of the navigation policy, and migrate the trained navigation policy to the physical robot.

[0059] The purpose and details of each step are described below:

[0060] Step one, design the neural network structure of the navigation policy and initialize it as a random policy

[0061] This embodiment uses a neural network to represent the navigation policy, and the structure of the neural network is as follows Figure 4As shown, the first three layers of 2D convolution and max pooling are used to extract the grid image information, the first convolution layer is 64-dimensional, the convolution kernel is 3x3 and the ReLU activation function, the second convolution layer is 128-dimensional, the convolution kernel is 3x3 and the ReLU activation function, and the third convolution layer is 256-dimensional, the convolution kernel is 3x3 and the ReLU activation function. The kernel of the three max pooling layers is 2x2 and the step is 2. The output of the convolution layer is converted to 512-dimensional after passing through a fully connected layer, and then spliced with the position of the target point. After passing through two 512-dimensional fully connected layers, a 2-dimensional data is obtained, and then after random sampling, ELU activation function, and 2-dimensional to 3-dimensional conversion in the trajectory parameter space, the linear velocity, angular velocity and execution time are finally obtained.

[0062] The parameters in the neural network are initialized by the xavier algorithm, and the corresponding initial strategy is a random strategy.

[0063] Step two, generate a random scene in the simulator, and collect experience according to the current strategy

[0064] As shown in FIG. 2, in each round of simulation, in order to make the scene and task more general, the positions of all obstacles and the starting point and target point of the robot are randomly set. The robot tries to reach the target point according to the current strategy. At each step, the robot can know its own state S, the current strategy generates a corresponding action A, and then after executing action A in the simulator, the robot obtains a certain value of reward R, and then reaches the next state S', and then the cycle is repeated until the robot reaches the target point or collision occurs. In this process, the (S, A, R) combination of each step is recorded and stored in the experience database.

[0065] Unlike traditional reinforcement learning navigation methods, since the action space of the embodiment is increased from two-dimensional to three-dimensional, the number of actions that need to be sampled during learning is greatly increased. In order to reduce the number of samples required, the three-dimensional action space is converted into a two-dimensional trajectory parameter space for experience storage, and the action is converted back to three-dimensional for actual execution.

[0066] The trajectory parameter space is a two-dimensional space, in which each point corresponds to a drivable trajectory of the robot. The basic idea is that a series of robot poses in three-dimensional space (x, y, θ) can be represented by a two-dimensional manifold in the trajectory family parameter space. In this invention, differential drive robots are considered, which can only move along circular trajectories due to their movement constraints. Then a parameterized circular trajectory can be defined using a 2D virtual action (v TP , ω TP ) and a fixed time τ TP . It is called a virtual action because v TP and ωTP is the virtual translational and rotational velocity in the TPspace, not limited by the actual robot's maximum translational velocity v m and rotational velocity ω m . This virtual motion can be converted to a unique 3D executable motion, converting (v TP , ω TP ) to (v, ω, d) as follows:

[0067]

[0068] where

[0069] Step three, update the navigation policy with the collected experience

[0070] Repeat step two, after collecting experience of 100 scenes, update the navigation policy with these experience, update the parameters of the navigation policy neural network model based on distributed proximal policy optimization algorithm (DPPO). In short, the principle of updating the parameters of the neural network is as follows: if the reward value obtained by performing action A in state S is larger, the probability of taking action A when the robot faces a similar state to state S next time will be larger; if the reward value obtained by performing action A in state S is smaller, the probability of taking action A when the robot faces a similar state to state S next time will be smaller.

[0071] If the probability of the robot reaching the end point is greater than 99% and the update amount of the neural network parameters is very small after a certain round of policy update, it means that the training has basically converged, and the robot has learned a navigation policy that can adaptively adjust the frequency, which can proceed to step four. Otherwise, use the updated policy to collect experience in the simulator again, and repeatedly perform steps two and three to update the policy neural network;

[0072] Step four, design various scenes in the simulator to verify the effectiveness of the navigation policy, and migrate the trained navigation policy to the physical robot

[0073] After the training of the policy network converges, we set up various test environments in the simulator to verify the effect of our algorithm. They are dense obstacle environment, back-shaped environment, and zigzag-shaped environment. Our robot successfully reached the end point in each test environment, and as Figure 4As shown in the boxed part, each origin on the robot trajectory represents that the robot plans an action at that position, and it can be seen that the robot automatically reduces the number of plans in the open area, reduces the frequency of decision-making, and reduces the computing overhead; while in the obstacle-dense area, the number of plans is increased, the frequency of decision-making is increased, and the safety of the navigation strategy is ensured. Thus, it is illustrated that the algorithm has the function of automatically adjusting the frequency according to the environment.

[0074] After verifying the effectiveness of the navigation strategy in the simulator, since the data of the laser sensor in the simulator is extremely similar to that in the real world, and a robust positioning algorithm, adaptive Monte Carlo localization, is also deployed for the robot, the trained navigation strategy can be directly migrated to the entity robot turtlebot2 as shown in (a). Figure 5

[0075] In summary, the application designs an adaptive frequency adjustment navigation method based on reinforcement learning. The navigation algorithm based on deep reinforcement learning can adapt to more diverse environments and obtain better generalization. In the environment with low scene complexity (such as (b) and (c) shown), the robot is controlled at a low frequency as much as possible, the computing overhead of the algorithm is reduced, and the smoothness of the trajectory is improved. In the environment with high scene complexity (such as (d) shown), the robot is controlled at a suitable high frequency to ensure its safety. The application provides a novel adaptive energy-saving scheme for indoor robot navigation. Figure 5 Figure 5 (d) shown), the robot is controlled at a suitable high frequency to ensure its safety. The application provides a novel adaptive energy-saving scheme for indoor robot navigation.

[0076] In another aspect, the application further discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to make the processor execute the steps of any of the above methods.

[0077] In another aspect, the application further discloses a computer device, which comprises a memory and a processor, and the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of any of the above methods.

[0078] In another embodiment provided in the application, a computer program product containing instructions is also provided, which makes a computer execute the steps of any of the above methods when the computer program product is run on the computer.

[0079] It can be understood that the system provided in the embodiments of the application corresponds to the method provided in the embodiments of the application, and the explanation, examples and beneficial effects of the related content can be referred to the corresponding part in the above method.

[0080] ​​Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0081] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0082] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A self-adapting frequency-modulated navigation method, characterized in that, Comprising the following steps, designing a robot navigation simulator close to real environment, mathematically modeling the navigation problem as a semi-Markov process; dimensionality reduction of the action space of the semi-Markov process using the trajectory parameter space, including designing the neural network structure of the navigation policy and initializing it as a random policy, specifically including: using a neural network to represent the navigation policy, the structure of the neural network including the first three layers of 2D convolution and max pooling for extracting grid image information, the first convolutional layer being 64-dimensional, the convolution kernel being 3x3 and the ReLU activation function, the second convolutional layer being 128-dimensional, the convolution kernel being 3x3 and the ReLU activation function, and the third convolutional layer being 256-dimensional, the convolution kernel being 3x3 and the ReLU activation function; the kernel of the three max pooling layers is 2x2 and the step is 2; the output of the convolutional layer is converted to 512-dimensional after passing through a fully connected layer, then spliced with the position of the target point, and then passed through two 512-dimensional fully connected layers to obtain a 2-dimensional data, then randomly sampled, ELU activated, and converted from 2-dimensional to 3-dimensional in the trajectory parameter space, and finally the linear velocity, angular velocity and execution time are obtained; the parameters in the neural network are initialized using the xavier algorithm, and the corresponding initial policy is a random policy; sampling from the dimensionality-reduced action space to obtain a large amount of interactive navigation experience, training the optimal frequency-modulated navigation policy according to the collected navigation experience, including generating a random scene in the simulator, collecting experience according to the current policy, and updating the navigation policy using the collected experience, specifically including: S2.1, set the positions of all obstacles and the starting point and target point of the robot to be randomly set, the robot tries to reach the target point according to the current policy, at each step the robot can know its own state S, the current policy will generate a corresponding action A, then the robot will obtain a certain value of reward R after executing the action A in the simulator, then reach the next state S', then loop until the robot reaches the target point or collision occurs; in this process, the (S, A, R) combination of each step is recorded and stored in the experience database; convert the three-dimensional action space to a two-dimensional trajectory parameter space for experience storage, and convert the action back to three-dimensional for actual execution; The trajectory parameter space is a two-dimensional space where each point corresponds to a robot drivable trajectory, the three-dimensional space A family of robot poses in the three-dimensional space is represented by a two-dimensional manifold in the trajectory family parameter space, using a 2D virtual motion and a fixed time to define a parameterized circular trajectory, called virtual motion, because and are virtual translational and rotational velocities in the TPspace, not subject to the actual robot maximum translational and rotational velocities This virtual 2D motion translates into a unique 3D executable motion, translating as follows: , , wherein ; S2.2, repeat step S2.1, collect experience for 100 scenes, and update the navigation policy using the experience, update the parameters of the navigation policy neural network model based on the distributed proximal policy optimization algorithm; migrate the trained navigation policy to the physical robot.

2. The adaptive frequency-modulated navigation method of claim 1, wherein: designing a robot navigation simulator close to real environment, mathematically modeling the navigation problem as a semi-Markov process, including: Let the state S of the robot contain two parts, which are the local map of the nearby area perceived by the laser sensor of the robot, and the relative position of the target point wherein denotes the coordinates, denotes the direction of the target point;​ The robot's action instruction A comprises three dimensions: linear velocity, angular velocity, and execution time. In each state S, after executing action A, a corresponding reward function R evaluates the appropriateness of action A; the more appropriate the action, the larger the value of R. The reward function at time step is defined as: ； wherein, is a reward term close to the target point, indicating the distance between the previous step and the target point minus the current distance from the target point and then multiplied by , indicating that the closer to the target point, the more rewarding, the farther away from the target point, the more punishment: , in the above formula indicates the target point of the robot, indicates the position of the robot at time indicates the position of the robot at time indicates the position of the robot at time indicates the position of the robot at time indicates the straight-line distance between the and positions; for When the reward value is reached, the robot reaches the target point. The hourly reward value is 500, that is Otherwise, this item is ,Right now ; For each time instant a collision penalty value is given, if a collision occurs for the robot at that time instant then a penalty of is given, i.e. , otherwise the term is i.e. a collision occurs, then ; For the action execution time penalty value at the moment is added a fixed value, which is set as , the longer the execution time of a single action, the greater the penalty of this item, which is used to make the robot reach the target point as soon as possible and reduce the number of planned actions as much as possible.

3. A computer-readable storage medium storing a computer program, the computer program being executed by a processor to cause the processor to perform the steps of the method of any one of claims 1 to 2.

Images