Robot obstacle avoidance neural network model training method and device and storage medium

Abstract

The embodiment of the invention relates to a robot obstacle avoidance neural network model training method and device and a storage medium. The method comprises the steps that in a preset robot obstacle avoidance simulation environment, based on a preset random elastic reward mechanism and a reinforcement learning neural network, a robot generates an action decision, a feedback operation state and a corresponding reward record in the obstacle avoidance process, and a random reward pool is formed; jointly training the reinforcement learning neural network by using the random reward pool and adding random noise of reinforcement learning actions, and updating network parameters; and in an actual operation environment, training the reinforcement learning neural network based on the random elastic reward mechanism according to the obtained action decision value generated by the robot in the obstacle avoidance process and the feedback operation state value, and obtaining an obstacle avoidance neural network model of the robot. According to the technical scheme of the embodiment of the invention, the reinforcement learning obstacle avoidance efficiency and practicability can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a robot obstacle avoidance neural network model training method, equipment and storage medium. Background Art

[0002] Robot obstacle avoidance algorithm is the fundamental guarantee for robots, such as wheeled robots, to operate autonomously. The quality of the obstacle avoidance algorithm directly determines the intelligence and usability of the robot. The development of efficient and fast obstacle avoidance algorithm can greatly improve the operating efficiency of the robot, especially in multi-robot collaboration scenarios.

[0003] Traditional robot obstacle avoidance algorithms are mainly based on numerical optimization methods, such as the DWA (Dynamic Window Approach) and TEB (Time Elastic Band) algorithms commonly used by ROS (Robot Operating System). Such algorithms usually require the establishment of a mathematical model of the robot system, and then use this mathematical model to estimate the robot's future collision state, so as to select a reasonable obstacle avoidance path. However, the computing power of the ROS robot chip is limited, and the optimization algorithm calculation process is complex, resulting in a very limited step length for obstacle avoidance estimation using numerical optimization methods, a prominent "short-sighted" phenomenon in robot obstacle avoidance, low obstacle avoidance efficiency in complex scenes, and even "detours" and "backtracking". Therefore, numerical optimization obstacle avoidance methods are difficult to meet the requirements for efficient obstacle avoidance in complex scenes.

[0004] Deep reinforcement learning is good at solving continuous sequence decision problems. By using action and feedback data to train deep neural networks, its best performance is comparable to numerical values ​​and can achieve real-time optimization scheduling of the system. More importantly, the trained reinforcement learning network does not require a time-consuming iterative solution process. However, the performance of reinforcement learning algorithms is greatly affected by the learning mechanism and reward and punishment functions. Traditional reinforcement learning algorithm network training has overfitting and insufficiency. It is difficult to achieve good control effects in actual use. It is necessary to design a reasonable training mechanism and reward function to complete efficient and accurate network training. Summary of the invention

[0005] The present invention provides a robot obstacle avoidance neural network model training method, equipment and storage medium, aiming to overcome the shortcomings of deep reinforcement learning and realize the reinforcement learning of wheeled robots with efficient obstacle avoidance operation.

[0006] In a first aspect, an embodiment of the present invention provides a robot obstacle avoidance neural network model training method, comprising:

[0007] In the preset robot obstacle avoidance simulation environment, the robot's action decisions, feedback of operating status and corresponding rewards generated during the obstacle avoidance process are recorded based on the preset random elastic reward mechanism and reinforcement learning neural network, and a random reward pool is formed;

[0008] Using the random reward pool and adding random noise of reinforcement learning actions, jointly training the reinforcement learning neural network and updating network parameters;

[0009] In an actual operating environment, based on the random elastic reward mechanism and the obtained action decision values ​​and feedback operating state values ​​generated by the robot during the obstacle avoidance process, the reinforcement learning neural network is trained to obtain the robot's obstacle avoidance neural network model.

[0010] In a second aspect, an embodiment of the present invention provides a robot obstacle avoidance neural network model training device, comprising:

[0011] The random reward pool module is formed to record the action decisions, feedback operation status and corresponding rewards generated by the robot in the obstacle avoidance process based on the preset random elastic reward mechanism and reinforcement learning neural network in the preset robot obstacle avoidance simulation environment, and form a random reward pool;

[0012] A first training module of a reinforcement learning neural network is used to utilize the random reward pool and add random noise of reinforcement learning actions to jointly train the reinforcement learning neural network and update network parameters;

[0013] The second training module of the reinforcement learning neural network is used to train the reinforcement learning neural network in an actual operating environment based on the random elastic reward mechanism according to the action decision values ​​and feedback operating state values ​​generated by the robot during the obstacle avoidance process to obtain the robot's obstacle avoidance neural network model.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0015] one or more processors;

[0016] A memory for storing one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the robot obstacle avoidance neural network model training method provided by any embodiment of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute a robot obstacle avoidance neural network model training method as provided in any embodiment of the present invention.

[0019] The embodiments of the present invention provide a robot obstacle avoidance neural network model training method, device and storage medium. By designing a random elastic reward mechanism, the policy network can achieve fast and efficient convergence, and the deep neural network can quickly give the correspondence between the continuous state of the system and the execution action, which solves the problem that the numerical optimization obstacle avoidance algorithm is difficult to meet the efficient obstacle avoidance requirements of complex scenes, and the problem that the traditional reinforcement learning algorithm network training has overfitting and insufficiency, and the control effect is not good in actual use. Compared with the traditional numerical optimization obstacle avoidance algorithm, it can better solve the efficient obstacle avoidance requirements of complex scenes, and at the same time can continuously use real-time operation data to cyclically update the reinforcement learning obstacle avoidance model, so that obstacle avoidance becomes more and more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a robot obstacle avoidance neural network model training method provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of a robot obstacle avoidance neural network model training process provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the structure of a robot obstacle avoidance neural network model training device provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0025] Figure 1 A flowchart of a robot obstacle avoidance neural network model training method provided by an embodiment of the present invention. This embodiment is applicable to a robot, such as a wheeled robot, which uses an obstacle avoidance neural network model in obstacle avoidance. The method can train the robot obstacle avoidance neural network model, and the method can be executed by a robot obstacle avoidance neural network model training device, which can be implemented by hardware and / or software and can generally be integrated in an electronic device, such as a computer device. The method specifically includes:

[0026] Step 110: In a preset robot obstacle avoidance simulation environment, the action decisions, feedback operation status and corresponding reward records generated by the robot during the obstacle avoidance process based on a preset random elastic reward mechanism and a reinforcement learning neural network are recorded to form a random reward pool;

[0027] Step 120: Utilize the random reward pool and add random noise of reinforcement learning actions to jointly train the reinforcement learning neural network and update network parameters;

[0028] Step 130: In an actual operating environment, based on the random elastic reward mechanism and the obtained action decision values ​​and feedback operating state values ​​generated by the robot during the obstacle avoidance process, the reinforcement learning neural network is trained to obtain an obstacle avoidance neural network model of the robot.

[0029] Among them, a random elastic reward mechanism is preset, and a robot mathematical model is preset. A robot obstacle avoidance simulation environment is established according to the robot mathematical model. The robot obstacle avoidance simulation environment is used to train the reinforcement learning neural network to form a random reward pool, and random actions are used to test the effectiveness of the robot obstacle avoidance simulation environment and the preset random elastic reward mechanism, and the network parameters and reward parameters are adjusted according to the test results. The reinforcement learning neural network is trained in the actual operating environment to obtain the robot's obstacle avoidance neural network model.

[0030] Optionally, setting the random elastic reward mechanism includes:

[0031] Set the random elastic reward mechanism evaluation function.

[0032] Optionally, the setting of the random elastic reward mechanism evaluation function includes:

[0033] Set the following function as the evaluation function of the random elastic reward mechanism:

[0034] G(v,w)=δ[k1(Theta(v,w)-Theta * )+k2|Pos(v,w)-

[0035] Pos * |+k3vel(v,w)+k4dis(v,w)]

[0036] Where: δ is the coefficient of the random elastic function, (v, w) is the robot's forward speed and rotation speed; Theta(v, w) is the azimuth function, which is used to evaluate the robot's predicted trajectory end direction and target point orientation at the current set speed. * The angle difference between them; Pos(v,w) is the position of the robot at the end point of the predicted trajectory, and its position relative to the local path target point Pos *distance deviation; vel(v,w) is the current linear velocity of the robot, in order to promote the robot to reach the target quickly; dis(v,w) is the distance between the robot and the nearest obstacle on the map when the robot is at the end point of the predicted trajectory. Penalize the sampling points close to the obstacle to ensure the robot's obstacle avoidance ability and reduce the probability of collision between the robot and the obstacle.

[0037] It also includes adjusting the parameters in the evaluation function of the random elastic reward mechanism during the training process of the reinforcement learning neural network, wherein the δ value remains unchanged during a single obstacle avoidance training, and a random value is obtained using a random function when the next obstacle avoidance training begins; and optimizing and adjusting k1, k2, k3, and k4 based on the time it takes for the robot to reach the target point and the effectiveness of obstacle avoidance.

[0038] Optionally, the robot obstacle avoidance neural network model training method also includes:

[0039] Set up the robot mathematical model for initial training of reinforcement learning neural network parameters.

[0040] in,

[0041] Taking the differential robot as an example, the motion coordinate calculation formula is:

[0042]

[0043] θ t =θ t +wΔt

[0044] Where: x is the distance the robot moves in the x-axis direction in the map coordinate system; y is the distance the robot moves in the y-axis direction in the map coordinate system; θ is the rotation angle of the robot around the rotation coordinate axis z perpendicular to the robot; v is the forward speed of the robot; w is the rotation speed of the robot; Δt is the model calculation step time.

[0045] When the robot generates a trajectory during obstacle avoidance, the size of the dynamic window is at least affected by the linear acceleration, angular acceleration and obstacles of the motor, and the constraints are as follows:

[0046] The robot is limited by its own maximum and minimum speeds:

[0047] v m ={(v,w)|v∈[v min ,v max ],w∈[w min ,w max ]}

[0048] Where: v min 、v max are the upper and lower limits of the robot's forward speed respectively; wmin 、w max The upper and lower limits of the robot's rotation speed.

[0049] The robot is affected by the motor performance: Due to the limited torque of the driving motor, there is a maximum acceleration and deceleration limit. During the forward simulation cycle of the robot's predicted trajectory, the target speed is the speed that the robot can actually achieve.

[0050] v d ={(v,w)|v∈[v c -v b Δt,v c +v a Δt],w∈[w c -w b Δt,w c +w a Δt]}Where: v d v is the robot's target forward speed; c is the current forward speed of the robot; v b v is the maximum forward speed difference that the robot can reduce within Δt time; a w is the maximum forward speed difference that the robot can increase within the time Δt; c is the current rotation speed of the robot; w b is the maximum rotation speed difference that can be reduced within the robot's Δt time; w a It is the maximum rotation speed difference that can be increased within the robot's Δt time.

[0051] Based on the safety restrictions of mobile robots: In order to be able to stop before hitting an obstacle, under the condition of maximum deceleration, the speed range is:

[0052]

[0053] Optionally, use the global planning algorithm to calculate the local path target point Pos * ,in,

[0054] The global path planning algorithm used is Dijkstra. Each vertex of the weighted graph is given a number, which is called the vertex label: the temporary label T represents the upper bound of the shortest path length from the starting vertex to the vertex, and the fixed label P represents the shortest path length from the starting vertex to the vertex. The steps are as follows:

[0055] (1) Give vertex v1, label P d(v1) = 0, give vertex v j (j=2,3,……,n)T label d(v j )=l 1j ; j is an integer;

[0056] (2) Take the minimum value among all T labels, for example, d(v j0 )=l 1j0 , then v j0 Change the T label of the vertex to P label, and recalculate the T labels of other vertices with T labels: Select vertex v j T label d(v j ) and d(v j0 )+l j0j The smaller one is v j The new T label;

[0057] (3) Repeat the above steps until the label of the target vertex is changed to P and the entire global optimal path is obtained;

[0058] According to the local cost map range, the global optimal path is intercepted to obtain the end point of the local optimal path, which is Pos * .

[0059] Optionally, training the reinforcement learning neural network includes:

[0060] Define the operating state S t {x(t),y(t),θ(t),dis(t)}, action decision a t {v,w}, reward r t {G(t)};

[0061] (1) Building the strategy network and value network of the reinforcement learning neural network, randomly initializing the neural network parameters, and establishing a robot obstacle avoidance simulation environment based on the robot mathematical model. The robot obstacle avoidance simulation environment outputs action a through the reinforcement learning agent. t , feedback reward r t and the state S at the next moment t+1 ;

[0062] (2) Simulation calculation loads the trained policy network, generates action, reward and state data sets through continuous iteration, and establishes an action-state experience pool and sets the capacity of the action-state experience pool to store training experience;

[0063] (3) Initialize the target network of the reinforcement learning policy network and the value network,

[0064] It includes a policy network and two value networks, denoted as θ targ ,

[0065] The target network is used to solve the overestimation problem that occurs during training;

[0066] (4) Adding action noise to help the reinforcement learning neural network find the global optimal solution;

[0067] (5) Determine whether the action-state experience pool is full. If it is full, use the gradient down

[0068] Update value network parameters by descending method

[0069]

[0070] Where: y(r t ,s t+1 ) represents the target value of the network; represents the gradient of the loss function; B represents the number of experience sequences extracted from the experience pool; α represents the learning rate of the value network;

[0071] (6) Delayed update of the action network through the gradient ascent algorithm:

[0072]

[0073] Among them, β represents the learning rate of the action network;

[0074] (7) Update target network parameters:

[0075] φ targ，i ←ρφ targ,i +(1-ρ)φ i for i=1,2

[0076] θ targ ←ρθ targ +(1-ρ)θ

[0077] (8) Stop when the number of training times is reached.

[0078] Optional, robot obstacle avoidance neural network model training, also includes:

[0079] In other actual operating environments, the reinforcement learning neural network is trained, wherein the training process is implemented according to steps (5)-(8) in training the reinforcement learning neural network.

[0080] In order to overcome the shortcomings of deep reinforcement learning and realize the efficient obstacle avoidance operation of wheeled robots through reinforcement learning, Figure 2As shown, this embodiment can first design a random elastic reward mechanism, calculate the reward value triggered by each action according to the running state fed back by the robot, record all the decision values ​​of each action of the machine during the obstacle avoidance process, the running state value fed back and the corresponding reward value, and form a random reward pool, and then randomly extract one set of data as the input and output of the neural network, train the policy network and action network of the reinforcement learning neural network, and speed up the training speed of reinforcement learning. At the same time, in order to avoid the algorithm from falling into the local optimum, random noise of the reinforcement learning action (decision variable) can also be appropriately introduced to explore the global optimum. The method first uses the simulation environment to calculate the random elastic reward, performs initial training on the reinforcement learning network, and then obtains the real-time data of the robot state during real-time operation, feeds back to the deep reinforcement learning and performs real-time training, which can realize the online update of the model parameters, making the obstacle avoidance result more accurate. The method can effectively improve the efficiency and practicality of reinforcement learning obstacle avoidance by designing a special reward mechanism, and at the same time, the method accelerates the convergence of reinforcement learning.

[0081] The reinforcement learning wheeled robot fast obstacle avoidance method based on random elastic rewards, with the help of the "precise experience" of random elastic rewards and the appropriate increase of action noise during the training process, the policy network can achieve fast and efficient convergence, and can better approach the global optimal obstacle avoidance effect. Secondly, the deep neural network can quickly give the corresponding relationship between the continuous state of the system and the execution action. Compared with the traditional numerical optimization obstacle avoidance algorithm, it can better solve the efficient obstacle avoidance requirements of complex scenes. At the same time, it can continuously use real-time operation data to cyclically update the reinforcement learning obstacle avoidance model, making obstacle avoidance more and more accurate.

[0082] The training method of the robot obstacle avoidance neural network model mainly includes the following steps: 1. A random elastic reward mechanism is designed according to the robot structure and usage environment characteristics;

[0083] 2. Establish mathematical model and simulation environment for robot obstacle avoidance;

[0084] 3. Establish the structure of the robot reinforcement learning network and randomize the initial parameters;

[0085] 4. Use random actions to test the effectiveness of the simulation environment and reward mechanism, and adjust network parameters and reward parameters;

[0086] 5. Use the simulation environment to preliminarily train the reinforcement learning network parameters, form a random reward pool, and continuously update the reward pool and the reinforcement learning network parameters;

[0087] 6. Retrain the reinforcement learning network parameters in the real-time running environment to enable the robot to form a more accurate obstacle avoidance neural network;

[0088] 7. In different operating environments, continuously optimize and improve network parameters to enable the robot obstacle avoidance network to have better generalization capabilities.

[0089] The method of this embodiment first designs a random elastic reward mechanism to calculate the reward value triggered by each action of the robot, establishes a robot mathematical model and an obstacle avoidance simulation environment, and uses the simulation environment to preliminarily train the robot's obstacle avoidance deep reinforcement learning network. In order to avoid the algorithm from falling into a local optimum, random noise of the reinforcement learning action (decision variable) is appropriately introduced, and the reinforcement learning network is trained again in the actual environment to obtain a more efficient and accurate obstacle avoidance strategy. In the actual use process, the obstacle avoidance parameters are continuously optimized and trained, so that the obstacle avoidance algorithm can continuously adapt to the new environment and improve the generalization ability of the network.

[0090] Figure 3 A schematic diagram of a robot obstacle avoidance neural network model training device provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the robot obstacle avoidance neural network model training device includes: a random reward pool module formation 310, a reinforcement learning neural network first training module 320 and a reinforcement learning neural network second training module 330, wherein:

[0091] A random reward pool module is formed 310, which is used to record the action decisions, feedback operation status and corresponding rewards generated by the robot in the obstacle avoidance process based on the preset random elastic reward mechanism and reinforcement learning neural network in the preset robot obstacle avoidance simulation environment, and form a random reward pool;

[0092] A first reinforcement learning neural network training module 320 is used to utilize the random reward pool and add random noise of reinforcement learning actions to jointly train the reinforcement learning neural network and update network parameters;

[0093] The second training module 330 of the reinforcement learning neural network is used to train the reinforcement learning neural network in an actual operating environment based on the random elastic reward mechanism according to the action decision values ​​and feedback operating state values ​​generated by the robot during the obstacle avoidance process to obtain the robot's obstacle avoidance neural network model.

[0094] Optional, robot obstacle avoidance neural network model training device, including:

[0095] The evaluation function setting module is used to set the evaluation function of the random elastic reward mechanism.

[0096] Optional, evaluation function setting module, specifically used for:

[0097] Set the following function as the evaluation function of the random elastic reward mechanism:

[0098] G(v,w)=δ[k1(Theta(v,w)-Theta * )+k2|Pos(v,w)-

[0099] Pos * |+k3vel(v,w)+k4dis(v,w)]

[0100] Where: δ is the coefficient of the random elastic function, (v, w) is the robot's forward speed and rotation speed; Theta(v, w) is the azimuth function, which is used to evaluate the robot's predicted trajectory end direction and target point orientation at the current set speed. * The angle difference between them; Pos(v,w) is the position of the robot at the end point of the predicted trajectory, and its position relative to the local path target point Pos * vel(v,w) is the current linear velocity of the robot; dis(v,w) is the distance between the robot and the nearest obstacle on the map when the robot is at the end point of the predicted trajectory;

[0101] It also includes adjusting the parameters in the evaluation function of the random elastic reward mechanism during the training process of the reinforcement learning neural network, wherein the δ value remains unchanged during a single obstacle avoidance training, and a random value is obtained using a random function when the next obstacle avoidance training begins; and optimizing and adjusting k1, k2, k3, and k4 based on the time it takes for the robot to reach the target point and the effectiveness of obstacle avoidance.

[0102] Optionally, the robot obstacle avoidance neural network model training device also includes:

[0103] The robot mathematical model setting module is used to set the robot mathematical model, wherein:

[0104] Taking the differential robot as an example, the motion coordinate calculation formula is:

[0105]

[0106] θ t =θ t +wΔt

[0107] Where: x is the distance the robot moves in the x-axis direction in the map coordinate system; y is the distance the robot moves in the y-axis direction in the map coordinate system; θ is the rotation angle of the robot around the rotation coordinate axis z perpendicular to the robot; v is the forward speed of the robot; w is the rotation speed of the robot; Δt is the model calculation step time.

[0108] When the robot generates a trajectory during obstacle avoidance, the size of the dynamic window is at least affected by the linear acceleration, angular acceleration and obstacles of the motor, and the constraints are as follows:

[0109] The robot is limited by its own maximum and minimum speeds:

[0110] v m ={(v,w)|v∈[v min ,v max ],w∈[w min ,w max ]}

[0111] Where: v min ,v max are the upper and lower limits of the robot's forward speed respectively; w min ,w max The upper and lower limits of the robot's rotation speed.

[0112] The robot is affected by the motor performance: there is a maximum acceleration and deceleration limit. During the forward simulation cycle of the robot's predicted trajectory, the target speed is the speed v that the robot can actually reach. d ={(v,w)|v∈[v c -v b Δt,v c +v a Δt],w∈[w c -w b Δt,w c +w a Δt]}Where: v d v is the robot's target forward speed; c is the current forward speed of the robot; v b v is the maximum forward speed difference that the robot can reduce within Δt time; a w is the maximum forward speed difference that the robot can increase within the time Δt; c is the current rotation speed of the robot; w b is the maximum rotation speed difference that can be reduced within the robot's Δt time; w a It is the maximum rotation speed difference that can be increased within the robot's Δt time.

[0113] Limitations based on mobile robot safety: Under maximum deceleration conditions, speed range:

[0114]

[0115] Optionally, the robot obstacle avoidance neural network model training device also includes:

[0116] The local path target point calculation module is used to calculate the local path target point Pos using the global planning algorithm * ,in,

[0117] The global path planning algorithm used is Dijkstra. Each vertex of the weighted graph is given a number, which is called the vertex label: the temporary label T represents the upper bound of the shortest path length from the starting vertex to the vertex, and the fixed label P represents the shortest path length from the starting vertex to the vertex. The steps are as follows:

[0118] (1) Give vertex v1, label P d(v1) = 0, give vertex v j (j=2,3,……,n)T label d(v j )=l 1j ;

[0119] (2) Take the minimum value among all T labels, d(v j0 )=l 1j0 , then v j0 Change the T label of the vertex to P label, and recalculate the T labels of other vertices with T labels: Select vertex v j T label d(v j ) and d(v j0 )+l j0j The smaller one is v j The new T label;

[0120] (3) Repeat the above steps until the label of the target vertex is changed to P and the entire global optimal path is obtained;

[0121] According to the local cost map range, the global optimal path is intercepted to obtain the end point of the local optimal path, which is Pos * .

[0122] Optionally, the reinforcement learning neural network second training module 330 is specifically used for:

[0123] Define the operating state S t {x(t),y(t),θ(t),dis(t)}, action decision a t {v,w}, reward r t {G(t)};

[0124] (1) Building the strategy network and value network of the reinforcement learning neural network, randomly initializing the neural network parameters, and establishing a robot obstacle avoidance simulation environment based on the robot mathematical model. The robot obstacle avoidance simulation environment outputs action a through the reinforcement learning agent. t , feedback reward r t and the state S at the next moment t+1 ;

[0125] (2) Simulation calculation loads the trained policy network, generates action, reward and state data sets through continuous iteration, and establishes an action-state experience pool and sets the capacity of the action-state experience pool to store training experience;

[0126] (3) Initialize the target network of the reinforcement learning policy network and value network, including one policy network and two value networks, denoted as θ targ , The target network is used to solve the overestimation problem that occurs during training;

[0127] (4) Adding action noise to help the reinforcement learning neural network find the global optimal solution;

[0128] (5) Determine whether the action-state experience pool is full. If so, update the value network parameters using the gradient descent method.

[0129]

[0130] Where: y(r t ,s t+1 ) represents the target value of the network; represents the gradient of the loss function; B represents the number of experience sequences extracted from the experience pool; α represents the learning rate of the value network;

[0131] (6) Delayed update of the action network through the gradient ascent algorithm:

[0132]

[0133] Among them, β represents the learning rate of the action network;

[0134] (7) Update target network parameters:

[0135] φ targ，i ←ρφ targ，i +(1-ρ)φ i for i=1,2

[0136] θ targ ←ρθ targ +(1-ρ)θ

[0137] (8) Stop when the number of training times is reached.

[0138] Optional, robot obstacle avoidance neural network model training, also includes:

[0139] In other actual operating environments, the reinforcement learning neural network is trained, wherein the training process is implemented according to steps (5)-(8) in training the reinforcement learning neural network.

[0140] The robot obstacle avoidance neural network model training device provided in the embodiment of the present invention can execute the robot obstacle avoidance neural network model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0141] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the electronic device includes a processor 410, a memory 420, an input device 430, and an output device 440; the number of the processor 410 in the electronic device can be one or more. Figure 4 A processor 410 is taken as an example; the processor 410, the memory 420, the input device 430 and the output device 440 in the electronic device can be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0142] The memory 420, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the robot obstacle avoidance neural network model training method in the embodiment of the present invention (for example, the random reward pool module formation 310, the reinforcement learning neural network first training module 320 and the reinforcement learning neural network second training module 330 in the robot obstacle avoidance neural network model training device). The processor 410 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 420, that is, realizes the above-mentioned robot obstacle avoidance neural network model training method.

[0143] The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 420 may further include a memory remotely arranged relative to the processor 410, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The input device 430 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 440 may include a display device such as a display screen.

[0145] An embodiment of the present invention further provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to execute a robot obstacle avoidance neural network model training method, including:

[0146] In the preset robot obstacle avoidance simulation environment, the robot's action decisions, feedback of operating status and corresponding rewards generated during the obstacle avoidance process are recorded based on the preset random elastic reward mechanism and reinforcement learning neural network, and a random reward pool is formed;

[0147] Using the random reward pool and adding random noise of reinforcement learning actions, jointly training the reinforcement learning neural network and updating network parameters;

[0148] In an actual operating environment, based on the random elastic reward mechanism and the obtained action decision values ​​and feedback operating state values ​​generated by the robot during the obstacle avoidance process, the reinforcement learning neural network is trained to obtain the robot's obstacle avoidance neural network model.

[0149] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present invention is not limited to the method operations described above, and can also execute related operations in the robot obstacle avoidance neural network model training method provided in any embodiment of the present invention.

[0150] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0151] It is worth noting that in the embodiment of the above-mentioned robot obstacle avoidance neural network model training device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0152] Although the present invention has been described in detail above by means of general description, specific implementation methods and tests, it is obvious to those skilled in the art that some modifications or improvements may be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of protection claimed by the present invention.

Claims

1. A robot obstacle avoidance neural network model training method, characterized in that: include: In the preset robot obstacle avoidance simulation environment, the robot's action decisions, feedback of operating status and corresponding rewards generated during the obstacle avoidance process are recorded based on the preset random elastic reward mechanism and reinforcement learning neural network, and a random reward pool is formed; Using the random reward pool and adding random noise of reinforcement learning actions, jointly training the reinforcement learning neural network and updating network parameters; In an actual operating environment, based on the random elastic reward mechanism and the obtained action decision values ​​and feedback operating state values ​​generated by the robot during the obstacle avoidance process, the reinforcement learning neural network is trained to obtain the robot's obstacle avoidance neural network model.

2. The method according to claim 1, characterized in that Setting the random elastic reward mechanism includes: Set the random elastic reward mechanism evaluation function.

3. The method according to claim 2, characterized in that The setting of the random elastic reward mechanism evaluation function includes: Set the following function as the evaluation function of the random elastic reward mechanism: G(v,w)=δ[k1(Theta(v,w)-Theta * )+k2|Pos(v,w)- Pos * |+k3vel(v,w)+k4dis(v,w)] Where: δ is the coefficient of the random elastic function, (v, w) is the robot's forward speed and rotation speed; Theta(v, w) is the azimuth function, which is used to evaluate the robot's predicted trajectory end orientation and target point orientation at the current set speed. * The angle difference between them; Pos(v,w) is the position of the robot at the end point of the predicted trajectory, and its position relative to the local path target point Pos * vel(v,w) is the current linear velocity of the robot; dis(v,w) is the distance between the robot and the nearest obstacle on the map when the robot is at the end point of the predicted trajectory; It also includes adjusting the parameters in the evaluation function of the random elastic reward mechanism during the training process of the reinforcement learning neural network, wherein the δ value remains unchanged during a single obstacle avoidance training, and when the next obstacle avoidance training begins, a random value with given upper and lower bounds is generated using a random function; based on the time it takes for the robot to reach the target point and the effectiveness of obstacle avoidance, k1, k2, k3, and k4 are optimized.

4. The method according to claim 3, characterized in that Also includes: Set up a robot mathematical model, where The calculation formula of robot motion coordinates is: i t =θ t +wΔt Where: x is the distance the robot moves in the x-axis direction in the map coordinate system; y is the distance the robot moves in the y-axis direction in the map coordinate system; θ is the rotation angle of the robot around the rotation coordinate axis z perpendicular to the robot; v is the forward speed of the robot; w is the rotation speed of the robot; Δt is the model calculation step time; When the robot generates a trajectory during obstacle avoidance, the size of the dynamic window is at least affected by the linear acceleration, angular acceleration and obstacles of the motor, and the constraints are as follows: The robot is limited by its own maximum and minimum speeds: v m ={(v,w)|v∈[v min ,v max ],w∈[w min ,w max ]} Where: v min 、v max are the upper and lower limits of the robot's forward speed respectively; w min 、w max The upper and lower limits of the robot's rotation speed; The robot is affected by the motor performance: there is a maximum acceleration and deceleration limit. During the forward simulation cycle of the robot's predicted trajectory, the target speed is the speed v that the robot can actually reach. d ={(v,w)|v∈[v c -v b Δt,v c +v a Δt],w∈[w c -w b Δt,w c +w a Δt]}Where: v d v is the robot's target forward speed; c is the current forward speed of the robot; v b v is the maximum forward speed difference that the robot can reduce within Δt time; a w is the maximum forward speed difference that the robot can increase within the time Δt; c is the current rotation speed of the robot; w b is the maximum rotation speed difference that can be reduced within the robot's Δt time; w a The maximum rotation speed difference that can be increased within the robot's Δt time; Limitations based on mobile robot safety: Under maximum deceleration conditions, speed range:

5. The method according to claim 3, characterized in that: Use the global planning algorithm to calculate the local path target point Pos * ,in, The global path planning algorithm used is Dijkstra. Each vertex of the weighted graph is given a number, which is called the vertex label: the temporary label T represents the upper bound of the shortest path length from the starting vertex to the vertex, and the fixed label P represents the shortest path length from the starting vertex to the vertex. The steps are as follows: (1) Give vertex v1, label P d(v1) = 0, give vertex v j , j=2,3,……,n, T mark d(v j )=l 1j ; (2) Take the minimum value among all T labels, d(v j0 )=l 1j0 , then v j0 Change the T label of the vertex to P label, and recalculate the T labels of other vertices with T labels: Select vertex v j T label d(v j ) and d(v j0 )+l j0j The smaller one is v j The new T label; (3) Repeat the above steps until the label of the target vertex is changed to P and the entire global optimal path is obtained; According to the local cost map range, the global optimal path is intercepted to obtain the end point of the local optimal path, which is Pos * .

6. The method according to claim 4, characterized in that Training the reinforcement learning neural network includes: Define the operating state S t {x(t),y(t),θ(t),dis(t)}, action decision a t {v,w}, reward r t {G(t)}; (1) Building the strategy network and value network of the reinforcement learning neural network, The neural network parameters are randomly initialized, and a robot obstacle avoidance simulation environment is established according to the robot mathematical model. The robot obstacle avoidance simulation environment outputs action a through reinforcement learning agent t , feedback reward r t and the state S at the next moment t+1 ; (2) Simulation calculation loads the trained policy network, generates action, reward and state data sets through continuous iteration, and establishes an action-state experience pool and sets the capacity of the action-state experience pool to store training experience; (3) Initialize the target network of the reinforcement learning policy network and value network, including one policy network and two value networks, denoted as θ targ , The target network is used to solve the overestimation problem that occurs during training; (4) Adding action noise to help the reinforcement learning neural network find the global optimal solution; (5) Determine whether the action-state experience pool is full. If so, update the value network parameters using the gradient descent method. Where: y(r t ,s t+1 ) represents the target value of the network; represents the gradient of the loss function; B represents the number of experience sequences extracted from the experience pool; α represents the learning rate of the value network; (6) Delayed update of the action network through the gradient ascent algorithm: Among them, β represents the learning rate of the action network; (7) Update target network parameters: f targ,i ←rf targ，i +(1-r)φ i for i=1,2 i targ ←ρθ targ +(1-p)θ (8) Stop when the number of training times is reached.

7. The method according to claim 6, characterized in that Also includes: In other actual operating environments, the reinforcement learning neural network is trained, wherein the training process is implemented according to steps (5)-(8) in claim 6.

8. A robot obstacle avoidance neural network model training device, characterized in that: include: The random reward pool module is formed to record the action decisions, feedback operation status and corresponding rewards generated by the robot in the obstacle avoidance process based on the preset random elastic reward mechanism and reinforcement learning neural network in the preset robot obstacle avoidance simulation environment, and form a random reward pool; A first training module of a reinforcement learning neural network is used to utilize the random reward pool and add random noise of reinforcement learning actions to jointly train the reinforcement learning neural network and update network parameters; The second training module of the reinforcement learning neural network is used to train the reinforcement learning neural network in an actual operating environment based on the random elastic reward mechanism according to the action decision values ​​and feedback operating state values ​​generated by the robot during the obstacle avoidance process to obtain the robot's obstacle avoidance neural network model.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the robot obstacle avoidance neural network model training method as described in any one of claims 1-7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions, when executed by a computer processor, are used to execute the robot obstacle avoidance neural network model training method as described in any one of claims 1-7.

Images