Differentiated task-oriented robot deep reinforcement learning motion planning method

Abstract

The invention relates to a robot deep reinforcement learning motion planning method for differentiated tasks. The working environment of a robot is reconstructed and analyzed in real time through a fast reconstruction neural network (FRNN) in a multi-angle visual sampling system. A robot carrying a dynamic entropy performer-commentator (DEAC) and a teaching-guidance mechanism (TG) trains trajectory planning in a virtual scene to complete a motion planning task, and when the scene changes, the proposed PP and IKM are activated to help the robot to quickly adapt to a new environment and meet optimal targets of trajectory and energy consumption at the same time. The problems that at present, three-dimensional space scene information is difficult to obtain, and the robot has the capacity of autonomously learning motion planning and rapidly adapting to unknown changes in the environment are solved, the robot can autonomously learn a motion planning strategy, when the environment changes, the planning strategy can be adjusted in real time, manual intervention is not needed, and the working efficiency is improved. And the task execution robustness and the working efficiency of the robot are improved.

Description

Technical Field

[0001] The present invention relates to a three-dimensional reconstruction and robot motion planning technology, and particularly to a robot deep reinforcement learning motion planning method for different tasks. Background Art

[0002] In the current era, the application of robot technology is booming and plays a key role in all aspects and fields. In the process of production and manufacturing, robots are indispensable key components. The application of robot technology can greatly simplify the originally time-consuming and laborious production activities, and greatly improve the production accuracy and efficiency. In addition, with advanced programming and control systems, they can perform repetitive operation tasks with extremely high accuracy.

[0003] Artificial intelligence technology is an emerging field of modern development and an important trend for future development. Artificial intelligence technology can endow robots with the ability to autonomously execute and implement tasks, greatly saving labor costs and helping people complete complex and high-precision operations. Deep reinforcement learning is an important branch in the field of artificial intelligence. Robots equipped with deep reinforcement learning can, without human intervention, analyze the observed state of the task through interaction with the environment and autonomously learn strategies to solve problems. This allows robots to help people improve operation efficiency in complex and unknown environments, and improve the production efficiency and quality of the industry.

[0004] In operations, the original deep reinforcement learning algorithm needs a large amount of exploration to obtain sufficient data to learn feasible strategies when dealing with high-dimensional robot tasks. Worse still, in complex environments or tasks, robot tasks often contain many criteria that need to be optimized, which greatly increases the training difficulty of the original reinforcement learning algorithm. Moreover, in a variable environment, the original deep reinforcement learning usually needs to be retrained according to the changed environment, which will lead to low task efficiency of the robot and even forced stagnation.

[0005] In addition, the reconstruction of a three-dimensional scene can provide key observation information for robots equipped with deep reinforcement learning. These information are indispensable factors for robots to recognize the environment and learn how to complete tasks. It is difficult to identify the accurate size of three-dimensional objects in the pictures taken by ordinary cameras, so it is difficult to extract and reconstruct the scene in three-dimensional space simply from two-dimensional plane photos. Moreover, although depth cameras or binocular cameras can capture the distance of the positions of objects in the environment, when there are multiple cameras, this relative numerical information is difficult to be analyzed and processed combinatorially to restore the true form of spatial objects. In most cases, objects in space will be mutually occluded and self-occluded, and directly distinguishing the form of objects from the numerical values of such depth images easily leads to errors or even invalidity.

[0006] Therefore, it is an urgent need in the current multi-field applications of artificial intelligence and robotics to explore methods that can endow robots with the ability to quickly adapt to these differences in a variable environment while meeting these optimization goals. Summary of the Invention

[0007] Aiming at the problems of difficult acquisition of three-dimensional space scene information, lack of the ability of robots to have autonomous learning motion planning and quickly adapt to unknown changes in the environment, a robot deep reinforcement learning motion planning method for differentiated tasks is proposed.

[0008] The technical solution of the present invention is as follows:

[0009] A robot deep reinforcement learning motion planning method for differentiated tasks, including the following steps:

[0010] S1. Through a simulation platform, a robot 3D model, a robot motion planning experimental scene, and a virtual-real real-time operation system are respectively constructed and associated as a real-time simulation training environment;

[0011] S2. A three-dimensional reconstruction algorithm FRNN is constructed through a fast reconstruction neural network based on a long short-term memory network. An omnidirectional visual sampling system is established through the three-dimensional reconstruction algorithm FRNN. Information on the real environment is obtained through the omnidirectional visual sampling system. In the virtual-real real-time operation system, an experimental scene that constructs and restores the real environment is configured; the distance and collision detection parameters between the robot and the obstacle are configured, and these parameters are transmitted to the teaching-guidance TG mechanism at the same time;

[0012] S3. In the virtual environment, a dynamic entropy actor-critic DEAC algorithm based on deep reinforcement learning combined with the teaching-guidance TG mechanism, namely DEAC-TG, is used to train the motion planning model of the robot to complete the robot motion planning task;

[0013] S4. When the real environment changes, the omnidirectional visual sampling system is used to detect the information of the changes occurring in the real environment and transmit it to the virtual-real real-time operation system, so that the virtual environment information and the real environment information in the system are synchronized in real time;

[0014] S5. If the virtual environment changes, the progressive planner PP and the inverse kinematics mapper IKM are activated; among them, PP is used to explore the optimal path in the changed area, and IKM maps this path from the three-dimensional space to the joint space of the robot;

[0015] S6. When the robot explores a feasible trajectory in the changed environment, the experience data related to the robot will be stored in the experience replay library, and this data will be used in the update after DEAC with the TG mechanism, namely DEAC-TG; through the experience data in these diverse environments, the finally obtained robot motion planning model can be generalized in diverse scenarios and can quickly adapt to new and different environments.

[0016] Further, in step S1, the real-time simulation training environment includes:

[0017] A robot: The robot is a 7-degree-of-freedom redundant robot located in the working environment, responsible for grasping an object block from the starting grasping area and then placing it in the target placement area, without colliding with obstacles midway, and completing the motion planning task;

[0018] The starting grasping area: The object can appear at random positions in this area;

[0019] The target placement area: The target placement position will be randomly designated in this area;

[0020] Randomly changing obstacles: In the working space of the robot, the obstacles in the scene will randomly appear or change;

[0021] The multi-angle visual sampling system: Multiple depth cameras are placed directly above and around the scene to form an image acquisition array, and the real-time image data collected is processed by FRNN and transmitted to the virtual-real real-time operation system to synchronize the virtual scene with the real scene;

[0022] The virtual-real real-time operation system: A centralized processing system is deployed on a computer group composed of several computers, which can interrelate the virtual scene, engineering files, camera array, and real scene, realize real-time data transmission and processing, and overall operation and management.

[0023] Further, in step S1, through FRNN to reconstruct and extract environmental information, a 1:1 restored virtual environment is built in CoppeliaSim, and is associated with the environment in CoppeliaSim through PyRep; several object blocks randomly appear in the starting grasping area, and the robot is trained to move from the initial pose to the grasping pose, grasp an object block, bypass the obstacles, and place this object block at a random target position; during this grasping and placing process, the virtual environment and the real environment remain in real-time synchronization; and the environment will change, and the robot quickly adapts to the environmental changes to complete new grasping and placing tasks; this process will be looped and repeated until all object blocks are transferred from the starting grasping area to the target placement area.

[0024] Further, in step S2, the FRNN algorithm is implemented as follows:

[0025] In the FRNN algorithm, a three-dimensional convolutional long short-term grid is set, and its equation:

[0026]

[0027] h t = tanh(e t ) + o t ☉ tanh(e t )

[0028] In the formula, x t represents the input at time t, * represents convolutional multiplication; v t , u t , o t represent the input gate, output gate, and forget gate respectively; e t and h t represent the memory element and the hidden layer respectively; W and U represent weights, b represents the bias, and tanh is the hyperbolic tangent function; i is denoted as the input gate, f is denoted as the forget gate, and 0 is denoted as the output gate;

[0029] The loss function of the network is defined as the sum of voxel cross-entropies, and the final output of each voxel (ι, η, ζ) is a Bernoulli distribution This loss function L FRNN is:

[0030]

[0031] In the formula, is a set, t is the current time, and T represents the total number of time steps; represents the true occupancy ratio; p(ι, η, ζ) represents the success ratio.

[0032] Further, step S3 is specifically implemented as follows:

[0033] In the virtual environment associated with PyRep and CoppeliaSim, DEAC-TG represents the DEAC algorithm equipped with the TG mechanism. Among them, TG is used to further process and optimize the successful data obtained by DEAC exploration in the early stage of training, and store these optimized data; in the middle stage of training, these optimized data are added as additional teacher data to the policy update process of DEAC to promote the efficiency of policy update; then, in the later stage of training, these optimized data are removed to ensure that the policy obtained after DEAC update is unbiased; the designed DEAC-TG algorithm is used to train the robot so that the robot autonomously learns the motion planning strategy;

[0034] Determine the definition of the robot's state and action parameters during training, where the state s is defined as:

[0035] s={θ,|pp target |,oo target |}

[0036] Where θ={θ1,θ2,...,θ D} is the set of joint angles of the robot, D is the total number of robot joints; |pp target | represents the current position p of the robot end and the target position p target The absolute value of the difference, |oo target | represents the current robot end posture o and the target posture o target The absolute value of the difference;

[0037] Action a is defined as:

[0038] a=Δθ={Δθ1,Δθ2,...,Δθ D}

[0039] where Δθ is the set of increments of the robot joint angles;

[0040] An attempt is divided into G rounds, each round has T moments; during the interaction between the robot and the environment, there is a state s at each moment t, that is, s t To describe the interaction between the robot and the environment at the current moment, the robot will receive the reward r at the current moment, that is, r t ; Status s t It is used as the input of the policy network, and then an action a is derived according to the policy network, that is, a t ; The next state of the robot after performing this action is represented as s t+1 ;

[0041] The deep reinforcement learning algorithm DEAC contains two experience replay libraries, namely the long-term experience replay library and the teacher experience replay library. The long-term experience replay library is used to store the experience data of the entire experimental process, and the teacher experience replay library is used to store the data of simplified trajectories.

[0042] DEAC adopts an actor-critic framework, which includes one actor network, two critic networks, one target actor network, and two target critic networks. The target actor network is a copy of the actor network several time ago and does not participate in updates. The target critic network is a copy of the critic network several time ago and does not participate in updates.

[0043] Experience data tuple (s t ,a t ,rt , s t+1 ) is stored in the long - term experience replay library; s t and a t are used as the input of the critic network; the update function of the critic network is:

[0044]

[0045] where represents the expectation, generically refers to the experience replay library, Q is the output of the critic network, r t+1 is the reward, τ is the decay factor, can take a value as an element in the constant set {1, 2}; Q target is the output of the target critic network; α is the temperature coefficient, which is used to assist in adjusting the value of entropy, S λ represents the th function in the entropy cluster function S;

[0046] then r t is used for the update of the actor network, s t+1 is the input of the target critic network, a t+1 is the output of the target policy network, and at the same time, s t is the input of the actor network, and its output is a t ; the update function of the actor network is:

[0047]

[0048] In this way, the actor network performs gradient update through observing the state s t and the output value Q of the critic network, and the critic network is updated according to the changes of the state s t , the output action a of the actor network t and the reward r t+1 ; at the same time, the input of the target actor network is s t+1 , its output a t+1 and the input s t+1 are used together as the input of the target critic network; then, the target critic network provides the output value Q target for the update of the critic network;

[0049] DEAC - TG is used to train the robot to interact with the environment, and in this process, it learns the optimal policy according to the above settings to complete the motion planning task;

[0050] Determine the trajectory progressive simplification algorithm, which is defined as follows:

[0051]

[0052] In the formula, at，t+n is an estimation action, a max is the maximum amplitude value of the action, is the action space;

[0053] From the state s at time t t attempts the action a with a time interval of n t，t+n , when a t，t+n exists in the action space , the state s at time t t will be directly bridged to the state s at time t + n by the action a t，t+n , and the generated experience data tuple (s t，t+n , a t , r t , s t , s t+n ) is stored in the teacher experience replay library, and the original experience data tuple (s t , a t , r t , s t+1 ) will be deleted from the long - term experience replay library to prevent confusion in state transitions;

[0054] Determine the calculation methods of the path and energy consumption:

[0055] The path length is denoted as L, and the energy consumption is represented by E:

[0056]

[0057] In the formula, ω represents the weight of the energy consumption of the robot joints; L and E are the path length of the robot end - effector and the energy consumption of the trajectory respectively, m ∈ {1, 2,..., M} represents the serial number of the discrete nodes of the robot joint angles or end - effector positions, and M is the total number of these nodes;

[0058] Determine the reward function r as:

[0059]

[0060] In the formula, the reward function r consists of two parts, namely the main reward function f and the additional reward function g. In the main reward function f, the position and orientation of the robot end - effector are inversely correlated with the position and orientation of the target. In addition, the additional reward function g is used to adjust the reward distribution, and the additional reward function g includes energy estimation;

[0061] Determine the reward function f(t), which is the main reward function that guides the robot to learn how to complete the task, and it is defined as:

[0062]

[0063] where A is a constant matrix; represents the distance between the robot body and the obstacle, is the maximum collision area distance considered; C obs and C col are sign functions, where if then C obs > 0, otherwise C obs = 0; if a collision occurs then C col < 0, otherwise C col = 0; ′ is the transpose symbol;

[0064] Determine the additional reward function g(t); when the robot explores a trajectory that can reach the target, this trajectory is recorded as a valuable trajectory, and this reward function is used to adjust the reward distribution in the environment. This additional reward is defined as:

[0065]

[0066] where represents the set of current robot joint angles θ t and the set of joint angles θ val on the valuable trajectory; clip(·) represents a numerical clipping function; C flag is a sign function, where if a valuable trajectory is explored then C flag > 0, otherwise C flag = 0; L max and E max are the maximum values of the path length and energy consumption respectively;

[0067] When a valuable trajectory is explored, the trajectory progressive simplification algorithm in DEAC-TG is used to simplify the state transition pairs (s t , s t+1 ) in this trajectory, and the simplified state transition pairs are put into the teacher experience replay library. During training, the experiences in the long-term experience replay library and the teacher experience replay library are sampled proportionally as training data; moreover, this ratio changes as the number of training rounds increases, that is, the experience data sampled from the teacher experience replay library will become less and less until it becomes 0, and the experience data sampled in the later stage of training all comes from the long-term memory library; after the ratio of the teacher experience data being sampled becomes 0, the guidance module is activated, that is, C flag > 0, the additional function g(t) takes effect, and the reward distribution is adjusted.

[0068] Further, step S4 is specifically implemented as follows: During the operation of the task, the environment changes; the FRNN algorithm in the multi-angle vision sampling system extracts and analyzes the environmental information, and transmits this environmental data to the virtual-real real-time operation system to synchronize the environmental changes in PyRep and CoppeliaSim.

[0069] Further, step S5 is specifically implemented as follows:

[0070] In the changed area, the surroundings of these obstacles are surrounded by meshed and randomized nodes, and every three nodes among these scattered points can be connected to form a triangle in three-dimensional space. The Voronoi diagram in PP traverses this set of triangles, calculates the circumcenters of each triangle, and saves them; assume the current triangle is Δ current , find the triangles Δ1, Δ2, and Δ3 that share sides with the three sides of Δ current . If these three co-edge triangles exist, that is, within the feasible area, then connect the circumcenters of these triangles with the circumcenter of Δ current and save them as the edges of the Voronoi diagram; if these triangles do not exist, then find the intersection points of the perpendicular bisector rays corresponding to each side of Δ current with the edge of the plannable space, and connect the circumcenter of Δ current to these intersection points and save them as the edges of the Voronoi diagram;

[0071] The set of nodes at both ends of the edges of the Voronoi diagram is used as the candidate node set, and the candidate node set is explored using the A-star path planning algorithm to find the path with the optimal path length; A-star contains an Open set and a Close set. The Open set contains candidate nodes, and the Close set will gradually be filled with feasible nodes until the path is found; during initialization, the starting node is loaded from the Open set into the Close set, and the distances to the reachable nodes adjacent to the starting node p start are calculated and updated, and the node closest to the starting node is selected according to the distance, assumed to be p near , which is deleted from the Open set and then added to the Close set; check one by one the path lengths from p start to p near and then to other nodes. If the path length from p start through p near to other nodes is the smallest, then update the parent node of the shortest path from p start to this node to be p near , otherwise do not update; from the nodes adjacent to p nearSample nodes one by one from the candidate nodes and update their parent nodes; until the candidate nodes that can reach the end point are traversed and the end point is added to the Close set, the path with the shortest length, i.e., the shortest path, is found;

[0072] The shortest path in the Cartesian space obtained by PP is represented as a set of nodes B = {b1, b2,... b M}, where M is the total number of nodes; the designed IKM is used to map this shortest path into the joint space of the robot;

[0073] Determine a mapper based on an artificial neural network, and the pose T of the robot m can be represented as:

[0074]

[0075] where m ∈ {1, 2,..., M}; v m and p m are the attitude matrix and position matrix at the m-th node, respectively;

[0076] The forward kinematics of the robot is represented as:

[0077]

[0078] where f k (·) represents the forward kinematics mapping of the robot;

[0079] The inverse kinematics of the robot is:

[0080]

[0081] where represents the inverse kinematics mapping;

[0082] This mapping can be approximately calculated by a solver Mapper based on an artificial neural network ANN. The loss function of Mapper is defined as:

[0083]

[0084] where represents the pose matrix estimated by Mapper, and T m is the target pose matrix; ρ and represent the ordinal numbers of the rows and columns of the matrix, respectively;

[0085] By reducing the value of the loss function L of the ANN-based Mapper f it is possible to fit to satisfy the target pose matrix T mInverse kinematics solution, by adjusting the input of the Mapper to find the inverse kinematics solution with optimal energy;

[0086] In the IKM, determine an energy optimization algorithm based on a mixture of normal distributions, which is used to find the optimal energy solution when the nodes on the current path are mapped to joint angles;

[0087] Initialize the parameters, including the mean set as x = {x1, x2,..., x Z}, Z is the total number of these means, and for each mean x z the corresponding variance is σ z , and z is a variable element from 1 to Z;

[0088] In each round f ∈ {1, 2,..., F}, where F is the total number of rounds; the mean x z is the input of the Mapper, and its output is the set of joint angles θ z of the robot; calculate the energy consumption according to the energy consumption formula, and the set of energy consumption is E = {E1, E2,..., E Z}; the mixture of normal distributions is:

[0089]

[0090] In the formula, is the z-th normal distribution, and the weight τ z is defined as

[0091] Determine the random distribution that the next sampled mean x z+1 follows, and the threshold is:

[0092]

[0093] In the formula, h α，β represents the distance between the means x α and x β , and H is the maximum distance between every two means; ZH is the product of the total number of means Z and the maximum distance H between every two means;

[0094] Generate a random number κ ∈ (0, 1), when κ ≤ ∈, the sampling of x z+1 follows the mixture of normal distributions Otherwise, the sampling of x z+1 follows a random distribution; this threshold ∈ will increase as the number of rounds increases and also has random fluctuations; x z+1 is used as the input of the Mapper, and the output is θ z+1 , the corresponding energy consumption is E z+1 , and add E z+1Load into E; sort E in ascending order, and delete the last element in it and its corresponding element in x; keep the values in set x fixed;

[0095] Loop the above steps until the number of rounds is used up. Obtain the first element E1 in E and its corresponding angle θ1, which are the minimum energy consumption value and the inverse kinematics solution with optimal energy.

[0096] Furthermore, is maintained under the condition that the robot can learn to keep a safe distance from the obstacle when bypassing the obstacle and reduce rubbing.

[0097] Furthermore, the trajectory experience data generated by PP and IKM are stored in the long-term experience replay library to provide new real-time data for subsequent training and updating.

[0098] Preferably, PyCharm is used as the integrated development environment, Python and C++ are used as the development languages, PyRep is used as the virtual environment function library, and CoppeliaSim is used as the virtual experiment platform.

[0099] The beneficial effects of the present invention are as follows:

[0100] The present invention designs a Fast Reconstruction Neural Network (FRNN) based on a Long Short-Term Memory Network (LSTM) and a deep reinforcement learning algorithm of Dynamic Entropy Actor-Critic (DEAC) for the robot motion planning task; in motion planning, a Tutoring-Guiding (TG) mechanism is established, which can convert the explored experience into heuristic experience, enabling the previous experience to be fully utilized to facilitate subsequent exploration. Among them, the designed TG mechanism can adjust the reward distribution according to valuable trajectories to help the robot make full use of exploration experience and learn the optimal strategy efficiently and stably. At the same time, in order to respond to different environments with differences in real time, a fast adaptation mechanism composed of a Progressive Planner (PP) and an Inverse Kinematics Mapper (IKM) is designed, enabling the previously trained strategy to quickly adapt to the new differentiated environment and enhancing the generalization ability of the policy model to the environment. The present invention enables the robot to autonomously learn the motion planning strategy. When the environment changes, the planning strategy can be adjusted in real time according to the differences between the pre-change and post-change environments without manual intervention, improving the robustness and working efficiency of the robot in performing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 is a 1:1 restored digital virtual system built by the present invention in CoppeliaSim;

[0102] Figure 2 is a flowchart of the DEAC-TG algorithm proposed by the present invention;

[0103] Figure 3 is a schematic diagram of the principle of the PP of the present invention;

[0104] Figure 4 is a flowchart of the IKM of the present invention;

[0105] Figure 5 is a schematic diagram of the alternative trajectories generated by the PP and IKM of the present invention;

[0106] Figure 6 is a flowchart of the composition framework of the entire system in the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0107] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented on the premise of the technical solution of the present invention, and provides a detailed implementation manner and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0108] The present invention reconstructs and analyzes the working environment of the robot in real time through a fast reconstruction neural network (FRNN) in a multi-angle visual sampling system. The robot equipped with a dynamic entropy actor-critic (DEAC) and a teaching-guidance (TG) mechanism trains trajectory planning in a virtual scenario to complete the motion planning task. When the scenario changes, the proposed PP and IKM will be activated to help the robot quickly adapt to the new environment while meeting the optimal goals of trajectory and energy consumption. The proposed method can accurately and quickly reconstruct the three-dimensional scene, providing key environmental information for the training of the robot motion planning model. This provides an efficient solution for the robot trajectory planning task, especially in tasks with differences.

[0109] A robot deep reinforcement learning motion planning method for differential tasks, comprising the following steps:

[0110] S1. To avoid equipment loss and personnel injury caused by operational errors during debugging and training, use SolidWorks to draw a 3D model of the equipment, and build a 1:1 restored robot motion planning experimental scene in the simulation environment CoppeliaSim. Use Python as the programming language and PyCharm as the integrated development environment (IDE) to build a virtual-real real-time operating system. Load the PyRep library in the program to associate it with CoppeliaSim as the real-time simulation training environment;

[0111] S2. Obtain information about the real environment through a multi-angle visual sampling system including the self-developed three-dimensional reconstruction algorithm FRNN. In the virtual-real real-time operating system, call the functions of the PyRep library (a robot motion control algorithm and command library based on the computer language Python) to associate with the CoppeliaSim (this method is based on this robot simulation software) virtual environment, and build an experimental scene that is 1:1 restored to the real environment. Configure the distance and collision detection parameters between the robot and the obstacles, and transmit these parameters to the self-developed TG mechanism at the same time;

[0112] S3. In the CoppeliaSim virtual environment, use the self-developed dynamic entropy actor-critic (DEAC) algorithm based on deep reinforcement learning combined with the teaching-guidance (TG) mechanism, namely DEAC-TG, to train the motion planning model of the robot to complete the robot motion planning task;

[0113] S4. When the environment changes, the self-developed multi-angle visual sampling system is used to detect the information of the changes in the environment and transmit it to the virtual-real real-time operation system, so that the virtual environment information in the system corresponds to the actual environment information in real time;

[0114] S5. If the environment changes, the self-developed progressive planner (PP) and inverse kinematics mapper (IKM) will be activated. Among them, PP is used to explore the optimal path in the changed area, and IKM maps this path from the three-dimensional space to the joint space of the robot;

[0115] S6. When the robot explores a feasible trajectory in the changed environment, the experience data related to the robot will be stored in the experience replay library, and these data will be used in the update after DEAC-TG (DEAC with TG mechanism). Through the experience data in these diverse environments, the finally obtained robot motion planning model can be generalized in diverse scenarios and can quickly adapt to new differentiated environments.

[0116] Specifically as follows:

[0117] The present invention is based on the designed and proposed three-dimensional reconstruction algorithm FRNN, the deep reinforcement learning algorithm DEAC-TG, and the fast adaptation mechanism composed of PP and IKM, as Figure 1 shown:

[0118] A robot ①: Preferably, the robot is a KUKA LBR iiwa 7R800, a 7-degree-of-freedom redundant robot, located in the working environment, responsible for grasping the object block from the starting grasping area and then placing it in the target placement area, without colliding with obstacles in the middle, and completing the motion planning task.

[0119] The starting grasping area ②: The object can appear at random positions in this area;

[0120] The target placement area ③: The target placement position will be randomly designated in this area;

[0121] Randomly changing obstacles ④: In the working space of the robot, the obstacles in the scene will randomly appear or change;

[0122] Preferably, the multi-angle visual sampling system ⑤: Multiple depth cameras are arranged directly above and around the scene to form an image acquisition array, and the collected real-time image data is processed by FRNN and transmitted to the virtual-real real-time operation system to synchronize the virtual scene with the real scene;

[0123] Preferably, a virtual-real real-time operation system ⑥: Deploy a centralized processing system on a computer group composed of several computers, which can correlate virtual scenes, Python or C++ project files, camera arrays, and real scenes to achieve real-time data transmission and processing, and overall operation and management.

[0124] The specific implementation method is as follows:

[0125] Step 1: To avoid possible equipment losses and personnel accidents that may occur during the training of physical robots, reconstruct and extract environmental information through FRNN, build a 1:1 restored virtual environment in CoppeliaSim (a dynamics robot simulator based on a physical engine), and associate it with the environment in CoppeliaSim through PyRep (a robot learning research toolkit developed based on CoppeliaSim).

[0126] In this method, several blocks randomly appear in the starting grasping area. The robot is trained to move from the initial pose to the grasping pose, grasp a block, bypass obstacles, and place this block at a random target position. During this grasping and placing process, the virtual environment and the real environment remain in real-time synchronization. And the environment will change, and the robot quickly adapts to the environmental changes to complete new grasping and placing tasks. This process will be looped and repeated until all blocks are transferred from the starting grasping area to the target placement area.

[0127] Step 2: Set a three-dimensional convolutional long short-term grid in the FRNN algorithm, and its equation:

[0128]

[0129] h t =tanh(e t )+o t ☉tanh(e t )

[0130] In the formula, x t represents the input at time t, and * represents the convolutional multiplication. v t , u t , o t represent the input gate, output gate, and forget gate respectively. e t and h t represent the memory cell and the hidden layer respectively. W and U represent weights, b represents the bias, and tanh is the hyperbolic tangent function. i is denoted as the input gate, f is denoted as the forget gate, and o is denoted as the output gate.

[0131] The incremental design of the input gate, output gate, forget gate, memory cell, and hidden layer can make this three-dimensional convolutional long short-term grid more sensitive to feature changes and remember more features.

[0132] The loss function of the network is defined as the sum of voxel cross - entropy, and the final output of each voxel (ι, η, ζ) is a Bernoulli distribution This loss function L FRNN is as follows:

[0133]

[0134] where, is a set, t is the current time, and T represents the total number of time steps. y (ι，η，ζ) ∈{0, 1} represents the true occupancy ratio. p(ι, η, ζ) represents the success ratio.

[0135] Step 3: In the virtual environment associated with PyRep and CoppeliaSim, as Figure 2 shown, DEAC - TG represents the DEAC algorithm equipped with the TG mechanism, where TG is used to further process and optimize the successful data obtained by DEAC exploration in the early stage of training, and store these optimized data. In the middle stage of training, these optimized data are added as additional teacher data to the policy update process of DEAC to promote the efficiency of policy update. Then, in the late stage of training, these optimized data are removed to ensure that the policy obtained after DEAC update is unbiased. The designed DEAC - TG algorithm is used to train the robot so that the robot autonomously learns the motion planning strategy.

[0136] Define the state and action parameters of the robot during training, where the definition of state s is:

[0137] s = {θ, |p - p target |, |o - o target |}

[0138] where, θ = {θ1, θ2,..., θ D} is the set of joint angles of the robot, and D is the total number of robot joints. |p - p target | represents the absolute value of the difference between the current position p of the robot end - effector and the target position p target , and |o - o target | represents the absolute value of the difference between the current orientation o of the robot end - effector and the target orientation o target .

[0139] The definition of action a is:

[0140] a = Δθ = {△θ1, △θ2,..., Δθ D}

[0141] where Δθ is the set of increments of the robot joint angles.

[0142] One attempt is divided into G rounds, and each round has T time instants. During the interaction between the robot and the environment, there is a state s at each time instant t, that is, s t to describe the interaction between the robot and the environment at the current time instant. Meanwhile, the robot obtains a reward r at the current time instant, that is, r t . The state s t is used as the input of the policy network, and then an action a is obtained according to the policy network, that is, a t . The next state after the robot executes this action is denoted as s t+1 .

[0143] The designed deep reinforcement learning algorithm DEAC contains two experience replay libraries, namely the long-term experience replay library and the teacher experience replay library. The long-term experience replay library is used to store the experience data during the whole experiment process, and the teacher experience replay library is used to store the data of the simplified trajectories.

[0144] DEAC adopts the actor-critic framework. Generally, it mainly includes 1 actor network, 2 critic networks, 1 target actor network and 2 target critic networks. Among them, the target actor network is a copy of the actor network several time instants ago, and it does not participate in the update. The target critic network is a copy of the critic network several time instants ago, and it does not participate in the update.

[0145] The experience data tuple (s t , a t , r t , s t+1 ) is stored in the long-term experience replay library. s t and a t are used as the input of the critic network. The update function of the critic network is:

[0146]

[0147] Among them, denotes the expectation, generally refers to the experience replay library, Q is the output of the critic network, r t+1 is the reward, τ is the decay factor, can take a value as an element in the constant set {1, 2}. Q<W target is the output of the target critic network. α is the temperature coefficient, which is used to assist in adjusting the value of entropy, and S λ represents the th function in the entropy cluster function S.

[0148] Then r t is used for the update of the actor network, s t+1 is the input of the target critic network, a t+1is the output of the target policy network, and s t is the input of the actor network, and its output is a t The update function of the executor network is:

[0149]

[0150] Thus, the actor network observes the state s t And the output value Q of the critic network is updated with gradient, and the critic network is updated according to the state s t , the output action a of the actor network t and reward r t+1 At the same time, the target executor network input is s t+1 , its output is a t+1 and input s t+1 Together as the input of the target critic network. Then, the target critic network provides the output value Q for the update of the critic network target .

[0151] DEAC-TG is used to train the robot to interact with the environment, and in the process learns the optimal strategy to complete the motion planning task based on the above settings.

[0152] Determine the trajectory progressive simplification algorithm, which is defined as follows:

[0153]

[0154] Where a t，t+n is an estimated action, a max is the maximum amplitude of the action, It is the action space.

[0155] From the state s at time t t Try action a for a time interval n t，t+n , when a t，t+n In the action space When there is, the state s at time t t Will be action a t,t+n Directly bridge to the state s at time t+n t,t+n , and the generated empirical data tuple (s t ,a t ,r t ,s t+n ) is stored in the teacher experience playback library, the original experience data tuple (s t ,a t ,r t ,s t+1 ) will be deleted from the long-term experience replay library to prevent confusion in state transitions.

[0156] Determine the calculation methods of path and energy consumption:

[0157] Let the path length be denoted as L and the energy consumption be represented by E:

[0158]

[0159] In the formula, ω represents the weight of the energy consumption of the robot joints. L and E are respectively the path length at the end of the robot and the energy consumption of the trajectory. m ∈ {1, 2,..., M} represents the serial number of the discrete nodes of the robot joint angle or the end position, and M is the total number of these nodes.

[0160] Determine the reward function r as:

[0161]

[0162] In the formula, the reward function r consists of two parts, namely the main reward function f and the additional reward function g. In the main reward function f, the position and attitude of the end of the robot are inversely correlated with the position and attitude of the target. In this way, it can promote the attitude of the end of the robot to continuously approach the attitude of the target to complete the task. In addition, the additional reward function g is used to adjust the reward distribution, so as to improve the utilization efficiency of valuable trajectories. And the additional reward function g includes energy estimation, so it can also promote the robot to explore trajectories with less energy consumption.

[0163] Determine the main reward function f(t). This reward function is the main reward function that guides the robot to learn how to complete the task, and it is defined as:

[0164]

[0165] In the formula, A is a constant matrix. represents the distance between the robot body and the obstacle, is the maximum collision area distance considered. C obs and C col are sign functions, where, if then C obs > 0, otherwise C obs = 0; if a collision occurs, then C col < 0, otherwise C col = 0; ′ is the transpose symbol.

[0166] As shown in f(t), when the end of the robot is closer to the target position, f(t) is larger, and vice versa, f(t) is smaller. When the distance between the robot body and the obstacle is less than a threshold When it is closer to the obstacle, the penalty is greater, and vice versa. This can help the robot learn to keep a distance from the obstacle and reduce scraping.

[0167] Determine the additional reward function g(t). When the robot explores a trajectory that can reach the target, this trajectory is recorded as a valuable trajectory. This reward function is used to adjust the reward distribution in the environment so that the previously explored valuable trajectories can be fully utilized by subsequent explorations, and it can also mitigate the negative impact of poor rewards on updates. This additional reward is defined as:

[0168]

[0169] In the formula, represents the set of joint angles θ of the current robot t and the set of joint angles θ on the valuable trajectory val The minimum distance. clip(·) represents the numerical clipping function. C flag is the sign function, where if a valuable trajectory is explored, then C flag > 0, otherwise C flag = 0. L max and E max are the maximum values of the path length and energy consumption respectively.

[0170] When a valuable trajectory is explored, the trajectory progressive simplification algorithm in DEAC-TG is used to simplify the state transition pairs (s t , s t+1 ) in this trajectory, and the simplified state transition pairs are put into the teacher experience replay library. During training, the experiences in the long-term experience replay library and the teacher experience replay library will be sampled proportionally as training data. Moreover, this ratio will change with the increase in the number of training rounds, that is, the experience data sampled from the teacher experience replay library will become less and less until it is 0. In the later stage of training, the sampled experience data all come from the long-term memory library to ensure the unbiasedness of the robot's motion planning model. After the ratio of the teacher experience data being extracted is 0, the guidance module is activated, that is, C flag > 0, the additional function g(t) takes effect, and the reward distribution is adjusted to prompt the robot to fully utilize the previous valuable trajectories and reduce the impact of poor rewards in updates.

[0171] is maintained under conditions so that the robot can learn to keep a safe distance from the obstacle when bypassing the obstacle and reduce scraping.

[0172] Step 4: During the operation, the environment changes. The FRNN algorithm in the multi-angle visual sampling system extracts and analyzes the environmental information, transmits this environmental data to the virtual-real real-time operation system, and synchronizes the environmental changes in PyRep and CoppeliaSim.

[0173] Since the environment has changed, the previous strategy cannot enable the robot to complete the motion planning task, that is, the robot will collide with obstacles in the environment during movement. PP is a path planning method in the Cartesian space, which is a method that combines a Voronoi diagram (used for sampling candidate nodes for spatial paths) and A-star (used for path planning). In the Cartesian space, PP is used to quickly plan a path that can bypass obstacles. IKM is a method for solving the inverse kinematics of the robot, which includes an optimization algorithm based on a mixture of normal distributions and an inverse kinematics mapper based on an ANN. IKM can map the path in the Cartesian space planned by PP to the joint space of the robot, forming a sequence of joint motion trajectories of the robot, enabling the robot to bypass obstacles.

[0174] As Figure 3 shown, in the changed area, the surroundings of these obstacles are surrounded by meshed and randomized nodes, and every three nodes among these scattered points can be connected to form a triangle in three-dimensional space. The Voronoi diagram in PP traverses this set of triangles, calculates the circumcenters of each triangle, and saves them. Assume the current triangle is Δ current , find the triangles Δ1, Δ2, and Δ3 that share sides with the three sides of Δ current . If these three co-edge triangles exist (i.e., within the feasible region), then connect the circumcenters of these triangles with the circumcenter of Δ current and save them as the edges of the Voronoi diagram; if these triangles do not exist, then find the intersection points of the perpendicular bisector rays corresponding to the respective sides of Δ current with the edge of the plannable space, and connect the circumcenter of Δ current to these intersection points and save them as the edges of the Voronoi diagram.

[0175] The set of nodes at both ends of the edges of the Voronoi diagram is used as the candidate node set, and the candidate node set is explored using the A-star path planning algorithm to find the path with the optimal path length. A-star contains an Open set and a Close set. The Open set contains candidate nodes, and the Close set will gradually be filled with feasible nodes until the path is found. At initialization, the starting node is loaded from the Open set into the Close set, and the distances to the reachable nodes adjacent to the starting node p start in the Open set are calculated and updated, and the node closest to the starting node is selected according to the distance size, assumed to be p near, delete it from the Open set and then add it to the Close set; check one by one the path lengths from p start to p near and then to other nodes. If the path length from p start through p near to other nodes is the smallest, then update the parent node of the shortest path from p start to this node to be p near , otherwise do not update; sample nodes one by one from the candidate nodes adjacent to p near and update their parent nodes; until the candidate nodes that can reach the end point are traversed and the end point is added to the Close set, then the path with the shortest length, i.e., the shortest path, is found.

[0176] The shortest path in the Cartesian space obtained by PP is represented as a set of nodes B = {b1, b2,... b M} and M is the total number of nodes. The designed IKM is used to map this shortest path into the joint space of the robot.

[0177] Determine a mapper based on an artificial neural network. The pose T m of the robot can be represented as:

[0178]

[0179] where m ∈ {1, 2,..., M}. v m and p m are the attitude matrix and position matrix at the m-th node respectively.

[0180] The forward kinematics of the robot is represented as:

[0181]

[0182] where f k (·) represents the forward kinematics mapping of the robot.

[0183] The inverse kinematics of the robot is:

[0184]

[0185] where represents the inverse kinematics mapping.

[0186] This mapping can be approximately calculated by a solver (Mapper) based on an artificial neural network (Artificial Neural Network, ANN). The loss function of the Mapper is defined as:

[0187]

[0188] Wherein, represents the pose matrix estimated by the Mapper, and T m is the target pose matrix. ρ and respectively represent the ordinal numbers of the rows and columns of the matrix.

[0189] By reducing the value of the loss function L of the ANN-based Mapper f the inverse kinematic solution satisfying the target pose matrix T m can be fitted. Although there are countless such inverse kinematic solutions, the inverse kinematic solution with optimal energy can be found by adjusting the input of the Mapper.

[0190] In the IKM, an energy optimization algorithm based on a mixture of normal distributions is determined, which is used to find the optimal energy solution when the nodes on the current path are mapped to joint angles.

[0191] As Figure 4 shown, initialize the parameters, including the mean set as x = {x1, x2,..., x Z}, Z is the total number of these means, and each mean x z corresponds to the variance σ z , and z is a variable element from 1 to Z.

[0192] In each round f ∈ {1, 2,..., F}, where F is the total number of rounds. The mean x z is the input of the Mapper, and its output is the set of joint angles θ of the robot z . Calculate the energy consumption according to the above energy consumption formula, and the set of energy consumption is E = {E1, E2,..., E Z}. The mixture of normal distributions is:

[0193]

[0194] Wherein, is the z-th normal distribution, and the weight τ z is defined as

[0195] Determine the random distribution followed by sampling the next mean x z+1 , and the threshold is:

[0196]

[0197] Wherein, h α，β represents the distance between the means x α and x β , and H is the maximum distance between every two means. ZH is the product of the total number of means Z and the maximum distance H between every two means.

[0198] Generate a random number κ ∈ (0, 1). When κ ≤ ∈, the sampling of x z+1 follows a mixture normal distribution Otherwise, the sampling of x z+1 follows a random distribution.

[0199] This threshold ∈ increases as the number of rounds increases, and it also has random fluctuations, that is, its possible values are [b / B, 1]. This means that in the early stage of the round, the probability of κ > ∈ is greater, and at this time the sampling of x z+1 follows a random distribution, that is, the sampling is more inclined to exploration. On the contrary, in the later stage of the round, the probability of κ ≤ ∈ is greater, and at this time the sampling of x z+1 follows a mixture normal distribution This distribution is calculated based on the previous exploration values, that is, the sampling is more inclined to exploitation. Thus, the balance between exploration and exploitation of the sampling is satisfied.

[0200] x z+1 As the input of the Mapper, the output is θ z+1 , and the corresponding energy consumption is E z+1 , and load E z+1 into E. Sort E from smallest to largest, and delete the last element in it and its corresponding element in x.

[0201] Keep the values in the set x fixed, which reduces the computational burden and also avoids being disturbed by low-quality data in the mixture normal distribution.

[0202] Loop the above steps until the number of rounds is used up, obtain the first element E1 in E and its corresponding angle θ1, which are the minimum energy consumption value and the inverse kinematic solution with optimal energy.

[0203] As Figure 5 shown, PP and IKM can quickly enable the robot to adapt to environmental changes and keep the robot working continuously without interruption. At the same time, the trajectory experience data generated by PP and IKM are stored in the long-term experience replay library, providing new real-time data for subsequent training and updating.

[0204] In summary, the present invention integrates technologies such as three-dimensional reconstruction, deep reinforcement learning, robot kinematics, path planning, and neural network non-linear fitting into the robot motion planning task at the same time. The entire system process is as Figure 6As shown in the figure. While maintaining real-time synchronization between the virtual and real environments, the path length of the robot's end effector and the energy consumption of the overall trajectory are optimized. At the same time, the robot can quickly adapt to environmental changes, thereby improving the operation efficiency. Combining the above, the present invention realizes the synchronization of virtual and real scenarios, the autonomous optimization of robot motion planning, and the rapid adaptation of the robot to different environments during operation. At the same time, in the present invention, PyRep and CoppeliaSim are associated as a virtual experimental platform. The real environment is reconstructed in real time in the virtual environment through the designed FRNN, and the motion planning model of the robot based on the designed DEAC-TG algorithm is trained. The designed PP and IKM combination is used to generate a new trajectory with optimal path and energy in different environments. The motion planning model of the robot is trained in a closed-loop manner using the real-time updated environmental information and valuable trajectory data. Finally, the trained optimal model is used to test the real environment, reducing the equipment loss of the real platform.

[0205] The present invention uses PyCharm as the integrated development environment, Python and C++ as the development languages, PyRep as the virtual environment function library to be called to improve the training efficiency, and CoppeliaSim as the virtual experimental platform to reduce the equipment loss of the real platform.

[0206] The above embodiments only represent one implementation manner of the present invention, and the description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the appended claims.

Claims

1. A robot deep reinforcement learning motion planning method for differentiated tasks, characterized in that, It includes the following steps: S1. Through a simulation platform, construct a 3D model of the robot, a robot motion planning experimental scenario, and a virtual-real real-time operation system respectively, and associate them as a real-time simulation training environment; S2. Construct a three-dimensional reconstruction algorithm FRNN through a fast reconstruction neural network based on a long short-term memory network. Establish a multi-angle visual sampling system through the three-dimensional reconstruction algorithm FRNN. Obtain information about the real environment through the multi-angle visual sampling system. In the virtual-real real-time operation system, construct and restore the experimental scenario of the real environment; Configure the distance and collision detection parameters between the robot and the obstacle, and at the same time transmit these parameters to the teaching-guidance TG mechanism; S3. In the virtual environment, use the dynamic entropy actor-critic DEAC algorithm based on deep reinforcement learning combined with the DEAC-TG of the teaching-guidance TG mechanism to train the motion planning model of the robot to complete the robot motion planning task; S4. When the real environment changes, use the multi-angle visual sampling system to detect the information about the changes in the real environment and transmit it to the virtual-real real-time operation system, so that the virtual environment information and the real environment information in the system are synchronized in real time; S5. If the virtual environment changes, activate the progressive planner PP and the inverse kinematics mapper IKM; among them, PP is used to explore the optimal path in the changed area, and IKM maps this path from the three-dimensional space to the joint space of the robot; S6. When the robot explores a feasible trajectory in the changed environment, the relevant experience data of the robot will be stored in the experience replay library, and these data will be used in the update after DEAC with the TG mechanism, namely DEAC-TG; Through the experience data in these diverse environments, the finally obtained robot motion planning model can be generalized in diverse scenarios and can quickly adapt to new and different environments.

2. The robot deep reinforcement learning motion planning method for differentiated tasks according to claim 1, wherein In step S1, the real-time simulation training environment includes: A robot: The robot is a 7-degree-of-freedom redundant robot located in the working environment, responsible for grasping the object block from the starting grasping area and then placing it in the target placement area, without colliding with obstacles midway, and completing the motion planning task; The starting grasping area: The object can appear at random positions in this area; The target placement area: The target placement position will be randomly specified in this area; Randomly changing obstacles: In the working space of the robot, the obstacles in the scene will randomly appear or change; The multi-angle visual sampling system: Multiple depth cameras are placed directly above and around the scene to form an image acquisition array, and the collected real-time image data is processed by FRNN and transmitted to the virtual-real real-time operation system to synchronize the virtual scene and the real scene; The virtual-real real-time operation system: Deploy a centralized processing system on a computer group composed of several computers. It can associate the virtual scene, engineering files, camera array, and real scene to achieve real-time data transmission and processing, and overall operation and management.

3. The method for robot deep reinforcement learning motion planning for differentiated tasks according to claim 1, wherein, In step S1, the environmental information is reconstructed and extracted by FRNN. A virtual environment that is restored 1:1 is built in CoppeliaSim and associated with the environment in CoppeliaSim through PyRep. A number of blocks randomly appear in the initial grasping area. The robot is trained to move from the initial pose to the grasping pose, grasp a block, bypass obstacles, and place this block at a random target position. During this grasping and placing process, the virtual environment and the real environment remain synchronized in real time. And the environment will change, and the robot quickly adapts to the environmental changes to complete new grasping and placing tasks. This process will be looped and repeated until all blocks are transferred from the initial grasping area to the target placement area.

4. The robot deep reinforcement learning motion planning method for differentiated tasks according to claim 1, characterized in that In step S2, the FRNN algorithm is implemented as follows: In the FRNN algorithm, a three-dimensional convolutional long short-term grid is set, and its equation: h t = tanh(e t ) + o t ⊙ tanh(e t ) where x t represents the input at time t, * represents the convolution multiplication; v t , u t , o t represent the input gate, output gate, and forget gate respectively; e t and h t represent the memory element and the hidden layer respectively; W and U represent weights, b represents the bias, tanh is the hyperbolic tangent function; i is denoted as the input gate, t is denoted as the forget gate, and 0 is denoted as the output gate; The loss function of the network is defined as the sum of voxel cross-entropies, and let the final output of each voxel (ι, η, ζ) be a Bernoulli distribution [1 - p (ι,η，ζ) , p (ι,η，ζ) , and this loss function L FRNN is: In the formula, is a set, t represents the current moment, and T represents the total number of moments; y (ι，η，ζ) ∈ {0, 1} represents the truth occupancy ratio; p(ι, η, ζ) represents the success ratio.

5. The robot deep reinforcement learning motion planning method for differentiated tasks according to claim 1, wherein Step S3 is specifically implemented as follows: In the virtual environment associated with PyRep and CoppeliaSim, DEAC-TG represents the DEAC algorithm equipped with the TG mechanism. Among them, TG is used to further process and optimize the successful data obtained by DEAC exploration in the early stage of training, and store these optimized data. In the middle stage of training, these optimized data are added as additional teacher data to the policy update process of DEAC to promote the efficiency of policy update. Then, in the later stage of training, these optimized data are removed to ensure that the policy obtained after DEAC update is unbiased. The designed DEAC-TG algorithm is used to train the robot so that the robot autonomously learns the motion planning strategy. Define the state and action parameters of the robot during training. The definition of state s is: s = {θ, |p - p target |, |o - o target |} Among them, θ = {θ1, θ2,..., θ D} is the set of the joint angles of the robot, and D is the total number of the robot joints; |p - p target | represents the absolute value of the difference between the position p of the current robot end and the target position p target ; |o - o target | represents the absolute value of the difference between the posture o of the current robot end and the target posture o target ; The definition of action a is: a = Δθ = {Δθ1, Δθ2,..., Δθ D} Where Δθ is the set of increments of the robot joint angles; One attempt is divided into G rounds, and each round has T moments; at each moment t during the interaction between the robot and the environment, there is a state s, that is, s t to describe the interaction between the robot and the environment at the current moment. Meanwhile, the robot will obtain the reward r at the current moment, that is, r t ; the state s t is used as the input of the policy network, and then an action a is obtained according to the policy network, that is, a t ; the next state after the robot executes this action is denoted as s t+1 ; The deep reinforcement learning algorithm DEAC contains two experience replay libraries, namely the long-term experience replay library and the teacher experience replay library. The long-term experience replay library is used to store the experience data during the entire experimental process, and the teacher experience replay library is used to store the data of the simplified trajectory. DEAC adopts an actor-critic framework, including 1 actor network, 2 critic networks, 1 target actor network and 2 target critic networks. Among them, the target actor network is a copy of the actor network several moments ago and it does not participate in the update. The target critic network is a copy of the critic network several moments ago and it does not participate in the update. The empirical data tuple (s t , a t , r t , s t+1 ) is stored in the long-term experience replay library; s t and a t are used as inputs to the critic network; the update function of the critic network is: Among them, represents the expectation, generally refers to the experience replay buffer, Q is the output of the critic network, r t+1 is the reward, τ is the decay factor, can take a value as an element in the constant set {1, 2}; Q target is the output of the target critic network; α is the temperature coefficient, which is used to assist in adjusting the value of the entropy, S λ represents the λ-th function in the entropy cluster function S; Then r t For the update of the actor network, s t+1 Is the input of the target critic network, a t+1 Is the output of the target policy network, and at the same time, s t Is the input of the actor network, and its output is a t ; The update function of the actor network is: Thus, the actor network updates its gradient by observing the state s t and the output value Q of the critic network. The critic network updates according to the changes in the state s t , the output action a of the actor network t and the reward r t+1 . Meanwhile, the target actor network takes s t+1 as input, and its output a t+1 and the input s t+1 are used together as the input to the target critic network. Then, the target critic network provides the output value Q target for the update of the critic network; DEAC-TG is used to train the robot to interact with the environment and learn the optimal strategy to complete the motion planning task according to the above settings during this process. Determine the trajectory progressive simplification algorithm, which is defined as follows: where a t，t+n is an estimated action, a max is the maximum amplitude value of the action, is the action space; From the state s at time t t Attempt to perform action a with a time interval of n t，t+n , when a t，t+n exists in the action space , the state s at time t t will be directly bridged to the state s at time t + n t,t+n , and the generated empirical data tuple (s t,t+n , a t , r t , s t , s t+n ) is stored in the teacher's experience replay library, and the original empirical data tuple (s t , a t , r t , s t+1 ) will be deleted from the long-term experience replay library to prevent confusion in state transitions; Determine the calculation methods of the path and energy consumption: The path length is denoted as L, and the energy consumption is represented by E: In the formula, ω represents the weight of the robot joint energy consumption; L and E are respectively the path length of the robot end and the energy consumption of the trajectory. m ∈ {1, 2,..., M} represents the serial number of the discrete nodes of the robot joint angle or end position, and M is the total number of these nodes. Determine the reward function r as: In the formula, the reward function r consists of two parts, namely the main reward function f and the additional reward function g. In the main reward function f, the position and orientation of the robot's end effector are inversely correlated with the position and orientation of the target. Additionally, the additional reward function g is used to adjust the reward distribution, and the additional reward function g includes energy estimation; Determine the reward function f(t), which is the main reward function that guides the robot to learn how to complete the task, and it is defined as: Where, A is a constant matrix; represents the distance between the robot body and the obstacle, is the maximum collision area distance considered; C obs and C col are sign functions, where, if then C obs > 0, otherwise C obs = 0; if a collision occurs, then C col < 0, otherwise C col = 0; ′ is the transpose symbol; Determine the additional reward function g(t); when the robot explores a trajectory that can reach the target, this trajectory is recorded as a valuable trajectory, and this reward function is used to adjust the reward distribution in the environment, and this additional reward is defined as: wherein, represents the set of joint angles θ of the current robot t and the minimum distance from the set of joint angles θ on the valuable trajectory val ; clip(·) represents the numerical clipping function; C falg is the sign function, where if a valuable trajectory is explored, then C flag > 0, otherwise C flag = 0; L max and E max are the maximum values of the path length and the energy consumption respectively; When a valuable trajectory is explored, the trajectory progressive simplification algorithm in DEAC-TG is used to simplify the state transition pairs (s t , s t+1 ) in this trajectory, and the simplified state transition pairs are put into the teacher experience replay library. During training, the experiences in the long-term experience replay library and the teacher experience replay library will be sampled proportionally as training data; moreover, this ratio will change as the number of training episodes increases, that is, the experience data sampled from the experience library in the teacher experience replay library will become less and less until it reaches 0, and the experience data sampled in the later stage of training all comes from the long-term memory library; after the ratio of the teacher experience data being sampled reaches 0, the guidance module is activated, that is, C flag > 0, the additional function g(t) takes effect, and the reward distribution is adjusted.

6. The robot deep reinforcement learning motion planning method for differentiated tasks according to claim 1, characterized in that, Step S4 is specifically implemented as follows: During the operation, the environment changes; the FRNN algorithm in the multi-angle vision sampling system extracts and analyzes the environmental information, and transmits this environmental data to the virtual-real real-time operation system to synchronize the environmental changes in PyRep and CoppeliaSim.

7. The method for robot deep reinforcement learning motion planning for differentiated tasks according to claim 1, wherein Step S5 is specifically implemented as follows: In the changed area, the surroundings of these obstacles are surrounded by meshed and randomized nodes, and every three nodes among these scattered points can be connected to form a triangle in three-dimensional space. The Voronoi diagram in PP traverses this set of triangles, calculates the circumcenters of each triangle, and saves them; assume the current triangle is Δ current , find the triangles Δ1, Δ2, and Δ3 that share sides with the three sides of Δ current . If these three co-side triangles exist, that is, within the feasible area, then connect the circumcenters of these triangles to the circumcenter of Δ current and save them as the edges of the Voronoi diagram; if these triangles do not exist, then find the intersection points of the perpendicular bisector rays corresponding to each side of Δ current with the edge of the plannable space, and save the connections between the circumcenter of Δ current and these intersection points as the edges of the Voronoi diagram; The set of nodes at both ends of the edges of the Voronoi diagram is used as the candidate node set, and the A-star path planning algorithm is used to explore the path with the optimal path length in the candidate node set; A-star contains an Open set and a Close set, where the Open set contains candidate nodes, and the Close set will gradually be filled with feasible nodes until the path is found; At the initialization, the starting node is loaded from the Open set into the Close set, and the distances to the adjacent reachable nodes are calculated and updated, and the node closest to the starting node is selected according to the distance magnitude, assumed to be p start . The distances to the adjacent reachable nodes are calculated and updated, and the node closest to the starting node is selected according to the distance magnitude, assumed to be p near , which is deleted from the Open set and then added to the Close set; Check the path lengths from p start to p near and then to other nodes one by one. If the path length from p start through p near to other nodes is the smallest, then update the parent node of the shortest path from p start to this node to be p near , otherwise do not update; Sample nodes one by one from the candidate nodes adjacent to p near and update their parent nodes; Until the candidate nodes that can reach the end point are traversed and the end point is added to the Close set, the shortest path, that is, the shortest path, is found; The shortest path in the Cartesian space obtained by PP is represented as a set of nodes B = {b1, b2,... b M}, where M is the total number of nodes; the designed IKM is used to map this shortest path into the joint space of the robot; Determine a mapper based on an artificial neural network, and the pose T of the robot m can be expressed as: where \(m\in\{1,2,\cdots,M\}\); \(v\) m and \(p\) m are the attitude matrix and the position matrix at the \(m\)-th node, respectively; The forward kinematics of the robot is expressed as: where f k (·) represents the forward kinematic mapping of the robot; The inverse kinematics of the robot is: wherein, represents the mapping of inverse kinematics; This mapping can be approximately calculated by a solver Mapper based on an artificial neural network ANN, and the loss function of Mapper is defined as: In the formula, represents the pose matrix estimated by the Mapper, and T m is the target pose matrix; ρ and represent the ordinal numbers of the rows and columns of the matrix, respectively; By reducing the value of the loss function L of the ANN-based Mapper f the numerical value can fit the inverse kinematic solution that satisfies the target pose matrix T m and by adjusting the input of the Mapper, an inverse kinematic solution with optimal energy can be found; In the IKM, determine an energy optimization algorithm based on a mixture of normal distributions, which is used to find the optimal energy solution when the nodes on the current path are mapped to joint angles; Initialize parameters, including the mean set as x = {x1, x2,..., x Z}, where Z is the total number of these means, and each mean x z corresponds to the variance σ z , and z is a variable element in the range from 1 to Z; In each round \(f\in\{1, 2,\cdots, F\}\), where \(F\) is the total number of rounds; the mean \(x\) z is the input of the Mapper, and its output is the set of joint angles \(\theta\) of the robot z ; calculate the energy consumption according to the energy consumption formula, and the set of energy consumption is \(E = \{E_1, E_2,\cdots, E\) Z \}; the mixture normal distribution is as follows: wherein, is the z-th normal distribution, and the weight τ z is defined as Determine the next mean value \(x\) for sampling z+1 The random distribution it follows, with the threshold being: where h α，β represents the distance between the mean value x α and x β , H is the maximum distance between every two mean values; ZH is the product of the total number of mean values Z and the maximum distance H between every two mean values; Generate a random number \(k\in(0,1)\). When \(k\leq\in\), the sampling of \(x\) z+1 follows a mixture normal distribution Otherwise, the sampling of \(x\) z+1 follows a random distribution; this threshold \(\in\) increases as the number of rounds increases and also has random fluctuations; \(x\) z+1 is used as the input of the Mapper, and the output is \(\theta\) z+1 , and the corresponding energy consumption is \(E\) z+1 , and load \(E\) z+1 into \(E\); sort \(E\) from smallest to largest, delete the last element in it and its corresponding element in \(x\); keep the values in the set \(x\) fixed; Loop the above steps until the number of rounds is used up, obtain the first element E1 in E and its corresponding angle θ1, which are the minimum energy consumption value and the inverse kinematics solution with optimal energy.

8. The robot deep reinforcement learning motion planning method for differentiated tasks according to claim 5, wherein, held under conditions such that the robot can learn to maintain a safe distance from obstacles when bypassing them and reduce the occurrence of scratches.

9. The method for robot deep reinforcement learning motion planning for differentiated tasks according to claim 6, wherein, The trajectory experience data generated by PP and IKM are stored in the long-term experience replay library to provide new real-time data for subsequent training and updating.

10. The method for robot deep reinforcement learning motion planning for differentiated tasks according to claim 6, characterized in that, Use PyCharm as the integrated development environment, Python and C++ as the development languages, PyRep as the virtual environment function library, and CoppeliaSim as the virtual experimental platform.

Images