Sub-target tree robot arm obstacle avoidance path planning method based on maximized curiosity

Abstract

The application discloses a kind of sub-target tree robot arm obstacle avoidance path planning methods based on maximization curiosity, as follows: step 1: create and register reinforcement learning environment;Step 2: construct the reward function of the planning strategy of sub-target tree;Step 3: define the evaluation method of curiosity, evaluate the curiosity size of sub-target;Step 4: design curiosity balance module;Step 5: set parameters;Step 6: train strategy in simulation environment;Step 7: save the parameter file trained;Step 8: import parameters, call model, recursively predict sub-target to generate sub-target sequence;Step 9: according to sub-target sequence, planning is carried out in turn, and the planning of path is completed.The application can more efficiently generate obstacle-free path in complex environment, and improve planning efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of path planning based on reinforcement learning, and specifically relates to a sub-goal tree robotic arm obstacle avoidance path planning method based on maximizing curiosity. Background Art

[0002] Path planning is a fundamental problem in robotic arm motion control. Traditional path planning methods are generally applicable to structured environments. However, in practice, robotic arms operate in diverse environments, making it more difficult to generate obstacle-free paths. In recent years, reinforcement learning-based path planning has provided an alternative approach to this problem. It enables robotic arms to autonomously learn and plan optimal paths in unstructured environments.

[0003] Due to sparse rewards, reinforcement learning is very inefficient for this type of task. To improve efficiency, some researchers have proposed a method that combines hierarchical reinforcement learning and planning. This method decomposes the obstacle avoidance task into a root-to-leaf path of subtasks on a search tree, where high-level tasks invoke lower-level tasks. However, this method has the following drawbacks: it is difficult to assess the cost and value of each subgoal in different environments. Often, there are multiple subgoals with the same cost, which makes it difficult for reinforcement learning to judge. Summary of the Invention

[0004] In response to the above-mentioned problems existing in the prior art, the present invention proposes a sub-goal tree robotic arm obstacle avoidance path planning method based on maximizing curiosity, which can more efficiently generate obstacle-free paths in complex environments and improve planning efficiency.

[0005] The present invention adopts the following technical solutions:

[0006] The obstacle avoidance path planning method for a robotic arm based on a sub-goal tree of maximizing curiosity is as follows:

[0007] Step 1: Create and register a reinforcement learning environment;

[0008] Step 2: Construct the reward function of the planning strategy of the sub-goal tree;

[0009] Step 3: Define the curiosity evaluation method and evaluate the curiosity level of the sub-goals;

[0010] Step 4: Design a curiosity balance module;

[0011] Step 5: Set parameters;

[0012] Step 6: Train the policy in a simulation environment.

[0013] Step 7: Save the trained parameter file;

[0014] Step 8: Import parameters, call the model, and recursively predict sub-goals to generate sub-goal sequences;

[0015] Step 9: Plan in sequence according to the sub-goal sequence to complete the path planning.

[0016] Preferably, step 1: Create and register a reinforcement learning environment. Rewrite OpenAI's gym interface, including methods such as reset, step, and render, to enable reinforcement learning in the simulation environment. The preferred embodiment of the present invention uses TD3 as the reinforcement learning method, and builds the simulation environment based on pybullet.

[0017] Preferably, step 2: design a reward function for the planning strategy of constructing the sub-goal tree. The reward function considers whether the current goal is a legal sub-goal, whether it encounters an obstacle, whether it exceeds the boundary, and whether it is the last sub-goal, denoted as r t e .

[0018] Preferably, in step 2, the sub-goal is to control the target joint angle of the manipulator joint to be reached, and the sub-goal is determined according to the sub-goal reward function; for each sub-goal s m , by evaluating s m Current location g reach With the starting position g begin and target position g end The Euclidean distance between and To determine whether the sub-goal is legal, d gap Represents the difference between the Euclidean distance of the sub-target and the starting position. When d gap <0.05 and no collision occurs, then the sub-goal is legal;

[0019]

[0020]

[0021]

[0022]

[0023] g midle =(g begin +g end ) / 2

[0024]

[0025] T free represents the free working space, T col Indicates a collision. Indicates the starting position gbegin and target position g end The Euclidean distance between midle The midpoint of the straight line connecting the starting position of the sub-target, the distance g of the sub-target midle The smaller the distance is, the better the sub-goal is. Indicates the midpoint g midle and the target position g end The Euclidean distance between

[0026]

[0027] When the distance between the sub-target and the target is less than d min When rewarded 100, d min Set to 0.1.

[0028] Preferably, step 3: define the curiosity evaluation method and use ICM to evaluate the curiosity of the sub-goal. t and the action a chosen by the agent t The forward model input to the ICM is used to predict the feature vector of the next observation state Predicted value Compared with the actual value φ(s t+1 ) as a curiosity reward signal r t i , η represents the scaling factor, and η is set to 0.5.

[0029] The calculation is as follows:

[0030]

[0031] Preferably, step 4: design curiosity balance module and plan strategy π p Predict sub-goal s given an observation state s m , the learning goal of the strategy is to maximize the curiosity of the selected action and select feasible actions to reach the appropriate sub-goal. The reward obtained by the agent at time t is curiosity reward r t i and the sub-goal reward is r t e , total rewards:

[0032] r t =r t e +r t i

[0033] At each update step, the ICM and policy are trained simultaneously using the same state-action pairs sampled from the experience replay buffer. To avoid perturbing the sample space, the curiosity reward is only used when updating the policy, and the reward values ​​for state-action pairs stored in the experience replay buffer only include the subgoal rewards for interacting with the environment.

[0034] Strategy π p Use TD3 algorithm for training, with parameter θ P The deep neural network representation policy π p (s t θ P ), input the observed state s of the agent at time t t , strategy π p According to s t Select the corresponding action a t ~π p (s t θ P ), and perform actions to optimize the parameters θ P To maximize the expected reward:

[0035]

[0036] But this also exacerbates the contradiction between exploration and utilization of reinforcement learning. Therefore, it is necessary to design a hyperparameter β t ≥ 0 to achieve the trade-off between exploration and exploitation at the tth epoch of training time step. The total reward is rewritten as:

[0037] r t =r t e +β t r t i

[0038] β t The settings are as follows:

[0039]

[0040] β0=max(r t e )

[0041] Among them, T curios and T end They represent the start time and end time of the curiosity mechanism, and ρ represents the decay rate of curiosity.

[0042] Preferably, in step 4, the output of the curiosity balancing module is a hyperparameter β t ≥0, used to achieve the trade-off between exploration and utilization at the t-th epoch of training time step; where Tend Set to β t When it is less than 0.01, ρ is set to 1.0×10 -5 , β0 is set to the maximum value that the sub-goal reward can reach; as the training process gradually deepens, the curiosity reward r t i Converge to 0, and let the strategy eventually point back to the original task;

[0043] T curios The time is calculated as follows, x[n] represents the test reward sequence of the first 100 epochs of the t-th epoch, y[n] is a 20-length all-1 sequence, and z[n] is a 100-length all-1 sequence;

[0044]

[0045] The purpose of this formula is to t [n] is smoothed, that is, each element in x is replaced by the average of the 20 elements around it. Calculation starts from the 100th epoch Then calculate it every 50 epochs right Calculate the variance. If the variance is less than 0.5, it is considered to be converged at that moment, recorded as T curios .

[0046] Preferably, step 5: parameter setting, specifically includes the forward model and backward model parameter settings of ICM, and the relevant parameter settings of the TD3 method.

[0047] Preferably, step 6: training strategy π in a simulation environment p , first input the initial environment state s and the goal g, recursively apply the path planning strategy π based on the sub-goal tree p Between s and g by predicting action a t To insert subgoals m , control the robotic arm to perform action a t To interact with the environment to obtain the reward r that measures the quality of this sub-goal t e and arrive at a new observation state s t+1 , observe the state transition process (s t ,a t ,s t+1 ,r t e ) is stored in the experience buffer pool. Then, a small batch of samples (s t ,a t ,s t+1 ,r t e) to update the strategy. The curiosity balance module updates the strategy according to r t e To calculate β t To balance the curiosity signal t i When β t When greater than 0, the curiosity signal generated by ICM is weighted as β t r t i When β t When it is equal to 0, no curiosity reward is given to the sub-goal and the TD3 update strategy is used directly. It should be noted that even if the sub-goal adds an additional curiosity reward, the strategy update is still based on the reward signal r of the sub-goal tree strategy. t e As the dominant factor, the goal of the strategy is to find a legitimate sub-goal that maximizes curiosity. To achieve this goal, the TD3 strategy is updated to maximize the expectation of reward

[0048] Preferably, in step 6, the strategy π is trained in a simulation environment p When the observation value s t Including the joint angles of the robot arm, joint coordinates, the size and position of the obstacle, and the position of the target; input to the strategy π p In the strategy π p Output action a τ That is, the change in the 6 joint angles of the robotic arm, calling the step function to execute a τ , reach the new state, calculate the environment reward r t e , and transform the state-action pair (s t ,a t ,s t+1 ,r t e )Save to the experience cache pool;

[0049] Call ICM to calculate curiosity reward r t i And update its forward network θ F and the backward network θ I ; The optimization goal of the forward model is to minimize the following loss function:

[0050]

[0051] The input of the reverse model is the observed state s t and s t+1 The feature vector φ(s t ) and φ(s t+1 ), the output is in s tand s t+1 Predicted actions between Defined as:

[0052]

[0053] The optimization goal of the inverse model is to minimize its prediction error:

[0054]

[0055] The curiosity reward and the environment reward are added together as the final reward, i.e. r t =r t e +β t r t i , which is used to plan the strategy π p Updates.

[0056] Preferably, step 7: After the training, the final parameter θ p Save as a .pth file for easy export and call later.

[0057] Preferably, step 8: calling the trained model to execute planning, first import the saved parameter file .pth of the reinforcement learning model.

[0058] Preferably, planning is a recursive process, the input is the observation s t , output an action a representing the change angle of the robotic arm t In the pybullet simulation environment, the robot arm is controlled to perform this action. The robot arm reaches the subgoal, saves the subgoal, obtains a new observation, and re-inputs the new observation into the model to continue predicting the next subgoal. The recursion is completed if the distance between every two subgoals is less than 0.1. If the number of saved subgoals exceeds 20, the planning is considered a failure and the recursion ends, resulting in a subgoal sequence.

[0059] Preferably, step 9: calling the OMPL library to perform planning in sequence according to the sub-goal sequence, and finally completing the planning of the entire path.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] (1) This paper designs a more intensive reward function for sub-goals, which can accelerate the convergence of the strategy.

[0062] (2) The present invention integrates the curiosity mechanism and provides additional evaluation indicators for sub-goals, which will promote the discovery of more challenging and diverse sub-goals and deepen the agent's understanding of the environment.

[0063] (3) The present invention designs a curiosity balancing module, which enables the intelligent agent to adapt to changes in the environment by completing courses from easy to difficult and then back to easy. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a flow chart of a method for obstacle avoidance path planning of a manipulator based on a sub-goal tree of maximizing curiosity in a preferred embodiment;

[0065] Figure 2 Schematic diagram of the sub-goal tree structure;

[0066] Figure 3 Schematic diagram of the obstacle avoidance path planning for the robotic arm based on the sub-goal tree of maximizing curiosity;

[0067] Figure 4 This is a diagram of the sub-goal tree structure based on maximizing curiosity;

[0068] Figure 5 This is the network structure diagram of ICM;

[0069] Figure 6 A schematic diagram of a simulation of the present invention in an example of climbing over a wall to reach a target;

[0070] Figure 7 This is a simulation diagram of the present invention in an example of avoiding five random obstacles in space to reach the target. DETAILED DESCRIPTION

[0071] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0072] like Figure 1 As shown in FIG, the present embodiment is based on the curiosity-maximizing sub-goal tree robot arm obstacle avoidance path planning method, which specifically includes the following steps:

[0073] Step 1: Create and register a reinforcement learning environment. Rewrite the OpenAI gym interface, including reset, step, render and other methods, to enable reinforcement learning in the simulation environment. This example uses the TD3 method. Implement the simulation environment based on pybullet, import the urdf file of the JAKA Zu5 6-axis collaborative robot, and set the collision information of obstacles according to different tasks. Figure 6 、 7 As shown, the pear shape is the target position, the cuboid and sphere are obstacles, and the space surrounded by thin lines is the working range of the robotic arm. Figure 6 The robot arm needs to climb over a wall to reach the target location. Figure 7 The goal is to avoid five obstacles of random size and random position in the workspace and reach the target position.

[0074] Step 2: Define the reward function of the planning strategy for constructing the sub-goal tree. The reward considerations include whether it is a legal sub-goal, whether it encounters an obstacle, whether it exceeds the boundary, and whether it is the last sub-goal. The sub-goal tree structure is as follows: Figure 2 As shown. For each sub-goal s m , by evaluating s m Current location g reach With the starting position g begin and target position g end The Euclidean distance between and To determine whether the sub-goal is legal, d gap Represents the difference between the Euclidean distance of the sub-target and the starting position. When d gap <0.05 and no collision occurs, the sub-goal is legal.

[0075]

[0076]

[0077]

[0078]

[0079] g midle =(g begin +g end ) / 2

[0080]

[0081] T free represents the free working space, T col Indicates a collision. Indicates the starting position g begin and target position g end The Euclidean distance between midle The midpoint of the straight line connecting the starting position of the sub-target, the distance g of the sub-target midle The smaller the distance is, the better the sub-goal is. Indicates the midpoint g midle and the target position g end The Euclidean distance between .

[0082]

[0083] When the distance between the sub-target and the target is less than d min When rewarded 100, d min Set to 0.1.

[0084] Step 3: Define the curiosity evaluation method. For the curiosity evaluation method, the present invention uses a method that is compatible with other curiosity mechanisms, such as RND, RE3, etc. In the preferred embodiment of the present invention, the ICM method is used. ICM consists of three modules (such as Figure 5 As shown in Figure 2, they are feature extraction module, forward model and backward model respectively. Input the current observation state s t and the observation state s at the next moment t+1 , the feature vector φ(s t ) and φ(s t+1 ) are input into the reverse model together to optimize the parameters of the feature extraction module. The current observation state s t and the action a chosen by the agent t Input forward model to predict the feature vector of the next observation state Predicted value Compared with the actual value φ(s t+1 ) as a curiosity reward signal r t i , η represents the scale factor, which is set to 0.5. The calculation formula is as follows:

[0085]

[0086] Step 4: Define the curiosity balancing module, strategy π p Predict sub-goal s given an observation state s m , the learning goal of the strategy is to maximize the curiosity of the selected action and select feasible actions to reach the appropriate sub-goal. The reward obtained by the agent at time t is curiosity reward r t i and the sub-goal reward is r t e , total rewards:

[0087] r t =r t e +r t i

[0088] At each update step, the ICM and policy are trained simultaneously using the same state-action pairs sampled from the experience replay buffer. To avoid perturbing the sample space, the curiosity reward is only used when updating the policy, and the reward values ​​for state-action pairs stored in the experience replay buffer only include the subgoal rewards for interacting with the environment.

[0089] Strategy π p Use TD3 algorithm for training, with parameter θ P The deep neural network representation policy π p(s t θ P ), input the observed state s of the agent at time t t , strategy π p According to s t Select the corresponding action a t ~π p (s t θ P ), and perform actions to optimize the parameters θ P To maximize the expected reward:

[0090]

[0091] But this also exacerbates the contradiction between exploration and utilization of reinforcement learning. Therefore, it is necessary to design a hyperparameter β t ≥ 0 to achieve the trade-off between exploration and exploitation at the tth epoch of training time step. The total reward is rewritten as:

[0092] r t =r t e +β t r t i

[0093] β t The settings are as follows:

[0094]

[0095] β0=max(r t e )

[0096] Among them, T curios Indicates that curiosity is introduced from this moment on, T end This means that curiosity has decayed to a small enough level that it can no longer affect the network update. Curiosity can be blocked to improve training efficiency and return to the original task. Therefore, T end Set to β t Less than 0.01. In the curiosity range, that is, t is greater than T curios and less than T end , β t An exponential decay method is used to encourage the agent to pay attention to novel states and actions during training, where ρ is the decay rate, which is set to 1.0×10 -5 , β0 represents the maximum value that the curiosity reward can reach. As the training process gradually deepens, the curiosity reward r t i It will converge to 0, allowing the strategy to eventually point back to the original task.

[0097] Tcurios At the moment the strategy begins to converge, when the average return tends to be stable during the training process and no longer changes or fluctuates significantly, the algorithm can be considered to have converged. Specifically, the average return of several consecutive training rounds (or time steps) can be calculated, and then the differences between these average return values ​​can be observed. When these differences are small to a certain extent and remain stable in subsequent training, the algorithm can be considered to have converged. Therefore, the calculation of the convergence moment is as follows:

[0098] x[n] represents the test reward sequence of the first 100 epochs of the t-th epoch, y[n] is a full-1 sequence of length 20, and z[n] is a full-1 sequence of length 100.

[0099]

[0100] The purpose of this formula is to t [n] is smoothed, that is, each element in x is replaced by the average of the 20 elements around it. Calculation starts from the 100th epoch Then calculate it every 50 epochs right Calculate the variance. If the variance is less than 0.5, it is considered to have converged at that moment, and the curiosity mechanism is introduced from that moment on.

[0101] Step 5: Parameter setting, ICM forward network, backward network parameter setting:

[0102] Table 1 ICM forward network and backward network parameter settings

[0103]

[0104] The TD3 algorithm parameter settings are as follows:

[0105] Table 2 TD3 algorithm related parameter settings

[0106]

[0107] Step 6: Train the policy π in simulation p , how to train the structure as Figure 4 As shown, the following is the specific process:

[0108] 6-1. Initialization strategy π p The neural network parameters θ p , initialize the experience buffer pool, initialize the forward network and reverse network parameters θ F ,θ I , the number of training epochs is initialized to 0.

[0109] 6-2. Randomly initialize the simulation environment, including initializing the joint angles of the robot arm, the size and position of the obstacles, and the target position.

[0110] 6-3. Use pybullet's built-in functions to obtain environmental information, including the joint angles and coordinates of the robot arm, the size and position of obstacles, and the position of the target. Input to the strategy π p In the strategy π p Output action a t (6 joint angle changes of the robotic arm), call the step function to execute a t , reach the new state, calculate the environment reward r t e , and transform the state-action pair (s t ,a t ,s t+1 ,r t e ) is saved to the experience cache pool.

[0111] 6-4. Determine whether this round of training has ended. If not, repeat step 6-3. The end condition is to repeat step 6-3 more than 20 times, encounter an obstacle, or reach the final sub-goal.

[0112] 6-5. Randomly sample a small batch of 256 samples from the experience buffer pool (s t ,a t ,s t+1 ,r t e ), according to r t e Calculating β t , if β t If it is greater than 0.01, ICM is called to calculate the curiosity reward r t i Otherwise, skip 6-5 and execute 6-6. Calculate the loss and update the forward network θ F and the backward network θ I .

[0113] The optimization goal of the forward model is to minimize the following loss function:

[0114]

[0115] The input of the reverse model is the observed state s t and s t+1 The feature vector φ(s t ) and φ(s t+1 ), the output is in s t and s t+1 Predicted actions between Define it as:

[0116]

[0117] The optimization goal of the inverse model is to minimize its prediction error:

[0118]

[0119] The prediction error of the reverse model is used to train both the reverse model and the feature extraction module.

[0120] 6-6. Add the curiosity reward and the environment reward as the final reward, r t =r t e +β t r t i .

[0121] 6-7, according to the above random small batch sampling 256 length samples (s t ,a t ,s t+1 ,r t ) Calculate the loss and update the neural network parameters θ p The deterministic policy of TD3 describes the relationship between states and actions as follows: π:S→A. Each episode starts with sampling from the initial state s0, and the agent moves according to the state s at each time step t. t Take action t After completing the operation, you will get a reward r t =(s t ,a t ), is the discounted sum of future returns, and γ∈[0,1] represents the discount factor of the environment. The goal of the agent is to maximize its expected E[R t ∣s t ,a t ], the optimal strategy can be defined as any strategy π * , so that for every s∈S,a∈A, any strategy π, there exists Q * is the Q function of the optimal strategy and satisfies the Bellman equation:

[0122]

[0123] 6-8. Use the updated neural network parameters θ p Perform tests on 10 randomly generated test tasks, calculate the success rate and save it, which is recorded as x[n].

[0124] 6-9. If the epoch is less than 2000, repeat steps 6-2 to 6-8, repeat 50 times epoch plus 1, and continuously iterate to update the neural network parameters θ p If epoch is equal to 2000, the training ends.

[0125] Step 7: After training, the final parameter θ p Save as a .pth file for easy export and call later;

[0126] Step 8: Call the trained model to perform planning. First, import the saved reinforcement learning model parameter file .pth. Planning is a recursive process. The input is the observation value s. t , output an action a representing the change angle of the robotic arm t In the pybullet simulation environment, the robot arm is controlled to perform this action. The robot arm reaches the sub-goal, saves the sub-goal, obtains a new observation, and re-inputs the new observation into the model to continue predicting the next sub-goal. The recursion is completed if the Euclidean distance between two adjacent sub-goals is less than 0.1. If the number of saved sub-goals is greater than 20, the planning is considered a failure and the recursion ends. Finally, the sub-goal sequence is obtained.

[0127] The recursive diagram is as follows Figure 3 As shown, first predict the first sub-target s between the starting point s and the target g m1 , then in s m1 Predict the next sub-goal s between g m2 , until s m2 The predicted sub-goal s between g m3 With s m2 The distance from g is less than 0.1, and the prediction s is returned. m1 With s m2 Subgoals between m4 , and then similarly, s m4 The distance to the nearby sub-target is also less than 0.1, so return. Continue to predict s and s m1 Similarly, after predicting the sub-goals between m5 After that, the distance between all sub-goals is less than 0.1, and the process ends. m5 ->s m1 ->s m4 ->s m2 ->s m3 ->gComplete the planning of the sub-goal to complete the planning of the entire path.

[0128] Step 9: Call the OMPL library to perform planning according to the sub-goal sequence, ultimately completing the planning of the entire path. First, import the URDF file for the JAKA Zu5 6-axis collaborative robot and configure the relevant OMPL parameters. Set the planning time limit to 5 seconds, the planning step limit to 20, and select BIT* as the planner. Then, create a planner and state space. The state space is the six joint angles of the robot arm. Create spatial information and use pybullet's check_body_pairs function to detect collisions between the robot arm and objects in space. Finally, input the robot arm joint angles corresponding to the state into the planner in the order of the sub-goal sequence. Execute the planning. If the planning is successful, the change sequence of the robot arm joint angles will be returned. Otherwise, the planning failure will be returned.

[0129] In summary, the present invention proposes a method for maximizing curiosity-based sub-goal tree obstacle avoidance path planning for a robotic arm, comprising the following steps: Step 1: Creating and registering a reinforcement learning environment to access reinforcement learning in a simulation environment. The reinforcement learning method is preferably TD3. Depending on the obstacle avoidance task, a corresponding simulation environment needs to be built based on pybullet; Step 2: Designing a reward function for the planning strategy that constructs the sub-goal tree; Step 3: Defining a curiosity evaluation method, preferably using ICM to evaluate the curiosity of sub-goals, encouraging the planning strategy to expand sub-goals in the direction of maximizing curiosity; Step 4: Designing a curiosity balancing module to balance exploration and utilization in reinforcement learning; Step 5: Setting relevant parameters, including the forward model and backward model parameters of the ICM, as well as relevant parameters of the TD3 method; Step 6: Training the planning strategy in a simulation environment; Step 7: After training, saving the parameter file; Step 8: Importing the parameter file, calling the model, and recursively predicting sub-goals to generate a sub-goal sequence; Step 9: Calling the OMPL library to perform planning in sequence according to the sub-goal sequence.

[0130] The above are merely preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A sub-goal tree robotic arm obstacle avoidance path planning method based on maximizing curiosity, characterized by: Follow these steps: Step 1: Create and register a reinforcement learning environment; Step 2: Construct the reward function of the planning strategy of the sub-goal tree; Step 3: Define the curiosity evaluation method and evaluate the curiosity level of the sub-goals; Step 4: Design a curiosity balance module; Step 5: Set parameters; Step 6: Train the policy in a simulation environment. Step 7: Save the trained parameter file; Step 8: Import parameters, call the model, and recursively predict sub-goals to generate sub-goal sequences; Step 9: Plan the path according to the sub-goal sequence. In step 2, the reward function considers whether the current target is a legal sub-target, whether it encounters an obstacle, whether it exceeds the boundary, and whether it is the last sub-target. The reward is recorded as In step 2, the sub-goal is to control the target joint angle of the manipulator joint to be reached, and the sub-goal is determined according to the sub-goal reward function; for each sub-goal s m , by evaluating s m Current location g reach With the starting position g begin and target position g end The Euclidean distance between and To determine whether the sub-goal is legal, d gap Represents the difference between the Euclidean distance of the sub-target and the starting position. When d gap <0.05 and no collision occurs, then the sub-goal is legal; T free represents the free working space, T col Indicates a collision. Indicates the starting position g begin and target position g end The Euclidean distance between midle The midpoint of the straight line connecting the starting position of the sub-target, the distance g of the sub-target midle The smaller the distance is, the better the sub-goal is. Indicates the midpoint g midle and the target position g end The Euclidean distance between When the distance between the sub-target and the target is less than d min When rewarded 100, d min Set to 0.1; In step 3, the current state s is observed t and the action a chosen by the agent t The forward model input to the ICM is used to predict the feature vector of the next observation state Predicted value Compared with the actual value φ(s t+1 ) as a curiosity reward signal η represents the scale factor, which is set to 0.5; the calculation formula is as follows: In step 4, the planning strategy π p Predict sub-goal s given an observation state s m The learning goal of the strategy is to maximize the curiosity of the selected action and select feasible actions to reach the appropriate sub-goal; the reward obtained by the agent at time t is curiosity reward and the sub-goal reward is Total rewards: At each update step, the ICM and the policy are trained simultaneously using the same state-action pairs sampled from the experience replay buffer; Strategy π p Use TD3 algorithm for training, with parameter θ P The deep neural network representation policy π p (s t θ P ), input the observed state s of the agent at time t t , strategy π p According to s t Select the corresponding action a t ~π p (s t θ P ), and perform actions to optimize the parameters θ P To maximize the expected reward: Design a hyperparameter β t ≥0 to achieve the trade-off between exploration and exploitation at the t-th epoch of training time step; the total reward is rewritten as: β t The settings are as follows: Among them, T curios and T end They represent the start time and end time of the curiosity mechanism, and ρ represents the decay rate of curiosity; In step 4, the output of the curiosity balancing module is a hyperparameter β t ≥0, used to achieve the trade-off between exploration and utilization at the t-th epoch of training time step; where T end Set to β t When it is less than 0.01, ρ is set to 1.0×10 -5 , β0 is set to the maximum value that the sub-goal reward can reach; as the training process gradually deepens, the curiosity reward r t i Converge to 0, and let the strategy eventually point back to the original task; T curios The time is calculated as follows, x[n] represents the test reward sequence of the first 100 epochs of the t-th epoch, y[n] is a 20-length all-1 sequence, and z[n] is a 100-length all-1 sequence; Counting starts from epoch 100 Then calculate it every 50 epochs right Calculate the variance. If the variance is less than 0.5, it is considered to be converged at that moment, recorded as T curios .

2. The curiosity-maximizing sub-goal tree robotic arm obstacle avoidance path planning method according to claim 1, characterized in that: In step 6, the strategy π is trained in the simulation environment p First, input the initial environment state s and the goal g, and recursively apply the path planning strategy π based on the sub-goal tree p Between s and g by predicting action a t To insert subgoals m , control the robotic arm to perform action a t To interact with the environment to obtain rewards that measure the quality of this sub-goal and arrive at a new observation state s t+1 , observe the state transition process Stored in the experience cache pool; Randomly sample mini-batches from the experience pool To update the strategy; the curiosity balance module based on To calculate β t To balance curiosity signals The scale of β t When it is greater than 0, the curiosity signal generated by ICM is weighted as When β t When it is equal to 0, no curiosity reward is given to the sub-goal and the TD3 update strategy is used directly; the TD3 strategy is updated to maximize the expectation of the reward 3. The curiosity-maximizing sub-goal tree robotic arm obstacle avoidance path planning method according to claim 2, characterized in that: In step 6, the strategy π is trained in the simulation environment p When the observation value s t Including the joint angles of the robot arm, joint coordinates, the size and position of the obstacle, and the position of the target; input to the strategy π p In the strategy π p Output action a τ That is, the change in the 6 joint angles of the robotic arm, calling the step function to execute a τ , reach the new state, calculate the environment reward And transform the state action into Save to the experience cache pool; Call ICM to calculate curiosity rewards And update its forward network θ F and the backward network θ I ; The optimization goal of the forward model is to minimize the following loss function: The input of the reverse model is the observed state s t and s t+1 The feature vector φ(s t ) and φ(s t+1 ), the output is in s t and s t+1 Predicted actions between Defined as: The optimization goal of the inverse model is to minimize its prediction error: The curiosity reward and the environment reward are added together as the final reward, i.e. Use it to plan strategy π p Updates.

4. The curiosity-maximizing sub-goal tree robot obstacle avoidance path planning method according to claim 3, characterized in that: In step 8, call the trained model to execute planning, and first import the saved reinforcement learning model parameter file .pth.

5. The curiosity-maximizing sub-goal tree robot obstacle avoidance path planning method according to claim 4, characterized in that: In step 8, planning is a recursive process. The recursive process is as follows: the input is the observation value s t , output an action a representing the change angle of the robotic arm t , control the robotic arm to perform this action in the pybullet simulation environment. The robotic arm reaches the sub-goal, saves the sub-goal, obtains new observations, and re-inputs the new observations into the model to continue predicting the next sub-goal. Ensure that the Euclidean distance between two adjacent sub-goals is less than 0.1 before the recursion is completed. If the number of saved sub-goals is greater than 20, the planning is considered a failure, the recursion ends, and finally the sub-goal sequence is obtained.

Images