Training method and device of improved TD3 model for path planning

Abstract

The invention provides an improved TD3 model training method and device for path planning, and the method comprises the steps: fusing an LSTM network in a conventional TD3 model, and forming an improved TD3 model which comprises an LSTM-current Actor network, an LSTM-target Actor network, an LSTM-current Critic network, and an LSTM-target Critic network. According to the method, the LSTM network in the improved TD3 model can be utilized to capture time sequence information of information in the environment, and the memory ability of the LSTM network enhances the utilization of the improved TD3 model for past information, so that the improved TD3 model can consider historical actions and states during decision making, thereby improving the ability of processing a sequence dependency problem, improving the model training speed and improving the model training efficiency. And the performance of the improved TD3 model is improved.

Description

Technical Field

[0001] This application relates to the field of robotic arm path planning technology, and in particular to a training method and apparatus for an improved TD3 model for path planning. Background Technology

[0002] In today's era of rapid technological advancement, robotic arms have been widely applied in numerous fields, particularly demonstrating their indispensable value in the military and aerospace sectors. Robotic arms are extensively used in military reconnaissance, bomb disposal, battlefield medical assistance, as well as on-orbit maintenance, assembly, and refueling.

[0003] Path planning is a key technical factor in these high-risk operations. Its purpose is to find an efficient and safe route for the robotic arm from the starting point to the target point, while avoiding various potential obstacles and risks. Because robotic arms typically operate in highly complex and unstructured environments, their path planning capabilities are subject to extremely stringent requirements. This not only requires the robotic arm to cope with changing environmental factors but also to have the ability to quickly adapt and learn to handle unknown and unpredictable challenges.

[0004] As the complexity of missions such as military and aerospace continues to increase, the need for intelligent robotic arms and autonomous path planning is becoming increasingly urgent. The rise of Deep Reinforcement Learning (DRL) has provided new possibilities for solving this problem, enabling robotic arms to learn autonomously and adapt to complex environments.

[0005] Among them, the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm is a heterogeneous policy deep reinforcement learning algorithm based on the Actor-Critic framework. It can handle problems in high-dimensional continuous action spaces and performs relatively stably on some tasks. Although TD3 can learn efficiently in many continuous state spaces, it requires a large number of samples to learn, especially when dealing with sparse rewards, which increases the amount of data processing during training. Summary of the Invention

[0006] This application provides a training method and apparatus for an improved TD3 model for path planning, in order to solve the technical problems mentioned in the background art.

[0007] Firstly, this application provides a training method for an improved TD3 model for path planning, characterized in that it is applied to the improved TD3 model, wherein the improved TD3 model includes: an LSTM-current Actor network, an LSTM-target Actor network, an LSTM-current Critic network, and an LSTM-target Critic network; the method includes: According to the preset sampling rules, a sample sampling sequence is obtained by sampling from the sample experience pool. The sample experience pool caches the sample path sequence obtained when the improved TD3 model to be trained interacts with the preset environment. The sample path sequence includes: the current sample state, sample action, and next sample state of the robotic arm corresponding to each step length. The sample sampling sequence is a sequence composed of a preset step length of the sample path sequence. The preset step length is less than or equal to the step length of the sample path sequence. Based on the current sample state of the robotic arm corresponding to each sample sampling sequence, the sample action, the next sample state corresponding to each sample sampling sequence, the LSTM-current Critic network, the LSTM-target Actor network, and the LSTM-target Critic network, update the parameters of the LSTM-current Critic network to obtain the updated LSTM-current Critic network. Based on the current sample state corresponding to each sample sampling sequence, the LSTM-current Actor network, and the updated LSTM-current Critic network, the updated LSTM-current Actor network is obtained; Based on the updated parameters of the current LSTM Actor network and the LSTM target Actor network, an updated LSTM target Actor network is obtained; and based on the updated parameters of the current LSTM Critic network and the LSTM target Critic network, an updated LSTM target Critic network is obtained. The improved TD3 model is obtained based on the updated LSTM-current Critic network, the updated LSTM-current Actor network, the updated LSTM-target Actor network, and the updated LSTM-target Critic network.

[0008] Optionally, the step of updating the parameters of the LSTM-current Critic network based on the current sample state of the robotic arm corresponding to each sample sampling sequence, the sample action, the next sample state corresponding to each sample sampling sequence, the next target action, the LSTM-current Critic network, the LSTM-target Actor network, and the LSTM-target Critic network to obtain the updated LSTM-current Critic network includes: The current sample state and sample action of the robotic arm corresponding to each sample sampling sequence are input into the LSTM-current Critic network in step order to obtain the first current q value and the second current q value. The next sample state corresponding to each sample sampling sequence is input into the LSTM-target Actor network in step order to obtain the next target action arranged in step order. The next sample state and the next target action corresponding to each sample sampling sequence are input into the LSTM-Target Critic network in step order to obtain the target Q value; Based on the first current q value, the second current q value, and the target Q value, the parameters of the LSTM-current Critic network are updated to obtain the updated LSTM-current Critic network.

[0009] Optionally, obtaining the updated LSTM-current Actor network based on the current sample state corresponding to each sample sampling sequence, the LSTM-current Actor network, and the updated LSTM-current Critic network includes: The current sample state corresponding to each sample sampling sequence is input into the LSTM-current Actor network in step order to obtain the current target action arranged in step order; According to the step size order, the current sample state and the current target action corresponding to each sample sampling sequence are input into the updated LSTM-current Critic network to obtain the first updated current q value and the second updated current q value; Based on the first or second update of the current q value, update the parameters of the LSTM-current Actor network to obtain the updated LSTM-current Actor network.

[0010] Optionally, the sample path sequence may further include: the actual location of the sample, the target location of the sample, the sample reward value, and a termination flag; Before obtaining the sample sampling sequence from the sample experience pool according to the preset sampling rules, the method further includes: Obtain preset environment parameters, and create the preset environment based on the preset environment parameters. The preset environment parameters include: the sample end position of the robotic arm, the sample target position, and the sample obstacle position. When the improved TD3 model to be trained interacts with the preset environment, the current sample state, sample action, sample actual position, sample target position, next sample state, sample reward value and termination flag are obtained for each step length of each path of the robotic arm. The sample reward value is obtained based on a sparse reward function. Based on the current sample state, sample action, sample actual position, sample target position, next sample state, sample reward value, and termination flag corresponding to each step length in each path, the sample path sequence is obtained and stored in the sample experience pool.

[0011] Optionally, after storing the sample path sequence into the sample experience pool, the method further includes: From the sample experience pool, at least one target sample path sequence is determined; Based on hindsight, the HER algorithm is resampled to obtain a new path sequence. The target position of each step in the new path sequence is obtained by sampling from the actual position of the sample corresponding to the step after the current step. The newly added path sequence is stored as a sample path sequence in the sample experience pool.

[0012] Optionally, determining at least one target sample path sequence from the sample experience pool includes: Based on the sample reward value corresponding to each step size, determine whether the path corresponding to the sample path sequence is a path to the sample target location; The sequence of sample paths that do not reach the target location of the sample is determined as the target sample path sequence.

[0013] Optionally, the sparse reward function is:

[0014] in, x end , y end , z end These are the three-dimensional coordinates corresponding to the end position of the robotic arm in each step. x object , y object, z object These are the three-dimensional coordinates corresponding to the target position of the sample.

[0015] Optionally, the step of inputting the next sample state corresponding to each sample sampling sequence into the LSTM-target Actor network in step-size order to obtain the next target action arranged in step-size order includes: The next sample state corresponding to each sample sampling sequence is input into the LSTM-target Actor network in step order to obtain the next preset action arranged in step order; The next preset action is subjected to noise processing to obtain the next target action.

[0016] Optionally, after obtaining the improved TD3 model based on the updated LSTM-current Critic network, the updated LSTM-current Actor network, the updated LSTM-target Actor network, and the updated LSTM-target Critic network, the method further includes: Based on the improved TD3 model, the target path corresponding to the movement of the robotic arm from the actual end position to the target position is obtained.

[0017] Secondly, this application provides a training apparatus for an improved TD3 model for path planning, comprising: The sampling module is used to obtain a sample sampling sequence from the sample experience pool according to a preset sampling rule. The sample experience pool caches the sample path sequence obtained when the improved TD3 model to be trained interacts with a preset environment. The sample path sequence includes: the current sample state, sample action, and next sample state of the robotic arm corresponding to each step length. The sample sampling sequence is a sequence composed of a preset step length of the sample path sequence. The preset step length is less than or equal to the step length of the sample path sequence. The training module is configured to update the parameters of the LSTM-current Critic network based on the current sample state of the robotic arm corresponding to each sample sampling sequence, the sample action, the next sample state corresponding to each sample sampling sequence, the LSTM-current Critic network, the LSTM-target Actor network, and the LSTM-target Critic network, thereby obtaining the updated LSTM-current Critic network; and to obtain the updated LSTM-current Activision network based on the current sample state corresponding to each sample sampling sequence, the LSTM-current Actor network, and the updated LSTM-current Critic network. or network; and for obtaining an updated LSTM-target Actor network based on the updated parameters of the current LSTM-Actor network and the parameters of the target LSTM-Actor network, and for obtaining an updated LSTM-target Critic network based on the updated parameters of the current LSTM-Critic network and the parameters of the target LSTM-Critic network; and for obtaining the improved TD3 model based on the updated current LSTM-Critic network, the updated current LSTM-Actor network, the updated target LSTM-Actor network and the updated target Critic network.

[0018] Optionally, the training module is specifically used for: The current sample state and sample action of the robotic arm corresponding to each sample sampling sequence are input into the LSTM-current Critic network in step order to obtain the first current q value and the second current q value. The next sample state corresponding to each sample sampling sequence is input into the LSTM-target Actor network in step order to obtain the next target action arranged in step order. The next sample state and the next target action corresponding to each sample sampling sequence are input into the LSTM-Target Critic network in step order to obtain the target Q value; Based on the first current q value, the second current q value, and the target Q value, the parameters of the LSTM-current Critic network are updated to obtain the updated LSTM-current Critic network.

[0019] Optionally, the training module is specifically used for: The current sample state corresponding to each sample sampling sequence is input into the LSTM-current Actor network in step order to obtain the current target action arranged in step order; According to the step size order, the current sample state and the current target action corresponding to each sample sampling sequence are input into the updated LSTM-current Critic network to obtain the first updated current q value and the second updated current q value; Based on the first or second update of the current q value, update the parameters of the LSTM-current Actor network to obtain the updated LSTM-current Actor network.

[0020] Optionally, the sample path sequence may further include: the actual location of the sample, the target location of the sample, the sample reward value, and a termination flag; The device further includes: a sample generation module; Before the training module obtains the sample sampling sequence from the sample experience pool according to the preset sampling rules, the sample generation module is used for: Obtain preset environment parameters, and create the preset environment based on the preset environment parameters. The preset environment parameters include: the sample end position of the robotic arm, the sample target position, and the sample obstacle position. When the improved TD3 model to be trained interacts with the preset environment, the current sample state, sample action, sample actual position, sample target position, next sample state, sample reward value and termination flag are obtained for each step length of each path of the robotic arm. The sample reward value is obtained based on a sparse reward function. Based on the current sample state, sample action, sample actual position, sample target position, next sample state, sample reward value, and termination flag corresponding to each step length in each path, the sample path sequence is obtained and stored in the sample experience pool.

[0021] Optionally, after storing the sample path sequence into the sample experience pool, the sample generation module is further used for: From the sample experience pool, at least one target sample path sequence is determined; Based on hindsight, the HER algorithm is resampled to obtain a new path sequence. The target position of each step in the new path sequence is obtained by sampling from the actual position of the sample corresponding to the step after the current step. The newly added path sequence is stored as a sample path sequence in the sample experience pool.

[0022] Optionally, when the sample generation module determines at least one target sample path sequence from the sample experience pool, it is specifically used for: Based on the sample reward value corresponding to each step size, determine whether the path corresponding to the sample path sequence is a path to the sample target location; The sequence of sample paths that do not reach the target location of the sample is determined as the target sample path sequence.

[0023] Optionally, the sparse reward function is:

[0024] in, x end , y end , z end These are the three-dimensional coordinates corresponding to the end position of the robotic arm in each step. x object , y object , z object These are the three-dimensional coordinates corresponding to the target position of the sample.

[0025] Optionally, when the training module inputs the next sample state corresponding to each sample sampling sequence into the LSTM-target Actor network in step-size order to obtain the next target action arranged in step-size order, it is specifically used for: The next sample state corresponding to each sample sampling sequence is input into the LSTM-target Actor network in step order to obtain the next preset action arranged in step order; The next preset action is subjected to noise processing to obtain the next target action.

[0026] Optionally, the device further includes: a path planning module; The route planning module is used for: Based on the improved TD3 model, the target path corresponding to the movement of the robotic arm from the actual end position to the target position is obtained.

[0027] Thirdly, this application provides an electronic device, including: a processor and a memory; The memory stores instructions that the computer executes; The processor executes computer execution instructions stored in memory, causing the processor to perform the method as described in any of the first aspects.

[0028] Fourthly, embodiments of this application provide a readable storage medium including a program or instructions that, when run on a computer, execute the method described in any of the first aspects above.

[0029] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects.

[0030] The training method and apparatus for the improved TD3 model for path planning provided in this application form an improved TD3 model by integrating an LSTM network into the traditional TD3 model. This improved TD3 model includes an LSTM-current Actor network, an LSTM-target Actor network, an LSTM-current Critic network, and an LSTM-target Critic network. Therefore, during model training, the LSTM network in the improved TD3 model can be used to capture time-series information from the environment. The memory capacity of the LSTM network enhances the improved TD3 model's utilization of past information, enabling it to consider historical actions and states when making decisions. This improves its ability to handle sequential dependency problems, thereby increasing the model training speed and improving the performance of the improved TD3 model. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart illustrating a training method for an improved TD3 model for path planning provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of an LSTM-current Critic network provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an LSTM-target Actor network provided in an embodiment of this application; Figure 4 This is a flowchart of a collision detection process provided in an embodiment of this application; Figure 5 A schematic diagram of the structure of a training device for an improved TD3 model for path planning provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application are described clearly and completely below. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are also within the scope of protection of this application.

[0034] Figure 1 This is a flowchart illustrating a training method for an improved TD3 model for path planning, provided as an embodiment of this application. Wherein, Figure 1 The execution subject of the method shown can be, for example, an electronic device such as a computer, laptop, or server. Figure 1 As shown, the method includes: S101. According to the preset sampling rules, sample the sample sequence from the sample experience pool.

[0035] The sample experience pool caches the sample path sequence obtained when the improved TD3 model to be trained interacts with the preset environment. The sample path sequence includes: the current sample state, sample action, and next sample state of the robotic arm corresponding to each step length. The sample sampling sequence is a sequence composed of a preset step length sampled from the sample path sequence. The preset step length is less than or equal to the step length of the sample path sequence.

[0036] In this step, the improved TD3 model includes: LSTM-current Actor network, LSTM-target Actor network, LSTM-current Critic network, and LSTM-target Critic network. Therefore, the input data needs to be sequence data so that LSTM can capture the features between the sequence data, thereby improving the training speed and performance of the improved TD3 model.

[0037] Before training the improved TD3 model, it is necessary to obtain the original experience, that is, to obtain the sample path sequence obtained when the improved TD3 model to be trained interacts with the preset environment. Optionally, the sample path sequence can be obtained in the following ways: S201. Obtain preset environment parameters and create a preset environment based on the preset environment parameters.

[0038] The preset environmental parameters include: the position of the robotic arm's sample end point, the position of the sample target, and the position of the sample obstacle.

[0039] Specifically, users input preset environment parameters through the corresponding human-computer interaction interface on laptops, servers, etc., to set up the environment for the planned path of the improved TD3 model to be trained, i.e., the preset environment. Therefore, the preset environment parameters include: the sample end position of the robotic arm, the sample target position, and the sample obstacle position.

[0040] Wherein, the sample end position of the robotic arm is the coordinate of the initial position of the end of the robotic arm, the sample target position is the coordinate of the position to be reached by the robotic arm, and the sample obstacle position is the coordinate of the position of the obstacle. The sample end position, sample target position and sample obstacle position of the robotic arm can be three-dimensional coordinates or two-dimensional coordinates. This embodiment uses three-dimensional coordinates.

[0041] S202. When the improved TD3 model to be trained interacts with the preset environment, obtain the current sample state, sample action, sample actual position, sample target position, next sample state, sample reward value and termination flag for each step length of each path of the robotic arm.

[0042] The sample reward value is obtained based on a sparse reward function.

[0043] Specifically, the improved TD3 model to be trained plans the path from the end-effector position of the robotic arm to the target position of the sample based on a preset environment. Within each path, the current sample state corresponding to each step size of the robotic arm's movement is recorded. s t Sample actions a t Actual location of the sample g t Sample target location goal Next sample state s t+1 Sample reward value r t and whether to terminate done Among them, the sample reward value r t The sparse reward function is used to determine when the robotic arm reaches the actual position of the sample at that step size. g t At that time, the sample reward value r t A positive reward is given when the robotic arm fails to reach the actual position of the sample within that step length. g t At that time, the sample reward value r t This is a negative reward.

[0044] Optionally, the sparse reward function for calculating sample reward values ​​is: Formula 1 in, x end , y end , z end These are the three-dimensional coordinates corresponding to the end effector position of the robotic arm in each step.x object , y object , z object These are the three-dimensional coordinates corresponding to the target location of the sample.

[0045] According to the path planning task, if the distance between the end position of the robotic arm and the target position is less than a given threshold, it means that the target point has been successfully reached and a positive reward is obtained. If a collision occurs, a negative reward is obtained.

[0046] The improved TD3 model to be trained can be decomposed into two sub-tasks when planning the robotic arm path: reaching the target point and avoiding obstacles. For these tasks, the state S is as follows: Formula 2 For each step, the input is the current state s of the sample, including the three-dimensional coordinates D of the robotic arm's end effector. x end , y end , z end The three-dimensional coordinates B of the sample target location in the environment. x object , y object , z object ), the centroid coordinates C of the obstacle ( x obs , y obs , z obs ).

[0047] The motion space is generally divided into two types: discrete and continuous. Considering the characteristics of robotic arm motion and path planning tasks, the continuous motion space of the robotic arm is as follows: Formula 3 Where A represents the next action to be taken by the end effector of the robotic arm, Δx, Δy, and Δz represent the changes in the end effector of the robotic arm in the x, y, and z directions in the next step, and [ ] represents the range of values ​​for Δx, Δy, and Δz.

[0048] S203. Based on the current sample state, sample action, sample actual position, sample target position, next sample state, sample reward value and termination flag corresponding to each step length in each path, obtain the sample path sequence and store the sample path sequence in the sample experience pool.

[0049] Specifically, for each path, the current sample state corresponding to all step sizes is... st Sample actions a t Actual location of the sample g t Sample target location goal Next sample state s t+1 Sample reward value r t and whether to terminate done As a sample path sequence, it is stored in the sample experience pool.

[0050] Among these issues, the improved TD3 model under training has difficulty guiding the robotic arm to the target location of the sample during path planning. goal If the improved TD3 model continues to output the robotic arm's actions, it affects the efficiency of generating sample path sequences. Therefore, a step size threshold is set to limit the maximum number of robotic arm actions. When the robotic arm's step size reaches the threshold, if it has not yet reached the target position, the action is cancelled. goal The robotic arm stops its movement and obtains the sequence of sample paths corresponding to that path, which is used as failure experience and stored in the sample experience pool; when the robotic arm reaches the sample target position before the step length reaches the step length threshold. goal If the path is successful, the corresponding sample path sequence will be stored as a successful experience in the sample experience pool.

[0051] When training the improved TD3 model to be trained, it is necessary to first sample from the sample experience pool to obtain sample sampling sequences. The preset sampling rules are set by the user and include: the number of sample sampling sequences and the preset step size. For example, two sample sampling sequences are sampled from the sample experience pool, and each sample sampling sequence contains a step size of 8.

[0052] It should be noted that the sample sampling sequence is a series of samples of a preset step length randomly sampled from each sample path sequence. For example, if the sample path sequence contains 14 steps, meaning the robotic arm performs 14 actions, the current sample state corresponding to the 3rd to 10th steps is randomly selected. s t Sample actions a t Actual location of the sample g t Sample target location goal Next sample state s t+1 Sample reward value r t and whether to terminate done This is the sampled sequence.

[0053] S102. Based on the current sample state, sample action, next sample state, LSTM-current Critic network, LSTM-target Actor network, and LSTM-target Critic network of the robotic arm corresponding to each sample sampling sequence, update the parameters of the LSTM-current Critic network to obtain the updated LSTM-current Critic network.

[0054] In this step, one specific implementation of S102 is as follows: S1021. Input the current sample state and sample action of the robotic arm corresponding to each sample sampling sequence into the LSTM-Current Critic network according to the step size order to obtain the first current Critic network. q value and second current q value.

[0055] In this step, for the LSTM-Current Critic network, the current sample state and sample action in each sample sampling sequence are sorted according to the step size and then used as input data into the LSTM-Current Critic network. The LSTM-Current Critic network processes each current sample state... s -Sample Actions a The evaluation is performed, where, since there are two LSTM-current Critic networks, each LSTM-current Critic network outputs the current... q Value, i.e., the first current value q value and second current q value.

[0056] It should be noted that, as Figure 2 As shown, when training the improved TD3 model for the first time, it is necessary to set the hidden state of the LSTM network in the current Critic network, i.e., initialize the hidden state. hc , will each current sample state s -Sample Actions a Initialize hidden state hc The input to the LSTM network is used as input, and the output is passed through two 256×256 fully connected layers and one 256×3 fully connected layer to generate... q Value. This can be expressed by the formula: q = f ( s , a , hc ) Formula 4 S1022. Input the next sample state corresponding to each sample sampling sequence into the LSTM-target Actor network in step order to obtain the next target action arranged in step order.

[0057] In this step, the next sample state corresponding to each step size in the sample sampling sequence is... s t+1 After being sorted according to step size, the data is input into the LSTM-Target Actor network as a single input. The LSTM-Target Actor network then processes the data based on the state of each next sample. s t+1 Output the corresponding next target action. a t+1 .

[0058] Optionally, upon obtaining the next target action... a t+1 At this time, the LSTM-target Actor network needs to process noise in the target action, therefore, it can be expressed by the formula: a t+1 =μ( s t+1 , ha )+ε formula 5 Wherein, ε follows a pruned normal distribution [(N(0, σ ),- c , c )], c >0.

[0059] It should be noted that, as Figure 3 As shown, during the first training of the improved TD3 model to be trained, it is necessary to set the hidden states of the LSTM network in the LSTM-target Actor network, that is, to initialize the hidden states. ha , for each next sample state s t+1 Initialize hidden state ha The input to the LSTM network is used as input, and the output is passed through two 256×256 fully connected layers and one 256×3 fully connected layer before the next target action is output. a t+1 .

[0060] S1023. Input the next sample state and the next target action corresponding to each sample sampling sequence into the LSTM-Target Critic network in the order of step size to obtain the target Q value.

[0061] In this step, the next sample state is the next sample state in the sample sampling sequence. s t+1The next target action is the next target action obtained in S103. a t+1 The next sample state will be sorted according to the step size. s t+1 Next target action a t+1 This data is fed into the LSTM-Target Critic network as initial input. There are two LSTM-Target Critic networks, each processing the next sample state for each pair of samples. s t+1 -Next target action a t+1 The evaluation is performed, and an evaluation value is obtained for each evaluation. The smallest one is selected as the target Q value, so that the final evaluation value is as close as possible to the final target Q value.

[0062] The target Q-value of the LSTM-target Critic network. The initialization process of the LSTM network in the LSTM-target Critic network is similar to that of the LSTM-current Critic network and will not be repeated here.

[0063] S1024, According to the first current q Value, Second Current q Given the target Q-value, update the parameters of the LSTM-current Critic network to obtain the updated LSTM-current Critic network.

[0064] In this step, after obtaining the target Q value, the sample reward value is used... r t Obtain the final target Q value y The formula is expressed as: y = r t + γQ Formula 6 in, γ It is the discount factor, and Q represents the target Q value.

[0065] According to the first current q Value and the final target Q value y Obtain the loss function, and apply gradient descent to the first current output. q The LSTM-current Critic network updates its parameters based on the second current value; q Value and the final target Q value y Obtain the loss function, and apply gradient descent to the second current output. qThe parameters of the current LSTM-critic network are updated to obtain the updated LSTM-current critic network. In this process, updating the parameters of the current LSTM-critic network also updates the parameters of the LSTM network itself.

[0066] The formula for the loss function can be: Formula 7 S103. Based on the current sample state corresponding to each sample sampling sequence, the LSTM-current Actor network, and the updated LSTM-current Critic network, obtain the updated LSTM-current Actor network.

[0067] In this step, one specific implementation of S103 is as follows: S1031. Input the current sample state corresponding to each sample sampling sequence into the LSTM-current Actor network in step order to obtain the current target action arranged in step order.

[0068] In this step, the current sample state corresponding to the sample sampling sequence is... s t According to the step size order, the data is input into the LSTM-Current Actor network as one input. The LSTM-Current Actor network then processes the data based on the current sample state. s t Output the corresponding current target action ,in, Corresponding to the sample sampling sequence a t different.

[0069] It should be noted that the current target action It was obtained after noise processing.

[0070] S1032. Input the current sample state and current target action corresponding to each sample sampling sequence into the updated LSTM-current Critic network according to the step size order to obtain the first updated current Critic network. q Value and second update current q value.

[0071] In this step, the current sample state corresponding to the sample sampling sequence is... s t Current target action Sort the data according to the compensation order and input it as one set of input data into the updated LSTM-Current Critic network. The updated LSTM-Current Critic network then processes each pair of current sample states... st -Current target action Evaluate and obtain the corresponding first update. q Value and second update current q value.

[0072] S1033, Update the current status based on the first update q Value or second update current q The value is updated to the parameters of the LSTM-current Actor network, and the updated LSTM-current Actor network is obtained.

[0073] In this step, when updating the parameters of the LSTM-current Actor network, the first option is to update the current one. q Value or second update current q Value, updated based on the first update. q Value or second update current q The loss function is calculated to update the parameters of the LSTM-current Actor network, resulting in the updated LSTM-current Actor network.

[0074] S104. Based on the updated parameters of the current LSTM Actor network and the parameters of the target LSTM Actor network, obtain the updated LSTM target Actor network.

[0075] In this step, after obtaining the updated parameters of the LSTM-current Actor network, the parameters of the LSTM-target Actor network are updated according to the current parameters of the LSTM-target Actor network and the updated parameters of the LSTM-current Actor network to obtain the updated LSTM-current Critic network.

[0076] S105. Based on the updated parameters of the current LSTM Critic network and the parameters of the target LSTM Critic network, obtain the updated LSTM target Critic network.

[0077] In this step, referring to S104, the parameters of the LSTM-target Critic network are updated based on the updated parameters of the current LSTM-Critic network and the current parameters of the LSTM-target Critic network to obtain the updated LSTM-target Critic network.

[0078] S106. Based on the updated LSTM-current Critic network, the updated LSTM-current Actor network, the updated LSTM-target Actor network, and the updated LSTM-target Critic network, obtain the improved TD3 model.

[0079] In this step, after updating the LSTM-current Actor network, LSTM-target Actor network, LSTM-current Critic network, and LSTM-target Critic network once, the training of the improved TD3 model to be trained is completed once.

[0080] Then, sample sequences are obtained again from the sample experience pool. The LSTM-current Actor network, LSTM-target Actor network, LSTM-current Critic network, and LSTM-target Critic network are updated in the above manner until the training termination condition is met, and the improved TD3 model is obtained.

[0081] Optionally, after S106, the method further includes: S107. Based on the improved TD3 model, obtain the target path corresponding to the movement of the robotic arm from the actual end position to the target position.

[0082] In this step, after obtaining the improved TD3 model, the improved TD3 model will be used in actual production to plan the path for the robotic arm to move from the actual end position to the target position.

[0083] In this embodiment, an improved TD3 model is formed by fusing an LSTM network into the traditional TD3 model, comprising an LSTM-current Actor network, an LSTM-target Actor network, an LSTM-current Critic network, and an LSTM-target Critic network. Therefore, during model training, the LSTM network in the improved TD3 model can be used to capture time-series information from the environment. The memory capacity of the LSTM network enhances the improved TD3 model's utilization of past information, enabling it to consider historical actions and states when making decisions, thereby improving its ability to handle sequence dependency problems, thus increasing the model training speed and improving the performance of the improved TD3 model.

[0084] Optionally, in the above embodiments, the sparse reward function for calculating sample reward values ​​makes policy learning difficult. Therefore, this application introduces the hindsight (HER) method to address the problem of sparse rewards. Thus, after S203, the method further includes: S301. Determine at least one target sample path sequence from the sample experience pool.

[0085] Specifically, when determining the target sample path sequence, one can randomly select one of the sample path sequences as the target sample path sequence, or the following method can be used to determine the target sample path sequence: S3011. Based on the sample reward value corresponding to each step, determine whether the path corresponding to the sample path sequence is a path to the sample target location. S3022. Determine the sample path sequence corresponding to the path that does not reach the target location of the sample as the target sample path sequence.

[0086] Specifically, the sample path sequence also includes: the actual sample position, the target sample position, the sample reward value, and whether it is terminated. In the target sample path sequence shown in S3011-S3012, the path that has not reached the target sample position is determined based on the difference between the actual sample position and the target sample position or the sample reward value corresponding to the last step.

[0087] By selecting paths that do not reach the target location of the sample, i.e., selecting failure experiences as target sample path sequences, and converting failure experiences into success experiences, the improved TD3 training process can learn not only success experiences but also failure experiences, thereby increasing the number of effective sample path sequences. This allows for rapid expansion of the scale of effective sample path sequences without increasing the original sample collection volume.

[0088] S302. Based on hindsight, the HER algorithm is resampled to obtain a new path sequence.

[0089] In the newly added path sequence, the target position of the sample corresponding to each step length is obtained by sampling from the actual position of the sample corresponding to the step length after the current step length.

[0090] S303. Store the newly added path sequence as a sample path sequence in the sample experience pool.

[0091] Specifically, for S301-S303, the specific processing procedure is as follows: First, initialize an empty buffer R; Then, in each set, initial sample states are uniformly sampled from the state space S and the target space G, respectively. s 0 and sample target location goal ; Then, in the steps t =1,2,3,…, T During the interaction with the environment, tuples are obtained ( s t , a t , r t , s t+1 , s goal ); Then, HER stores the state for each step. ξ =( s 0, s 1, s 2, … , s T ), and combine each tuple with the original target. goal They are stored together in the playback buffer R.

[0092] Finally, add the extra tuples and ,in, From state ξ =( s 0, s 1, s 2, … , s T Obtained by uniform sampling within ) Even if the target is not reached. goal HER also provides supplementary rewards, which increases the utilization rate of samples and solves the problem of difficulty in learning caused by sparse rewards. g ， This indicates the target location of the sample corresponding to each step size in the newly added path sequence.

[0093] In this embodiment, the post-experience replay technique (HER) is integrated into the improved TD3 model, which solves the problem of the difficulty in learning sparse reward functions in the environment. By transforming the experience of not reaching the target position into the experience of reaching the new target, the sample is utilized more effectively.

[0094] Optionally, collision detection is a crucial step in robotic arm path planning to ensure path safety and feasibility. This paper uses bounding boxes to simplify the spatial description of obstacles and determine whether the robotic arm's path will collide with surrounding obstacles. Although this method occupies some space, it simplifies the collision detection process to a certain extent, thereby significantly improving the efficiency of obstacle avoidance for the robotic arm.

[0095] Figure 4This is a flowchart of a collision detection process provided in one embodiment of this application. To achieve obstacle avoidance by the robotic arm, the posture of the robotic arm's end effector must be clearly defined. For a six-DOF robotic arm with a specified pose and spatial coordinates, the maximum number of inverse kinematics angle solutions is eight. Correspondingly, each of these eight solutions contains six joint angles. During each training stage, the robotic arm's end effector pose is initialized, and the feasible solution set for the robotic arm to reach the target point is calculated based on inverse kinematics. During the robotic arm's movement, the end effector may collide with the connecting links. In this case, the current solution is marked, and a new solution is selected from the eight solutions. The system then checks whether a collision has occurred. If no collision has occurred, the current solution corresponds to a collision-free path, and the result is fed back to the reward function. Furthermore, if a collision occurs after traversing all eight solutions, it proves that the paths corresponding to the current solution are all unsafe paths, and the robotic arm's end effector pose is reset.

[0096] Figure 5 This is a schematic diagram of the structure of a training device for an improved TD3 model for path planning, provided in an embodiment of this application. Figure 5 As shown, the training device for the improved TD3 model used for path planning includes a sampling module 501 and a training module 502. The device also includes a sample generation module 503 and a path planning module 504.

[0097] The sampling module 501 is used to obtain a sample sampling sequence from the sample experience pool according to a preset sampling rule. The sample experience pool caches the sample path sequence obtained when the improved TD3 model to be trained interacts with a preset environment. The sample path sequence includes: the current sample state, sample action, and next sample state of the robotic arm corresponding to each step length. The sample sampling sequence is a sequence composed of a preset step length of the sample path sequence. The preset step length is less than or equal to the step length of the sample path sequence. Training module 502 is used to update the parameters of the LSTM-current Critic network based on the current sample state of the robotic arm corresponding to each sample sampling sequence, the sample action, the next sample state corresponding to each sample sampling sequence, the LSTM-current Critic network, the LSTM-target Actor network, and the LSTM-target Critic network, and to obtain the updated LSTM-current Critic network. Based on the current sample state corresponding to each sample sampling sequence, the LSTM-current Actor network, and the updated LSTM-current Critic network, an updated LSTM-current Actor network is obtained; and an updated LSTM-target Actor network is obtained based on the parameters of the updated LSTM-current Actor network and the parameters of the LSTM-target Actor network, and an updated LSTM-target Critic network is obtained based on the parameters of the updated LSTM-current Critic network and the parameters of the LSTM-target Critic network; and the improved TD3 model is obtained based on the updated LSTM-current Critic network, the updated LSTM-current Actor network, the updated LSTM-target Actor network, and the updated LSTM-target Critic network.

[0098] Optionally, training module 502 is specifically used for: The current sample state and sample action of the robotic arm corresponding to each sample sampling sequence are input into the LSTM-current Critic network in step order to obtain the first current q value and the second current q value. The next sample state corresponding to each sample sampling sequence is input into the LSTM-target Actor network in step order to obtain the next target action arranged in step order. The next sample state and the next target action corresponding to each sample sampling sequence are input into the LSTM-Target Critic network in step order to obtain the target Q value; Based on the first current q value, the second current q value, and the target Q value, the parameters of the LSTM-current Critic network are updated to obtain the updated LSTM-current Critic network.

[0099] Optionally, training module 502 is specifically used for: The current sample state corresponding to each sample sampling sequence is input into the LSTM-current Actor network in step order to obtain the current target action arranged in step order; According to the step size order, the current sample state and the current target action corresponding to each sample sampling sequence are input into the updated LSTM-current Critic network to obtain the first updated current q value and the second updated current q value; Based on the first or second update of the current q value, update the parameters of the LSTM-current Actor network to obtain the updated LSTM-current Actor network.

[0100] Optionally, the sample path sequence may further include: the actual location of the sample, the target location of the sample, the sample reward value, and a termination flag; Before the training module 502 obtains the sample sampling sequence from the sample experience pool according to the preset sampling rules, the sample generation module 503 is used for: Obtain preset environment parameters, and create the preset environment based on the preset environment parameters. The preset environment parameters include: the sample end position of the robotic arm, the sample target position, and the sample obstacle position. When the improved TD3 model to be trained interacts with the preset environment, the current sample state, sample action, sample actual position, sample target position, next sample state, sample reward value and termination flag are obtained for each step length of each path of the robotic arm. The sample reward value is obtained based on a sparse reward function. Based on the current sample state, sample action, sample actual position, sample target position, next sample state, sample reward value, and termination flag corresponding to each step length in each path, the sample path sequence is obtained and stored in the sample experience pool.

[0101] Optionally, after storing the sample path sequence into the sample experience pool, the sample generation module 503 is further used for: From the sample experience pool, at least one target sample path sequence is determined; Based on hindsight, the HER algorithm is resampled to obtain a new path sequence. The target position of each step in the new path sequence is obtained by sampling from the actual position of the sample corresponding to the step after the current step. The newly added path sequence is stored as a sample path sequence in the sample experience pool.

[0102] Optionally, when the sample generation module 503 determines at least one target sample path sequence from the sample experience pool, it is specifically used for: Based on the sample reward value corresponding to each step size, determine whether the path corresponding to the sample path sequence is a path to the sample target location; The sequence of sample paths that do not reach the target location of the sample is determined as the target sample path sequence.

[0103] Optionally, the sparse reward function is:

[0104] in, x end , y end , z end These are the three-dimensional coordinates corresponding to the end position of the robotic arm in each step. x object , y object , z object These are the three-dimensional coordinates corresponding to the target position of the sample.

[0105] Optionally, when the training module 502 inputs the next sample state corresponding to each sample sampling sequence into the LSTM-target Actor network in step order to obtain the next target action arranged in step order, it is specifically used for: The next sample state corresponding to each sample sampling sequence is input into the LSTM-target Actor network in step order to obtain the next preset action arranged in step order; The next preset action is subjected to noise processing to obtain the next target action.

[0106] Optional, the route planning module 504 is used for: Based on the improved TD3 model, the target path corresponding to the movement of the robotic arm from the actual end position to the target position is obtained.

[0107] The training device for the improved TD3 model for path planning provided in this application embodiment can be referred to the above method embodiment for its specific implementation process. The implementation principle and technical effect are similar, and will not be repeated here.

[0108] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may be a server, such as... Figure 6 As shown, the electronic device includes a processor 601 and a memory 602.

[0109] The memory 602 stores computer-executed instructions.

[0110] The processor 601 executes the computer execution instructions stored in the memory 602, causing the processor 601 to perform the method described in any of the above embodiments.

[0111] The electronic device provided in this application embodiment can be referred to the above method embodiment for its specific implementation process. The implementation principle and technical effect are similar, and will not be repeated here.

[0112] In the above Figure 6 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0113] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0114] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0115] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method shown in the above-described method embodiments.

[0116] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0117] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0118] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A training method for an improved TD3 model for path planning, characterized in that, The method is applied to the improved TD3 model, wherein the improved TD3 model includes: LSTM-current Actor network, LSTM-target Actor network, LSTM-current Critic network, and LSTM-target Critic network; the method includes: According to the preset sampling rules, a sample sampling sequence is obtained by sampling from the sample experience pool. The sample experience pool caches the sample path sequence obtained when the improved TD3 model to be trained interacts with the preset environment. The sample path sequence includes: the current sample state, sample action, and next sample state of the robotic arm corresponding to each step length. The sample sampling sequence is a sequence composed of a preset step length of the sample path sequence. The preset step length is less than or equal to the step length of the sample path sequence. Based on the current sample state of the robotic arm corresponding to each sample sampling sequence, the sample action, the next sample state corresponding to each sample sampling sequence, the LSTM-current Critic network, the LSTM-target Actor network, and the LSTM-target Critic network, update the parameters of the LSTM-current Critic network to obtain the updated LSTM-current Critic network. Based on the current sample state corresponding to each sample sampling sequence, the LSTM-current Actor network, and the updated LSTM-current Critic network, the updated LSTM-current Actor network is obtained; Based on the updated parameters of the current LSTM Actor network and the parameters of the target LSTM Actor network, the updated target LSTM Actor network is obtained; Based on the updated parameters of the current LSTM Critic network and the parameters of the target LSTM Critic network, the updated target LSTM Critic network is obtained. The improved TD3 model is obtained based on the updated LSTM-current Critic network, the updated LSTM-current Actor network, the updated LSTM-target Actor network, and the updated LSTM-target Critic network.

2. The method according to claim 1, characterized in that, The step of updating the parameters of the LSTM-current Critic network based on the current sample state of the robotic arm corresponding to each sample sampling sequence, the sample action, the next sample state corresponding to each sample sampling sequence, the next target action, the LSTM-current Critic network, the LSTM-target Actor network, and the LSTM-target Critic network to obtain the updated LSTM-current Critic network includes: The current sample state and sample action of the robotic arm corresponding to each sample sampling sequence are input into the LSTM-current Critic network in step order to obtain the first current q value and the second current q value. The next sample state corresponding to each sample sampling sequence is input into the LSTM-target Actor network in step order to obtain the next target action arranged in step order. The next sample state and the next target action corresponding to each sample sampling sequence are input into the LSTM-Target Critic network in step order to obtain the target Q value; Based on the first current q value, the second current q value, and the target Q value, the parameters of the LSTM-current Critic network are updated to obtain the updated LSTM-current Critic network.

3. The method according to claim 1, characterized in that, The step of obtaining the updated LSTM-current Actor network based on the current sample state corresponding to each sample sampling sequence, the LSTM-current Actor network, and the updated LSTM-current Critic network includes: The current sample state corresponding to each sample sampling sequence is input into the LSTM-current Actor network in step order to obtain the current target action arranged in step order; According to the step size order, the current sample state and the current target action corresponding to each sample sampling sequence are input into the updated LSTM-current Critic network to obtain the first updated current q value and the second updated current q value; Based on the first or second update of the current q value, update the parameters of the LSTM-current Actor network to obtain the updated LSTM-current Actor network.

4. The method according to claim 1, characterized in that, The sample path sequence further includes: the actual location of the sample, the target location of the sample, the sample reward value, and a termination flag; before obtaining the sample sampling sequence from the sample experience pool according to the preset sampling rules, it also includes: Obtain preset environment parameters, and create the preset environment based on the preset environment parameters. The preset environment parameters include: the sample end position of the robotic arm, the sample target position, and the sample obstacle position. When the improved TD3 model to be trained interacts with the preset environment, the current sample state, sample action, sample actual position, sample target position, next sample state, sample reward value and termination flag are obtained for each step length of each path of the robotic arm. The sample reward value is obtained based on a sparse reward function. Based on the current sample state, sample action, sample actual position, sample target position, next sample state, sample reward value, and termination flag corresponding to each step length in each path, the sample path sequence is obtained and stored in the sample experience pool.

5. The method according to claim 4, characterized in that, After storing the sample path sequence into the sample experience pool, the method further includes: From the sample experience pool, at least one target sample path sequence is determined; Based on hindsight, the HER algorithm is resampled to obtain a new path sequence. The target position of each step in the new path sequence is obtained by sampling from the actual position of the sample corresponding to the step after the current step. The newly added path sequence is stored as a sample path sequence in the sample experience pool.

6. The method according to claim 5, characterized in that, Determining at least one target sample path sequence from the sample experience pool includes: Based on the sample reward value corresponding to each step size, determine whether the path corresponding to the sample path sequence is a path to the sample target location; The sequence of sample paths that do not reach the target location of the sample is determined as the target sample path sequence.

7. The method according to any one of claims 4-6, characterized in that, The sparse reward function is: ,in, x end , y end , z end These are the three-dimensional coordinates corresponding to the end position of the robotic arm in each step. x object , y object , z object These are the three-dimensional coordinates corresponding to the target position of the sample.

8. The method according to claim 2, characterized in that, The step of inputting the next sample state corresponding to each sample sampling sequence into the LSTM-target Actor network in step-size order to obtain the next target action arranged in step-size order includes: The next sample state corresponding to each sample sampling sequence is input into the LSTM-target Actor network in step order to obtain the next preset action arranged in step order; The next preset action is subjected to noise processing to obtain the next target action.

9. The method according to any one of claims 1-8, characterized in that, After obtaining the improved TD3 model based on the updated LSTM-current Critic network, the updated LSTM-current Actor network, the updated LSTM-target Actor network, and the updated LSTM-target Critic network, the process further includes: Based on the improved TD3 model, the target path corresponding to the movement of the robotic arm from the actual end position to the target position is obtained.

10. An electronic device, characterized in that, include: Processor and memory; The memory stores the instructions that the computer executes; The processor executes computer execution instructions stored in memory, causing the processor to perform the method according to any one of claims 1-9.

Images