Trajectory planning inverse reinforcement learning method, system and product

Abstract

The trajectory planning inverse reinforcement learning method, system and product belong to the technical field of robots, and solve the technical problems that the existing technology has weak strategy generalization ability and insufficient environment state perception when facing more complex compliant placement tasks. The execution environment and execution parameters of the mechanical arm are initialized; the current state is loaded, and demonstration data is collected; the hierarchical coordinate perception neural network features are extracted according to the current state; the strategy function is constructed to generate the current execution action; the current total reward is calculated based on the current state, the current execution action and the demonstration data; the state value is calculated, and the value advantage of the current execution action is obtained in combination with the current total reward; after the total loss is used to update the corresponding strategy parameters and value parameters, the mechanical arm executes the current action, it is judged whether the current action reaches the target position, and the optimal strategy is output. The present application is used for realizing the human-like string fruit compliant placement of the mechanical arm.

Description

Technical Field

[0001] This invention relates to the field of robotics technology, specifically to trajectory planning inverse reinforcement learning methods, systems, and products. Background Technology

[0002] Fruit harvesting robots have become a core element of smart agriculture due to their unique advantage of replacing manual labor. Developing and applying fresh fruit harvesting robots can meet the production and application needs of standardized greenhouse and orchard planting scenarios. However, existing harvesting robots exhibit significant limitations when faced with more complex compliant placement tasks.

[0003] In the prior art, Chinese patent document CN116985150A discloses a "method and apparatus for planning the trajectory of compliant fruit collection". This method includes: determining a target spatial point set on the collection path of the fruit to be collected, the target spatial point set including start and end spatial points and at least one intermediate spatial point; using the target spatial point set to segment the reference trajectory of the fruit collection, obtaining multiple segmented trajectories and a reference trajectory segment corresponding to each segmented trajectory; using each segmented trajectory to perform imitation learning calculations on its corresponding reference trajectory segment to determine the optimal learning trajectory corresponding to each segmented trajectory, and splicing together each optimal learning trajectory to generate the optimal collection trajectory of the fruit to be collected. However, this technical solution is a motion trajectory morphology description method based on teaching data, emphasizing the structural expression and reproduction of the trajectory. When facing more complex compliant placement tasks, it exhibits weak strategy generalization ability and insufficient environmental state perception.

[0004] In the prior art, Chinese patent document CN118504808A discloses "a trajectory planning method based on inverse reinforcement learning." First, an expert database is collected and established; second, the expert dataset is processed using CTGAN to increase the diversity and coverage of the training data; finally, the reward function and motion strategy are continuously updated and optimized using a maximum entropy inverse reinforcement learning algorithm based on an improved PPO, and real-time cooperative motion of multiple robots is achieved through algorithm evaluation, adjustment, and iteration. However, this technical solution lacks scenario specificity and fine-grained operational considerations. Although it discloses robot inverse reinforcement learning, it only focuses on macroscopic paths and cooperation. Its trajectory planning mainly refers to spatial path obstacle avoidance and efficiency, without addressing the requirements of compliant contact of the end effector with vulnerable objects and fine-grained posture adjustment.

[0005] In summary, existing technologies suffer from weak strategy generalization ability and insufficient environmental state awareness when faced with more complex compliant placement tasks. Summary of the Invention

[0006] This invention addresses the problems of weak strategy generalization ability and insufficient environmental state awareness in existing technologies when facing more complex compliant placement tasks.

[0007] The trajectory planning inverse reinforcement learning method of the present invention includes the following steps:

[0008] Step 1: Initialize the robotic arm's execution environment and execution parameters;

[0009] Step 2: Obtain the current status and load sample data;

[0010] Step 3: Construct a hierarchical coordinate-aware neural network feature extractor to extract features from the hierarchical coordinate-aware neural network based on the current state;

[0011] Step 4: Construct a policy function, input the features of the hierarchical coordinate perception neural network to obtain the probability distribution of the robotic arm's movements, and generate the current action based on the probability distribution of the robotic arm's movements;

[0012] Step 5: Construct a neural reward network and calculate the current total reward based on the current state, the current action, and the demonstration data;

[0013] Step 6: Construct a value function, calculate the state value, and combine it with the current total reward to obtain the value advantage of the current action.

[0014] Step 7: Based on the value advantage of the current action and the probability distribution of the robotic arm's actions, calculate the total loss. After updating the policy parameters of the policy function and the value parameters of the value function using the total loss, use the robotic arm to execute the current action and determine whether the current action has reached the target position. If yes, output the optimal policy; otherwise, repeat steps 2 to 7.

[0015] Furthermore, in one embodiment of the present invention, the hierarchical coordinate perception neural feature extractor in step 3 includes a main network and a secondary network. The main network is used to extract features from the current state to obtain global features. The secondary network is used to extract features from any two of the x, y, and z coordinates of the end effector of the robotic arm in the current state to obtain coordinate features. The hierarchical coordinate perception neural network features are obtained by combining the global features and the coordinate features.

[0016] Furthermore, in one embodiment of the present invention, the main network includes a main input layer, a first main hidden layer, a second main hidden layer, a third main hidden layer, and a main output layer. The first main hidden layer consists of 128 neurons with ReLU and BN, the second main hidden layer consists of 64 neurons with ReLU and BN, and the third main hidden layer consists of 32 neurons with ReLU.

[0017] Furthermore, in one embodiment of the present invention, the sub-network includes a sub-input layer, a first hidden layer, a second hidden layer, a third hidden layer, and a sub-output layer. The first hidden layer consists of 128 neurons with ReLU, the second hidden layer consists of 64 neurons with ReLU and BN, and the third hidden layer consists of 8 neurons with ReLU and BN.

[0018] Furthermore, in one embodiment of the present invention, the strategy function in step 4 adopts a deep neural network structure, which performs nonlinear mapping on the features of the hierarchical coordinate perception neural network and outputs the probability distribution of the robotic arm's actions through a Gaussian distribution.

[0019] Furthermore, in one embodiment of the present invention, step 5, which calculates the current total reward based on the current state, the currently executed action, and the demonstration data, specifically involves:

[0020] Based on the current state and the currently executed action, an agent action reward is obtained. Based on the current state and demonstration data, a demonstration similarity reward is obtained. The agent action reward and the demonstration similarity reward are weighted and fused to obtain the current total reward.

[0021] Furthermore, in one embodiment of the present invention, step 6, in combination with the current total reward, yields the value advantage of the currently executed action, specifically as follows:

[0022] ;

[0023] in, The value advantage of the current action, As a discount factor, This is the current total reward. For state value, For interval Time index, For time-series correlation weights, The function for calculating the Pearson correlation coefficient. This is the current state.

[0024] Furthermore, in one embodiment of the present invention, step 7, which calculates the total loss based on the value advantage of the currently executed action and the probability distribution of the robotic arm's actions, specifically involves:

[0025] Calculate the strategy loss and value loss:

[0026] ;

[0027] ;

[0028] ;

[0029] in, For strategic losses, For strategy parameters, As an experience buffer, For the experience buffer Mid-sampling Find the expected value. For the currently executing action, This represents the probability distribution of the robotic arm's actions. Weights are used to constrain motion smoothness. For the first Rate of change of movement of the step For motion smoothness constraints, For policy entropy weights, For policy entropy, Weighting for the smoothness of movement. For the loss of value, As a value parameter, Entropy regularization;

[0030] Combining strategy loss and value loss, we obtain the total loss. .

[0031] The trajectory planning inverse reinforcement learning system of the present invention is constructed based on the above method and includes the following modules:

[0032] The initialization module initializes the robotic arm's execution environment and parameters.

[0033] Load the module, obtain the current state, and load sample data;

[0034] The feature extraction module constructs a hierarchical coordinate-aware neural network feature extractor to extract features from the hierarchical coordinate-aware neural network based on the current state.

[0035] The action generation module constructs a policy function, inputs the features of the hierarchical coordinate perception neural network, obtains the probability distribution of the robotic arm's actions, and generates the currently executed action based on the probability distribution of the robotic arm's actions.

[0036] The reward module constructs a neural reward network to calculate the current total reward based on the current state, the current action, and the demonstration data.

[0037] The value estimation module constructs a value function, calculates the state value, and combines it with the current total reward to obtain the value advantage of the current action.

[0038] The update and execution module calculates the policy loss and value loss based on the value advantage of the current action and the probability distribution of the robotic arm's actions. After updating the corresponding policy parameters and value c parameters using the policy loss and value loss, the robotic arm executes the current action and determines whether the current action has reached the target position. If so, the optimal policy is output; otherwise, the loading module is re-executed to the update and execution module.

[0039] The present invention provides a computer program product comprising a computer program or instructions which, when executed by a processor, implement the trajectory planning inverse reinforcement learning method as described in any of the preceding claims.

[0040] This invention addresses the problems of weak strategy generalization and insufficient environmental state awareness in existing technologies when facing more complex compliant placement tasks. Specific beneficial effects include:

[0041] 1. This invention proposes a trajectory planning inverse reinforcement learning method. By integrating hierarchical coordinate perception and inverse reinforcement learning, the robotic arm can mimic the compliant operation characteristics of humans. After grasping a bunch of fruits, it can accurately control the placement trajectory to avoid squeezing, bumping, or breaking of the fruit stems, while ensuring the regularity of the fruit placement. This method focuses on trajectory generation strategies under state perception, addresses the trajectory planning problem in the compliant operation process, and emphasizes decision-making mechanisms and policy autonomy. It avoids the problems of weak policy generalization ability and insufficient environmental state perception when facing more complex compliant placement tasks.

[0042] 2. This invention proposes an inverse reinforcement learning method for trajectory planning, which adopts a hierarchical coordinate-aware feature extraction mechanism to overcome the limitations of traditional single coordinate system feature extraction. By combining the outputs of the main network and the secondary network, the feature extractor provides a comprehensive representation of the state space, ensuring that it includes both overall information (captured by the main network) and specific details (captured by the secondary network), thereby providing a more comprehensive understanding of the environment.

[0043] 3. This invention proposes a trajectory planning inverse reinforcement learning method, which constructs a dual-branch neural reward network, synchronously inputs agent action-state and demonstration action-state pairs, and balances demonstration similarity and task goal achievement through weighted fusion, avoiding policy bias of single reward orientation;

[0044] 4. This invention proposes a trajectory planning inverse reinforcement learning method. When evaluating temporal correlation advantages, a Pearson temporal correlation term is introduced into the traditional advantage function to quantify the temporal correlation of states, ensure the continuity of the robotic arm's action sequence, and solve the problem of sudden action changes in dynamic tasks.

[0045] 5. This invention proposes a trajectory planning inverse reinforcement learning method, which samples and fuses smoothness constraint policies. A custom action smoothness constraint term is embedded in the policy loss calculation. The smoothness of human operation is simulated by the dual constraints of the difference and absolute value of the action change rate. At the same time, policy entropy regularization is introduced to balance the policy exploration and utilization, avoid training from getting stuck in local optima, and the value loss is superimposed with the entropy regularization term to improve the generalization ability of state evaluation.

[0046] This invention relates to a robotic arm for human-like compliant placement of bunched fruits, focusing on the placement of bunched fruits after harvesting. The core of the invention is to solve the problem of low-damage placement of bunched fruits (such as grapes, cherry tomatoes, etc.) after harvesting or during processing. Attached Figure Description

[0047] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0048] Figure 1 This is a schematic diagram of the hierarchical coordinate sensing neural feature extractor described in Implementation Method 2;

[0049] Figure 2 This is a schematic diagram of the neural reward network described in Implementation Method 3;

[0050] Figure 3 It is the same deep neural network structure diagram used by the policy function and value function described in Implementation Method 3. Detailed Implementation

[0051] Various embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. The embodiments described with reference to the drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0052] Implementation Method 1: Trajectory Planning Inverse Reinforcement Learning Method, including the following steps:

[0053] Step 1: Initialize the robotic arm's execution environment and execution parameters;

[0054] The initialization of the robotic arm's execution environment and parameters specifically includes: building a physical environment model for the robotic arm's operation, clarifying environmental information such as workspace boundaries and obstacle distribution; setting basic operating parameters for the robotic arm, such as joint range of motion, upper limit of movement speed, load threshold, and control cycle; and initializing the core parameters of the robotic arm's intelligent agent, including the initial weights of the hierarchical coordinate perception neural feature extractor, policy function, neural reward network, and value function, as well as key parameters such as the capacity of the experience buffer, laying the foundation for the subsequent stable operation and learning process of the robotic arm.

[0055] Step 2: Obtain the current status and load sample data;

[0056] The current state refers to the set of observation information of the robotic arm and its end effector at the current moment.

[0057] The example data The sample data represents the state of the robotic arm's end effector at each time point, and is a set of all possible actions. This data was collected in advance and provided as part of the training dataset.

[0058] Step 3: Construct a hierarchical coordinate-aware neural network feature extractor to extract features from the hierarchical coordinate-aware neural network based on the current state;

[0059] Step 4: Construct a policy function, input the features of the hierarchical coordinate perception neural network to obtain the probability distribution of the robotic arm's movements, and generate the current action based on the probability distribution of the robotic arm's movements;

[0060] Step 5: Construct a neural reward network and calculate the current total reward based on the current state, the current action, and the demonstration data;

[0061] Step 6: Construct a value function, calculate the state value, and combine it with the current total reward to obtain the value advantage of the current action.

[0062] Step 7: Based on the value advantage of the current action and the probability distribution of the robotic arm's actions, calculate the total loss. After updating the policy parameters of the policy function and the value parameters of the value function using the total loss, use the robotic arm to execute the current action and determine whether the current action has reached the target position. If yes, output the optimal policy; otherwise, repeat steps 2 to 7.

[0063] Existing technologies have significant limitations in handling the compliant placement of clustered fruits. Traditional methods, employing trajectory segmentation and imitation learning based on taught data, can reproduce preset trajectory patterns, but they heavily rely on expert demonstration data and lack real-time perception of dynamic environments. This results in poor strategy generalization and difficulty adapting to complex conditions such as changes in fruit posture, occlusion, or contact force disturbances. Furthermore, current inverse reinforcement learning frameworks focus on macroscopic path obstacle avoidance and multi-machine collaborative efficiency, failing to establish a multi-layered spatial coordinate perception mechanism. This prevents the simultaneous capture of global pose and local contact states, leading to a disconnect between action decisions and actual interaction, which can easily cause problems such as fruit compression, stem breakage, or misplacement.

[0064] To address the aforementioned issues, this implementation proposes a human-like compliant placement of a string of fruits using a robotic arm based on hierarchical coordinate-aware inverse reinforcement learning. The core process includes: environment and network initialization → expert data loading → hierarchical coordinate-aware neural network feature extraction → policy network action generation → neural reward network → value estimation → iterative termination and optimal policy output. Each step is executed sequentially, and parameter constraints ensure the compliance, low-damage, and human-like nature of the placement action. The core innovation lies in designing a dedicated hierarchical coordinate-aware feature extractor, a contextualized neural reward network, action value estimation, and policy optimization, forming a proprietary technical solution completely independent of the general spatial control reinforcement learning framework. This solves the problems of weak policy generalization and insufficient environmental state awareness that exist in existing technologies when facing more complex compliant placement tasks.

[0065] Implementation Method 2: This implementation method further defines the trajectory planning inverse reinforcement learning method described in Implementation Method 1. In step 3, the hierarchical coordinate perception neural feature extractor includes a main network and a secondary network. The main network is used to extract features from the current state to obtain global features. The secondary network is used to extract features from any two of the x, y, and z coordinates of the end effector of the robotic arm in the current state to obtain coordinate features. The hierarchical coordinate perception neural network features are obtained by combining the global features and the coordinate features.

[0066] This implementation employs a hierarchical coordinate-aware feature extraction mechanism, constructing a hierarchical coordinate-aware neural feature extractor. It extracts features from the hierarchical coordinate-aware neural network based on the current state, overcoming the limitations of traditional single-coordinate-system feature extraction. By combining the outputs of the main network and secondary networks, the feature extractor provides a comprehensive representation of the state space, ensuring that both overall information (captured by the main network) and specific details (captured by the secondary network) are included, thus providing a more comprehensive understanding of the environment. Figure 1 As shown, this is a neural network that preprocesses the input observations before they are received. The hierarchical coordinate-aware network extracts state features, focusing on capturing relevant features from the observations, which can significantly enhance the learning process. When the x and y coordinates of the robotic arm's end effector in the current state are used as inputs to the sub-network, the feature expression of the hierarchical coordinate-aware neural network is:

[0067] ;

[0068] in, This is the current state. The global features output by the main network. For the current state of the robotic arm's end effector coordinates and coordinate, The coordinate features are output by the secondary network.

[0069] The hierarchical coordinate-aware neural feature extractor consists of a main network and subnetworks. Both the main and subnetworks are composed of fully connected layers (FC), rectified linear units (ReLU), and batch normalization (BN), designed to extract task-relevant global information. FC establishes complete connections between all neurons in the previous layer and all neurons in the current layer. ReLU introduces non-linearity, effectively mitigating the vanishing gradient problem and enabling the network to learn complex patterns. BN normalizes the output of each layer to improve convergence speed and stability.

[0070] The main network processes the entire observation space, capturing macroscopic global information. The secondary network processes the x and y coordinates of the observations, representing spatial location information or parameters crucial to the task. By processing these key coordinates separately, the secondary network can extract more detailed spatial features, significantly improving the policy's sensitivity to fine-grained information in the environment and helping the agent better understand relationships within the task.

[0071] 1) The main network architecture is as follows:

[0072] Input layer: encompasses the entire observation space As input, the observation space is the set of all possible states.

[0073] Hidden Layers: The first main hidden layer consists of 128 neurons with ReLU (Rectified Luminous Interference) and BN (Batch Normalization) functions to ensure stable learning. Data processed by the first main hidden layer is input into the second main hidden layer, which has 64 neurons and incorporates ReLU and BN functions. Data processed by the second main hidden layer is then input into the third main hidden layer, which contains 32 ReLU neurons.

[0074] Output layer: The output is a 256-dimensional global feature.

[0075] 2) The sub-network architecture is as follows:

[0076] Input layer: Inputs the x and y coordinates of the observation space. As input.

[0077] Hidden Layers: The first hidden layer consists of 128 ReLU neurons. Data processed by the first hidden layer is input into the second hidden layer, which has 64 neurons, including ReLU and BN neurons. Data processed by the second hidden layer is then input into the third hidden layer, which contains 8 ReLU and BN neurons.

[0078] Output layer: The output is a 256-dimensional coordinate feature.

[0079] By combining the outputs of the main and secondary networks, the hierarchical coordinate-aware neural feature extractor provides a comprehensive representation of the state space. This ensures a more holistic understanding of the environment, including both overall information (captured by the main network) and specific details (captured by the secondary network).

[0080] Implementation Method 3: This implementation method further defines the trajectory planning inverse reinforcement learning method described in Implementation Method 1. In step 4, the policy function adopts a deep neural network structure, which performs nonlinear mapping on the features of the hierarchical coordinate perception neural network and outputs the probability distribution of the robotic arm's actions through a Gaussian distribution.

[0081] In this embodiment, the extracted hierarchical coordinate perception neural network features are input into a pre-constructed policy function, which outputs the probability distribution of the robotic arm's movements based on the input features.

[0082] The policy function employs a deep neural network structure, performing nonlinear mapping on features through fully connected layers or convolutional layers, and ultimately outputting a probability density function of continuous actions through a Gaussian distribution, i.e., the probability distribution of the robotic arm's actions. Based on the output probability distribution of the robotic arm's actions, the current action performed by the robotic arm is obtained through random sampling. This enables feature-based action generation.

[0083] The value function adopts the same deep neural network structure as the policy function in this embodiment.

[0084] In this implementation, both the strategy function and the value function are constructed using an Actor-Critic (participant-evaluator) architecture, such as... Figure 3 As shown, both share a feature extraction layer and have their own dedicated output layers, collaboratively achieving policy optimization and state value assessment, thus solving the problems of parameter redundancy and training instability in existing technologies. Based on this architecture, the policy network (Actor) is responsible for outputting the probability distribution of the robotic arm's actions to guide the agent's actions, while the value network (Critic) is responsible for estimating the value of the environmental state to calculate the action's value advantage. The two networks complete parameter iteration through collaborative training. Specifically:

[0085] Input layer: 256-dimensional global features obtained by a hierarchical coordinate-aware neural feature extractor.

[0086] Hidden layers: The first hidden layer consists of 128 neurons with ReLU, and the second hidden layer has 64 neurons containing ReLU.

[0087] Output layer: The output of the policy is the action probability distribution, and the output of the value is the state value scalar.

[0088] Implementation Method Four: This implementation method further defines the trajectory planning inverse reinforcement learning method described in Implementation Method One. In step 5, the current total reward is calculated based on the current state, the current action, and the demonstration data, specifically as follows:

[0089] Based on the current state and the currently executed action, an agent action reward is obtained. Based on the current state and demonstration data, a demonstration similarity reward is obtained. The agent action reward and the demonstration similarity reward are weighted and fused to obtain the current total reward.

[0090] This implementation constructs a neural reward network, such as Figure 2 As shown, state-action pairs (including current state-agent action pairs) are used. Current state - demonstration action pair Using as input, the agent's action rewards are calculated respectively. Similarity to the demonstration reward The current total reward is then obtained through a weighted fusion strategy, and is expressed as:

[0091] ;

[0092] in, This is the current total reward. The reward weighting coefficient is used to balance the priority between demonstration similarity and goal achievement. To calculate the current state of the agent With demonstration actions The reward for similarity of the examples received To calculate the current state Actions generated by the policy network The reward for the agent's actions.

[0093] The exemplary reward architecture in the neural reward network shown is as follows:

[0094] Input layer: The current state and demonstration actions As input.

[0095] Hidden Layers: The first demonstration hidden layer consists of 128 neurons with ReLU and a 25% dropout rate to stabilize learning. Data processed by the first demonstration hidden layer is fed into the second demonstration hidden layer, which has 64 neurons, ReLU, and a 25% dropout rate. Data processed by the second demonstration hidden layer is fed into the third demonstration hidden layer, which has 32 neurons, ReLU, and a 25% dropout rate. Data processed by the third demonstration hidden layer is fed into the fourth demonstration hidden layer, which has 16 neurons, ReLU, and a 25% dropout rate.

[0096] Output layer: The output is the similarity reward value of the demonstration.

[0097] The agent action reward architecture is as follows:

[0098] Input layer: The current state and agent actions As input.

[0099] Hidden Layers: The first agent hidden layer consists of 128 neurons with ReLU and a 25% dropout rate for stable learning. Data processed by the first agent hidden layer is input into the second agent hidden layer, which has 64 neurons with ReLU and a 25% dropout rate. Data processed by the second agent hidden layer is input into the third agent hidden layer, which contains 32 neurons with ReLU and a 25% dropout rate. Data processed by the third agent hidden layer is input into the fourth agent hidden layer, which contains 16 neurons with ReLU and a 25% dropout rate.

[0100] Output layer: The output is the reward value for the agent's actions.

[0101] Ultimately, the total reward is the sum of the demonstration reward and the agent action reward, based on the reward weighting coefficient.

[0102] This implementation method simultaneously inputs agent action-state and demonstration action-state pairs, and balances demonstration similarity with task goal achievement through weighted fusion, avoiding policy bias driven by a single reward.

[0103] Implementation Method 5: This implementation method further defines the trajectory planning inverse reinforcement learning method described in Implementation Method 1. In step 6, the value advantage of the current action is obtained by combining the current total reward, specifically as follows:

[0104] ;

[0105] in, The value advantage of the current action, As a discount factor, This is the current total reward. For state value, For interval Time index, For time-series correlation weights, The function for calculating the Pearson correlation coefficient. This is the current state.

[0106] This implementation method is a time-series-related advantage function estimation. Based on the current total reward and the state value output by the value function, it calculates the advantage function value and quantifies the value advantage of the current action. Among them, time-series correlation weight Used to control the impact of temporal feature associations on advantage assessment The Pearson correlation coefficient is a function used to measure... Time and The temporal correlation of moments ensures the continuity of the action sequence.

[0107] Implementation Method Six: This implementation method further defines the trajectory planning inverse reinforcement learning method described in Implementation Method One. In step 7, the policy loss and value loss are calculated based on the value advantage of the currently executed action and the probability distribution of the robotic arm's actions. Specifically:

[0108] ;

[0109] ;

[0110] ;

[0111] in, For strategic losses, For strategy parameters, As an experience buffer, For the experience buffer Mid-sampling Find the expected value. For the currently executing action, This represents the probability distribution of the robotic arm's actions. Weights are used to constrain motion smoothness. For the first Rate of change of movement of the step For motion smoothness constraints, For policy entropy weights, For policy entropy, Weighting for the smoothness of movement. For the loss of value, As a value parameter, For entropy regularization.

[0112] This implementation calculates the strategy loss to maximize expected returns while ensuring the new strategy does not significantly deviate from the old strategy. It calculates the strategy loss of the strategy function by combining the value advantage of the current action and the probability distribution of the robotic arm's actions. The strategy parameters... The control strategy outputs the probability of actions based on the state, and the experience buffer is used. Store data generated during the interaction between the intelligent agent and the environment, and use it as an experience buffer. Mid-sampled state-action pairs Find the expected value Eliminate single-sample noise, ensure statistical stability of loss calculation, and avoid parameter oscillations and rate of change of action. Measuring motion change is the core quantitative indicator of smoothness constraints; motion smoothness constraint terms Constraining abrupt changes in actions, simulating the consistency of human operations, and policy entropy. It measures the uncertainty of the probability distribution of actions (the higher the entropy value, the stronger the exploratory nature).

[0113] When calculating the value loss, a simple and efficient multilayer perceptron (MLP) structure is used, and a regularization term is added to avoid overfitting.

[0114] The total loss function is a unified optimization objective used to simultaneously update policy parameters and value parameters, and it calculates the total loss:

[0115] ;

[0116] in, This represents the total loss value.

[0117] Implementation Method Seven: The trajectory planning inverse reinforcement learning system described in this implementation method is constructed based on the method described in Implementation Method One, and includes the following modules:

[0118] The initialization module initializes the robotic arm's execution environment and parameters.

[0119] Load the module, obtain the current state, and load sample data;

[0120] The feature extraction module constructs a hierarchical coordinate-aware neural network feature extractor to extract features from the hierarchical coordinate-aware neural network based on the current state.

[0121] The action generation module constructs a policy function, inputs the features of the hierarchical coordinate perception neural network, obtains the probability distribution of the robotic arm's actions, and generates the currently executed action based on the probability distribution of the robotic arm's actions.

[0122] The reward module constructs a neural reward network to calculate the current total reward based on the current state, the current action, and the demonstration data.

[0123] The value estimation module constructs a value function, calculates the state value, and combines it with the current total reward to obtain the value advantage of the current action.

[0124] The update and execution module calculates the total loss based on the value advantage of the current action and the probability distribution of the robotic arm's actions. After updating the policy parameters of the policy function and the value parameters of the value function using the total loss, the robotic arm executes the current action and determines whether the current action has reached the target position. If so, the optimal policy is output; otherwise, the loading module is re-executed and the update and execution module is re-executed.

[0125] Implementation Method Eight: A computer program product according to this implementation method includes a computer program or instructions, characterized in that the computer program or instructions, when executed by a processor, implement any of the above-described trajectory planning inverse reinforcement learning methods.

[0126] The trajectory planning inverse reinforcement learning method, system, and product proposed in this invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A trajectory planning inverse reinforcement learning method, characterized in that, Includes the following steps: Step 1: Initialize the robotic arm's execution environment and parameters. Step 2: Obtain the current status and load sample data; Step 3: Construct a hierarchical coordinate-aware neural network feature extractor to extract features from the hierarchical coordinate-aware neural network based on the current state; In step 3, the hierarchical coordinate perception neural feature extractor includes a main network and a secondary network. The main network is used to extract features from the current state to obtain global features. The secondary network is used to extract features from any two of the x, y, and z coordinates of the end effector of the robotic arm in the current state to obtain coordinate features. The global features and coordinate features are combined to obtain the hierarchical coordinate perception neural network features. The main network includes a main input layer, a first main hidden layer, a second main hidden layer, a third main hidden layer, and a main output layer. The first main hidden layer consists of 128 neurons with ReLU and BN, the second main hidden layer consists of 64 neurons with ReLU and BN, and the third main hidden layer consists of 32 neurons with ReLU. Step 4: Construct a policy function, input the features of the hierarchical coordinate perception neural network to obtain the probability distribution of the robotic arm's movements, and generate the current action based on the probability distribution of the robotic arm's movements; Step 5: Construct a neural reward network and calculate the current total reward based on the current state, the current action, and the demonstration data; Step 6: Construct a value function, calculate the state value, and combine it with the current total reward to obtain the value advantage of the current action. Step 7: Based on the value advantage of the current action and the probability distribution of the robotic arm's actions, calculate the total loss. After updating the policy parameters of the policy function and the value parameters of the value function using the total loss, use the robotic arm to execute the current action and determine whether the current action has reached the target position. If yes, output the optimal policy; otherwise, repeat steps 2 to 7.

2. The trajectory planning inverse reinforcement learning method according to claim 1, characterized in that, The subnetwork includes a sub-input layer, a first hidden layer, a second hidden layer, a third hidden layer, and a sub-output layer. The first hidden layer consists of 128 neurons with ReLU, the second hidden layer consists of 64 neurons with ReLU and BN, and the third hidden layer consists of 8 neurons with ReLU and BN.

3. The trajectory planning inverse reinforcement learning method according to claim 1, characterized in that, In step 4, the strategy function adopts a deep neural network structure, which performs nonlinear mapping on the features of the hierarchical coordinate perception neural network and outputs the probability distribution of the robotic arm's actions through a Gaussian distribution.

4. The trajectory planning inverse reinforcement learning method according to claim 1, characterized in that, In step 5, the current total reward is calculated based on the current state, the current action being performed, and the demonstration data, specifically as follows: Based on the current state and the currently executed action, an agent action reward is obtained. Based on the current state and demonstration data, a demonstration similarity reward is obtained. The agent action reward and the demonstration similarity reward are weighted and fused to obtain the current total reward.

5. A trajectory planning inverse reinforcement learning system, said system being constructed based on the method described in claim 1, characterized in that, Includes the following modules: The initialization module initializes the robotic arm's execution environment and parameters. Load the module, obtain the current state, and load sample data; The feature extraction module constructs a hierarchical coordinate-aware neural network feature extractor to extract features from the hierarchical coordinate-aware neural network based on the current state. The action generation module constructs a policy function, inputs the features of the hierarchical coordinate perception neural network, obtains the probability distribution of the robotic arm's actions, and generates the currently executed action based on the probability distribution of the robotic arm's actions. The reward module constructs a neural reward network to calculate the current total reward based on the current state, the current action, and the demonstration data. The value estimation module constructs a value function, calculates the state value, and combines it with the current total reward to obtain the value advantage of the current action. The update and execution module calculates the total loss based on the value advantage of the current action and the probability distribution of the robotic arm's actions. After updating the policy parameters of the policy function and the value parameters of the value function using the total loss, the robotic arm executes the current action and determines whether the current action has reached the target position. If so, the optimal policy is output; otherwise, the loading module is re-executed and the update and execution module is re-executed.

6. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the trajectory planning inverse reinforcement learning method as described in any one of claims 1-4.

Images