A robot long-sequence operation task execution method, system and medium

Abstract

The application discloses a kind of robot long time sequence operation task execution method, system and medium, belong to artificial intelligence and robot control technical field;Including the following steps: long time sequence operation task is decomposed into multiple time sequence dependent operation stages, the pose of previous stage end is as the pose of next stage start;For each stage, an independent action decision network is constructed, and a shared value evaluation network is constructed, and the value prediction is output according to the sensing data and stage identification;Sequentially alternating iterative training: if substage fails, it is reset to the first stage and retrained, and it is promoted step by step after success;After all sub-stages are completed, a new global action decision network is created, and its training is guided using the shared value evaluation network until the overall task success rate meets the standard;After training, discard the sub-stage network and evaluation network, and only keep the global action decision network for controlling the robot to execute the task;The application solves the problems of long time sequence task training difficulty and slow convergence, and improves the task success rate and generalization ability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and robot control technology, and in particular to a method, system and medium for executing long-term operation tasks of a robot. Background Technology

[0002] Long-sequence robotic tasks refer to complex tasks consisting of multiple temporally dependent operation stages, such as shaft-hole assembly and object stacking, where the ending state of the previous operation stage serves as the starting state of the next operation stage. These tasks have wide applications in industrial automation and service robotics, but traditional reinforcement learning methods face numerous challenges in solving them.

[0003] End-to-end reinforcement learning methods (such as PPO) treat the entire long-term task as a whole, using a single policy network to directly learn the mapping from state to action. Due to the long task chain, effective reward signals often only appear upon final success, leading to a lack of gradient guidance in the early training stages, making policy convergence difficult. Furthermore, the evaluation network's estimation variance for long-term rewards is large, easily causing the policy to get trapped in local optima. Imitation learning methods based on expert demonstrations (such as GAIL and AIRL) learn policies by introducing teaching data; however, acquiring high-quality demonstration data for long-term tasks is extremely costly, and noise or suboptimal operations in the data can limit policy performance. When the system encounters uncovered states, errors accumulate and amplify over time, resulting in poor robustness. Hierarchical reinforcement learning methods decompose the task into sub-tasks for separate training, but ignore the correlation between sub-tasks. Independent training can easily lead to instability in the policies of preceding sub-tasks, affecting the initial states of subsequent sub-tasks, and it cannot utilize global objective information to stabilize value evaluation. Multi-task reinforcement learning methods achieve joint learning by sharing network parameters. However, when sub-tasks are weakly correlated, sharing knowledge can lead to negative interference, and the performance degradation of preceding sub-tasks can propagate along the timeline, accumulating errors. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, and medium for executing long-term sequential tasks of robots, which solves the problems of difficult training and slow convergence of long-term sequential tasks, and improves the success rate and generalization ability of tasks.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a method for executing long-duration robot operations, comprising the following steps: The long-sequential operation task of the robot to be trained is decomposed into multiple time-dependent operation stages. The robot's pose at the end of the previous operation stage is used as the starting pose of the next operation stage, and a pose attainment threshold is preset for each operation stage. An independent action decision network is constructed for each operation stage, and a shared value evaluation network is constructed for all operation stages. The shared value evaluation network outputs the predicted value of the future cumulative reward under the current pose of the robot in the current operation stage based on the robot's sensor data and the stage number identifier. Training is performed iteratively in the order of the operation stages. Starting from the first operation stage, the action decision network corresponding to each operation stage is trained sequentially. At the same time, the shared value evaluation network is updated using the interaction data collected in the current operation stage and all previous operation stages. If the pose of the current operation stage reaches the preset threshold, the next operation stage is entered. Otherwise, the robot environment is reset to the starting pose of the first operation stage and training is restarted. After the pose of the last operation stage reaches the preset threshold, a new global action decision network is created, and the shared value evaluation network is used to guide the training of the global action decision network. Training stops when the global action decision network achieves a preset threshold in the execution success rate of the robot's long-term operation task for multiple consecutive rounds. After training, the action decision networks and shared value evaluation networks of each operation stage are discarded, and only the trained global action decision network is retained to control the robot to perform target long-term operation tasks.

[0006] In some optional embodiments, the action decision network adopts the Actor network in the proximal policy optimization algorithm, and the shared value evaluation network adopts the Critic network in the proximal policy optimization algorithm.

[0007] In some alternative embodiments, the shared value evaluation network participates in every update step, while the action decision network for each operational stage is updated only a limited number of times in each round of training, making the learning frequency of the shared value evaluation network higher than that of any single action decision network.

[0008] In some optional embodiments, a composite reward mechanism is used during training, which includes a general reward and a specific reward; the general reward increases with the operation stage to incentivize the completion of the operation process; the specific reward is designed according to the specific characteristics of each operation stage to provide a refined evaluation of single-step actions.

[0009] In some optional embodiments, the target robot long-term operation task and the robot long-term operation task to be trained have the same operation phase decomposition structure, but the operation starting pose is different.

[0010] In some optional embodiments, when the robot's working environment is reset to the operation start state of the first operation stage, the operation start state is randomly generated within a preset range.

[0011] Embodiments of the present invention also provide a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described robot long-sequence operation task execution method.

[0012] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when run by a processor, is capable of executing the above-described method for executing long-term robot operation tasks.

[0013] The robot long-term operation task execution method provided by the present invention has at least the following beneficial effects: This invention decomposes long-sequential robot operation tasks into multiple time-dependent operation stages and employs a two-stage learning architecture: independent training of sub-tasks and guidance from a global policy network. First, during iterative training of alternating sub-tasks, a shared value evaluation network learns accurate state value priors. These priors are then used as evaluation benchmarks to guide the rapid convergence of the global action decision network. This effectively overcomes the exploration difficulties and slow convergence problems caused by the large action space and sparse rewards in traditional end-to-end reinforcement learning. Simultaneously, this method ensures sufficient learning and accurate state transitions in preceding operation stages through a failure reset mechanism and a step-by-step advancement strategy, avoiding skill forgetting and error accumulation, and completely eliminating the need for expert demonstration data. Experiments show that this invention achieves excellent high success rates in various complex tasks such as shaft-hole assembly, pushing, position swapping, and stacking, and possesses good generalization ability from simulation environments to real robots, significantly improving the training efficiency and execution robustness of long-sequential operation tasks. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0015] Figure 1 This is a flowchart of a robot long-sequential task generalization execution method based on a partially shared architecture according to an embodiment of the present invention; Figure 2 This is a schematic diagram of shaft hole assembly according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the delivery according to an embodiment of the present invention; Figure 4 This is a schematic diagram of position exchange provided according to an embodiment of the present invention; Figure 5This is a stacking diagram provided according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the shaft and hole assembly process according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the push execution process provided according to an embodiment of the present invention; Figure 8 This is a schematic diagram of a location exchange execution process according to an embodiment of the present invention; Figure 9 This is a schematic diagram of a stacked execution process provided according to an embodiment of the present invention; Figure 10 This is a schematic diagram of the PCRL architecture provided according to an embodiment of the present invention; Figure 11 This is a flowchart of the progressive alternating iterative training of PCRL according to an embodiment of the present invention.

[0016] Figure 12 This is a real-world robotic arm experiment scene diagram of the PCRL algorithm provided according to an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0018] One embodiment of the present invention relates to a method for executing long-term operation tasks of a robot. The implementation details of this method are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0019] The specific flow of a robot long-sequence operation task execution method in this embodiment can be as follows: Figure 1 As shown, it includes: Step 101: Decompose the long-sequential operation task of the robot to be trained into multiple time-dependent operation stages. The robot's pose at the end of the previous operation stage is used as the starting pose of the next operation stage, and a pose attainment threshold is preset for each operation stage. Specifically, a complex long-term task is first decomposed into M operation stages with temporal dependencies, denoted as {T1, T2, ..., T...}. mThe robot pose at the end of the i-th operation phase is the starting pose of the (i+1)-th operation phase. The long-sequence tasks of this invention include, but are not limited to, shaft-hole assembly, pushing, position exchange, and stacking tasks.

[0020] Shaft and hole assembly as follows Figure 2 As shown in the figure, the robotic arm completes a high-precision assembly task of aligning and inserting a pin into a square hole under random initial positions of the pin and the square hole; (The last sentence appears to be incomplete and possibly refers to a different process.) Figure 3 As shown in the figure, the robotic arm performs an obstacle avoidance operation, pushing a target object to a designated area under conditions of random number and location of obstacles; position swapping is as follows... Figure 4 As shown in the figure, the robotic arm performs a two-object manipulation task by moving and swapping the positions of two objects, given that their initial positions are random; the objects are stacked as follows. Figure 5 As shown in the figure, the robotic arm performs a multi-step assembly task by stacking objects into a specified structure under the condition that the initial positions and positional relationships of multiple objects are random.

[0021] Furthermore, the aforementioned long-sequential tasks can all be decomposed into several operational stages with temporal dependencies according to their operational logic. For example, the shaft-hole assembly task can be decomposed into two operational stages: alignment and insertion; the position exchange task can be decomposed into multiple operational stages such as grasping, placing, re-grabbing, and stacking; and the stacking task can be decomposed into multiple grasping and stacking operational stages. Through the above decomposition method, the original long-sequential tasks can be transformed into a sequence of operational stages with clear stage boundaries and sequential dependencies, providing a foundation for the subsequent construction of a partially shared training architecture and progressive alternating iterative training.

[0022] Taking the shaft-hole assembly task as an example, this task consists of a square shaft and a corresponding hole. Alignment errors include translational deviations along the x, y, and z axes, as well as rotational deviations around the z-axis. This task is decomposed into two operation stages: "search" and "insertion." When the distance between the shaft and the hole is less than 4 mm, the pose is considered acceptable; if the shaft is inserted into the hole to a depth exceeding 4 cm, the pose is also considered acceptable. The input states for these two operation stages include the three-dimensional position of the robot's end effector and the three-dimensional position of the target.

[0023] Taking the push-to-reach task as an example, this task refers to a scenario where the target object is surrounded by obstacles, and a pushing action is needed to create a favorable grasping space. This task is divided into two phases: "push" and "reach." A successful push is considered successful when there are fewer than one obstacle (distance < 4cm); a successful reach is considered successful when the distance between the end effector and the target object is within 1cm. In the "push" phase, the input states include the two-dimensional position of the robot's end effector and four-dimensional information about the presence or absence of obstacles in four directions around the target object. In the "reach" phase, the input states are the three-dimensional position of the robot's end effector and the three-dimensional position of the target object. Both phases are weakly correlated, meaning the dynamic processes differ but the goal is the same.

[0024] Taking a position-swapping task as an example, this task involves exchanging the positions of two objects and is broken down into four steps: grasping, placing, grasping again, and stacking. A grasp is considered successful if the object is successfully lifted. For placing and stacking, success is defined as follows: the grasped object's horizontal deviation from the target position is less than 1 cm, and its vertical deviation is less than 2 cm. The input states for all four operation stages include the three-dimensional position of the robot's end effector and the three-dimensional position of the target object.

[0025] Taking the stacking task as an example, this task involves stacking four objects in a specified order from an initial state, and is divided into eight steps: grasp, place, grasp, stack, grasp, stack, grasp, and stack. The success criteria and input states for the operation phases are consistent with those for the position exchange task. In summary, shaft-hole assembly, position exchange, and stacking can be considered highly similar tasks because they share the same position-related state information. Push-to-reach is a cross-domain task relative to other tasks.

[0026] The shaft and hole assembly process is as follows: Figure 6 As shown in the figure, the two-step operation process of the robotic arm moving the pin from its initial position and precisely inserting it into the square hole is illustrated; the push-to-execute process is as follows: Figure 7 As shown in the figure, the process of a robotic arm avoiding obstacles and pushing a target object to a designated area is illustrated; the position exchange execution process is as follows: Figure 8 As shown in the figure, the process of a robotic arm sequentially grasping and swapping the positions of two objects is illustrated in four steps; the stacking process is as follows: Figure 9 As shown, the eight-step long-term operation process of the robotic arm sequentially grasping different objects and completing multi-layer stacking is demonstrated. They intuitively present the process of the PCRL method gradually advancing and completing the complex operation stage sequence in different long-term tasks.

[0027] Step 102: Construct an independent action decision network for each operation stage, and construct a shared value evaluation network for all operation stages. The shared value evaluation network outputs the predicted value of the future cumulative reward under the current pose of the robot in the current operation stage based on the robot's sensor data and the stage number identifier. Inspired by knowledge sharing between operation phases, this invention proposes a novel PCRL method, which is equipped with a single priority evaluator and a single action decision network. A schematic diagram of the PCRL architecture is shown below. Figure 10 As shown, the priority evaluator is constructed using only the shared value evaluation network through a progressively alternating iterative training mechanism, enabling it to maintain a training advantage over the Actor throughout the entire learning process of the original long-duration task. Specifically, the shared value evaluation network participates in every training step and uses a shared value evaluation network designed for decomposition operation stages to perform a unified value evaluation for each operation stage. In contrast, the action decision network is only updated synchronously with the shared value evaluation network.

[0028] Furthermore, the shared value assessment network outputs the state value estimate of the current operation stage state in the form of a state value function. For an interactive data quadruple consisting of the current state, the action to be performed, the immediate reward, and the next state, the current state and the next state are first input into the shared value assessment network to obtain the corresponding state value estimate. Then, an advantage function or a target state value is constructed by combining the immediate reward and the discount factor. The advantage function is used for policy gradient updates in the action decision network, and the target state value is used for error backpropagation and parameter updates in the shared value assessment network. Through this method, the shared value assessment network not only performs unified value assessment but also directly participates in the parameter update process of each action decision network.

[0029] Furthermore, to mitigate negative interference between action phases, we employ a shared value evaluation network mechanism consisting of a shared value evaluation network and multiple action sub-policies. Each action sub-policy independently learns its corresponding action phase, while the shared value evaluation network evaluates all action decision networks through knowledge sharing, thereby further enhancing its evaluation capability. Moreover, unlike standard multi-task reinforcement learning (MTRL), the action phases decomposed in PCRL are completely equivalent to the original long-term task. This unique structural correspondence enables the shared value evaluation network to provide strong and consistent support for other shared value evaluation networks, thus facilitating the construction of a unified and high-performance priority evaluator.

[0030] Step 103: Perform alternating iterative training according to the order of operation stages. Starting from the first operation stage, train the action decision network corresponding to each operation stage in sequence. At the same time, update the shared value evaluation network using the interaction data collected in the current operation stage and all previous operation stages. If the pose of the current operation stage reaches the preset threshold, proceed to the next operation stage. Otherwise, reset the robot environment to the starting pose of the first operation stage and start training again. Continue until the pose of the last operation stage reaches the preset threshold. Then, create a new global action decision network and use the shared value evaluation network to guide the training of the global action decision network. Stop training when the global action decision network achieves the preset threshold for the success rate of the robot's long-term operation task for multiple consecutive rounds. The PCRL training architecture is trained using progressively alternating iterative training, rather than synchronous iterative training. "Progressive" means learning step-by-step according to temporal relationships, while "alternating" means executing the task sequence cyclically. This training paradigm prioritizes ensuring that earlier operation stages are fully learned, making it well-suited for operation stages with temporal dependencies. As the task progresses, the importance of each operation stage gradually decreases. Furthermore, this paradigm enables PCRL to learn and recognize the currently executing operation stage.

[0031] The flowchart of PCRL progressive alternating iterative training is as follows: Figure 11 As shown. Specifically, the decomposed operational stages and the original long-term task are simultaneously incorporated into the learning process. Taking the i-th operational stage as an example, we use the i-th action decision network to interact with the environment and collect data. The data is then stored in the corresponding experience replay buffer. When the amount of data exceeds a threshold, the i-th action decision network and the shared value evaluation network are updated simultaneously. This process is repeated until the current operation phase is successfully completed.

[0032] It is worth noting that although the i-th action decision network completes the current operation stage, this does not mean that it has been fully learned; rather, it only indicates that it has established a certain learning foundation. Subsequently, the ending pose of the i-th operation stage is used as the starting pose of the (i+1)-th operation stage, and the learning of subsequent operation stages continues. If the i-th operation stage fails, the robot environment is reset to the starting pose of the first operation stage. It should be noted that setting the operation stage failure judgment condition is only used to determine whether the current operation stage meets the advancement conditions, not to exhaustively list or diagnose the cause of failure. The technical purpose of resetting the operation is to maintain the state connection between the preceding and following operation stages, so that the starting pose of the subsequent stage when retraining comes from the ending pose of the previous stage after actual execution, rather than an idealized state. Training failures caused by systemic reasons such as reward function design defects, improper hyperparameters, or the inability to complete the task itself can be solved by adjusting the task parameters, which is not a core technical problem of this patent method. This process continues until the final operation stage is completed.

[0033] Specifically, a new global action decision network (i.e., the policy model to be deployed) is created at this stage. The initial parameters of this global action decision network can be randomly initialized or inherited from the parameters of the action decision networks in each operational stage. During training, the aforementioned shared value evaluation network continues to be used, which already possesses accurate state value estimation capabilities during the sub-task training phase. This shared value evaluation network provides value guidance for the global action decision network; that is, the state value estimate and advantage function output by the shared value evaluation network guide the policy gradient update of the global action decision network. Throughout the global training process, the parameters of the shared value evaluation network will continue to be updated, but its main role is to provide stable, low-variance value guidance for the global action decision network, thereby accelerating its convergence. Training stops when the execution success rate of the global action decision network on the robot's long-term operational tasks reaches a preset threshold (e.g., 95%) for several consecutive rounds.

[0034] Step 104: After training is completed, discard the action decision network and shared value evaluation network of each operation stage, and retain only the trained global action decision network to control the robot to perform the target long-term operation task.

[0035] After training, the action decision networks and shared value evaluation networks for each operational stage are discarded and no longer used for deployment. Only the trained global action decision network is retained as the final policy model. This global action decision network can directly execute the complete long-term robot operation task from the initial state without the participation of the evaluator network. The target long-term task and the long-term task to be trained have the same operational stage decomposition structure, but the starting pose of the operation can be different, thus demonstrating the generalization ability.

[0036] Furthermore, a composite reward mechanism is used during training, comprising a general reward and a specific reward. The general reward incentivizes the agent to complete the task and increases with each operational stage. For example, a general reward of r = 2500 × i is given for successfully executing the i-th operational stage; a general penalty (e.g., r = −10) is given for failed attempts. The failure criteria are: the number of exploration steps reaches the maximum value, the robot moves beyond a designated area, or the operational stage termination state does not meet the objective. The specific reward is designed based on the specific characteristics of each operational stage and is used to evaluate single-step actions. For example, in position-related tasks, the negative value of the distance between the current position and the target position is used as the single-step reward; in push-to-reach tasks, a positive reward is given as the number of obstacles around the target object decreases.

[0037] In the training process of this invention, the proximal policy optimization algorithm is used as the reinforcement learning model to implement PCRL. The input states at different operation stages can be set according to the task characteristics. The corresponding loss function is designed as follows:

[0038] The loss function of the Actor network is: ; in It is the importance weight of the old strategy relative to the new strategy, and ε is a hyperparameter used to limit its range of variation. It is the dominance function, expressed as:

[0039] ; in, For instant rewards, This is the current evaluator network's estimate of the state value. This is the discount factor.

[0040] Since all operational stages share a value assessment network, and this shared value assessment network outputs a state value estimate, its loss function covers the state value estimation error for all stages: ; Where the target state value Calculated using the time-difference (TD) method: ; Here, λ is used to control the trade-off between bias and variance.

[0041] In one specific embodiment, the specific training process of the present invention is illustrated using a shaft and hole assembly task.

[0042] The shaft-hole assembly task includes two operation phases: "search" and "insertion". The pose qualification threshold for the search operation phase is that the distance between the shaft and the hole is less than 4 mm, and the pose qualification threshold for the insertion operation phase is that the depth of the shaft inserted into the hole exceeds 4 cm. The input states for both operation phases include the three-dimensional position of the robot end effector and the three-dimensional position of the target.

[0043] During the training phase of the search operation phase, the action decision network corresponding to the search operation phase receives the current state s. t Output the execution action a of the robot's end effector t After the robot moves according to the action, the environment returns an immediate reward r. t And generate the next state s t+1 This yields a training sample for a search operation phase, which includes at least the current operation phase identifier and the current state s. t Execute action a t Instant rewards t and the next state s t+1 And store it in the experience replay pool corresponding to the search operation stage.

[0044] When the number of samples in the experience replay pool reaches a preset threshold during the search operation phase, training samples are retrieved from the experience replay pool for network updates. During the update, the action decision network corresponding to the search operation phase adjusts its state s according to the current state. t Re-output action a t The shared value assessment network receives the current operation stage identifier and the current status. t And action a t Output the state-action value evaluation result for this search operation phase. This evaluation result is compared with the immediate reward r. t Next state s t+1 Both are used to calculate the advantage function or target value, and the gradient is calculated using a proximal policy optimization algorithm. During the search operation phase, the action decision network updates its parameters based on the advantage function, and the shared value evaluation network updates its parameters based on the error between its output evaluation result and the target value.

[0045] Once the pose attainment threshold is met during the search phase, the robot's pose at the end of the search phase is used as the starting pose for the insertion phase. The action decision network corresponding to the insertion phase receives this initial state and outputs the insertion action. After the robot executes the insertion action, it also receives an immediate reward and the next state, forming a training sample containing the current phase identifier, current state, executed action, immediate reward, and next state. The samples from the insertion phase are entered into the experience replay pool corresponding to the insertion phase. When the number of samples reaches a preset threshold, updates are performed according to the same data flow as the search phase: the action decision network corresponding to the insertion phase outputs the action, the shared value evaluation network evaluates the value of the state-action state under the insertion phase, and then the parameters of the action decision network and the shared value evaluation network corresponding to the insertion phase are updated based on the proximal policy optimization algorithm.

[0046] In the above process, the shared value evaluation network uses the same set of network parameters in both the search and insertion phases. The input includes the current phase identifier, and the same physical state is treated as a state-action evaluation object in different phases. Training samples from the search phase are used to update the state-action evaluation relationship in the search phase, and training samples from the insertion phase are used to update the state-action evaluation relationship in the insertion phase.

[0047] After the search and insertion phases are completed, the trained shared value evaluation network is used to construct a priority evaluator, and the action decision network for the entire original long-term task of shaft and hole assembly is trained. During the training of the original long-term task, the action decision network outputs actions based on the current state, while the shared value evaluation network evaluates the value of the original long-term task. The shared value evaluation network provides evaluation support from the decomposition phase to the action decision network, and together they form the priority evaluator to provide value guidance for updating the parameters of the action decision network. When the original shaft and hole assembly task is completed, one round of training ends, and then the next round of progressively alternating iterative training begins again from the search phase until the action decision network for the original long-term task converges.

[0048] In this embodiment, the shearing parameter ε of the proximal policy optimization algorithm is set to 0.2, the GAE coefficient λ is set to 0.97, and the discount factor γ is set to 0.995; the experience replay pool capacity of both the Actor network and the shared value evaluation network is set to 2048, and the learning rate of both the Actor network and the shared value evaluation network is set to 0.0003.

[0049] The technical effects of the present invention will be explained below with reference to specific experimental design and verification.

[0050] In terms of experimental setup, a simulation environment was constructed using Pybullet, and a Panda robot was selected as the actuator. Due to the use of incremental position control via the end effector and the independence of the input state from joint information, the framework of this invention does not depend on a specific robot configuration. Therefore, in real-world experiments, the UR10 was used as the robotic arm. In the simulation environment, state information is directly derived from the environment; in the real world, state information is measured by the robot. Specifically, the pose of the robot's end effector is directly read, and other positional information (such as object-related data) can also be measured through the robot's end effector. For each long-term task, its initial state is randomly generated within a 4cm × 4cm range. Since the focus of this invention is to improve the convergence of reinforcement learning on long-term tasks, the input state of reinforcement learning is simplified to eliminate interference from the state feature extraction process. To simulate errors in real-world scenarios, random Gaussian noise with a distribution of (0, 0.2) is added to the simulated state recognition.

[0051] In the comparative experiments, the method of this invention was compared with two types of methods: benchmark methods commonly used for long-term tasks and methods based on multi-task reinforcement learning. The former includes PPO, GAIL, AIRL, and HRL; while the latter includes MTRL-PPO, SC-AIRL, SA-RL, and SC-RL (the latter is a variant that extends SC-AIRL to the Actor-Critic architecture in reinforcement learning).

[0052] In the comparison between PCRL and benchmark methods for long-duration tasks, the success rates of two-stage, four-stage, and eight-stage operations after 200,000, 300,000, and 500,000 training steps were evaluated. The experimental results of PCRL and the benchmark methods are shown in Table 1.

[0053] Table 1 Comparison of experimental results between PCRL and benchmark methods AIRL and GAIL are almost incapable of completing any long-term tasks, primarily due to their over-reliance on demonstration data, which limits their execution capabilities. PPO shows a slight increase in success rate in the first operational phase of all long-term tasks, but still cannot effectively complete the entire long-term task, mainly because the large exploration space presents inherent challenges to trial-and-error learning. HRL achieves a significant improvement in success rate across all tasks. This improvement is attributed to HRL's hierarchical strategy, where skill strategies for each operational phase can be effectively learned at the operational phase level. However, independent skill learning ignores the relationships between operational phases, leading to performance limitations.

[0054] The method of this invention is based on a hierarchical strategy and further utilizes a shared value evaluation network mechanism, thus achieving a success rate that is superior to or comparable to HRL while relying solely on a single Actor-Critic architecture.

[0055] In the comparison between PCRL and multi-task-based reinforcement learning methods, the experimental results of PCRL and multi-task-based reinforcement learning methods are shown in Table 2.

[0056] Table 2 Comparison of experimental results between PCRL and multi-task reinforcement learning-based methods. It is evident that MTRL-PPO achieves higher success rates than HRL in shaft-hole assembly, position swapping, and stacking tasks, relying on only a single model, thanks to full knowledge sharing during the learning process. However, MTRL-PPO's success rate in the "push" task is significantly lower than HRL's (77% vs. 82%). This is because the "push" task includes weakly correlated operation stages, leading to negative interference during MTRL-PPO's learning process and ultimately impairing the model's performance in task execution. In contrast, SC-AIRL achieves comparable success rates to MTRL-PPO on intra-domain tasks, a result stemming from its combination of partial sharing mechanisms and a hierarchical strategy. For the "push" task, SC-AIRL achieves a significant performance improvement (7%) because the partial sharing mechanism effectively mitigates the negative interference effects encountered in MTRL-PPO. However, as the number of operation stages increases, SC-AIRL fails to achieve satisfactory performance due to its dependence on demonstration data. In comparison, SA-RL and SC-RL achieve high success rates across all tasks, and their performance does not significantly decrease with increasing number of operation stages. This is because both employ a no-demonstration approach and incorporate a partial sharing mechanism. Compared to SA-RL and SC-RL, the PCRL method of this invention still achieves a considerable success rate using only a single Actor-Critic architecture.

[0057] Furthermore, to further illustrate the characteristics of the method of this invention in terms of the number of models and parameters, the number of Actors and the total number of Actor parameters for each method were statistically analyzed, and the results are shown in Table 3. Since the evaluator network does not participate in the policy execution during the deployment phase, only the number of Actor networks and the number of parameters are compared.

[0058] Table 3 Comparison of the number of Actors and total number of Actor parameters for each method AIRL 1(21.7K) 1(21.7K) 1(21.7K) 1(21.7K) 4(86.8K) GAIL 1(21.7K) 1(21.7K) 1(21.7K) 1(21.7K) 4(86.8K) PPO 1(21.7K) 1(21.7K) 1(21.7K) 1(21.7K) 4(86.8K) HRL 2(43.4K) 2(43.4K) 4(86.8K) 8(173.6K) 16(347.2K) SC-AIRL 2(43.4K) 2(43.4K) 4(86.8K) 8(173.6K) 16(347.2K) SC-RL 2(43.4K) 2(43.4K) 4(86.8K) 8(173.6K) 16(347.2K) MTRL-PPO 1(21.7K) 1(21.7K) 1(21.7K) 1(21.7K) 4(86.8K) SA-RL 1(21.7K) 1(21.7K) 1(21.7K) 1(21.7K) 4(86.8K) PCRL 1(21.7K) 1(21.7K) 1(21.7K) 1(21.7K) 4(86.8K) As shown in Table 3, HRL, SC-AIRL, and SC-RL all require multiple Actor networks for each decomposed operation stage. Therefore, as the number of operation stages increases, the number of models and parameters increases significantly. In contrast, MTRL-PPO, SA-RL, and the PCRL method of this invention require only one Actor network for each individual long-term task. Furthermore, the method of this invention maintains a high success rate while relying solely on a single Actor-Critic architecture to complete the training and execution of long-term tasks, thereby reducing the number of models and parameters.

[0059] In terms of real-world experiments, PCRL was directly migrated from the simulation environment to the real world without any fine-tuning. The real-world experiments still focused on four long-time-series tasks (shaft-hole assembly, push-to-reach, position swapping, and stacking). The real-world robotic arm experimental scenario using the PCRL algorithm is shown in the figure below. Figure 12 As shown in the figure, the real experimental platform and object configuration for four tasks—shaft-hole assembly, obstacle pushing, position swapping, and multi-object stacking—are used to verify the transfer effect and actual operation performance of the PCRL algorithm from the simulation environment to the real robotic arm.

[0060] The initial state for each task was randomly generated. Specifically, for the push task, the number and location of surrounding obstacles were randomly determined; while for other tasks, the initial position of the object or pin was randomly assigned within a 4cm × 4cm area. Each task was tested 20 times, and the success rate of PCRL for a single long-term task in both simulated and real-world scenarios is shown in Table 4.

[0061] Table 4. Success rate of PCRL for a single long-term task in simulated and real-world scenarios. Simulation 92% 90% 90% 82% Reality 85% 85% 80% 75% Evaluation results show that PCRL can effectively complete all tasks, with success rates of 85%, 85%, 80%, and 75% in shaft-hole assembly, push-to-send, position exchange, and stacking tasks, respectively. Compared with the results in the simulation environment, the success rate is slightly lower but does not exceed 10%, which is still within a reasonable range. This is attributed to the introduction of Gaussian noise into the input state to simulate errors in real-world scenarios.

[0062] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.

[0063] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0064] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0065] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A method for executing long-term sequential operation tasks of a robot, characterized in that, The method includes: The long-sequential operation task of the robot to be trained is decomposed into multiple time-dependent operation stages. The robot's pose at the end of the previous operation stage is used as the starting pose of the next operation stage, and a pose attainment threshold is preset for each operation stage. An independent action decision network is constructed for each operation stage, and a shared value evaluation network is constructed for all operation stages. The shared value evaluation network outputs the predicted value of the future cumulative reward under the current pose of the robot in the current operation stage based on the robot's sensor data and the stage number identifier. Training is performed iteratively in the order of the operation stages. Starting from the first operation stage, the action decision network corresponding to each operation stage is trained sequentially. At the same time, the shared value evaluation network is updated using the interaction data collected in the current operation stage and all previous operation stages. If the pose of the current operation stage reaches the preset threshold, the next operation stage is entered. Otherwise, the robot environment is reset to the starting pose of the first operation stage and training is restarted. After the pose of the last operation stage reaches the preset threshold, a new global action decision network is created, and the shared value evaluation network is used to guide the training of the global action decision network. Training stops when the global action decision network achieves a preset threshold in the execution success rate of the robot's long-term operation task for multiple consecutive rounds. After training, the action decision networks and shared value evaluation networks of each operation stage are discarded, and only the trained global action decision network is retained to control the robot to perform target long-term operation tasks.

2. The robot long-sequence operation task execution method as described in claim 1, characterized in that, The action decision network adopts the Actor network in the proximal policy optimization algorithm, and the shared value evaluation network adopts the Critic network in the proximal policy optimization algorithm.

3. The robot long-sequence operation task execution method as described in claim 1, characterized in that, The shared value evaluation network participates in every update step, while the action decision network for each operation stage is updated only a limited number of times in each round of training, making the learning frequency of the shared value evaluation network higher than that of any single action decision network.

4. The robot long-sequence operation task execution method as described in claim 1, characterized in that, The training process employs a composite reward mechanism, which includes general rewards and specific rewards. The general rewards increase with each operation stage to incentivize the completion of the operation. The specific rewards are designed based on the specific characteristics of each operation stage to provide a refined evaluation of individual actions.

5. The robot long-sequence operation task execution method as described in claim 1, characterized in that, The target robot's long-term operation task and the robot's long-term operation task to be trained have the same operation phase decomposition structure, but the operation starting poses are different.

6. The robot long-time operation task execution method as described in claim 1, characterized in that, When the robot's working environment is reset to the operation start state of the first operation stage, the operation start state is randomly generated within a preset range.

7. A computer system, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the robot long-sequence operation task execution method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, is capable of performing the robot long-term operation task execution method as defined in any one of claims 1 to 6.

Images