Robot strategy model training method, device and storage medium

Abstract

The application provides a robot strategy model training method and device and a storage medium, wherein the method comprises the following steps: training a first target teacher strategy by decomposing a task into single object task through constructing a scene data set containing continuous cycle behavior sequence and the task; training a second target teacher strategy from a handover state after execution, and obtaining a target strategy model through distillation. Through double teacher strategy division training and unified distillation, the problems of long time domain error accumulation, weak cross-scene generalization, non-standard state recovery and the like are solved, continuous operation without reset and stable stage transition are realized, the long time domain robustness and generalization ability of the model in a cluttered scene are significantly improved, and a single end-to-end strategy which can be directly deployed is obtained.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and more specifically, to a method, device, and storage medium for training a robot strategy model. Background Technology

[0002] Humanoid robots autonomously perform continuous full-body operations such as walking, grasping, carrying, and placing in chaotic and unstructured environments. This is the core direction of embodied intelligence. In such scenarios, it is necessary to balance control, obstacle avoidance navigation, and multi-task long-term execution, which is crucial for the practical application.

[0003] Existing solutions are based on motion synthesis, physical interaction, and room-scale operation. For example, they are based on reinforcement learning and imitation learning to achieve single-step / single-object interaction and fixed scene control. Some methods introduce VLA models (Vision-Language-Action Model) and skill libraries to improve the robot's basic operation and interaction capabilities.

[0004] However, current methods suffer from drawbacks such as long-term time-series error accumulation, poor cross-scenario generalization, and difficulty in balancing end-to-end and task decomposition. They also lack non-standard pose recovery and stable phase transition mechanisms, and rely heavily on privileged observations, making them difficult to implement in real-world scenarios. Summary of the Invention

[0005] The purpose of this application is to provide a robot strategy model training method, device and storage medium to address the shortcomings of the prior art, in order to solve the problems of long-term error accumulation, poor cross-scene generalization, difficulty in balancing end-to-end and task decomposition, lack of non-standard posture recovery and stable phase transition mechanism, and reliance on privileged observation, which makes it difficult to implement in real-world scenarios.

[0006] To achieve the above objectives, the technical solution adopted in this application is as follows: Firstly, this application provides a method for training a robot policy model, the method comprising: Construct a scene dataset and an initial task corresponding to the scene dataset. The scene dataset includes multiple scene data. The initial task is used to instruct the robot to perform a cyclical behavior sequence on each operation object in the scene within a single consecutive round in the task scene. The cyclical behavior sequence includes: walking, grasping, obstacle avoidance and transportation, placing, releasing objects, and retreating. The initial task is decomposed to obtain a single-object task, and the first initial teacher policy is trained based on the scene dataset and the single-object task to obtain the first target teacher policy. The robot is controlled to execute the initial tasks corresponding to each of the scenario data according to the first target teacher strategy, and the handover state after the robot executes the initial tasks is obtained. Starting from the handover state, the second initial teacher strategy is trained according to the scenario data and the initial tasks to obtain the second target teacher strategy. The first target teacher strategy and the second target teacher strategy are distilled to obtain the target strategy model of the robot.

[0007] Optionally, training the first initial teacher policy based on the scene dataset and the single-object task to obtain the first target teacher policy includes: The first initial teacher policy is pre-trained based on the scene dataset and the single object task to obtain the first intermediate teacher policy. Fine-tuning training of the first intermediate teacher strategy during the release and withdrawal phase yields the first target teacher strategy.

[0008] Optionally, the step of pre-training the first initial teacher policy based on the scene dataset and the single-object task to obtain the first intermediate teacher policy includes: The first initial teacher strategy is trained using target-conditional reinforcement learning. During the training process, the robot is controlled to perform walking, grasping, obstacle avoidance, transportation and placement behaviors in the single object task corresponding to each scene data according to the first initial teacher strategy, so as to obtain the task execution result. The task success reward will be determined based on the task execution results. Determine the prior style reward for adversarial sports; The first initial teacher policy is iteratively modified based on the task success reward and the adversarial movement prior style reward, and the first initial teacher policy after iterative training is used as the first intermediate teacher policy.

[0009] Optionally, the prior style reward for adversarial motion is calculated using a style reward function, which is:

[0010] in, For discriminator, These are the weighting coefficients. Let t represent the environmental state and posture characteristics of the robot at time step t. t represents the environmental state and posture characteristics of the robot at time step t+1. Let t be the action of the robot at time t.

[0011] Optionally, the fine-tuning training of the first intermediate teacher strategy during the release and withdrawal phase to obtain the first target teacher strategy includes: After the object to be operated on is successfully placed, the robot is controlled to release the object to be operated on and retreat to a safe distance according to the first intermediate teacher strategy, and the distance information between the robot and the object to be operated on is determined. The release and withdrawal reward value is determined based on the preset release and withdrawal reward function and the distance information, and the parameters of the first intermediate teacher strategy are optimized based on the release and withdrawal reward value to obtain the first target teacher strategy.

[0012] Optionally, the distance information includes: first distance information between the robot's torso and the placed object, second distance information between the robot's hand and the placed object, and third distance information between the placed object and the target position; The step of determining the release / retreat reward value based on a preset release / retreat reward function and the distance information includes: The placement result is determined based on the third distance information; If the placement result indicates successful placement, then the release and withdrawal reward value is determined based on the first distance information, the second distance information, and the release and withdrawal reward function; If the placement result indicates placement failure, the release / retreat reward value is set to a preset value.

[0013] Optionally, starting from the handover state, training the second initial teacher strategy based on the scenario data and the initial task to obtain the second target teacher strategy includes: The robot is controlled by target conditional reinforcement learning to perform stable recovery of the handover state, target reorientation, and scene obstacle avoidance navigation under each scene data according to the second initial teacher strategy, and then to complete the behavior sequence of grasping, obstacle avoidance and transportation, and placement of the next operation object, and obtain the execution result of the behavior sequence. The second initial teacher strategy is iteratively optimized based on the execution results of the behavior sequence to obtain the second target teacher strategy.

[0014] Optionally, the step of distilling the first target teacher strategy and the second target teacher strategy to obtain the robot's target strategy model includes: Distillation is performed on the first target teacher strategy and the second target teacher strategy to obtain the student strategy; The student strategy is distilled to obtain the target strategy model of the robot.

[0015] Secondly, this application provides an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the robot strategy model training method described above.

[0016] Thirdly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the robot strategy model training method described above.

[0017] The beneficial effects of this application are as follows: By constructing diverse scene datasets and continuous cyclical behavior tasks, long-term multi-object tasks are decomposed into single-object tasks to train the first teacher policy, and then the second teacher policy is trained from the handover state. Finally, a single-objective policy model is obtained through distillation. The second-objective teacher policy can take over from non-standard postures, so the final single-objective policy model also has the ability to continue executing tasks from non-standard postures. This effectively solves problems such as long-term error accumulation, unstable stage transitions, and the inability to recover from non-standard postures. The dual-teacher structure covers the standard initial state and the non-standard handover state respectively, expanding the state coverage and robustness; distillation forms a single model to eliminate hard-coded boundaries and achieve continuous execution without reset; release and retreat behaviors ensure safe stage transitions. This application enables robots to have stronger continuous operation capabilities, cross-scene generalization capabilities, and long-term stability in cluttered scenes. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating an application scenario provided by an embodiment of this application is shown; Figure 2 A flowchart of a robot strategy model training method provided in an embodiment of this application is shown; Figure 3 A flowchart illustrating a training strategy for a first-target teacher, as provided in an embodiment of this application, is shown. Figure 4 A flowchart illustrating a strategy for obtaining a first intermediate teacher, as provided in an embodiment of this application, is shown. Figure 5A flowchart illustrating a strategy for obtaining a first target teacher, as provided in an embodiment of this application, is shown. Figure 6 A flowchart illustrating the determination of a release withdrawal reward provided in an embodiment of this application is shown; Figure 7 A flowchart illustrating a strategy for obtaining a second intermediate teacher, as provided in an embodiment of this application, is shown. Figure 8 This application provides a flowchart of obtaining a target strategy model according to an embodiment of the present application; Figure 9 This paper shows a schematic diagram of the structure of a robot strategy model training device provided in an embodiment of this application; Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0021] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0023] Existing humanoid robot mobility manipulation technologies generally use methods such as reinforcement learning (RL) and imitation learning to train models, achieving motion control, object interaction, and room-scale operation. However, they generally simplify the task time domain or scene distribution, and most rely on fixed skill libraries and privileged state observations. They cannot simultaneously handle long-term continuous operation, cross-scene generalization, and unified end-to-end control, and are also difficult to adapt to chaotic and unstructured environments.

[0024] Specifically, the existing technology has five technical shortcomings. First, it accumulates errors over long time domains, leading to stage transition failures. Second, it has poor cross-scenario generalization, with performance plummeting under unfamiliar layouts. Third, it presents contradictions in RL training, with end-to-end convergence being difficult and layered methods being fragile. Fourth, it lacks the ability to recover from non-standard states, making it impossible to restart the task from abnormal postures. Fifth, it relies on privileged observations and lacks a lightweight deployment solution for VLA, making it difficult to implement in real-world scenarios.

[0025] Based on this, this application proposes a robot policy model training method. By constructing a scene dataset and task containing continuous cyclic behavior sequences, the task is decomposed into a single-object task to train a first target teacher policy; a second target teacher policy is trained from the post-execution handover state, and then the target policy model is obtained through distillation. Through dual-teacher policy training and unified distillation, problems such as long-term error accumulation, weak cross-scene generalization, and inability to recover from non-canonical states are addressed. This achieves continuous operation without reset and stable phase transitions, significantly improving the model's long-term robustness and generalization ability in cluttered scenes, and obtaining a single, directly deployable end-to-end policy.

[0026] First, the application scenarios of the robot in this application will be described, such as... Figure 1 As shown, taking a humanoid robot as an example, the robot is in a long-term cluttered scene, which includes multiple objects to be operated on and multiple obstacles. The robot needs to avoid obstacles during its movement. First, it walks from the starting point to the operation area of ​​the first object to be operated on, then grabs the object to be operated on and moves it to the designated location. After placing the object to be operated on at the designated location, it continues to walk to the next object to be operated on and continues the above process until all objects to be operated on in the scene are operated on.

[0027] The method described in this application involves a robot performing... Figure 1 Prior to the application shown, the process of training the model deployed in the robot should be understood. During robot training, it is possible to base it on... Figure 1 The model is trained on the scenario shown, and the trained model can be deployed on the robot or other control devices to control the robot in similar situations. Figure 1 Perform tasks in the specified scenarios.

[0028] Next, combine Figure 2This paper describes the robot strategy model training method of this application. The execution subject of this method can be an electronic device with model training and data processing capabilities. The electronic device can be deployed inside the robot as a robot controller or communicate with the robot. The specific method is not limited here. Figure 2 As shown, the method includes: S201. Construct a scene dataset and the corresponding initial task. The scene dataset includes multiple scene data. The initial task is used to instruct the robot to perform a cyclical behavior sequence on each operation object in the scene within a single consecutive round in the task scene. The cyclical behavior sequence includes: walking, grasping, obstacle avoidance and transportation, placing, releasing objects, and retreating.

[0029] The scene dataset refers to a collection of scenes consisting of various room types, cluttered layouts, and different operational objects, including information such as scene structure, obstacles, object positions, and target poses. The initial task refers to the complete operational task that the robot is to complete within a single consecutive round.

[0030] Scene data refers to a complete description of a single scene, including layout, obstacles, operable objects, initial pose, and target pose. A scene dataset consists of multiple independent scene data sets, each of which can be stored as a configuration file, scene description text, simulation environment parameters, etc.

[0031] A single continuous round refers to a complete execution process from robot startup to completion of all tasks, without resetting the environment or restarting the process. The object being manipulated refers to an object in the scene that can be grasped, moved, or placed by the robot. A cyclical behavior sequence refers to a complete chain of actions that is repeatedly executed on multiple objects in sequence.

[0032] The initial task requires the robot to perform a fixed sequence of actions on multiple objects in a single, uninterrupted, and unreset process, achieving continuous multi-target operations. This can be defined through task instructions, target sequences, language commands, or by specifying the objects to be operated on sequentially.

[0033] A cyclical behavior sequence includes walking, grasping, obstacle avoidance and transport, placing, releasing the object, and retreating. For example, when a robot needs to manipulate object 1 in a scene, it first walks to the vicinity of object 1, grasps object 1, then transports object 1 to a designated location, avoiding obstacles during the transport process. After reaching the designated location, it places object 1, releases object 1, and retreats to a safe position, no longer touching object 1, thus completing a complete cyclical behavior sequence. The robot can execute cyclical behavior sequences according to a fixed time sequence and guide behavior switching through state machines, reward functions, target conditions, etc.

[0034] S202. Decompose the initial task to obtain a single-object task, and train the first initial teacher policy based on the scene dataset and the single-object task to obtain the first target teacher policy.

[0035] A single-object task refers to a simplified task that deals with only one object and consists of only one round of behavior sequence. The initial task can be split according to the number of objects, task stage, or target point to obtain single-object tasks. This allows long-term tasks involving multiple objects to be broken down into multiple independent single-object tasks, reducing the initial training difficulty.

[0036] Optionally, reinforcement learning, imitation learning, or target-conditional learning can be used to train the first target teacher strategy, enabling the robot to learn a complete sequence of behaviors.

[0037] S203. Control the robot to execute the initial tasks corresponding to each scenario data according to the first target teacher strategy, and obtain the handover state after the robot executes the initial tasks. Starting from the handover state, train the second initial teacher strategy according to the scenario data and the initial tasks to obtain the second target teacher strategy.

[0038] The handover state refers to the non-standard, non-initial posture and position state of the robot after the first teacher's strategy has been executed.

[0039] Optionally, the first target teacher strategy can be executed first to obtain a handover state that can be used for further training. Then, the training of the second initial teacher strategy can begin from the handover state, so that the second initial teacher strategy can learn from the handover state and the resulting second target teacher strategy can also have the ability to continue to complete the task from a non-standard posture.

[0040] In one possible implementation, when training the second initial teacher strategy, the training can continue along the trajectory of the first teacher strategy, using the same or similar reward function during the training process, and strengthening the posture recovery and redirection capabilities.

[0041] S204. Distill the first target teacher strategy and the second target teacher strategy to obtain the robot's target strategy model.

[0042] Distilling the first-objective teacher strategy and the second-objective teacher strategy can integrate the capabilities of the two teacher strategies into a single strategy model, enabling the robot to complete the entire continuous task using a single model.

[0043] In one possible implementation, distillation can be performed using methods such as behavior cloning, DAgger, or knowledge distillation, with a finite state machine distributing monitoring signals during the process.

[0044] In this embodiment, by constructing diverse scene datasets and continuous cyclical behavior tasks, the long-term multi-object task is decomposed into a single-object task to train the first teacher policy, and then the second teacher policy is trained from the handover state. Finally, a single-objective policy model is obtained through distillation. The second target teacher policy can take over from non-standard postures, so the final single-objective policy model also has the ability to continue executing tasks from non-standard postures. This effectively solves problems such as long-term error accumulation, unstable stage transitions, and the inability to recover from non-standard postures. The dual-teacher structure covers the standard initial state and the non-standard handover state respectively, expanding the state coverage and robustness; distillation forms a single model to eliminate hard-coded boundaries and achieve continuous execution without reset; release and retreat behaviors ensure safe stage transitions. This application enables robots to have stronger continuous operation capabilities, cross-scene generalization capabilities, and long-term stability in cluttered scenes.

[0045] The following is a further explanation of how the first initial teacher policy was trained based on the scene dataset and single-object task to obtain the first target teacher policy, as follows: Figure 3 As shown, the above step S202 includes: S301. Pre-train the first initial teacher policy based on the scene dataset and single object task to obtain the first intermediate teacher policy.

[0046] Optionally, the diverse environments provided by the scene dataset can be used as the training background, and the decomposed single-object tasks can be used as the learning objectives to conduct preliminary training on the first initial teacher strategy, so that the first initial teacher strategy can master the complete basic operation process of a single object and obtain the first intermediate teacher strategy with basic capabilities.

[0047] S302. Fine-tune the first intermediate teacher strategy during the release and withdrawal phase to obtain the first target teacher strategy.

[0048] Once the basic operations of the first intermediate teacher strategy have been mastered, fine-tuning training can be conducted specifically for the handover process of releasing objects and retreating. This will enable the first intermediate teacher strategy to learn stable and safe transition behaviors, ultimately resulting in a usable first target teacher strategy.

[0049] This application embodiment enables the initial teacher strategy to master basic single-object operations through pre-training, followed by fine-tuning training for the release and withdrawal phases, thus improving the strategy's capabilities step by step. Pre-training ensures the strategy possesses complete walking, grasping, carrying, and placing capabilities, while fine-tuning training strengthens stable handover and safe withdrawal after placement, avoiding object interference and abnormal posture. Step-by-step training reduces learning difficulty, improves convergence speed and strategy stability, enabling the initial target teacher strategy to not only complete object operations but also provide a reliable handover state for subsequent tasks, ensuring smooth continuous operations over long time domains.

[0050] The following is a further explanation of how the first initial teacher policy was pre-trained based on the scene dataset and the single-object task to obtain the first intermediate teacher policy. Figure 4 As shown, the above step S301 includes: S401. The first initial teacher strategy is trained using target-conditional reinforcement learning. During the training process, the robot is controlled to perform walking, grasping, obstacle avoidance, transportation and placement behaviors in the single object task corresponding to each scene data according to the first initial teacher strategy, and the task execution results are obtained.

[0051] Among them, target-conditional reinforcement learning is a reinforcement learning method oriented towards achieving a preset goal, where the strategy adjusts actions based on the difference between the target state and the current state.

[0052] In one possible implementation, a reward function can be constructed based on the target state, with the target position, object pose, and completion progress as conditional inputs, and trained using online or offline reinforcement learning.

[0053] S402. Determine the task success reward based on the task execution results.

[0054] The task execution result refers to the state achieved by the robot after performing its actions, including whether the target has been reached, whether the grasping was successful, and whether the placement was in place. The task success reward refers to the incentive signal given based on the degree of task completion, used to guide the strategy to optimize in the direction of success.

[0055] During training, the robot can output actions based on the initial teacher's strategy, sequentially completing walking, grasping, obstacle avoidance, transportation, and placement, ultimately generating task execution results that can be used for evaluation. Rewards are set for the task execution results based on distance error, grasping stability, and placement accuracy, or sparse rewards or shaping rewards are used, or rewards are given in stages and steps.

[0056] S403, Determine the prior style reward for adversarial sports.

[0057] Adversarial motion priors refer to regularization mechanisms used to constrain robot motion to more closely resemble natural human motion. Style rewards refer to incentives used to encourage more natural, stable, and human-like robot movements.

[0058] Introducing adversarial motion prior style rewards during training can make robot movements more stable and natural, improving motion rationality and robustness. Optionally, a discriminator can be used to distinguish between natural and unnatural movements.

[0059] S404. Based on the task success reward and the prior style reward of adversarial movement, iteratively modify the first initial teacher policy, and use the first initial teacher policy after iterative training as the first intermediate teacher policy.

[0060] Optionally, the task success reward and the prior style reward of adversarial movement can be weighted and integrated to jointly guide the parameter iteration and update of the first initial teacher policy. When the iterative training converges or reaches the preset condition, training stops and the current policy is set as the first intermediate teacher policy.

[0061] In one possible implementation, the total reward can be obtained by weighted summation of task success reward and style reward, and the first initial teacher policy can be updated using algorithms such as policy gradient and Actor-Critic. The number of training rounds, convergence threshold or success rate index can be preset. Once the conditions are met, the first intermediate teacher policy is saved and used for subsequent fine-tuning training.

[0062] Optionally, the total pre-training reward can be represented as ,in, Rewards for successful task completion. To counteract the prior style reward of the movement.

[0063] The reward for a successful task can be calculated by weighting multiple sub-rewards, as shown below. .

[0064] The sub-reward items include a speed reward for the robot approaching the target object: Reward for robots that are close to the target object: The proximity reward between the hand and the object's point cloud: Reward for the height the object is lifted: Speed ​​bonus for an object approaching a target position: Rewards for objects approaching the target location from a greater distance: Precise reward for close proximity between the object and the target location: Object orientation alignment bonus: .

[0065] in , , These represent the robot, the object, and the target location, respectively. , For the corresponding speed; This is the unit direction vector pointing towards the current target; This refers to the position of the robot's hand. Points on the object point cloud; , , These represent the object's current, initial, and desired heights, respectively. This represents the rotational error between the orientation of the object and the target.

[0066] In this embodiment, a target-conditional reinforcement learning training strategy is adopted. Combining task success rewards ensures operational accuracy, while combining adversarial movement prior style rewards enhances the naturalness and stability of movement. The dual-reward joint iteration enables the first intermediate teacher strategy to efficiently complete basic operations such as walking, grasping, carrying, and placing, while also possessing a stable and natural movement pattern. This reduces the difficulty of subsequent training, provides a reliable foundation for fine-tuning release and withdrawal, and improves the robustness and generalization ability of the overall strategy.

[0067] Optionally, the prior style reward for adversarial motion is calculated using a style reward function, which is:

[0068] in, For discriminator, These are the weighting coefficients. Let t represent the environmental state and posture characteristics of the robot at time step t. t represents the environmental state and posture characteristics of the robot at time step t+1. Let t be the action of the robot at time t.

[0069] Optionally, the robot's state can be composed of position, posture, and velocity, and the robot's actions can be composed of joint targets.

[0070] The following is a further explanation of the first target teacher strategy, derived from the fine-tuning training of the release and withdrawal phase of the first intermediate teacher strategy. Figure 5 As shown, step S302 above includes: S501. After the object is successfully placed, the robot releases the object and retreats to a safe distance according to the first intermediate teacher strategy, and determines the distance information between the robot and the object.

[0071] When the object being manipulated is moved to the vicinity of the target position and meets the preset positional accuracy requirements, the object can be considered to have been successfully placed. At this time, the robot's hand releases its grip, disengages from the object, and controls the robot's torso and hand to actively move away from the placed object to avoid collision interference.

[0072] During the release and retraction process, the distance data between the robot and the placed object is calculated in real time as the basis for reward calculation. For example, the Euclidean distance can be calculated directly from the simulated coordinates, the distance can be calculated from visual points, or the relative distance between the robot's position and the position of the manipulated object can be estimated through the attitude sensor.

[0073] S502. Determine the release and withdrawal reward value based on the preset release and withdrawal reward function and distance information, and optimize the parameters of the first intermediate teacher strategy based on the release and withdrawal reward value to obtain the first target teacher strategy.

[0074] The release / retreat reward function is a dedicated reward function used to encourage the robot to perform release and retreat behaviors. The release / retreat reward value is a reward amount used to optimize the strategy and guide the retreat behavior. Using the release / retreat reward as the optimization objective, the parameters of the first intermediate teacher's strategy are iteratively updated, allowing the robot to learn stable and reliable release and retreat behaviors.

[0075] This embodiment of the application ensures the correct task sequence by confirming successful placement before releasing and retracting. It optimizes the strategy using distance information and a dedicated reward function, allowing the robot to stably detach from the object and retreat to a safe position. This effectively avoids problems such as accidental collisions and continuous clamping after placement, providing a stable and interference-free handover state for subsequent tasks and improving the reliability and robustness of long-term continuous operation.

[0076] Optionally, the distance information includes: first distance information between the robot's torso and the placed object, second distance information between the robot's hand and the placed object, and third distance information between the placed object and the target position.

[0077] The first distance information refers to the spatial distance between the center of the robot's torso and the placed object. The second distance information refers to the minimum spatial distance between the robot's hand and the placed object. The third distance information refers to the spatial distance between the actual position of the placed object and the preset target position.

[0078] The process described above for determining the release / retreat reward value based on a preset release / retreat reward function and distance information is as follows: Figure 6 As shown, it includes: S601. Determine the placement result based on the third distance information.

[0079] Optionally, a distance threshold can be preset; if the distance is less than the threshold, placement is considered successful. Alternatively, interval judgment or hierarchical judgment can be used to determine the placement result. The placement result is used to indicate whether the object has been successfully placed in the target area.

[0080] S602. If the placement result indicates successful placement, then determine the release / retreat reward value based on the first distance information, the second distance information, and the release / retreat reward function.

[0081] S603. If the placement result indicates that the placement failed, the release retreat reward value will be set to the default value.

[0082] The release and withdrawal reward function can be defined as:

[0083]

[0084]

[0085] When the distance between the robot's torso and the placed object Exceeding the threshold hour, Saturation is 1; when the hand is at a distance Exceeding the threshold hour, Saturation is 1. This applies when the distance between the object and the target position is... Exceeding the tolerance threshold When the placement fails, both rewards are reset to zero to prevent premature withdrawal; the default value can be 0.

[0086] The final release-retreat reward function is expressed as follows: in, This is a balance coefficient between torso distance and hand distance rewards.

[0087] This application's embodiments achieve precise reward control through three types of distance information: the distance between the object and the target determines placement success or failure, and the distance between the torso and hand monitors release and withdrawal behavior, ensuring the execution logic of placing first, then releasing, and finally withdrawing. This effectively prevents problems such as premature withdrawal, unstable release, and collision interference, making stage transitions more stable, providing a clean and safe initial state for subsequent tasks, and improving the success rate and robustness of continuous operations.

[0088] The following is a further explanation of how, starting from the handover state, the second initial teacher strategy is trained based on scenario data and the initial task to obtain the second target teacher strategy. Figure 7 As shown, step S203 above includes: S701. The robot is controlled by target conditional reinforcement learning to perform stable recovery of handover state, target reorientation and scene obstacle avoidance navigation under each scene data according to the second initial teacher strategy, and then to complete the behavior sequence of grasping, obstacle avoidance and transportation and placement of the next operation object, and obtain the execution result of the behavior sequence.

[0089] Optionally, the robot can be controlled to start from the handover state according to the second initial teacher strategy, with balance constraints and posture errors as optimization objectives. A reward function guides the robot to adjust its center of gravity and joint angles, recovering from tilted or crouching states to a stable standing posture. The target direction is calculated based on the position of the next object to be manipulated, and the robot's orientation is adjusted through a turning reward constraint, ensuring the robot faces the object to be manipulated. Using obstacle distance as the observation input, collision penalties are set, allowing the robot to actively avoid obstacles during movement, employing local path adjustment or dynamic obstacle avoidance methods for safe navigation. After posture recovery, orientation, and navigation are completed, the robot continues to grasp, avoid obstacles, transport, and place the next object, without resetting the environment or re-initializing. Finally, information such as posture recovery effect, orientation deviation, obstacle avoidance status, object position error, and grasping status is recorded to form execution results that can be used for strategy optimization.

[0090] S702. Based on the execution results of the behavior sequence, iteratively optimize the second initial teacher strategy to obtain the second target teacher strategy.

[0091] Optionally, the execution results of the behavioral sequence can be used as a basis to continuously adjust the second initial teacher strategy through reward feedback, so that it can stably complete the subsequent tasks and finally obtain the second target teacher strategy.

[0092] During training, each round first executes the first teacher policy until completion; then, starting from the generated non-canonical terminal state, a second teacher policy is adopted, using the same objective conditional single-object reward function as the pre-training policy for optimization. Although the reward function is the same, the effective training distribution is significantly more difficult due to the lack of a reset long-term setting, making the second teacher policy robust to non-canonical initial states and enabling it to learn implicit obstacle-aware navigation capabilities.

[0093] In this embodiment, target-condition reinforcement learning enables the second teacher strategy to specifically learn recovery from non-standard handover states, reorientation, obstacle avoidance navigation, and subsequent object operations, effectively solving problems such as abnormal posture, misalignment, and obstacle interference in long-term tasks. Iterative optimization gives the strategy strong recovery and continuation capabilities, ensuring uninterrupted and unfailed continuous execution of multiple objects, thereby improving the stability and generalization of the overall process.

[0094] The following is a further explanation of the distillation process performed on the first and second target teacher strategies to obtain the robot's target strategy model, as follows: Figure 8 As shown, step S204 above includes: S801. Distill the first-objective teacher strategy and the second-objective teacher strategy to obtain the student strategy.

[0095] Optionally, methods such as DAgger (Dataset Aggregation), behavior cloning, and knowledge distillation can be used to provide action supervision by corresponding teachers according to the task stage, and use finite state machines to match the supervision sources to distill the two teacher policies into a single end-to-end student policy. This student policy executes a complete long-term round in a single continuous rollout without any intermediate resets or hard-coded stage boundaries.

[0096] In one possible implementation, DAgger can be used for distillation. During data aggregation, the first objective teacher policy provides action supervision in the first object cycle phase (walk-grab-carry-place-release-retreat); the second objective teacher policy provides action supervision in subsequent object cycle phases. At each time step, a finite state machine selects the appropriate supervising teacher based on the round phase, providing phase-adaptive supervision signals while training a single unified policy. The final student policy directly outputs the actions for all phases of the entire round during testing, without requiring any external phase switching logic.

[0097] S802. Distill the student's strategy to obtain the robot's target strategy model.

[0098] Optionally, the unified student policy can be further distilled into a vision-language-action (VLA) model, conditioned on natural language instructions and first-person RGB observations. The VLA model replaces privileged state observations with deployable perceptual modalities. The model's inputs include: text instructions specifying the currently placed target and a first-person RGB video stream from the robot.

[0099] The same DAgger principle as step S801 above can be used to unify the student policy and provide action supervision. The VLA model is trained to mimic these actions only from RGB and language inputs. The multimodal fusion architecture of the VLA model includes: an EfficientNet visual feature extraction backbone network, a small MLP language feature projector, and normalized ontological receptive features. The fused representation is decoded into control actions through a residual MLP.

[0100] This application's embodiments employ a two-stage distillation process to fuse the dual-teacher strategy into a single student strategy, which is then further optimized into the final target strategy model, achieving dual-teacher capabilities and single-model deployment. This retains the high success rate and stability of segmented training while eliminating hard-coded stage switching, enabling uninterrupted and reset-free continuous long-term domain operations. The final strategy exhibits stronger generalization and simpler deployment, stably handling cluttered scenes, non-standard poses, and continuous multi-object tasks.

[0101] Based on the same inventive concept, this application also provides a robot strategy model training device corresponding to the robot strategy model training method. Since the principle of the device in this application is similar to the robot strategy model training method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0102] Reference Figure 9 The diagram shown is a schematic of a robot strategy model training device provided in an embodiment of this application. The device includes: a construction module 901, a first training module 902, a second training module 903, and a distillation module 904.

[0103] Module 901 is used to build a scene dataset and the corresponding initial task. The scene dataset includes multiple scene data. The initial task is used to instruct the robot to perform a cyclic behavior sequence on each operation object in the scene within a single consecutive round in the task scene. The cyclic behavior sequence includes: walking, grasping, obstacle avoidance and transportation, placing, releasing objects, and retreating. The first training module 902 is used to decompose the initial task to obtain a single-object task, and to train the first initial teacher policy based on the scene dataset and the single-object task to obtain the first target teacher policy. The second training module 903 is used to control the robot to execute the initial tasks corresponding to each scene data according to the first target teacher strategy, and to obtain the handover state after the robot executes the initial tasks. Starting from the handover state, the second initial teacher strategy is trained according to the scene data and the initial tasks to obtain the second target teacher strategy. Distillation module 904 is used to distill the first target teacher strategy and the second target teacher strategy to obtain the robot's target strategy model.

[0104] Optionally, the first training module 902 is specifically used for: The first initial teacher policy is pre-trained based on the scene dataset and single-object task to obtain the first intermediate teacher policy. By fine-tuning the first intermediate teacher strategy during the release and withdrawal phase, the first target teacher strategy was obtained.

[0105] Optionally, the first training module 902 is specifically used for: The first initial teacher strategy is trained using target-conditional reinforcement learning. During the training process, the robot is controlled to perform walking, grasping, obstacle avoidance, transportation and placement behaviors in the single object task corresponding to each scene data according to the first initial teacher strategy, and the task execution results are obtained. The reward for a successful task will be determined based on the task execution results. Determine the prior style reward for adversarial sports; The first initial teacher policy is iteratively revised based on the task success reward and the prior style reward of the adversarial movement, and the first initial teacher policy after iterative training is used as the first intermediate teacher policy.

[0106] Optionally, the prior style reward for adversarial motion is calculated using a style reward function, which is:

[0107] in, For discriminator, These are the weighting coefficients. The environmental state and posture characteristics of the robot at time step t. t represents the robot's environmental state and posture characteristics at time step t+1. Let t be the robot's action at time t.

[0108] Optionally, the first training module 902 is specifically used for: After the object is successfully placed, the robot releases the object and retreats to a safe distance according to the first intermediate teacher strategy, and determines the distance information between the robot and the object. The release and withdrawal reward value is determined based on the preset release and withdrawal reward function and distance information. The parameters of the first intermediate teacher strategy are then optimized based on the release and withdrawal reward value to obtain the first target teacher strategy.

[0109] Optionally, the distance information includes: first distance information between the robot's torso and the placed object, second distance information between the robot's hand and the placed object, and third distance information between the placed object and the target position. Optionally, the first training module 902 is specifically used for: The placement result is determined based on the third distance information; If the placement result indicates successful placement, the release and withdrawal reward value is determined based on the first distance information, the second distance information, and the release and withdrawal reward function. If the placement result indicates placement failure, the release retreat bonus value will be set to the default value.

[0110] Optionally, the second training module 903 is specifically used for: The robot is controlled by target-conditional reinforcement learning. According to the second initial teacher strategy, it performs stable recovery of handover state, target reorientation and obstacle avoidance navigation in each scenario data, and then continues to complete the behavior sequence of grasping, obstacle avoidance and transportation and placement of the next operation object, and obtains the execution result of the behavior sequence. The second initial teacher strategy is iteratively optimized based on the execution results of the behavior sequence to obtain the second target teacher strategy.

[0111] Optionally, the distillation module 904 is specifically used for: Distillation is performed on the teacher strategies for the first objective and the teacher strategies for the second objective to obtain the student strategies; The student's strategy is distilled to obtain the robot's target strategy model.

[0112] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0113] This application's embodiments decompose long-term multi-object tasks into single-object tasks by constructing diverse scene datasets and continuous cyclical behavior tasks. A first teacher policy is trained on a single-object task, and a second teacher policy is trained from the handover state. Finally, a single-objective policy model is obtained through distillation. The second teacher policy can take over from non-standard postures, thus the final single-objective policy model also has the ability to continue task execution from non-standard postures. This effectively solves problems such as long-term error accumulation, unstable stage transitions, and the inability to recover from non-standard postures. The dual-teacher structure covers both the standard initial state and the non-standard handover state, expanding state coverage and robustness. Distillation forms a single model, eliminating hard-coded boundaries and achieving continuous execution without reset. Release and retreat behaviors ensure safe stage transitions. This application enables robots to have stronger continuous operation capabilities, cross-scene generalization capabilities, and long-term stability in cluttered scenes.

[0114] This application also provides an electronic device, such as... Figure 10 The diagram shown is a schematic representation of an electronic device structure provided in an embodiment of this application, including a processor 101, a memory 102, and a bus. The memory 102 stores machine-readable instructions executable by the processor 101. When the computer device is running, the processor 101 and the memory 102 communicate via the bus. When the machine-readable instructions are executed by the processor 101, the aforementioned robot strategy model training method is performed.

[0115] This application also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the robot strategy model training method described above.

[0116] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for training a robot policy model, characterized in that, include: Construct a scene dataset and an initial task corresponding to the scene dataset. The scene dataset includes multiple scene data. The initial task is used to instruct the robot to perform a cyclical behavior sequence on each operation object in the scene within a single consecutive round in the task scene. The cyclical behavior sequence includes: walking, grasping, obstacle avoidance and transportation, placing, releasing objects, and retreating. The initial task is decomposed to obtain a single-object task, and the first initial teacher policy is trained based on the scene dataset and the single-object task to obtain the first target teacher policy. The robot is controlled to execute the initial tasks corresponding to each of the scenario data according to the first target teacher strategy, and the handover state after the robot executes the initial tasks is obtained. Starting from the handover state, the second initial teacher strategy is trained according to the scenario data and the initial tasks to obtain the second target teacher strategy. The first target teacher strategy and the second target teacher strategy are distilled to obtain the target strategy model of the robot.

2. The method according to claim 1, characterized in that, The process of training the first initial teacher policy based on the scene dataset and the single-object task to obtain the first target teacher policy includes: The first initial teacher policy is pre-trained based on the scene dataset and the single object task to obtain the first intermediate teacher policy. Fine-tuning training of the first intermediate teacher strategy during the release and withdrawal phase yields the first target teacher strategy.

3. The method according to claim 2, characterized in that, The step of pre-training the first initial teacher policy based on the scene dataset and the single-object task to obtain the first intermediate teacher policy includes: The first initial teacher strategy is trained using target-conditional reinforcement learning. During the training process, the robot is controlled to perform walking, grasping, obstacle avoidance, transportation and placement behaviors in the single object task corresponding to each scene data according to the first initial teacher strategy, so as to obtain the task execution result. The task success reward will be determined based on the task execution results. Determine the prior style reward for adversarial sports; The first initial teacher policy is iteratively modified based on the task success reward and the adversarial movement prior style reward, and the first initial teacher policy after iterative training is used as the first intermediate teacher policy.

4. The method according to claim 3, characterized in that, The prior style reward for adversarial motion is calculated using a style reward function, which is: in, For discriminator, These are the weighting coefficients. Let t represent the environmental state and posture characteristics of the robot at time step t. t represents the environmental state and posture characteristics of the robot at time step t+1. Let t be the action of the robot at time t.

5. The method according to claim 2, characterized in that, The fine-tuning training of the first intermediate teacher strategy during the release and withdrawal phase to obtain the first target teacher strategy includes: After the object to be operated on is successfully placed, the robot is controlled to release the object to be operated on and retreat to a safe distance according to the first intermediate teacher strategy, and the distance information between the robot and the object to be operated on is determined. The release and withdrawal reward value is determined based on the preset release and withdrawal reward function and the distance information, and the parameters of the first intermediate teacher strategy are optimized based on the release and withdrawal reward value to obtain the first target teacher strategy.

6. The method according to claim 5, characterized in that, The distance information includes: first distance information between the robot's torso and the placed object, second distance information between the robot's hand and the placed object, and third distance information between the placed object and the target position. The step of determining the release / retreat reward value based on a preset release / retreat reward function and the distance information includes: The placement result is determined based on the third distance information; If the placement result indicates successful placement, then the release and withdrawal reward value is determined based on the first distance information, the second distance information, and the release and withdrawal reward function; If the placement result indicates placement failure, the release / retreat reward value is set to a preset value.

7. The method according to claim 1, characterized in that, Starting from the handover state, the second initial teacher strategy is trained based on the scenario data and the initial task to obtain the second target teacher strategy, including: The robot is controlled by target conditional reinforcement learning to perform stable recovery of the handover state, target reorientation, and scene obstacle avoidance navigation under each scene data according to the second initial teacher strategy, and then to complete the behavior sequence of grasping, obstacle avoidance and transportation, and placement of the next operation object, and obtain the execution result of the behavior sequence. The second initial teacher strategy is iteratively optimized based on the execution results of the behavior sequence to obtain the second target teacher strategy.

8. The method according to claim 1, characterized in that, The process of distilling the first target teacher strategy and the second target teacher strategy to obtain the robot's target strategy model includes: Distillation is performed on the first target teacher strategy and the second target teacher strategy to obtain the student strategy; The student strategy is distilled to obtain the target strategy model of the robot.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the robot strategy model training method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the robot strategy model training method as described in any one of claims 1-8.

Images