Robots, robot control methods, systems, and readable storage media

Abstract

This invention provides a robot, a robot control method, a system, and a readable storage medium. The method includes: acquiring the robot's pose before and after performing two consecutive target tasks in a preset environment; the robot performing the first target task based on a first control model and the second target task based on a second control model; training the first control model and the second control model based on a preset reward function; collecting control sample pairs of the robot during the training of the first and second control models, and training a robot control model based on the control sample pairs; wherein the control sample pairs include observations and motion quantities of the robot performing the target tasks. This method ensures the robot's motion stability during operation.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control technology, specifically relating to a robot, a robot control method, a system, and a readable storage medium. Background Technology

[0002] Robots are now widely used for moving objects. However, current strategies for robot learning struggle to address the challenges of balance constraints and full-body coordination during robot movement.

[0003] Therefore, it is necessary to improve the existing technology to overcome the aforementioned defects. Summary of the Invention

[0004] The purpose of this invention is to provide a robot, robot control method, system, and readable storage medium that can ensure the motion stability of the robot during operation.

[0005] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0006] In a first aspect, the present invention provides a method for training a robot control model, comprising:

[0007] The robot's pose before and after performing two consecutive target tasks in a preset environment is obtained. The target tasks include grasping and placing target objects. The robot performs the previous target task based on a first control model and the next target task based on a second control model.

[0008] The first control model and the second control model are trained based on a preset reward function, wherein the preset reward function includes at least one of the following: Euclidean distance reward between the robot end effector and the target object, error reward between the expected pose and the current pose of the target object, deviation reward between the robot torso pose and the standard pose, distance reward between the robot and obstacles in the preset environment, and state error reward between the terminal state of the robot after completing the previous task and the initial state of the robot after completing the next task.

[0009] Collect control sample pairs of the robot during the training of the first control model and the second control model, and train the robot control model based on the control sample pairs;

[0010] The control sample pair includes the observations and motion quantities of the robot when it performs the target task.

[0011] In one or more embodiments of the present invention, the method calculates the value of the preset reward function in at least one of the following ways:

[0012] Obtain the Euclidean distance between the robot's end effector and the target object, as well as the error between the target object's desired pose and its current pose, and calculate the weighted sum; calculate the task reward based on the weighted sum.

[0013] Obtain the weighted sum of squares of the robot's torso tilt angle and angular velocity; based on the weighted sum of squares, calculate the deviation reward between the robot's torso posture and the standard posture;

[0014] Obtain the minimum distance between the robot and obstacles in the preset environment, and determine the distance interval to which the minimum distance belongs; determine the reward value corresponding to the distance interval to which the minimum distance belongs as the distance reward between the robot and obstacles in the preset environment; wherein, the distance interval is constructed based on a safe distance threshold;

[0015] Obtain the terminal state of the robot after completing the previous task and the initial state for the next task; calculate the state error reward based on the state error norm of the terminal state and the initial state.

[0016] In one or more embodiments of the present invention, the method further includes:

[0017] Weights are assigned to the task reward, the deviation reward between the robot's torso posture and the standard posture, the distance reward between the robot and obstacles in the preset environment, and the state error reward, respectively. The weights of the task reward and the distance reward between the robot and obstacles in the preset environment are inversely correlated with the number of time steps in which the target task is executed, while the weights of the deviation reward between the robot's torso posture and the standard posture, and the state error reward are positively correlated with the number of time steps in which the target task is executed.

[0018] In one or more embodiments of the present invention, the method further includes:

[0019] Within a preset time window after the task begins, the weight of the task reward is maintained at the maximum value among the weight coefficients.

[0020] In one or more embodiments of the present invention, the two consecutive target tasks include grasping and placing a first target object and grasping and placing a second target object.

[0021] After training the first control model and the second control model, the method further includes:

[0022] The first control model is adjusted based on a safe evacuation reward function, wherein the safe evacuation reward function includes a distance reward between the robot's torso and end effector and the first target object after the robot places the first target object.

[0023] In one or more embodiments of the present invention, the method further includes:

[0024] Determine a loss function for training the robot control model, the loss function being associated with at least one of behavior cloning loss and reinforcement learning loss;

[0025] Wherein, the behavior cloning loss is the expected mean square error between the control command output by the robot control model and the control command output by the first control model or the second control model in the control sample pair; the reinforcement learning loss is a weighted combination of the cut-and-substitute target loss, the value function loss and the policy entropy regularization term of the proximal policy optimization algorithm.

[0026] In one or more embodiments of the present invention, the method further includes:

[0027] Based on the robot control model, the robot is controlled to perform two consecutive target tasks in a preset environment, and the robot's state information, control commands output by the robot control model, and corresponding robot first-view RGB images and natural language commands are collected when the two consecutive target tasks are performed.

[0028] Based on the robot's state information, the control commands output by the robot control model, and the corresponding first-person view RGB image and natural language commands, an interactive model is trained.

[0029] Secondly, the present invention provides a robot control model training system, comprising:

[0030] The acquisition module is used to acquire the pose of the robot before and after performing two consecutive target tasks in a preset environment. The target tasks include grasping and placing target objects. The robot performs the previous target task based on a first control model and the next target task based on a second control model.

[0031] The first training module is used to train the first control model and the second control model based on a preset reward function. The preset reward function includes at least one of the following: Euclidean distance reward between the robot end effector and the target object, error reward between the expected pose and the current pose of the target object, deviation reward between the robot torso pose and the standard pose, distance reward between the robot and obstacles in the preset environment, and state error reward between the terminal state of the robot after completing the previous task and the initial state of the robot after completing the next task.

[0032] The second training module is used to collect control sample pairs of the robot during the training of the first control model and the second control model, and to train the robot control model based on the control sample pairs; wherein, the control sample pairs include the observations and motion quantities of the robot when performing the target task.

[0033] Thirdly, the present invention provides a robot comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform robot control model training.

[0034] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the robot control model training method described above.

[0035] Compared with existing technologies, the robot control model training method provided by this invention improves the stability of robot control by configuring multi-dimensional reward functions. At the same time, since each reward term is a monotonically decreasing convex function, its gradient has a unique global optimal solution, which ensures the global convergence of the control model training and solves the problem of non-convergence of end-to-end learning in long-term tasks.

[0036] Furthermore, the weight values ​​of each reward item are associated with the time step of the target task execution. Based on different task stages, the influence of different reward items on the preset reward function can be dynamically adjusted, improving the adaptability of the robot control model training method of this invention. In addition, this invention sets up staged training for the first control model and the second control model, effectively reducing the task difficulty and exploration space of a single training session. This is beneficial for improving the success rate of long-term tasks. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of a robot control model training method in one embodiment of the present invention;

[0039] Figure 2 This is a structural block diagram of a robot control model training system according to another embodiment of the present invention;

[0040] Figure 3This is a structural block diagram of a robot according to one embodiment of the present invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0042] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0043] While current mainstream end-to-end reinforcement learning methods possess a certain degree of autonomous learning capability, enabling them to directly map environmental observations to control actions to complete movement tasks, these methods face severe efficiency bottlenecks during the exploration process due to the extended task duration and rapid expansion of the state space, making it difficult for model training to converge when dealing with complex tasks that are long-term, multi-stage, and rich in contact points.

[0044] Specifically, industrial robots, in real-world scenarios such as industrial operations, scene inspections, and even home-like environments, need to perform continuous full-body control, including movement, grasping, handling, and placement. The core requirements are cross-scenario generalization ability, continuous execution of long-duration tasks, and end-to-end control capabilities. Existing technologies for movement operations mainly fall into three categories, all of which suffer from technical limitations in adapting to long-duration full-body movement operations in diverse and complex environments:

[0045] The first category is traditional motion control methods, such as optimal controller models based on inverse kinematics (IK). These methods can achieve high-precision single-step motion control, but they lack self-learning and generalization capabilities. They require manual parameter tuning for specific scenarios and tasks, cannot adapt to dynamic obstacles and diverse object interaction needs in cluttered environments, and are difficult to complete long-term multi-step continuous tasks.

[0046] The second category is end-to-end reinforcement learning methods, which achieve movement operations by directly mapping environmental observations to control actions. Although they can learn task strategies autonomously, they suffer from low exploration efficiency in long-term, contact-rich tasks, are prone to non-convergence during training, have poor generalization ability, and experience a significant performance drop in unseen scenes and object configurations.

[0047] The third type is the modular skill library approach, which breaks down movement operations into independent skills such as walking, grasping, and carrying, and builds a skill library. The task is completed by calling skills through a state machine. Although this improves the reliability of individual skills, the switching between skills depends on hard-coded stage boundaries, lacks seamless long-term collaboration capabilities, and the construction of the skill library requires a large amount of manually labeled demonstration data. The adaptation cost to new scenes and new objects is high, and the generalization ability is relatively average.

[0048] In recent years, the development of vision-language-action (VLA) models has provided new possibilities for the natural interaction and environmental adaptation of robots. However, general VLA models lack modeling of the robot's whole-body dynamics and balance constraints, and direct application can easily lead to motion instability and operation failure. At the same time, existing mobile operation research focuses on short-term, single-object interaction tasks and has not designed a dedicated learning framework for long-term, multi-object continuous operation. There is still no operation solution that takes into account cross-scene generalization, continuous long-term execution and end-to-end control.

[0049] In summary, existing technologies cannot achieve full-body control of continuous long-term tasks in diverse and chaotic environments, and have core problems such as weak cross-scene generalization ability, discontinuous switching of modular skill libraries, and lack of dynamic constraints in VLA models.

[0050] Based on this, the inventors developed a completely new technical approach. To solve the training convergence problem, they constructed a multi-dimensional composite reward function to replace the single-task reward. Each sub-item in this system is designed as a monotonically convex function closely related to the robot's operational stability and task completion. This ensures that the gradient space of the entire reward function has a unique globally optimal solution, thus providing a fundamental theoretical guarantee for the global convergence of the control model training.

[0051] Secondly, to adapt to the complex requirements of long-term, multi-object movement operations, the macroscopic long-term task is decomposed into multiple target tasks with logical connections, and multiple control models are configured accordingly for targeted training. This effectively compresses the exploration space of a single learning iteration, ensuring that the strategy at each stage can stably converge to a local optimum. Simultaneously, to bridge the hard-coded stage boundaries that may be introduced by task decomposition, this invention integrates the knowledge and capabilities of multiple control models into a single robot control model through further training. This process not only eliminates explicit switching gaps between task stages but also enables the model to learn smooth transition capabilities across stages, thereby achieving truly seamless execution of long-term continuous movement operations. Furthermore, it effectively inherits and enhances generalization performance when facing diverse scenarios and object configurations.

[0052] Please refer to Figure 1The diagram shown is a flowchart of a robot control model training method according to an embodiment of the present invention. The robot control model training method specifically includes the following steps:

[0053] S101: Obtain the robot's pose before and after performing two consecutive target tasks in a preset environment;

[0054] To address the limitations of existing datasets, such as limited scenario variety and short task durations, this module constructs a large-scale, diverse, long-duration motion operation benchmark dataset. This provides ample training and generalization samples for policy learning, constraining the policy learning space to realistic, cluttered environment task distributions. The module outputs complete configuration information for diverse cluttered scenes / tasks and multiple object classes (including manipulable targets), including scene layout, object attributes (mass / shape / color), initial / target poses, and natural language commands. The core module comprises three sub-modules:

[0055] Multi-room scene construction submodule: Constructs different types of typical scenes. Each scene contains randomly distributed static obstacles (furniture, shelves) and dynamic operable objects to simulate a realistic cluttered environment layout. The arrangement of obstacles and the position of objects in the scene are randomly generated to enhance the diversity of the scene.

[0056] The diverse object configuration submodule defines multiple types of objects with different attributes, including movable manipulable targets, covering object types with different masses (light / medium / heavy), shapes (regular / irregular), and sizes. In each task, two target objects are randomly selected, requiring the robot to complete a continuous grasp-transport-placement cycle to realize the construction of long-term tasks.

[0057] Of course, depending on the robot's final application scenario, the pre-set environment for robot training can also be configured as an open environment for other suitable scenarios. For example, for cargo robots, a logistics warehouse scenario can be constructed, including multi-layer shelves, conveyor belts, and randomly stacked packages. Static obstacles include shelf columns of different heights, forklift lane markings, and weighing platforms fixed to the ground, while dynamic manipulable objects include cardboard boxes of different sizes and irregularly shaped bags. As another example, for home companion robots, a scenario simulating a family living room can be constructed. Static obstacles include sofas, coffee tables, TV cabinets, and floor lamps, while dynamic manipulable objects include water cups of different materials, remote controls (flat and irregularly shaped), and books (cuboids). In these open environments, scene layout parameters (such as shelf spacing and furniture placement) and initial object poses (such as the stacking method of packages on shelves and the distribution range of water cups on the coffee table) can be randomly generated using a preset algorithm to ensure that the robot faces different environmental states each time it is trained, thereby improving the model's adaptability to complex real-world scenarios.

[0058] It should be noted that the target task includes operations such as grasping and placing multiple specified target objects. During the grasping and placing process, operations such as moving the target objects may also be combined. This application does not impose any restrictions on this.

[0059] For example, objective task A is to pick up object T from shelf a to workbench b, and objective task B is to pick up object T from workbench b to conveyor belt c. In this scenario, the end point of objective task A can be considered as object T being picked up to workbench b and the robot maintaining a stable non-standard state. The start point of objective task B can be considered as the robot starting to perform the picking action of object T from the aforementioned stable non-standard state. The objects operated on in the two consecutive objective tasks can also be different objects. For example, objective task A is to pick up object T1 from shelf a to workbench b, and objective task B is to pick up object T2 from workbench b to conveyor belt c.

[0060] In one embodiment, the robot executes a previous target task based on a first control model and a subsequent target task based on a second control model; the first control model and / or the second control model can output action instructions adapted to the current task target by receiving environmental information, the current pose of the machine and target task information.

[0061] Starting from the non-standard terminal state output by the first control model, the robot can complete the execution of the target task corresponding to the second control model without environmental reset, thus solving the problems of motion recovery, reorientation, and obstacle avoidance in the non-standard initial state. In the non-standard initial state, the robot can orient itself toward the corresponding target object by rotating in place and walking along curved paths without human intervention.

[0062] Optionally, motion commands may include, but are not limited to, robot target joint positions, target joint torque commands, or speed commands, to directly adapt to different robot underlying control interfaces and improve the framework's compatibility with different models of industrial robots and humanoid robots.

[0063] In one embodiment of the present invention, in order to provide a stable initial action to the second control model that is executed subsequently, after the first control model that is executed first completes the corresponding target task, the robot can be guided to retreat to a safe position after placing the object by distance reward, thereby generating a stable non-standard terminal state, limiting the error accumulation of long-term tasks and ensuring the continuity of the strategy.

[0064] The retreat to a safe position can be achieved by fine-tuning the first control model trained based on the safe retreat reward function. The safe retreat reward can include the distance reward between the robot's torso and end effector and the first target object after the robot places the first target object.

[0065] For example, after training the first control model, the task reward can be replaced with a safe evacuation target reward. This rewards the robot for maintaining a safe distance between its torso and hands and the object after successfully placing it, while also maintaining its balance to avoid collisions with the placed object during subsequent operations, thus generating a stable terminal state. The details of the task reward and distance reward are described in S102.

[0066] S102: Train the first control model and the second control model based on a preset reward function;

[0067] It should be noted that the preset reward function is the optimization objective of both the first and second control models, used to guide the models to learn correct movement operation behaviors. Simultaneously, the first and second control models reuse the preset reward function to ensure consistency with the strategy of operating the first object. Furthermore, through long-term training without resetting, the state coverage of the control models is expanded, improving robustness to non-standard states.

[0068] The preset reward function may include at least one of the following: Euclidean distance reward between the robot end effector and the target object; error reward between the expected pose and the current pose of the target object; deviation reward between the robot's torso pose and the standard pose; distance reward between the robot and obstacles in the preset environment; and state error reward between the robot's terminal state after completing the previous task and the initial state after completing the next task.

[0069] Specifically, the value of the aforementioned preset reward function can be calculated using at least one of the following implementation methods.

[0070] In one embodiment, the Euclidean distance between the robot end effector and the target object, as well as the error between the target object's desired pose and its current pose, can be obtained, and a weighted sum can be calculated; based on the weighted sum, the task reward can be calculated.

[0071] Task rewards, which measure the degree of approximation between the robot and the target object, and between the object and the target pose, can be expressed mathematically as follows:

[0072]

[0073] in, The Euclidean distance between the robot's end effector and the target object; This represents the error between the actual pose of the object and the pose of the target object. , All are greater than 0, representing the distance penalty coefficient.

[0074] In one embodiment, a weighted sum of squares of the robot's torso tilt angle and angular velocity can be obtained; based on the weighted sum of squares, a deviation reward between the robot's torso posture and the standard posture is calculated.

[0075] Deviation reward measures the deviation of the robot's torso posture from the standard stable posture, and can be expressed mathematically as follows:

[0076]

[0077] in, The tilt angle of the robot's torso. The time indicates that the robot is fully upright; This refers to the robot's angular velocity. The variance is a Gaussian distribution used to control the reward decay rate.

[0078] In one embodiment, the minimum distance between the robot and an obstacle in the preset environment can be obtained, and the distance interval to which the minimum distance belongs can be determined; the reward value corresponding to the distance interval to which the minimum distance belongs can be determined as the distance reward between the robot and the obstacle in the preset environment; wherein, the distance interval is constructed based on a safe distance threshold.

[0079] Safety rewards are used to measure the safe distance between the robot and obstacles or placed objects, and can be expressed as the following mathematical formula:

[0080]

[0081] in, This represents the minimum distance between the robot and the obstacle. This is the safe distance threshold; This is the safety penalty coefficient.

[0082] In one embodiment, the terminal state of the robot after completing the previous task and the initial state for the next task can be obtained; the state error reward is calculated based on the state error norm of the terminal state and the initial state.

[0083] Continuous rewards can measure the degree of matching between the terminal state of the previous stage and the initial state of the next stage, and can be expressed as the following mathematical formula:

[0084]

[0085] in, The norm of the state error vector between adjacent task stages; This is the continuous penalty coefficient.

[0086] It is understandable that the preset reward function can include one or more of the different types of reward items mentioned above. If the preset reward function is constructed based on multiple of the above reward items, since the physical dimensions, numerical ranges, and sensitivity to change of each reward item are different, in order to avoid the value of a certain reward item dominating the entire preset reward function value, in one embodiment, a one-to-one weight coefficient can be set for each reward item. Then, a weighted sum can be performed based on the weight values ​​corresponding to each reward item.

[0087] For example, the reward function can be expressed as:

[0088]

[0089] in, The reward for completing the task is related to the Euclidean distance between the robot and the target object, the error between the expected pose of the target object and the current pose of the target object, and ensures that the robot completes the core task of grasping, moving and placing. The reward for motion stability is related to the tilt angle of the robot's torso relative to the stable state and the robot's angular velocity, ensuring the robot's overall balance and posture compliance. The safety reward is linked to the distance between the robot and obstacles to ensure that the robot does not collide with the environment or objects. For task continuity rewards, the error vector norm is associated with the robot state after task completion and the robot state when the adjacent subsequent task begins, ensuring seamless connection of long-term multi-stage operations.

[0090] Since the requirements for robot behavior differ at different stages during the execution of the target task, in another embodiment of the present invention, time constraints can be introduced into the weight calculation of each reward item to achieve adaptive adjustment of the weight of sub-reward items as the task progresses.

[0091] Specifically, in the preset reward function, the weights of the Euclidean distance from the robot to the target object, the error between the target object's expected pose and its current pose, and the distance between the robot and the obstacle can be configured to be inversely correlated with the number of time steps in the target task execution; the weights of the robot's torso tilt angle, robot angular velocity, and the error vector norm of the robot's state after the task is completed and the robot's state when the adjacent subsequent task begins execution can be configured to be positively correlated with the number of time steps in the target task execution.

[0092] For example, to adapt to the different reward emphases at different stages of long-duration tasks, a dynamic weight adjustment mechanism is designed, with the weights changing according to the task progress. change:

[0093]

[0094] Based on this, in the early stages of the mission Take the maximum value among the weighting coefficients and prioritize single-object operations; in the later stages of the task... Increase the size to enhance stability and continuity, and suppress error accumulation.

[0095] It should be noted that, in one embodiment, the task progress... , This is used to characterize the ratio of the number of time steps that the target task has already been executed to the number of time steps expected to complete the target task, in order to quantify the current progress of the task.

[0096] In another embodiment, the number of time steps that the target task has already been executed can be directly used as a time constraint to dynamically adjust the weight value of the reward item in the preset reward function. This embodiment of the invention does not limit this.

[0097] S103: Collect control sample pairs of the robot during the training of the first control model and the second control model, and train the robot control model based on the control sample pairs.

[0098] The control sample includes the observations and motion quantities of the robot when it performs the target task.

[0099] In an exemplary embodiment, after the first control model and the second control model have been trained, sample data from their execution can be collected and the same robot control model can be further trained to eliminate hard-coded stage boundaries, achieve seamless continuous execution of long-term continuous movement operations, and obtain the robot control model.

[0100] Specifically, in one implementation, when the first control model and the second control model execute the corresponding target task, data including the robot's pose and the control commands output by the corresponding first or second control model are collected. Based on the collected sample pairs as training data, the first and second control models are trained. The inputs to the first and second control models are robot body perception (joint angles, velocities) and object / target states; the outputs are the corresponding control commands. Through iterative training, the trained robot control model gradually approximates the performance of the first and second control models when executed sequentially, achieving end-to-end unified control for long-term tasks while ensuring the generalization ability of the control model.

[0101] It should be noted that the loss function used in training the robot control model is not limited in this embodiment of the invention.

[0102] For example, the loss function can be defined as:

[0103]

[0104]

[0105]

[0106] in: The cut-and-replace objective function of PPO ensures that the policy update steps are small and the training is stable, steadily improving the task reward without drastically changing the policy. This is the loss term in the value function, used to estimate the quality of the state. It is the value loss weighting coefficient, which is greater than 0; This is a policy entropy regularization term that encourages policies to explore more and improves generalization ability. It is the entropy coefficient, which is greater than 0; To monitor the loss weight, it can be reduced from large to small as the number of time steps of the target task is executed; This is the distillation loss function.

[0107] Furthermore, to enhance the robot's actual deployment and human-computer interaction capabilities, in one implementation, the robot control model can be further trained into an interactive model based on first-person RGB observation and natural language commands, replacing the original state input and adopting deployment-friendly multimodal observation.

[0108] Specifically, when the aforementioned robot control model is executing the target task, robot state information, control commands output by the robot control model, and corresponding robot first-view RGB images and natural language commands can be collected. Based on the collected information, an interactive model is trained until the difference between the success rate of the interactive model in executing the target task and the success rate of the robot control model is less than a second preset threshold.

[0109] Based on this, the robot control model can be further trained into an interactive model. The interactive model takes RGB images from the robot's first-person perspective and natural language commands as input data, and control commands as output data.

[0110] For example, a multimodal observation input can be constructed, consisting of first-person perspective RGB images and natural language commands. The RGB images represent the environment captured by the robot's head camera, while the natural language commands describe the corresponding operation loop. Lightweight robot propagation perception information is also integrated to ensure the completeness of the observation. Depth images and force / torque sensors can be added to enhance perception capabilities. The action output of a unified robot control model is used as the supervision signal to train an interactive model.

[0111] The interactive model employs a multimodal fusion encoder + action decoder structure. It extracts visual features from RGB images using a VisionTransformer (ViT) network, projects semantic features of natural language onto a small MLP, and fuses these features with normalized ontology perception features. The resulting data is then decoded into joint control actions for the robot using a residual MLP. This model supports continuous natural language command input, enabling multi-round movement operation cycles and long-term language-driven control.

[0112] On the other hand, to support efficient training, validation and performance evaluation of the framework, a high-fidelity simulation platform based on a physics engine can be built, which can utilize the parallel computing capabilities of GPUs to achieve synchronous training in large-scale environments, improve training efficiency, and provide standardized performance evaluation metrics.

[0113] For example, a dynamic model consistent with the robot can be constructed to reproduce the motion constraints, balance characteristics, and contact dynamics of all joints; accurate modeling can be performed on four types of room scenes and various objects in the dataset module to reproduce the physical properties (mass, coefficient of friction) and collision characteristics of the objects; ensuring the real-time performance and accuracy of the simulation. Through efficient training and evaluation submodules, synchronous training in a parallel simulation environment is supported, utilizing high-performance GPUs to provide computing resources, significantly improving the efficiency of model training; standardized performance evaluation metrics are defined, including single-loop success rate, full-task success rate, and placement error, enabling quantitative evaluation of the control model's performance, while also supporting qualitative motion visualization for intuitive analysis of the control model's execution effect.

[0114] Please refer to Figure 2 As shown, based on the same inventive concept as the aforementioned robot control model training method, one embodiment of the present invention provides a robot control model training system 200, including: an acquisition module 201, a first training module 202, and a second training module 203.

[0115] It should be noted that the robot control model training method of this embodiment can be applied to the robot control model training system 200 of this embodiment.

[0116] Specifically, the acquisition module 201 is used to acquire the pose of the robot before and after performing two consecutive target tasks in a preset environment. The target tasks include grasping and placing target objects. The robot performs the previous target task based on a first control model and the next target task based on a second control model.

[0117] The first training module 202 is used to train the first control model and the second control model based on a preset reward function, wherein the preset reward function includes at least one of the following: Euclidean distance reward between the robot end effector and the target object, error reward between the expected pose and the current pose of the target object, deviation reward between the robot torso pose and the standard pose, distance reward between the robot and obstacles in the preset environment, and state error reward between the robot's terminal state after completing the previous task and the initial state after completing the next task.

[0118] The second training module 203 is used to collect control sample pairs of the robot when training the first control model and the second control model, and to train the robot control model based on the control sample pairs; wherein, the control sample pairs include the observations and motion quantities of the robot when performing the target task.

[0119] Please refer to Figure 3 As shown, embodiments of the present invention also provide a robot 300, which includes at least one processor 301, a memory 302 (e.g., non-volatile memory), a main memory 303, and a communication interface 304, and the at least one processor 301, memory 302, main memory 303, and communication interface 304 are connected together via an internal bus 305. The at least one processor 301 is used to invoke at least one program instruction stored or encoded in the memory 302, so that the at least one processor 301 performs various operations and functions of the robot control model training method described in the various embodiments of this specification.

[0120] This invention also provides a computer-readable medium carrying computer-executable instructions. When executed by a processor, these instructions can be used to implement various operations and functions of the robot control model training methods described in the various embodiments of this specification.

[0121] The computer-readable medium in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0122] In this invention, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0123] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, systems, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

[0126] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0127] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for training a robot control model, characterized in that, include: The robot's pose before and after performing two consecutive target tasks in a preset environment is obtained. The target tasks include grasping and placing target objects. The robot performs the previous target task based on a first control model and the next target task based on a second control model. The first control model and the second control model are trained based on a preset reward function, wherein the preset reward function includes at least one of the following: Euclidean distance reward between the robot end effector and the target object, error reward between the expected pose and the current pose of the target object, deviation reward between the robot torso pose and the standard pose, distance reward between the robot and obstacles in the preset environment, and state error reward between the terminal state of the robot after completing the previous task and the initial state of the robot after completing the next task. Collect control sample pairs of the robot during the training of the first control model and the second control model, and train the robot control model based on the control sample pairs; The control sample pair includes the observations and motion quantities of the robot when it performs the target task.

2. The robot control model training method according to claim 1, characterized in that, The method calculates the value of the preset reward function in at least one of the following ways: Obtain the Euclidean distance between the robot's end effector and the target object, as well as the error between the target object's desired pose and its current pose, and calculate the weighted sum. Calculate the task reward based on the weighted sum value; Obtain the weighted sum of squares of the robot's torso tilt angle and angular velocity; based on the weighted sum of squares, calculate the deviation reward between the robot's torso posture and the standard posture; Obtain the minimum distance between the robot and obstacles in the preset environment, and determine the distance interval to which the minimum distance belongs; determine the reward value corresponding to the distance interval to which the minimum distance belongs as the distance reward between the robot and obstacles in the preset environment; wherein, the distance interval is constructed based on a safe distance threshold; Obtain the terminal state of the robot after completing the previous task and the initial state for the next task; calculate the state error reward based on the state error norm of the terminal state and the initial state.

3. The robot control model training method according to claim 2, characterized in that, The method further includes: Weights are assigned to the task reward, the deviation reward between the robot's torso posture and the standard posture, the distance reward between the robot and obstacles in the preset environment, and the state error reward, respectively. The weights of the task reward and the distance reward between the robot and obstacles in the preset environment are inversely correlated with the number of time steps in which the target task is executed, while the weights of the deviation reward between the robot's torso posture and the standard posture, and the state error reward are positively correlated with the number of time steps in which the target task is executed.

4. The robot control model training method according to claim 3, characterized in that, The method further includes: Within a preset time window after the task begins, the weight of the task reward is maintained at the maximum value among the various weight coefficients.

5. The robot control model training method according to claim 1, characterized in that, The two consecutive target tasks include grasping and placing a first target object and grasping and placing a second target object. After training the first control model and the second control model, the method further includes: The first control model is adjusted based on a safe evacuation reward function, wherein the safe evacuation reward function includes a distance reward between the robot's torso and end effector and the first target object after the robot places the first target object.

6. The robot control model training method according to claim 1, characterized in that, The method further includes: Determine a loss function for training the robot control model, the loss function being associated with at least one of behavior cloning loss and reinforcement learning loss; Wherein, the behavior cloning loss is the expected mean square error between the control command output by the robot control model and the control command output by the first control model or the second control model in the control sample pair; the reinforcement learning loss is a weighted combination of the cut-and-substitute target loss, the value function loss and the policy entropy regularization term of the proximal policy optimization algorithm.

7. The robot control model training method according to claim 1, characterized in that, The method further includes: Based on the robot control model, the robot is controlled to perform two consecutive target tasks in a preset environment, and the robot's state information, control commands output by the robot control model, and robot first-view RGB images and natural language commands corresponding to the target tasks are collected when the two consecutive target tasks are performed. Based on the robot's state information, the control commands output by the robot control model, and the corresponding first-person view RGB image and natural language commands, an interactive model is trained.

8. A robot control model training system, characterized in that, include: The acquisition module is used to acquire the pose of the robot before and after performing two consecutive target tasks in a preset environment. The target tasks include grasping and placing target objects. The robot performs the previous target task based on a first control model and the next target task based on a second control model. The first training module is used to train the first control model and the second control model based on a preset reward function. The preset reward function includes at least one of the following: Euclidean distance reward between the robot end effector and the target object, error reward between the expected pose and the current pose of the target object, deviation reward between the robot torso pose and the standard pose, distance reward between the robot and obstacles in the preset environment, and state error reward between the terminal state of the robot after completing the previous task and the initial state of the robot after completing the next task. The second training module is used to collect control sample pairs of the robot during the training of the first control model and the second control model, and to train the robot control model based on the control sample pairs; wherein, the control sample pairs include the observations and motion quantities of the robot when performing the target task.

9. A robot, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the robot control model training method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the robot control model training method according to any one of claims 1-7.

Images