A method for constructing a robot teleoperation embodied large model and a robot teleoperation system

Abstract

The application provides a construction method of a robot teleoperation body model and a robot teleoperation system, and the method comprises the following steps: fine-tuning a pre-training body model by using master-slave robot control data to obtain a fine-tuned body model; deploying the fine-tuned body model to provide robot control services and collecting real machine execution data; obtaining scores of the real machine execution data, and forming reinforcement learning training data pairs of the real machine execution data and corresponding scores, and adding the reinforcement learning training data pairs to a reinforcement training data set; and performing reinforcement learning training on the fine-tuned body model by using the reinforcement training data set. The pre-training model is fine-tuned, the cost of manual intervention is effectively reduced, the reinforcement learning training data is collected by deploying the fine-tuned body model, the reinforcement learning method and the pre-training body model are organically combined, the advantages of the two are fully utilized, and efficient learning and performance improvement of the robot in a complex environment are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot teleoperation, and in particular to a robot teleoperation embodied large model construction method and robot teleoperation system. BACKGROUND

[0002] In recent years, robot teleoperation systems have been widely used in industries, medical care, services and other fields. In particular, full-body teleoperation technology of dual-arm robots is one of the research hotspots. This technology aims to remotely control robots to imitate human full-body movements and perform complex tasks.

[0003] Embodied intelligence is a research hotspot in the field of artificial intelligence in recent years, which is the combination of robots and artificial intelligence, emphasizing that intelligent agents dynamically interact with the environment through physical bodies to achieve the integration of perception, cognition, decision-making and action. Providing embodied intelligence services for robot teleoperation systems can remotely and autonomously operate robot systems. Existing robot teleoperation embodied large models still have some obvious defects:

[0004] 1. Since the emergence of large language models, pre-training-fine-tuning-reinforcement learning has become a standard large model training paradigm. There is no mature implementation solution for transferring this paradigm to the robot field. The data used for training large language models can be obtained from the Internet, and the collection cost is relatively low. However, the data used to train robot teleoperation embodied large models can only be manually constructed through simulation or human demonstration, which has a high collection cost.

[0005] 2. The existing robot teleoperation embodied large model has poor adaptability to specific robot devices, device usage environments and task purposes, and its control effect is difficult to meet the demand.

[0006] To solve or at least partially solve the defects of robot teleoperation embodied large model training data being difficult to obtain, model training effect being poor, and model adaptability to specific robot devices, device usage environments and task purposes in the prior art, the present application provides a robot teleoperation embodied large model construction method and robot teleoperation system. SUMMARY

[0007] The present application provides a robot teleoperation embodied large model construction method and robot teleoperation system to solve or at least partially solve the defects of robot teleoperation embodied large model training data being difficult to obtain, model training effect being poor, and model adaptability to specific robot devices, device usage environments and task purposes in the prior art.

[0008] The present application provides a robot teleoperation embodied large model construction method, which comprises:

[0009] The pre-trained embodied large model is fine-tuned by using master-slave robot control data to obtain a fine-tuned embodied large model;

[0010] The fine-tuned embodied large model is deployed to provide a robot control service, and real machine execution data is collected;

[0011] The real machine execution data is scored, and the real machine execution data and the corresponding score form a reinforcement learning training data pair, which is added to a reinforcement training data set;

[0012] The fine-tuned embodied large model is trained by using the reinforcement training data set.

[0013] Optionally, before the pre-trained embodied large model is fine-tuned by using master-slave robot control data to obtain a fine-tuned embodied large model, the method further comprises:

[0014] An initial embodied large model is trained by using open source robot control data to obtain the pre-trained embodied large model.

[0015] Further, the initial embodied large model includes a visual language model and an action head model, the visual language model is used to extract features from input slave robot scene data and task instruction text, and the action head model processes the extracted features and slave robot state data to output slave robot control data.

[0016] Optionally, the master-slave robot control data is collected during the process of remotely operating a slave robot by a master arm, and the master-slave robot control data includes task instruction text, slave robot control data output by the master arm, and slave robot state data and slave robot scene data.

[0017] Optionally, the process in which the fine-tuned embodied large model provides a robot control service includes:

[0018] A control request sent by a currently accessed master arm host is received;

[0019] The slave robot scene data, task instruction text, and slave robot state data carried in the control request are input to the fine-tuned embodied large model to obtain slave robot control data corresponding to the control request;

[0020] The slave robot control data corresponding to the control request is sent to the currently accessed master arm host.

[0021] Optionally, the real machine execution data includes slave robot scene data, task instruction text, slave robot state data, and slave robot control data output by the fine-tuned embodied large model.

[0022] Further, the reinforcement learning training data pair comprises: slave robot scene data, task instruction text, slave robot state data, slave robot control data, and score.

[0023] Optionally, the score of the real machine execution data is obtained by manual scoring of the execution result in the process of providing robot control service by the fine-tuned body model.

[0024] Optionally, the score of the real machine execution data is obtained by:

[0025] The slave robot scene data and / or the slave robot state data in the real machine execution data are processed based on a preset judgment rule to obtain a score, wherein the preset judgment rule is determined according to a task scene.

[0026] Optionally, the reinforcement training data set further comprises a reinforcement learning training data pair generated according to the master-slave robot control data, and the score corresponding to the master-slave robot control data is a preset score; and / or the reinforcement training data set further comprises a reinforcement learning training data pair generated according to the artificial intervention data, and the score corresponding to the artificial intervention data is a preset score.

[0027] Optionally, the score of the real machine execution data is obtained by:

[0028] The real machine execution data is input into a scoring model to obtain a corresponding score, wherein the scoring model is trained by using the scored master-slave robot control data and / or the scored artificial intervention data and / or the scored real machine execution data.

[0029] The application further provides a teleoperation robot system comprising a master robot and a slave robot, wherein the master robot comprises a master arm and a master arm host, and the slave robot comprises a slave arm, a slave arm host, and a scene acquisition module.

[0030] The master arm host can receive user input control data through the master arm, process and generate slave robot control data, and send the slave robot control data to the slave arm host.

[0031] The slave arm host can control the slave arm to move according to the slave robot control data, and the slave arm host can also acquire slave robot movement scene data through the scene acquisition module.

[0032] The master arm host or the slave arm host or a cloud end is deployed with a fine-tuned body model, which can provide robot control service for the teleoperation robot system.

[0033] The remote operation system collects data according to the construction method of the robot remote operation body model of any one of the preceding, and performs reinforcement learning training on the fine-tuned body model.

[0034] Optionally, the system further comprises a control handle connected to the master arm host, when the control handle is in a first state, the slave robot control data is sent to the master robot by the fine-tuned body model to control the slave arm, and when the control handle is in a second state, the slave arm is controlled by the master arm.

[0035] Optionally, the system is further used to collect master-slave robot control data for training and fine-tuning the pre-trained body model to obtain the fine-tuned body model.

[0036] Optionally, the master arm is in the form of a mechanical arm or an exoskeleton device or an optical motion capture device.

[0037] Optionally, the master arm and the slave arm have the same initial joint posture and consistent joint motion mode.

[0038] Optionally, the scene acquisition module comprises a depth camera and / or a wide-angle lens and / or a laser radar and / or an inertial sensor.

[0039] The construction method of the robot remote operation body model and the robot remote operation system provided by the present application at least have the following beneficial effects:

[0040] 1. The master-slave robot control data collected during user operation is used to fine-tune the pre-trained model, so that the model has better adaptability to specific robot devices, device use environments, and tasks to be executed, avoids causing the slave robot to move chaotically, effectively reduces the cost of manual intervention, provides robot control services by deploying the fine-tuned body model, collects real machine execution data and real machine execution data scores in real environments while providing control services, and further continues reinforcement learning training on the fine-tuned body model, organically combines reinforcement learning methods with pre-trained body models, fully utilizes the advantages of both, and realizes efficient learning and performance improvement of robots in complex environments.

[0041] 2. The robot can more efficiently learn using data by using the pre-trained VLA model and the efficient reinforcement learning algorithm, reducing the data demand and time cost in the learning process. Compared with the traditional method of learning from scratch, the learning efficiency can be improved by several times, and a good performance level can be achieved in a shorter time.

[0042] 3、The flexibility and real-time adjustment capability of the system enable it to better adapt to different task requirements and environmental changes. The robot can adjust the action planning and execution strategy in real time according to the feedback information during task execution, ensuring the smooth completion of the task. This strong adaptability enables the robot to play a greater role in diversified application scenarios, such as industrial production, logistics transportation, medical rehabilitation, etc.

[0043] 4、Through the human-in-the-loop mechanism and real-time monitoring and adjustment function, the system can ensure the safety of the robot during operation. The real-time intervention of human operators can prevent the robot from dangerous behavior or wrong operation, while the rapid response and adjustment capability of the system to abnormal situations can also effectively avoid potential safety risks, ensuring the safety of human-robot collaboration. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0045] Figure 1 is a structural schematic diagram of a robot teleoperation system provided by the present application;

[0046] Figure 2 is a flowchart of a construction method of a robot teleoperation body model provided by the present application;

[0047] Figure 3 is a structural schematic diagram of an initial body model in the construction method of a robot teleoperation body model provided by the present application;

[0048] Figure 4 is a system structure schematic diagram of a robot teleoperation system provided by the present application, which is deployed with a fine-tuned body model;

[0049] Figure 5 is a construction flowchart of a robot teleoperation body model provided by the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0051] The application is described below Figures 1-5 A method for constructing a robot teleoperation body model and a robot teleoperation system are provided.

[0052] To facilitate understanding of the technical solutions, the structure of a robot teleoperation system is briefly described first. As shown in Figure 1 The robot teleoperation system includes a master robot and a slave robot. The master robot includes a master arm, a master arm host, the slave robot includes a slave arm, a slave arm host, and a scene collection module for collecting the motion scene of the slave arm. A user can input control data by operating the master arm, and the slave robot is remotely controlled through a data transmission line of "master arm-master arm host-slave arm host-slave arm". The master arm can have a single arm, or two or more arms. The master arm has a gripper at the end (the gripper can also be regarded as a joint). The master arm and the slave arm have the same degrees of freedom. It should be noted that the same degrees of freedom here means that the master arm should have the same number of degrees of freedom as the slave arm, and the types of degrees of freedom and the adjacency relationship of the degrees of freedom at the corresponding joints are also the same, but the specific sizes and shapes of the joints do not necessarily need to be exactly the same.

[0053] With reference to Figure 2 The application provides a method for constructing a robot teleoperation body model, which includes the following steps.

[0054] S1. A pre-trained body model is fine-tuned by using master-slave robot control data to obtain a fine-tuned body model.

[0055] S2. The fine-tuned body model is deployed to provide robot control services, and real machine execution data is collected.

[0056] S3. The real machine execution data is scored, and the real machine execution data and the corresponding scores are used to form a reinforcement learning training data pair, which is added to a reinforcement training data set.

[0057] S4. The fine-tuned body model is trained by reinforcement learning using the reinforcement training data set.

[0058] Specifically, the pre-trained model is fine-tuned in step S1 using master-slave robot control data, so that the model can respond to the control request issued by the robot and output control data for the robot. The master-slave robot control data is data generated in the process of controlling the driven robot by directly operating the master robot by the user (operator), and is collected in a real environment, so that the model has better adaptability to specific robot equipment, the use environment of the equipment, and the task to be executed, for example, the motion path of the driven robot is more concise, the motion amplitude is smaller, the motion safety is higher, and the robot can better control the robot to achieve the task goal. The pre-trained embodied large model is fine-tuned and trained by the master-slave robot control data in S1, which can avoid the poor adaptability of the pre-trained embodied large model deployed directly to the application environment, avoid causing the driven robot to act chaotically, and effectively reduce the cost of manual intervention.

[0059] Step S2 deploys the fine-tuned embodied large model trained in step S1 to a service program to provide online control services for the teleoperation robot, that is, the teleoperation robot system can use the services provided by the fine-tuned embodied large model to control the robot autonomously, without the user frequently manually teleoperating the driven arm through the master arm. The fine-tuned embodied large model can be deployed on the master arm host, the driven arm host, or an independent control center, and the present application does not make any limitation.

[0060] In the process of deploying the model to provide online control services in a real environment in step S2, real machine execution data is also collected, which includes control data output by the model and execution related data of the robot. Step S3 gives a score to the real machine execution data, which is used to evaluate the "good" or "bad" of the execution result, which can be used as reward and punishment data to cooperate with the real machine execution data to perform reinforcement learning training on the fine-tuned embodied large model in step S4. Reinforcement learning is a branch of machine learning, which mainly studies how an agent learns a strategy in an environment through interaction with the environment to maximize some accumulated reward. The core idea of reinforcement learning is to learn through trial and error. The agent affects the environment by performing actions, and obtains feedback from the environment, and then adjusts its strategy to optimize long-term rewards.

[0061] Steps S2-S4 provide online control services in a real environment through the deployed fine-tuned embodied large model, and collect real machine execution data while providing control services, give a score to the execution data, and use it as reinforcement learning data to continue reinforcement learning training on the fine-tuned embodied large model, further improving the performance of the model.

[0062] The embodiment fine-tunes the pre-trained model by collecting master-slave robot control data during user operation, so that the model has better adaptability to specific robot devices, device use environments, and tasks to be executed, avoids causing confusion in the action of the driven robot, effectively reduces the cost of manual intervention, and provides robot control services by deploying the fine-tuned embodied large model. While providing control services, real machine execution data and real machine execution data scores in a real environment are also collected for further reinforcement learning training of the fine-tuned embodied large model. The reinforcement learning method and the pre-trained embodied large model are organically combined to fully play the advantages of both, and efficient learning and performance improvement of the robot in a complex environment are realized.

[0063] Based on the previous embodiment, in some embodiments, before S1, the method further comprises:

[0064] S0, training an initial embodied large model by using open-source robot control data to obtain a pre-trained embodied large model.

[0065] The robot control data is composed of robot control data, robot state data, robot scene data, and task instruction text. The robot control data is the expected future action of the driven robot output by the master robot, which can be a single-step action or an action in the future period (for example, joint angles). The robot state data can be the joint angle, end gripper width, joint current, etc. of the controlled robot. The robot scene data is the scene data in the motion process of the driven robot, such as images and videos taken of the driven robot and the environment, and point cloud data collected from the driven robot and the environment. The task instruction text is a description of the task to be completed by the current control robot, such as picking up an apple or picking up a part and placing it into a processing device in the required posture. Currently, several organizations or companies have opened source robot datasets for community use, such as the Open X-Embodiment dataset collected by Google DeepMind in cooperation with 21 international well-known institutions, the AgiBotWorld dataset of Zhiyuan Robot, and the RoboMind dataset of Guodai Innovation Center. These open-source robot control data can be used for pre-training of the robot large model, so that the embodied large model has the basic ability of the robot teleoperation scene, reducing the time cost and data collection cost of the next step of training.

[0066] Based on any embodiment, in some embodiments, the initial embodied large model includes a visual language model and an action head model. The visual language model is used to extract features from the input driven robot scene data and task instruction text, and the action head model processes the extracted features and the driven robot state data to output the driven robot control data.

[0067] Specifically, with reference toFigure 3 The initial embodied large model (Vision language action, VLA) is composed of a vision language model (Vision Language Model, VLM) and an action head model. The vision language model (VLM) can extract and fuse features from the scene data of the slave robot and the task instruction text obtained by the robot, and provide semantic information for subsequent action decision. The scene data of the slave robot is the scene data during the movement of the slave robot, such as images and videos taken of the slave robot and the environment, and point cloud data collected from the slave robot and the environment. The task instruction text is a description of the task that the robot needs to complete, such as holding a tool to complete a specific trajectory action, or picking up a part and placing it into a processing device in a required posture. The above-mentioned vision language model (VLM) is, for example, a QWEN-VL model, or a variant of the QWEN-VL model. The action head model can be a diffusion model-based action head. The robot state and noise and the features extracted by the vision language model (VLM) are input into the action head model, and through an inverse diffusion process, the action that the robot should take under the current state (i.e., generating a reasonable action sequence) can be iteratively predicted to complete the specified task.

[0068] Overall, the input of the initial embodied large model is [scene data of the slave robot, task instruction text, state data of the slave robot], and the output is the control data of the slave robot. The control data of the slave robot can be a single-step action, or an action block, that is, the action of the robot in the future period of time (such as the joint angle in the future period of time).

[0069] During the pre-training of S0 and the fine-tuning training of S1, the pre-processed scene data of the slave robot and the task instruction text are input into the embodied large model (VLA), the vision language model (VLM) extracts and fuses features from the scene data of the slave robot and the task instruction text, and generates a feature representation containing semantic information. The action head predicts the control data of the slave robot according to the features output by the vision language model (VLM) and the state data of the slave robot, and continuously optimizes and adjusts the parameters of the vision language model (VLM) and the action head model according to the deviation (i.e., the loss function) between the predicted data and the control data of the slave robot in the training sample, until the trained embodied large model (VLA) is obtained. It can be understood that during the training of the embodied large model (VLA) in S0 and S1, the two sub-models are trained jointly.

[0070] Based on any of the embodiments, in some embodiments, the master-slave robot control data is data collected in the process of teleoperation of the master arm on the slave robot, and the master-slave robot control data includes: task instruction text, slave robot control data output by the master arm, and slave robot state data and slave robot scene data.

[0071] Specifically, the master-slave robot control data is data generated in the process of controlling the slave robot by the user operating the master robot, and is data collected in a real environment. The process of collecting the master-slave robot control data is described below with a preferred example:

[0072] The task instruction text is a description of the task that needs to be completed by the current control robot, such as grabbing an apple, and for example, picking up a part and placing it into a processing device in a posture required for processing. The master arm host is specially configured (with corresponding software modules) to be able to adapt different types of slave arms for the master arm. Specifically, the master arm host has built-in control protocols and communication protocols of multiple types of slave arms, and the master host determines the type of the currently connected slave arm in advance. After obtaining the operation data of the user on the master arm, it is converted into slave robot control data that conforms to the control protocol and communication protocol of the slave arm, such as converted into instructions suitable for a RiMan (slave) robot arm, and for example, converted into instructions suitable for a Songling (slave) robot arm, so that the master arm can be adapted to multiple slave arms, improving compatibility and further reducing costs. Of course, in order to improve the performance of the fine-tuned embodied large model, different types of slave robots can also be connected to the master robot, and master-slave robot control data can be collected based on different types of robots to improve the compatibility of the trained embodied large model for different types of robots.

[0073] The slave arm performs corresponding actions according to the corresponding slave robot control data, and the slave arm also records data (i.e., slave robot state data) when it performs corresponding actions, such as angles of each joint measured by sensors built-in the slave arm, and spatial positions of the joints.

[0074] The scene collection module collects scene data (i.e., slave robot scene data) when the slave arm performs corresponding actions, such as taking images of the process of "clamping jaws clamping apples", and for example, collecting point cloud perception of the spatial positions of the clamping jaws and apples through a laser point cloud collector.

[0075] During the process of the user operating the master arm to demonstrate, the above task instruction text, slave robot control data, slave robot state data, and slave robot scene data are stored in association as master-slave robot control data. For example, ([grasp apple], [master arm left arm, master arm left gripper, master arm right arm, master arm right gripper, chassis], [slave arm left arm, slave arm left gripper, slave arm right arm, slave arm right gripper, chassis], [slave arm front view image, slave arm left hand image, slave arm right hand image]) are taken as master-slave robot control data for subsequent training of the embodied AI model.

[0076] The embodiment trains the pre-trained embodied AI model using the master-slave robot control data, so that the trained AI model has better adaptability to specific robot devices, device use environments, and tasks to be executed, and avoids causing the slave robot to move chaotically, effectively reducing the cost of manual intervention. In particular, by embedding multiple types of control protocols and communication protocols of slave arms in the master arm host, the host can convert the operation data of the user on the master arm into slave robot control data that conforms to the control protocol and communication protocol of the slave arm according to the specific type of the slave arm, thereby improving compatibility.

[0077] Further, after collecting the slave robot scene data, the collected data is preprocessed, such as image cropping, scaling, and normalization, etc., to facilitate subsequent feature extraction and processing.

[0078] Based on any embodiment, in some embodiments, the process of fine-tuning the embodied AI model to provide robot control services in S2 includes:

[0079] receiving a control request sent by a currently connected master arm host;

[0080] inputting the slave robot scene data, task instruction text, and slave robot state data carried in the control request into the fine-tuned embodied AI model to obtain slave robot control data corresponding to the control request;

[0081] sending the slave robot control data corresponding to the control request to the currently connected master arm host.

[0082] Specifically, referring to the example of Figure 4 , the fine-tuned embodied AI model is deployed in a cloud server. The cloud server not only provides inference services (robot control services) to robots through a service interface, but also collects real machine execution data for subsequent continuous training and iteration of the embodied AI model (VLA), i.e., constitutes an “inference-data collection system”.

[0083] The "reasoning-data collection system" is divided into cloud and robot side (edge). The cloud side deploys the pre-trained, fine-tuned embodied large model (VLA), and provides model reasoning services to the robot side through the service interface. The robot side collects the driven robot scene data, driven robot state data and task instruction text at the same time, and initiates a service request to the service interface. After receiving the request data, the service end sends the [driven robot scene data, driven robot state data, task instruction text] data pair to the embodied large model (VLA) for reasoning, obtains the driven robot control data, and returns the driven robot control data to the robot side. At the same time, the cloud stores [driven robot scene data, driven robot state data, task instruction text, driven robot control data] in the data cache pool for subsequent training.

[0084] Based on any embodiment, in some embodiments, the real machine execution data includes: driven robot scene data, task instruction text, driven robot state data, and driven robot control data output by the fine-tuned embodied large model.

[0085] Specifically, the fine-tuned embodied large model (VLA) is deployed in the cloud to provide online control robot services, and then a reasoning request is started on the robot side. The embodied large model (VLA) in the cloud outputs the driven robot state data according to the reasoning request, and the robot (active arm host) directly executes the corresponding action on the real machine after receiving the action block obtained by model reasoning. Generally speaking, the action block output by the model is a fixed-length action sequence used to control the joint angle of the robot.

[0086] Based on the previous embodiment, in some embodiments, the reinforcement learning training data pair in S3 includes: driven robot scene data, task instruction text, driven robot state data, driven robot control data, and score.

[0087] Specifically, the above real machine execution data is given a score, which is used to evaluate the "good" or "bad" of the execution result, which can be used as reward and punishment data to cooperate with the real machine execution data to perform reinforcement learning training on the fine-tuned embodied large model in step S4. For example, according to the execution result, the score of the real machine execution data is divided into [-1, 0, 1], which are respectively marked as failed data, process data and successful data. Taking the task of grabbing an apple as an example, if the current action block makes the robot grab the apple, the data is marked as 1; if the current action block makes the robot collide, move away from the target, grab other objects, etc., it is marked as -1; if the current action block makes the robot move, but there is no obvious failure or success trend, it is marked as 0.

[0088] Based on any of the embodiments, in some embodiments, the score of the real machine execution data in S3 is obtained, including: in the process of fine-tuning the embodied large model to provide robot control service, the score given by the human to the execution result.

[0089] Specifically, for the execution result of each action block, a human rater gives a score of [-1, 0, 1], marking failed data, process data and successful data. Taking the task of picking up an apple as an example, if the current action block makes the robot just pick up the apple when the model executes the "pick up the apple" reasoning, the data is marked as 1; if the current action block makes the robot collide, move away from the target, pick up other objects, etc., it is marked as -1; if the current action block makes the robot produce motion, but there is no obvious failure or success trend, it is marked as 0.

[0090] It should be noted that in the embodiment, in the process of providing robot control service by the embodied large model (VLA), the rater (user) can score online according to the execution result, and the score will be collected and stored, for example, an artificial scoring function is added to the robot teleoperation system, for example, by connecting the score button on the control handle of the master arm host. When the program runs, the robot first collects the slave robot scene data, slave robot state data, task instruction text, and then sends it to the cloud for reasoning, and then gets the reasoning result in the real machine execution, the human scores the execution result, and the score is transmitted back to the cloud. The [image, language instruction, state, action, score] data pair is collected. [slave robot scene data, task instruction text, slave robot state data, slave robot control data, score]. Correspondingly, the reinforcement training data set includes reinforcement learning training data pairs formed by online collection and artificial scoring.

[0091] The embodiment utilizes the Human-in-the-Loop mechanism to introduce human operators to participate in the learning process of the robot in real time, provides key guidance and correction, accelerates the learning process of the robot, improves the learning effect, and ensures that the robot learns a more human-intended and safety-required action strategy.

[0092] Based on any of the embodiments, in some embodiments, the score of the real machine execution data in S3 is obtained, including:

[0093] Based on a preset determination rule, the slave robot scene data and / or the slave robot state data in the real machine execution data are processed to obtain the score; wherein the preset determination rule is determined according to the task scene.

[0094] Specifically, according to a specific task scene, a corresponding determination rule can be set, and after the robot performs a corresponding action, the execution result can be judged as "good" or "bad" by using related data in combination with the determination rule. For example, whether the task is successful is determined by processing the slave robot scene data using a target detection model. For example, whether the object is placed at the specified position is determined by processing the image collected during the movement of the slave arm. For another example, the joint current or force of the robot is detected, and if a mutation occurs, it is determined that a collision exists, and a corresponding penalty is performed.

[0095] Taking the task of picking up an apple as an example, when the model performs "picking up an apple" reasoning, if the current action block causes the robot to just pick up the apple, this data is marked as 1; if the current action block causes the robot to collide, move away from the target, pick up other objects, etc., it is marked as -1; if the current action block causes the robot to move, but there is no obvious failure or success trend, it is marked as 0.

[0096] Based on any of the embodiments, in some embodiments, the reinforcement training data set further includes a pair of reinforcement learning training data generated according to master-slave robot control data, the score corresponding to the master-slave robot control data being a preset score; and / or, the reinforcement training data set further includes a pair of reinforcement learning training data generated by artificial intervention data, the score corresponding to the artificial intervention data being a preset score.

[0097] Specifically, the master-slave robot control data (demonstration data) is data generated during the process of directly controlling the master arm to remotely operate the slave robot by the operator, and the scores of these data can all be set to 1.

[0098] Similarly, during the process of providing control services by the embodied large model (VLA), there can be a case of large execution error, at which time human intervention is needed to take over the master arm to control the slave arm (for example, the user switches the control right by pressing the button on the control handle), and the human intervention data generated at this time is recorded.

[0099] The embodiment can continuously obtain reinforcement learning data in the real machine control process through the above-mentioned manner.

[0100] Based on any of the embodiments, in some embodiments, the score of the real machine execution data obtained in S3 includes:

[0101] The real machine execution data is input into a scoring model to obtain a corresponding score, wherein the scoring model is trained by using the scored real machine execution data.

[0102] Specifically, the scored real robot execution data can be obtained in the manner of the above two embodiments. The scored real robot execution data [from the robot scene data, the task instruction text, the slave robot state data, the slave robot control data, the score] is used to train a scoring model, which can be a visual language action model (VLA) or a small model (such as a convolutional neural network, a Transformer, etc.). Specifically, the scoring model can be represented as R(s, a), s state includes [from the robot scene data, the task instruction text, the slave robot state data] information, a slave robot control data, input (s, a) into the scoring model, the model can output a score [-1, 0, 1], thereby evaluating the action under the current state.

[0103] Using the scoring model of the present embodiment, after the embodied large model provides robot control service to collect real robot execution data, the real robot execution data can be sent to the scoring model for scoring, and the real robot execution data and the corresponding score can be further used for reinforcement learning training of the embodied large model.

[0104] Based on any embodiment, in some embodiments, the reinforcement learning algorithm used in the reinforcement learning training process of step S4 is selected from RLPD algorithm, Soft Actor-Critic (SAC) algorithm or Proximal Policy Optimization (PPO) algorithm. These algorithms may exhibit different advantages under different task scenarios and data conditions, and can be selected and adjusted according to specific needs.

[0105] Taking the RLPD (Efficient online reinforcement learning with offline data) reinforcement learning algorithm as an example: in each training step, a batch of training data is randomly sampled in the reinforcement training data set, and then the parameters of the target function Q φ (s, a) and the VLA large model π θ (s) are updated respectively.

[0106] The loss functions are respectively represented as:

[0107] ;

[0108]

[0109] R(s, a) represents the immediate reward of the current step, γ is a discount factor, is in the next state , according to the current policy , the action and the expected Q value obtained by the target Q network. is the entropy of the policy, i.e. encouraging the policy to be more random, more exploratory, and a is the weight of the entropy.

[0110] Using RLPD and other efficient reinforcement learning algorithms can make full use of prior data and online data, improve the learning efficiency and stability of the policy network, and further improve the learning performance of the robot.

[0111] The following refers to Figure 5 , an optimal embodiment of the method for constructing a robot teleoperation embodied large model is described:

[0112] S0, using open source robot control data to train the initial embodied large model, and obtaining a pre-trained embodied large model.

[0113] The open source robot control data is, for example, the OpenX-Embodiment dataset collected by Google DeepMind in cooperation with 21 international well-known institutions, the AgiBotWorld dataset of the Zhiyuan robot, the RoboMind dataset of the Guodai Innovation Center, etc.

[0114] S1, using master-slave robot control data to fine-tune the pre-trained embodied large model, and obtaining a fine-tuned embodied large model.

[0115] Optimize and reform the robot teleoperation system to more conveniently collect robot data: set the master arm and the slave arm to have the same degrees of freedom, but the joint lengths and proportions can be inconsistent, and the type of the slave arm can also be multiple (selected from different manufacturers), the master arm has multiple types of control protocols and communication protocols for the slave arm built-in, the master arm determines the type of the slave arm currently connected in advance, after obtaining the operation data of the user on the master arm, convert it into slave robot control data that conforms to the control protocol and communication protocol of the slave arm, for example, convert it into instructions suitable for the Ruierman (slave) robot arm, and for example, convert it into instructions suitable for the Songling (slave) robot arm, so that the master arm can be adapted to multiple sets of slave arms, improving compatibility and further reducing costs. Of course, in order to improve the performance of the fine-tuned embodied large model, different types of slave robots can also be connected to the master robot, and master-slave robot control data can be collected based on different types of robots to improve the compatibility of the embodied large model obtained by training for different types of robots.

[0116] The master-slave robot control data is the data generated in the process of the user (operator) directly operating the master robot to control the slave robot, that is Figure 5The master-slave robot control data in the demonstration data includes: task instruction text, slave robot control data output by the master arm, and slave robot state data and slave robot scene data. The pre-trained embodied large model is trained by using the master-slave robot control data, so that the trained large model has better adaptability to specific robot equipment, use environment of the equipment and tasks to be executed, and effectively reduces the cost of manual intervention.

[0117] S2, deploy the fine-tuned embodied large model to provide robot control service and collect real machine execution data.

[0118] Referring to Figure 4 Examples, the fine-tuned embodied large model is deployed in a cloud server. The cloud server not only provides inference service (robot control service) to the robot through a service interface, but also collects real machine execution data for subsequent continuous training and iteration of the embodied large model (VLA), that is, constitutes a "inference-data collection system".

[0119] The cloud end deploys the pre-trained and fine-tuned embodied large model (VLA) to provide model inference service to the robot side through a service interface. The robot collects the slave robot scene data, slave robot state data and task instruction text at the same time, and initiates a service request to the service interface. After receiving the request data, the server inputs [slave robot scene data, slave robot state data, task instruction text] data pairs into the embodied large model (VLA) for inference to obtain slave robot control data. At the same time, [slave robot scene data, slave robot state data, task instruction text, slave robot control data] are stored in a data cache pool for subsequent training.

[0120] S3, obtain the score of the real machine execution data, and form a reinforcement learning training data pair of the real machine execution data and the corresponding score, and add it to the reinforcement training data set.

[0121] The fine-tuning body large model is deployed on a cloud server. The cloud server not only provides inference services (robot control services) to the robot through a service interface, but also allows scorers to give scores to the model during the control service process by controlling the handle to execute data on the real machine. The scoring scores are divided into [-1, 0, 1], which respectively mark failed data, process data and successful data. These scoring data and other scoring data are used to train the scoring model. Other scoring methods include automatic scoring based on preset rules. For example, a target detection model is used to process the scene data of the driven robot to determine whether the task is successful. For example, images collected during the movement of the driven arm are processed to determine whether the object is placed in the specified position. For another example, the joint current or force of the robot is detected. If there is a mutation, it can be determined that there is a collision, and the corresponding punishment is performed. For another example, the scoring of the master-slave robot control data (demonstration data) and the artificial intervention data is automatically set to 1.

[0122] The scoring model is trained using the scored demonstration data, artificial intervention data and real machine execution data [driven robot scene data, task instruction text, driven robot state data, driven robot control data, score]. The scoring model can be a visual language action model (VLA) or a small model (such as a convolutional neural network, Transformer, etc.). Specifically, the scoring model can be represented as R(s, a), s state includes [driven robot scene data, task instruction text, driven robot state data] information, a driven robot control data, input (s, a) into the scoring model, the model can output the score [-1, 0, 1], thereby evaluating the action under the current state.

[0123] The scoring model can continue to score the collected real machine execution data during the subsequent process of providing robot control services by the model in combination with the above artificial scoring and rule-based scoring. The real machine execution data and the corresponding score form a reinforcement learning training data pair, which is added to the reinforcement training data set.

[0124] S4, reinforcement learning training of the fine-tuned body large model using the reinforcement training data set.

[0125] The fine-tuned body large model is trained using the reinforcement learning algorithm RLPD (Efficient online reinforcement learning with offline data): in each training step, a batch of training data is randomly sampled from the reinforcement training data set, and then the parameters of the target function Q φ (s, a) and the VLA large model π θ (s) are updated respectively.

[0126] The loss function is represented as:

[0127] ;

[0128]

[0129] R(s,a) represents the immediate reward at the current step, and gamma is the discount factor, is the next state , according to the current policy sample an action and get the expected Q value through the target Q network. is the entropy of the policy, that is, to encourage the policy to be more random and more exploratory, and alpha is the weight of the entropy.

[0130] Using efficient reinforcement learning algorithms such as RLPD can make full use of prior data and online data, improve the learning efficiency and stability of the policy network, and further improve the learning performance of the robot.

[0131] Below, a teleoperation robot system provided by the application will be described, and the teleoperation robot system described below can be referred to the method of constructing a robot teleoperation body model described above.

[0132] The teleoperation robot system provided by the application comprises a master robot and a slave robot, the master robot comprises a master arm, a master arm host, the slave robot comprises a slave arm, a slave arm host, and a scene acquisition module.

[0133] The master arm host can receive user input control data through the master arm, process and generate slave robot control data, and send the slave robot control data to the slave arm host.

[0134] The slave arm host can control the slave arm to move according to the slave robot control data, and the slave arm host can also acquire slave robot motion scene data through the scene acquisition module.

[0135] The master arm host or the slave arm host or the cloud end is deployed with a fine-tuned body model, which can provide robot control services for the teleoperation robot system.

[0136] The teleoperation system collects data according to the method of constructing a robot teleoperation body model according to any one of the preceding methods, and performs reinforcement learning training on the fine-tuned body model.

[0137] Specifically, the master arm can have a single arm, or it can have two or more arms, and the master arm has a gripper at the end of the arm (the gripper can also be considered as a joint). The master arm has the same degrees of freedom as the slave arm, and preferably, the master arm has at least 12 degrees of freedom. It should be noted that the same degrees of freedom here means that the master arm should have the same number of degrees of freedom as the slave arm, and the types of degrees of freedom at the corresponding joints and the adjacency relationship of the degrees of freedom are also the same, but the specific joint size and shape do not necessarily need to be exactly the same.

[0138] The master arm can use a general teleoperation device, and the master arm should support real-time reading of the angle data of each joint servo, and can move to a specified position under a control signal. Each robot arm is equipped with a high-performance controller that can receive instructions from a software system and accurately control the movement of the joints. The controller supports multiple control modes, including position control, speed control, and force control, to adapt to different task requirements.

[0139] The master arm host is equipped with a software system controller (the full-size model controls the master robot and the slave robot through this part), which is used to issue control signals to the robot controller, including position control, speed control, and force control.

[0140] The slave arm has the same degrees of freedom as the master arm, and the model of the slave arm can be selected from multiple manufacturers. Each arm has high-precision joints and sensors that can control the angle of each joint in real time and read the joint state, and the end uses a two-finger gripper that can control the opening and closing width.

[0141] The slave arm is equipped with a scene acquisition module. In one example, the slave arm is equipped with three high-resolution cameras, including a camera mounted on the robot wrist and a camera fixed on the head. The wrist camera provides a close-up view for precise observation of the operating object, and the main camera provides a first-person view of the robot to observe the entire work scene. Of course, a depth camera, an RGB-D camera, or a multi-view camera can also be considered. These devices can provide more rich visual information, such as depth information and stereo vision, which can help improve the robot's perception of the environment and objects, and thus better complete complex dual-arm operation tasks.

[0142] The above-mentioned teleoperation robot system supports users to remotely control the slave robot through the "master arm - master arm host - slave arm host - slave arm" data transmission line, and further, other types of human-computer interaction devices can also be used, such as gesture recognition systems, 3D mice, VR, voice interaction systems, or brain-computer interfaces, etc. These devices can provide a more natural and convenient interaction method for human operators, further improving the efficiency and flexibility of human-robot collaboration.

[0143] In addition to controlling the robot by the user, the system also supports autonomous control of the robot to perform the corresponding task by the embodied large model, and the system can continuously collect real machine execution data, give scores to the real machine execution data manually and / or automatically, continuously reinforce learning training of the embodied large model, and realize efficient learning and performance improvement of the robot in a complex environment.

[0144] Based on the previous embodiment, in some embodiments, the system further comprises a control handle connected to the master arm host, when the control handle is in a first state, the slave robot control data is sent to the slave arm by the fine-tuned embodied large model, and when the control handle is in a second state, the slave arm is controlled by the master arm.

[0145] For example, a button, knob or switch is provided on the control handle, and in the first state, the master arm and slave arm are controlled by the large model, and in the second state, human intervention is indicated, and the master arm no longer receives motion instructions from the software controller, at which time the operator takes over the master arm and controls the slave arm. Further, in combination with the previous embodiment, three buttons are provided on the control handle as scoring buttons, corresponding to scores [-1, 0, 1], respectively, and the operator can give corresponding scores to the execution data of the human or the model.

[0146] Based on any of the embodiments, in some embodiments, the system is further used to collect master-slave robot control data, and the master-slave robot control data is used to train and fine-tune the pre-trained embodied large model to obtain the fine-tuned embodied large model.

[0147] The master-slave robot control data includes: task instruction text, slave robot control data output by the master arm, and slave robot state data and slave robot scene data. During the operation of the master arm by the user, the system collects the task instruction text uploaded by the master arm host, the slave robot control data, the slave robot state data uploaded by the slave arm host, and the slave robot scene data, and stores these data in association as master-slave robot control data. For example, ([grasp apple], [master arm left arm, master arm left gripper, master arm right arm, master arm right gripper, chassis], [slave arm left arm, slave arm left gripper, slave arm right arm, slave arm right gripper, chassis], [slave arm front view image, slave arm left hand image, slave arm right hand image]) is used as master-slave robot control data for subsequent training of the embodied large model.

[0148] Based on any of the embodiments, in some embodiments, the form of the active arm is a mechanical arm, and the operation data of the active arm is obtained through sensors (such as angle sensors at the joints). In other embodiments, the form of the active arm is an exoskeleton device, and the user wears the exoskeleton device to perform the target task, and the operation data of the active arm is obtained through sensors (such as angle sensors at the joints) on the exoskeleton device; in other embodiments, the form of the active arm is an optical motion capture device (such as an inertial navigation glove, an infrared tracking kit, etc.), and the operation data of the active arm is obtained by capturing the joint position through light sensors.

[0149] Based on any of the embodiments, in some embodiments, the active arm has the same initial joint posture as the driven arm, and the same joint movement mode.

[0150] Specifically, the active arm has the same initial joint posture as the driven arm, and the same joint movement mode, so that the coordination difficulty between the active arm and the driven arm is reduced, and the usability is improved.

[0151] Based on any of the embodiments, in some embodiments, the active arm adopts a modular design and is made through 3D printing, and the active arm can adaptively adjust the number of joints according to the type of the currently connected driven arm.

[0152] Specifically, the active arm adopts a modular and lightweight design, and supports 3D printing for rapid preparation, and can adaptively adjust the number of joints and the adjacency relationship of different joints according to the type of the currently connected driven arm. The whole system structure is compact and flexible to deploy, reduces the occupation of physical space, and significantly reduces the use threshold, providing a high cost-effective solution for scientific research experiments, industrial training, and special task environments. In addition, the sizes and shapes of the joints of the active arm and the driven arm do not necessarily need to be completely the same. In this way, a set of active arms based on 3D printing can be as small as possible in size and can be adapted to multiple sets of driven arms.

[0153] Based on any of the embodiments, in some embodiments, the scene acquisition module includes a depth camera and / or a wide-angle lens and / or a laser radar and / or an inertial sensor.

[0154] Specifically, in the present embodiment, the scene acquisition module is independently arranged away from the driven arm, and the scene acquisition module can be a wide-angle lens, a depth camera, or a 360° panoramic camera system to meet the demand for visual perception ability under different tasks.

[0155] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of constructing a robot teleoperation body model, characterized by, The method comprises the following steps: training and fine-tuning a pre-trained embodied large model by using master-slave robot control data to obtain a fine-tuned embodied large model; deploying the fine-tuned embodied large model to provide robot control services and collecting real machine execution data; the real machine execution data comprises control data output by the fine-tuned embodied large model and execution-related data of the robot; obtaining scores of the real machine execution data, and constructing a reinforcement learning training data pair of the real machine execution data and the corresponding scores, and adding the reinforcement learning training data pair to a reinforcement training data set; performing reinforcement learning training on the fine-tuned embodied large model by using the reinforcement training data set; the process of providing robot control services by the fine-tuned embodied large model comprises the following steps: receiving a control request sent by a currently connected master arm host; inputting scene data of a slave robot, task instruction text and state data of the slave robot carried in the control request into the fine-tuned embodied large model to obtain slave robot control data corresponding to the control request; sending the slave robot control data corresponding to the control request to the currently connected master arm host.

2. The method of constructing a robotic teleoperated avatar model according to claim 1, wherein, Before the step of training and fine-tuning a pre-trained embodied large model by using master-slave robot control data to obtain a fine-tuned embodied large model, the method further comprises the following step: training an initial embodied large model by using open-source robot control data to obtain the pre-trained embodied large model.

3. The method of claim 2, wherein the robot teleoperated avatar model is constructed based on a plurality of images of the user. The initial embodied large model comprises a visual language model and an action head model; the visual language model is used for feature extraction of input scene data of a slave robot and task instruction text; and the action head model processes the extracted features and state data of the slave robot to output slave robot control data.

4. The method of claim 1, wherein the robot teleoperated avatar model is constructed based on a plurality of images of a user. The master-slave robot control data is collected in the process of teleoperation of a master arm on a slave robot, and comprises task instruction text, slave robot control data output by the master arm, state data of the slave robot and scene data of the slave robot.

5. The method of claim 1, wherein the robot teleoperated avatar model is constructed based on a plurality of images of a user. The real machine execution data comprises scene data of the slave robot, task instruction text, state data of the slave robot and slave robot control data output by the fine-tuned embodied large model.

6. The method of constructing a robotic telepresence avatar model according to claim 5, wherein, The reinforcement learning training data pair comprises scene data of the slave robot, task instruction text, state data of the slave robot, slave robot control data and scores.

7. The method of constructing a robotic teleoperated avatar model of claim 1, wherein, The step of obtaining scores of the real machine execution data comprises the following steps:

8. The method of claim 1, wherein the robot teleoperated avatar model is constructed based on a plurality of images of a user. in the process of providing robot control services by the fine-tuned embodied large model, manually giving scores of execution results by a person. The step of obtaining scores of the real machine execution data comprises the following steps: processing scene data of the slave robot and / or state data of the slave robot in the real machine execution data based on a preset judgment rule to obtain scores; wherein the preset judgment rule is determined according to a task scene.

9. The method of claim 1, wherein the robot teleoperated avatar model is constructed based on a plurality of images of a user. The reinforcement training data set further includes a reinforcement learning training data pair generated according to the master-slave robot control data, and the score corresponding to the master-slave robot control data is a preset score; and / or the reinforcement training data set further includes a reinforcement learning training data pair generated by artificial intervention data, and the score corresponding to the artificial intervention data is a preset score.

10. The method of claim 1, wherein the robot teleoperated avatar model is constructed based on a plurality of images of a user. The score of the real machine execution data is obtained by: The real machine execution data is input into a scoring model to obtain a corresponding score, wherein the scoring model is trained using scored master-slave robot control data and / or scored artificial intervention data and / or scored real machine execution data.

11. A robotic teleoperation system, characterized by The system includes an active robot and a driven robot, the active robot includes an active arm and an active arm host, the driven robot includes a driven arm, a driven arm host, and a scene acquisition module. The active arm host can receive user input control data through the active arm, process and generate driven robot control data, and send the data to the driven arm host. The driven arm host can control the movement of the driven arm according to the driven robot control data, and the driven arm host can also acquire driven robot movement scene data through the scene acquisition module. The active arm host or the driven arm host or the cloud is deployed with a fine-tuned body large model, which can provide robot control services for the robot teleoperation system. The robot teleoperation system collects data according to the construction method of the robot teleoperation body large model of any one of claims 1-10, and performs reinforcement learning training on the fine-tuned body large model.

12. A robotic teleoperation system according to claim 11, wherein, The system further includes a control handle connected to the active arm host, when the control handle is in a first state, the fine-tuned body large model sends driven robot control data to the active robot to control the driven arm, and when the control handle is in a second state, the active arm controls the driven arm.

13. The robotic teleoperation system of claim 11, wherein, The system is also used to collect master-slave robot control data, which is used to train and fine-tune the pre-trained body large model to obtain the fine-tuned body large model.

14. The robotic teleoperation system of claim 11, wherein, The active arm is in the form of a robotic arm or an exoskeleton device or an optical motion capture device.

15. The robotic teleoperation system of claim 11, wherein, The active arm and the driven arm have the same initial joint posture and consistent joint movement mode.

16. The robotic teleoperation system of claim 11, wherein, The scene acquisition module includes a depth camera and / or a wide-angle lens and / or a laser radar and / or an inertial sensor.

Images