Method, device, storage medium and electronic device for predicting robot action

Abstract

The present disclosure relates to a method, device, storage medium and electronic equipment for robot action prediction. The method comprises: obtaining a target task to be executed by a robot and current state information; generating a plurality of candidate action control signals according to the target task and the current state information; inputting the current state information and the candidate action control signals into a robot action rehearsal model to obtain a task execution result of a candidate action corresponding to each candidate action control signal under the current state information; and the robot action rehearsal model is trained according to human operation video data and robot operation data. The robot action rehearsal model is trained using the human operation video data and the robot operation data, the interaction prior and the physical common sense contained in the massive human operation video are fully utilized, the dependence on the robot real machine operation data is reduced, and the generalization ability and the task success rate of the model are improved.

Description

Technical Field

[0001] This disclosure relates to the field of robot control technology, and more specifically, to a method, apparatus, storage medium, and electronic device for predicting robot actions. Background Technology

[0002] With the development of embodied intelligence technology, robots are evolving from performing fixed, single tasks to engaging in complex and open scenarios such as home, service, and industrial collaboration. Current robot capability acquisition primarily relies on imitation learning, reinforcement learning, and specialized rule-based strategies. While these methods perform reasonably well in limited scenarios, they generally depend on large amounts of ontological data, dedicated data collection processes, and repetitive parameter tuning and deployment, resulting in high training costs, weak generalization, and low transfer efficiency. In contact-intensive, continuous-action tasks such as grasping, assembly, and handling, models tend to be strongly bound to specific objects, workstations, and processes, leading to performance degradation with even slight changes in the scenario or task.

[0003] Especially in tasks involving frequent contact, continuous actions, and changing environments, such as grasping, assembling, transporting, opening and closing doors, organizing, and tool use, robots have high spatial dimensions and complex physical interactions. Training solely based on real-machine trajectories captured from vertical scenes often only learns specialized capabilities strongly bound to specific objects, workstations, perspectives, and processes. Once the target object changes, the environmental layout changes, the task sequence is adjusted, or the robot's hardware changes, existing strategy models are prone to significant performance degradation, action failures, and poor recovery capabilities. Furthermore, large-scale internet videos and human operation videos contain extremely rich interaction priors, physical common sense, operation sequences, object availability, and failure recovery patterns. If these low-cost, massive human video resources can be effectively utilized, and the knowledge about "how to interact with the world" can be transferred to robot systems, it is expected to significantly reduce the reliance on real-machine data for building general robot capabilities. Summary of the Invention

[0004] The purpose of this disclosure is to provide a method, apparatus, storage medium, and electronic device for predicting robot actions, in order to solve the problems existing in the related art.

[0005] To achieve the above objectives, the first aspect of this disclosure provides a method for predicting robot actions, the method comprising: The robot acquires the target task to be executed and its current state information, including environmental information, robot body state information, and semantic information of the target task. Based on the target task and the current state information, multiple candidate action control signals are generated using a visual language action model; The current state information and the candidate action control signal are input into the robot action pre-simulation model to obtain the task execution result of executing the candidate action corresponding to each candidate action control signal under the current state information; The robot motion pre-simulation model was trained based on human operation video data and robot operation data.

[0006] Optionally, the method further includes: Based on the task execution results of multiple candidate actions, the target action is determined, and the robot is controlled to execute the target action. Update the current state information based on the task execution result of the target action, and return to the step of generating multiple candidate action control signals using a visual language action model based on the target task and the current state information.

[0007] Optionally, the robot motion pre-simulation model is obtained through the following training method, which includes: Acquire multimodal data, including the human operation video data and the robot operation data; Training samples are determined based on the multimodal data. The training samples include current sample state information, sample action control signals, and sample task execution results of executing the sample action corresponding to the sample action control signals under the current sample state information. The current sample state information and the sample action control signal are used as model input parameters, and the sample task execution result is used as model output parameters. The initial model is trained to obtain a trained robot action pre-simulation model.

[0008] Optionally, determining training samples based on the multimodal data includes: For the operation data of each task in the multimodal data, the operation data of the task is aligned to align the sample action control signals and sample actions in the multimodal data to obtain aligned data. Keyframes are extracted from the aligned data to obtain key action frame data. For each key action frame data, determine the sample state information when the sample action in the key action frame data begins to be executed, and determine the sample task execution result of executing the sample action corresponding to the sample action control signal under the sample state information based on the key action frame data and the previous action frame data.

[0009] Optionally, the initial model includes an action representation module and a task execution result generation module; the step of training the initial model by using the current sample state information and the sample action control signal as model input parameters and the sample task execution result as model output parameters to obtain a trained robot action pre-simulation model includes: The robot action representation vector is determined based on the robot operation data, and the human action representation vector is determined based on the human operation video data; The robot motion representation vector and the human motion representation vector are mapped to the same potential motion space to establish a motion representation system, which is used to characterize the correspondence between motion control signals and motion representation vectors. The sample action control signal is used as the input parameter of the action representation module, and the action representation vector corresponding to the sample action control signal is used as the output parameter of the action representation module. The action representation module is trained to obtain the trained action representation module. The sample action control signal is input into the trained action representation module to obtain the sample action representation vector output by the action representation module; The sample action representation vector and the current sample state information are input into the task execution result generation module to obtain the predicted task execution result output by the task execution result generation module; The parameters of the task execution result generation module are adjusted based on the predicted task execution result and the sample task execution result to obtain the trained task execution result generation module; The trained motion representation module and the trained task execution result generation module are identified as the trained robot motion pre-simulation model.

[0010] Optionally, the training method further includes: Acquire specialized skills knowledge in a vertical field and convert that specialized technical knowledge into structured text; The structured text is embedded into the trained robot motion pre-model or the initial model; The specialized skill knowledge includes standard operating procedures and task rules, and the standard operating procedures and task rules are the constraints of the robot action pre-model or the initial model.

[0011] Optionally, after obtaining the trained robot motion pre-simulation model, the training method further includes: The trained robot motion pre-simulation model is subjected to model distillation to obtain a lightweight robot motion pre-simulation model; the lightweight robot motion pre-simulation model is deployed on the robot body.

[0012] A second aspect of this disclosure provides an apparatus for predicting robot actions, the apparatus comprising: The first acquisition module is used to acquire the target task to be executed by the robot and the current state information, wherein the state information includes environmental information, robot body state information and semantic information of the target task; The first determining module is used to determine candidate action control signals based on the target task and the current state information; The second acquisition module is used to input the current state information and the candidate action control signal into the robot action pre-simulation model to obtain the task execution result output by the robot action pre-simulation model. The task execution result is the result of executing the candidate action corresponding to the candidate action control signal under the current state information. The robot motion pre-simulation model was trained based on human operation video data and robot operation data.

[0013] A third aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method described in the first aspect of this disclosure.

[0014] A fourth aspect of this disclosure provides an electronic device, comprising: a memory having a computer program stored thereon; and a processor for executing the computer program in the memory to implement the steps of the method described in the first aspect of this disclosure.

[0015] By employing the above technical solution, and integrating environmental information, robot body state information, and semantic information of the target task, a comprehensive state basis is provided for the generation of candidate action control signals, improving the rationality and task adaptability of the candidate action control signals. Furthermore, multiple candidate action control signals are generated using a visual language action model, enhancing the robot's action tolerance and selection space in open scenarios. The task execution result is predicted using a robot action pre-simulation model, enabling the prediction of action effects before physical execution, effectively avoiding invalid actions, collisions, and execution failures, significantly improving robot operation safety and task success rate. Moreover, the model is trained not only using robot operation data but also using human operation video data and robot operation data, fully leveraging the interaction priors and physical common sense contained in massive amounts of human operation videos, reducing reliance on real robot operation data, and improving the model's generalization ability.

[0016] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0017] The accompanying drawings are provided to further understand the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof.

[0018] Figure 1 This is a method for predicting robot actions according to an exemplary embodiment.

[0019] Figure 2 This is a block diagram illustrating a device for predicting robot actions according to an exemplary embodiment.

[0020] Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0021] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0022] Figure 1 This is a method for predicting robot actions according to an exemplary embodiment. For example... Figure 1 As shown, a method for predicting robot actions may include the following steps.

[0023] In step S11, the target task to be executed by the robot and the current state information are obtained. The state information includes environmental information, robot body state information and semantic information of the target task.

[0024] The robot can be a single-arm robot, a dual-arm robot, a mobile manipulation robot, or a service robot equipped with an end effector. The target task refers to any task performed by the robot. For example, in an industrial collaboration scenario, the target task could be tasks such as parts picking, precision assembly, workpiece handling, equipment switching, tool usage, and parts organization. In a home setting, the target task could be tasks such as picking and placing items, opening and closing doors and windows, organizing tableware, and moving small items. In a commercial service scenario, the target task could be tasks such as shelf organization, goods handling, material sorting, and equipment start-up and shutdown.

[0025] The current state information includes environmental information, robot body state information, and semantic information of the target task. After determining the target task to be executed, the environmental information, body state information, and semantic information of the target task in the robot's scene are acquired, and the above three types of information are feature-encoded to form a unified state representation. For example, temporal fusion, cross-modal attention fusion, graph structure representation, object-level state representation, or hierarchical state representation can be used to construct the fused state information.

[0026] The environmental information reflects the location, spatial relationships, contact changes, and environmental dynamics of the target object. This information can be obtained through images captured by image acquisition devices, such as cameras, deployed on the robot itself. For example, environmental information can be obtained by recognizing images. It should be understood that, in this disclosure, the object refers to the entity that the robot directly manipulates, interacts with, acts upon, or processes during the execution of the target task; it is the target carrier on which the robot's actions are applied. For example, if the target task is to grasp a cup, then the object is the cup.

[0027] The robot's overall state information may include, but is not limited to, robot joint information, end effector (the execution component that directly interacts with the object, such as grippers, suction cups, power tools, etc.) information, and semantic information of the target task. Joint information may include the angles, positions, angular velocities, angular accelerations, torques, loads, and limits of each joint, reflecting the robot's motion posture and mobility. End effector information may include the three-dimensional position and orientation of the end effector, the gripper's opening and closing degree and clamping force, and the suction cup's vacuum level and suction state. Current action state information may include the type of action currently being performed (grasping, moving, placing, assembling, waiting, etc.), the action's progress / stage (initial, in motion, in position, completed, abnormal), and the deviation between the target pose and the actual pose. Semantic information of the target task reflects the current task objective, operational constraints, and stage requirements; that is, it indicates what the robot should do, on whom it should do it, and to what extent it should be considered complete.

[0028] In step S12, multiple candidate action control signals are generated using a visual language action model based on the target task and the current state information.

[0029] The visual language action model can be a pre-trained model, which can be trained using existing training methods. This disclosure does not impose specific limitations on the training method of the visual language action model.

[0030] In this disclosure, the target task and current state information are input into a visual language action model (VLAMA), resulting in multiple candidate action control signals output by the VLAMA. These control signals, output by the VLAMA, are action commands that can be directly used by the robot's underlying controller to drive the robot to complete a specified operation. For example, assuming the target task is to pick up a cup from a table, the multiple candidate action control signals generated by the VLAMA may include, but are not limited to, pose control signals, motion control signals, gripper control signals, etc.

[0031] In step S13, the current state information and candidate action control signals are input into the robot action pre-simulation model to obtain the task execution result of executing the candidate action corresponding to each candidate action control signal under the current state information.

[0032] For each candidate action signal, the current state information and the candidate action signal can be input into a pre-trained robot action pre-simulation model to obtain the task execution result of executing the candidate action corresponding to the candidate action control signal under the current state information. For example, the robot action pre-simulation model can output the task execution result in the form of video, image or text, and this disclosure does not limit it.

[0033] Task execution results include, but are not limited to, the probability of task success, changes in object state, execution stability, and whether task constraints are violated. Furthermore, robot motion pre-simulation models can predict future scene changes, task progress, and potential failure risks resulting from candidate actions.

[0034] Human operation video data contains extremely rich prior knowledge of interactions, physical common sense, operation timing, object operability, and fault recovery patterns. Fully utilizing this type of low-cost, large-scale data to transfer behavioral knowledge of human-environment interactions to robotic systems can significantly reduce the reliance on real-machine interaction data for building general robot capabilities. Therefore, in this disclosure, training can be performed based on human operation video data and robot operation data. The specific training methods will be described in detail below.

[0035] By employing the above technical solution, and integrating environmental information, robot body state information, and semantic information of the target task, a comprehensive state basis is provided for the generation of candidate action control signals, improving the rationality and task adaptability of the candidate action control signals. Furthermore, multiple candidate action control signals are generated using a visual language action model, enhancing the robot's action tolerance and selection space in open scenarios. The task execution result is predicted using a robot action pre-simulation model, enabling the prediction of action effects before physical execution, effectively avoiding invalid actions, collisions, and execution failures, significantly improving robot operation safety and task success rate. Moreover, the robot action pre-simulation model is trained using human operation video data and robot operation data, fully utilizing the interaction priors and physical common sense contained in massive amounts of human operation videos, reducing reliance on real robot operation data, and improving the model's generalization ability.

[0036] In one embodiment, the method further includes: Based on the task execution results of multiple candidate actions, the target action is determined, and the robot is controlled to execute the target action. Update the current state information based on the task execution result of the target action, and return to the step of generating multiple candidate action control signals using a visual language action model based on the target task and the current state information.

[0037] After obtaining the task execution result of each candidate action control signal (hereinafter referred to as the candidate action task execution result) according to the above scheme, the optimal action, i.e., the target action, can be determined from the multiple candidate actions corresponding to multiple candidate action control signals based on the task execution result, and the robot can be controlled to execute the target action. This can be achieved by using a value function-based screening method, a predefined rule-based screening method, a multimodal model state determination method, or a human-in-the-loop method to select the action that best matches the task objective, is safer, or is more stable as the target action from multiple task execution results. After determining the target action, the target action or the corresponding control signal is converted into a control command executable by the robot body, and the robot is driven to complete the corresponding operation according to the control command.

[0038] In addition, control commands can be control commands for robot joints, control commands for end-effector pose, control commands for skill invocation actions, or control commands for task phase switching.

[0039] After the robot completes the target action, the environmental information and the robot's own state information change. To ensure the generation of accurate multiple candidate action control signals and the obtaining of accurate task execution results in the next moment, after determining the target action, the current state information needs to be updated based on the task execution result of the target action. The updated state information is then used as the current state information, and the process returns to step S12. The generation of candidate action control signals for the next moment is then continued based on the updated state information, and this process is iterated until the target task is completed. In this way, through a time-sequential iterative execution method, the robot can advance step by step and make decisions action by action according to the task progress, achieving continuous, closed-loop, and task-oriented time-sequential execution.

[0040] In this way, multiple candidate action control signals are first obtained using a visual model. Then, the task execution result of the candidate action corresponding to each candidate action control signal is predicted. After that, the target action to be executed by the robot is selected from the multiple predicted task execution results. This allows the robot to predict the progress of the task and the risk of failure before execution, thereby improving the stability, accuracy and success rate of action selection, and thus improving the stability of determining the robot's actions and the efficiency of completing the task.

[0041] The training method for the robot motion pre-simulation model is described below. In this disclosure, the robot motion pre-simulation model is obtained through the following training method, which includes: Acquire multimodal data, including the human operation video data and the robot operation data; Training samples are determined based on the multimodal data. The training samples include current sample state information, sample action control signals, and sample task execution results of executing the sample action corresponding to the sample action control signals under the current sample state information. The current sample state information and the sample action control signal are used as model input parameters, and the sample task execution result is used as model output parameters. The initial model is trained to obtain a trained robot action pre-simulation model.

[0042] The robot operation data can be actual machine data of the robot performing actions involving historical figures. This actual machine data can include the robot's actual operation trajectory, teleoperation records, task execution logs, task descriptions, scene perception data, etc. The human operation video data can be first-person perspective video data, third-person perspective video data, dual-perspective video data, or data paired with video and text descriptions, etc.

[0043] After acquiring the multimodal data, it is processed to obtain training samples. The training samples include the current sample state information, sample action control signals, and the sample task execution results of executing the sample action corresponding to the sample action control signal under the current sample state information.

[0044] In one implementation of this embodiment, determining training samples based on the multimodal data includes: For the operation data of each task in the multimodal data, the operation data of the task is aligned to align the sample action control signals and sample actions in the multimodal data to obtain aligned data. Keyframes are extracted from the aligned data to obtain key action frame data. For each key action frame data, determine the sample state information when the sample action in the key action frame data begins to be executed, and determine the sample task execution result of executing the sample action corresponding to the sample action control signal under the sample state information based on the key action frame data and the previous action frame data.

[0045] For example, for each task's operation data included in both human-operated video data and robot-operated data, sample motion control signals and sample actions in the multimodal data are aligned. It should be understood that robot-operated data itself includes sample motion control signals and sample actions, while human-operated video data typically does not. Therefore, it is necessary to identify human actions as sample actions based on two or more adjacent frames in the human-operated video data, and generate corresponding sample motion control signals based on the identified human actions. Then, the sample actions and sample motion control signals are aligned. For example, the control signal used to instruct robot joint A to rotate 10° clockwise is aligned with the action of rotating joint A clockwise by 10°. In this way, aligned data can be obtained for both human-operated video data and robot-operated data.

[0046] Next, keyframe extraction is performed on the aligned data, that is, key action frame data is extracted. The key action frame data includes key sample actions and corresponding sample action control signals.

[0047] In addition, key action frame data can be processed through action segmentation, task stage annotation, object identification standardization, and text description normalization. Action segmentation refers to switching sample actions within the key action frame data to further refine the actions. Task stage annotation identifies which stage of the task the action belongs to: initialization, preparation, execution, adjustment, or completion. Object identification standardization refers to uniformly identifying the carrier on which the action is applied; for example, water cups, regular cups, and teacups are all uniformly identified as "cup."

[0048] After the above processing, the scattered human operation video data and robot operation data are unified into the same data system, serving as the basis for subsequent model training.

[0049] Finally, for each key action frame data, determine the sample state information when the sample action in the key action frame data begins to be executed, and determine the sample task execution result of the sample action corresponding to the sample action control signal under the sample state information based on the key action frame data and the previous action frame data.

[0050] For example, sample state information can include environmental information when the sample action in the key action frame data begins to be executed, robot body state information, and semantic information of the historical task (sample task in multimodal data). In this way, data from different sources, different modalities, and different robot platforms in multimodal data can be described as sample state information using the same encoding method, that is, using the same state description method.

[0051] After obtaining the training samples in the above manner, the sample state information and sample action control signals in the training samples are used as model input parameters, and the sample task execution results are used as model output parameters to train the initial model and obtain the trained robot action pre-simulation model.

[0052] The initial model can be an autoregressive prediction model, a diffusion prediction model, a latent space dynamics model, or a combination of multiple models, and this disclosure does not limit it.

[0053] In one implementation, the initial model includes an action representation module and a task execution result generation module. The action representation module unifies the robot's action representation vector and the human's action representation vector into a single action representation vector. The task execution result generation module pre-determines the result of executing the action based on the current sample state information and the action representation vector, i.e., it predicts the task execution result.

[0054] In this embodiment, the current sample state information and the sample action control signal are used as model input parameters, and the sample task execution result is used as model output parameters to train the initial model, thereby obtaining a trained robot action pre-simulation model, which may include: The robot action representation vector is determined based on the robot operation data, and the human action representation vector is determined based on the human operation video data; The robot motion representation vector and the human motion representation vector are mapped to the same potential motion space to establish a motion representation system, which is used to characterize the correspondence between motion control signals and motion representation vectors. The sample action control signal is used as the input parameter of the action representation module, and the action representation vector corresponding to the sample action control signal is used as the output parameter of the action representation module. The action representation module is trained to obtain the trained action representation module. The sample action control signal is input into the trained action representation module to obtain the sample action representation vector output by the action representation module; The sample action representation vector and the current sample state information are input into the task execution result generation module to obtain the predicted task execution result output by the task execution result generation module; The parameters of the task execution result generation module are adjusted based on the predicted task execution result and the sample task execution result to obtain the trained task execution result generation module; The trained motion representation module and the trained task execution result generation module are identified as the trained robot motion pre-simulation model.

[0055] Currently, the field of robot capability acquisition typically relies on direct mapping from "task to action," such as predicting corresponding actions directly based on a task. This lacks an intermediate representation layer that can uniformly integrate human and robot actions. Human operation videos often lack explicit action labels that strictly correspond to robot control systems, and significant differences exist in joint definitions, end-effector types, control frequencies, and action semantics across different robot platforms. This makes it difficult to directly translate operational knowledge from human videos into executable robot capabilities. In this implementation, a motion representation module is trained to encode both robot actions and implicit actions from human videos, mapping them to a unified latent motion space.

[0056] First, robot motion representation vectors are determined based on robot operation data, and human motion representation vectors are determined based on human operation video data. For example, the robot motion representation vector is a fixed-dimensional, normalized structured vector generated by extracting multi-dimensional features from the robot operation data, including control signals, end-effector pose changes, contact changes, and execution process characteristics, through feature fusion and encoding. The human motion representation vector is a standardized vector with the same dimension as the robot motion representation vector, generated by extracting temporal features of human or hand postures, the hand's trajectory relative to the target object, hand-object contact relationships, object state changes, interaction sequence, and action semantic features from the human operation video data, through video feature encoding. Since human operation videos typically do not contain explicit robot joint control quantities, human joint movements are not directly mapped to robot joint movements. Instead, transferable motion effect information is recovered based on hand movement trends, contact events, object displacement, object posture changes, environmental state changes, and task progress, and this motion effect information is represented as a latent motion vector at the object center. The cross-embodied action latent space is used to constrain the robot action representation vector and the human action representation vector to have a closer distance under the same task objective, the same interaction stage, or the same object state change, and a greater distance under different task objectives, different contact methods, or different object change results. The action representation vector can be a continuous vector, discrete action code, action token, hierarchical action unit, or a hybrid continuous-discrete representation; alternatively, the action representation vector can also be a short-time action primitive, a staged skill unit, or a multi-step action fragment, not limited to single-step action variables. Through this method, even if there are no explicit control signals in the human video, a unified action representation compatible with the robot action representation can be generated based on the action results, contact relationships, and object state changes during the human operation process.

[0057] Next, the robot motion representation vector and the human motion representation vector are mapped to the same potential motion space to establish a motion representation system, which is used to characterize the correspondence between motion control signals and motion representation vectors.

[0058] After obtaining the robot motion representation vectors and human motion representation vectors, a cross-embodied motion alignment network based on temporal cyclic consistency is used to map the two heterogeneous motion vectors to the same latent motion space. For example, firstly, the robot motion representation vectors and human motion representation vectors undergo temporal window segmentation, dimensional unification, coordinate system transformation, and distribution normalization to ensure uniformity in time length, feature dimension, and numerical range. Specifically, the robot end-effector pose change and the human hand motion trajectory can both be converted into object center motion features relative to the target object, operating area, or initial state. Subsequently, temporal features are extracted using robot motion encoders and human motion encoders respectively, and both are projected into latent motion vectors of the same dimension through a shared projection network. During training, a one-to-one correspondence between human motion frames and robot motion frames is not directly required. Instead, weak alignment is performed through temporal cyclic consistency constraints: for a given human motion segment, the closest robot motion segment is retrieved in the latent motion space, and then the human motion sequence is retrieved back from that robot motion segment. If the back-retrieval result is still close to the original human motion segment, the two types of actions are considered to be consistent in task progress and motion semantics. Furthermore, by combining contrastive learning constraints and object state prediction constraints, human and robot action segments performing the same task, at the same interaction stage, or producing the same object state changes can be placed closer together in the potential action space, while action segments with different task objectives, contact methods, or object change outcomes can be placed further apart. Through this approach, the system can achieve a unified semantic, spatial, and temporal representation of human manual operations and robot mechanical actions based on action objectives, motion trends, contact relationships, object state changes, and task stage progress, without relying on an explicit correspondence between human and robot joints.

[0059] Next, the sample motion control signals are used as input parameters to the motion representation module, and the corresponding motion representation vectors are used as output parameters to train the motion representation module, resulting in a trained motion representation module. In this way, the trained motion representation module can represent motion in a unified manner based on state change results, independent of specific robot joint definitions, enabling human operation video data to be used for robot capability learning.

[0060] Next, the sample action control signal is input into the trained action representation module to obtain the sample action representation vector output by the action representation module. This sample action representation vector and the current sample state information are then input into the task execution result generation module to obtain the predicted task execution result output by the task execution result generation module. Based on the predicted task execution result and the sample task execution result, the parameters of the task execution result generation module are adjusted to obtain the trained task execution result generation module. In this way, the trained task execution result generation module can predict the task execution result after executing the action corresponding to the action representation vector using the action representation vector and the current sample state information.

[0061] Finally, the trained motion representation module and the trained task execution result generation module are identified as the trained robot motion pre-simulation model.

[0062] In the inference process, firstly, the candidate action control signal is input into the trained action representation module to obtain the action representation vector output by the action representation module. Then, the action representation vector and the current state information are input into the trained task execution result generation module to obtain the task execution result output by the task execution result generation module.

[0063] In this way, instead of directly hard-aligning the robot motion representation vectors and human motion representation vectors, the robot motion representation vectors and human motion representation vectors are mapped to the same latent motion space. This establishes a unified standard between robot operation data and human operation video data. As a result, robot operation data and human operation video data can be used as training samples. This allows the robot motion pre-playing model to learn the ability to perform tasks from both robot operation data and human operation video data, significantly improving the feasibility of capability transfer across subjects, platforms, and tasks.

[0064] After obtaining the trained robot motion pre-simulation model in the above manner, the robot motion pre-simulation model can be directly applied to the inference stage to predict robot motion.

[0065] However, considering the vast amount of specialized skills, processes, operating rules, and human experience accumulated in vertical domain robotic systems, often using knowledge such as scripts, task trees, labeled trajectories, textual SOPs, video teaching, and teleoperation records, this knowledge can enhance the model's understanding and execution accuracy of tasks in the vertical domain. Therefore, in one embodiment, the training method further includes: Acquire specialized skills knowledge in a vertical field and convert that specialized technical knowledge into structured text; The structured text is embedded into the trained robot motion pre-model or the initial model; The specialized skill knowledge includes standard operating procedures and task rules, and the standard operating procedures and task rules are the constraints of the robot action pre-model or the initial model.

[0066] In this implementation, specialized skills knowledge within a vertical domain can be transformed into general capabilities. For example, the specialized skills knowledge within the vertical domain is converted into structured text. Then, the structured text is embedded into a trained robot motion pre-simulation model. For example, this structured text can serve as state change samples, action samples, and task constraint information to enhance the robot motion pre-simulation model's understanding and execution accuracy of tasks within the vertical domain.

[0067] Specialized skills knowledge can include standard operating procedures and task rules. In addition, specialized skills knowledge can also include historical experience notes, successful execution cases, and unsuccessful execution cases.

[0068] For example, specialized knowledge from the assembly domain, such as insertion sequence, contact constraints, force direction, and tolerance boundaries, can be introduced as constraints into the trained robot motion pre-model or initial model. Similarly, specialized knowledge from the grasping domain, such as graspable area, obstacle avoidance rules, and grasping stability requirements, can be introduced as constraints into the trained robot motion pre-model or initial model.

[0069] In this way, existing standard operating procedures and task rules in the vertical domain are injected into the trained robot motion pre-model or initial model. This allows the model to learn general interaction rules and achieve capability generalization while fully retaining high-precision operational capabilities in the vertical scenario. This ensures that the transfer of the model's general capabilities does not deviate from the needs of specific business scenarios, achieving a balance between general generalization capabilities and the high-precision requirements of vertical scenarios. This approach prevents the transfer of the model's general capabilities from being a vague generalization detached from business scenarios. Instead, it enables capability extension based on existing skill assets in the vertical domain, improving the flexibility of general interaction, the operational accuracy in vertical scenarios, and business compliance, effectively avoiding the problem of general capabilities being disconnected from specific business scenarios.

[0070] In this embodiment, the specialized skills knowledge of the vertical domain can be converted into structured text before training and the structured text can be embedded into the initial model. Alternatively, the structured text can be embedded into the model as a constraint during training.

[0071] In one embodiment, after obtaining the trained robot motion pre-simulation model, the training method further includes: The trained robot motion pre-simulation model is subjected to model distillation to obtain a lightweight robot motion pre-simulation model; the lightweight robot motion pre-simulation model is deployed on the robot body.

[0072] In this disclosure, an initial model can be trained on a server to obtain a trained robot motion pre-simulation model. Since the computational cost of the trained robot motion pre-simulation model is typically high, while the robot's memory and computing resources are limited, model distillation can be performed on the trained robot motion pre-simulation model before deploying it onto the robot body to obtain a lightweight robot motion pre-simulation model. Finally, the lightweight robot motion pre-simulation model is deployed onto the robot body.

[0073] The lightweight robot motion pre-model deployed on the robot itself is used to predict robot movements online, while the robot motion pre-model trained on the server can continue to be responsible for capability updates, policy optimization, and retraining in complex scenarios. In this way, the strong generalization ability of the server-side model can be preserved while meeting the actual needs of low-latency control of the robot itself.

[0074] Furthermore, after obtaining the trained robot motion pre-simulation model, it can be deployed on a server, with the robot itself only responsible for uploading perception data and receiving task execution results. Here, the perception data is used to determine state information and the target task.

[0075] Based on the same inventive concept, this disclosure provides a device for predicting robot movements. Figure 2 This is a block diagram illustrating a device for predicting robot movements according to an exemplary embodiment. Figure 2 As shown, the device 200 for predicting robot actions may include: The first acquisition module 201 is used to acquire the target task to be executed by the robot and the current state information, wherein the state information includes environmental information, robot body state information and semantic information of the target task; The first determining module 202 is used to determine candidate action control signals based on the target task and the current state information; The second acquisition module 203 is used to input the current state information and the candidate action control signal into the robot action pre-simulation model to obtain the task execution result output by the robot action pre-simulation model. The task execution result is the result of executing the candidate action corresponding to the candidate action control signal under the current state information. The robot motion pre-simulation model was trained based on human operation video data and robot operation data.

[0076] Optionally, the device 200 for predicting robot actions may further include: The second determining module is used to determine the target action based on the task execution results of multiple candidate actions, and control the robot to execute the target action; The update module is used to update the current state information according to the task execution result of the target action, and return the step of generating multiple candidate action control signals using a visual language action model based on the target task and the current state information.

[0077] Optionally, the robot motion pre-simulation model is obtained through the following training method, which includes: Acquire multimodal data, including the human operation video data and the robot operation data; Training samples are determined based on the multimodal data. The training samples include current sample state information, sample action control signals, and sample task execution results of executing the sample action corresponding to the sample action control signals under the current sample state information. The current sample state information and the sample action control signal are used as model input parameters, and the sample task execution result is used as model output parameters. The initial model is trained to obtain a trained robot action pre-simulation model.

[0078] Optionally, determining training samples based on the multimodal data includes: For the operation data of each task in the multimodal data, the operation data of the task is aligned to align the sample action control signals and sample actions in the multimodal data to obtain aligned data. Keyframes are extracted from the aligned data to obtain key action frame data. For each key action frame data, determine the sample state information when the sample action in the key action frame data begins to be executed, and determine the sample task execution result of executing the sample action corresponding to the sample action control signal under the sample state information based on the key action frame data and the previous action frame data.

[0079] Optionally, the initial model includes an action representation module and a task execution result generation module; the step of training the initial model by using the current sample state information and the sample action control signal as model input parameters and the sample task execution result as model output parameters to obtain a trained robot action pre-simulation model includes: The robot action representation vector is determined based on the robot operation data, and the human action representation vector is determined based on the human operation video data; The robot motion representation vector and the human motion representation vector are mapped to the same potential motion space to establish a motion representation system, which is used to characterize the correspondence between motion control signals and motion representation vectors. The sample action control signal is used as the input parameter of the action representation module, and the action representation vector corresponding to the sample action control signal is used as the output parameter of the action representation module. The action representation module is trained to obtain the trained action representation module. The sample action control signal is input into the trained action representation module to obtain the sample action representation vector output by the action representation module; The sample action representation vector and the current sample state information are input into the task execution result generation module to obtain the predicted task execution result output by the task execution result generation module; The parameters of the task execution result generation module are adjusted based on the predicted task execution result and the sample task execution result to obtain the trained task execution result generation module; The trained motion representation module and the trained task execution result generation module are defined as the trained robot motion pre-simulation model.

[0080] Optionally, the training method further includes: Acquire specialized skills knowledge in a vertical field and convert that specialized technical knowledge into structured text; The structured text is embedded into the trained robot motion pre-model or the initial model; The specialized skill knowledge includes standard operating procedures and task rules, and the standard operating procedures and task rules are the constraints of the robot action pre-model or the initial model.

[0081] Optionally, after obtaining the trained robot motion pre-simulation model, the training method further includes: The trained robot motion pre-simulation model is subjected to model distillation to obtain a lightweight robot motion pre-simulation model; the lightweight robot motion pre-simulation model is deployed on the robot body.

[0082] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0083] Based on the same inventive concept, this disclosure also provides an electronic device, including: a memory having a computer program stored thereon; A processor for executing the computer program in the memory to implement the steps of the method provided in this disclosure.

[0084] Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Figure 3 As shown, the electronic device 700 may include a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0085] The processor 701 controls the overall operation of the electronic device 700 to complete all or part of the steps in the method for predicting robot actions described above. The memory 702 stores various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, and application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 703 may include a screen and audio components. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 702 or transmitted via communication component 705. The audio component also includes at least one speaker for outputting audio signals. I / O interface 704 provides an interface between processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication may include Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof; therefore, the corresponding communication component 705 may include a Wi-Fi module, a Bluetooth module, or an NFC module.

[0086] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method for predicting robot actions.

[0087] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the method for predicting robot actions described above. For example, the computer-readable storage medium may be the memory 702 including program instructions described above, which may be executed by the processor 701 of the electronic device 700 to complete the method for predicting robot actions described above.

[0088] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a processor, which, when executed by the processor, implements the steps of the method for predicting robot actions described above.

[0089] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0090] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0091] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A method for predicting robot actions, characterized in that, The method includes: The robot acquires the target task to be executed and its current state information, including environmental information, robot body state information, and semantic information of the target task. Based on the target task and the current state information, multiple candidate action control signals are generated using a visual language action model; The current state information and the candidate action control signal are input into the robot action pre-simulation model to obtain the task execution result of executing the candidate action corresponding to each candidate action control signal under the current state information; The robot motion pre-simulation model was trained based on human operation video data and robot operation data; The robot motion pre-simulation model is obtained through the following training methods, which include: Acquire multimodal data, including the human operation video data and the robot operation data; Training samples are determined based on the multimodal data. The training samples include current sample state information, sample action control signals, and sample task execution results of executing the sample action corresponding to the sample action control signals under the current sample state information. The current sample state information and the sample action control signal are used as model input parameters, and the sample task execution result is used as model output parameters. The initial model is trained to obtain a trained robot action pre-simulation model. The initial model includes an action representation module and a task execution result generation module; the initial model is trained by using the current sample state information and the sample action control signal as model input parameters, and the sample task execution result as model output parameters, to obtain a trained robot action pre-simulation model, including: The robot action representation vector is determined based on the robot operation data, and the human action representation vector is determined based on the human operation video data; The robot motion representation vector and the human motion representation vector are mapped to the same potential motion space to establish a motion representation system, which is used to characterize the correspondence between motion control signals and motion representation vectors. The sample action control signal is used as the input parameter of the action representation module, and the action representation vector corresponding to the sample action control signal is used as the output parameter of the action representation module. The action representation module is trained to obtain the trained action representation module. The sample action control signal is input into the trained action representation module to obtain the sample action representation vector output by the action representation module; The sample action representation vector and the current sample state information are input into the task execution result generation module to obtain the predicted task execution result output by the task execution result generation module; The parameters of the task execution result generation module are adjusted based on the predicted task execution result and the sample task execution result to obtain the trained task execution result generation module; The trained motion representation module and the trained task execution result generation module are identified as the trained robot motion pre-simulation model.

2. The method according to claim 1, characterized in that, The method further includes: Based on the task execution results of multiple candidate actions, the target action is determined, and the robot is controlled to execute the target action. Update the current state information based on the task execution result of the target action, and return to the step of generating multiple candidate action control signals using a visual language action model based on the target task and the current state information.

3. The method according to claim 1, characterized in that, The step of determining training samples based on the multimodal data includes: For the operation data of each task in the multimodal data, the operation data of the task is aligned to align the sample action control signals and sample actions in the multimodal data to obtain aligned data. Keyframes are extracted from the aligned data to obtain key action frame data. For each key action frame data, determine the sample state information when the sample action in the key action frame data begins to be executed, and determine the sample task execution result of executing the sample action corresponding to the sample action control signal under the sample state information based on the key action frame data and the previous action frame data.

4. The method according to claim 1, characterized in that, The training methods also include: Acquire specialized skills knowledge in a vertical field and convert that specialized skills knowledge into structured text; The structured text is embedded into the trained robot motion pre-model or the initial model; The specialized skill knowledge includes standard operating procedures and task rules, and the standard operating procedures and task rules are the constraints of the robot action pre-model or the initial model.

5. The method according to claim 1, characterized in that, After obtaining the trained robot motion pre-simulation model, the training method further includes: The trained robot motion pre-simulation model is subjected to model distillation to obtain a lightweight robot motion pre-simulation model; the lightweight robot motion pre-simulation model is deployed on the robot body.

6. A device for predicting robot actions, characterized in that, The device includes: The first acquisition module is used to acquire the target task to be executed by the robot and the current state information, wherein the state information includes environmental information, robot body state information and semantic information of the target task; The first determining module is used to determine candidate action control signals based on the target task and the current state information; The second acquisition module is used to input the current state information and the candidate action control signal into the robot action pre-simulation model to obtain the task execution result output by the robot action pre-simulation model. The task execution result is the result of executing the candidate action corresponding to the candidate action control signal under the current state information. The robot motion pre-simulation model was trained based on human operation video data and robot operation data; The robot motion pre-simulation model is obtained through the following training methods, which include: Acquire multimodal data, including the human operation video data and the robot operation data; Training samples are determined based on the multimodal data. The training samples include current sample state information, sample action control signals, and sample task execution results of executing the sample action corresponding to the sample action control signals under the current sample state information. The current sample state information and the sample action control signal are used as model input parameters, and the sample task execution result is used as model output parameters. The initial model is trained to obtain a trained robot action pre-simulation model. The initial model includes an action representation module and a task execution result generation module; the initial model is trained by using the current sample state information and the sample action control signal as model input parameters, and the sample task execution result as model output parameters, to obtain a trained robot action pre-simulation model, including: The robot action representation vector is determined based on the robot operation data, and the human action representation vector is determined based on the human operation video data; The robot motion representation vector and the human motion representation vector are mapped to the same potential motion space to establish a motion representation system, which is used to characterize the correspondence between motion control signals and motion representation vectors. The sample action control signal is used as the input parameter of the action representation module, and the action representation vector corresponding to the sample action control signal is used as the output parameter of the action representation module. The action representation module is trained to obtain the trained action representation module. The sample action control signal is input into the trained action representation module to obtain the sample action representation vector output by the action representation module; The sample action representation vector and the current sample state information are input into the task execution result generation module to obtain the predicted task execution result output by the task execution result generation module; The parameters of the task execution result generation module are adjusted based on the predicted task execution result and the sample task execution result to obtain the trained task execution result generation module; The trained motion representation module and the trained task execution result generation module are identified as the trained robot motion pre-simulation model.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-5.

8. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-5.

Images