Embodied robot action generation method, system, device and medium based on structured action representation

Abstract

The application relates to a somatic robot action generation method, system, device and storage medium based on structured action representation. The method comprises the following steps: inputting current observation information obtained by observing a current scene into a visual-language-action model based on a OneModel architecture, and extracting latent policy features containing scene perception information and task execution intention; determining a local coordinate system based on the latent policy features; generating a local execution action corresponding to the current time in the local coordinate system based on the latent policy features and a shared prototype action library; converting the local execution action into an initial execution action in a world coordinate system; and determining a final execution action of the somatic robot at the current time based on the initial execution action and a supplementary control parameter. Thus, the OneModel architecture can be used to realize the cooperation between the task execution intention and the structured action representation, and the action generation stability and generalization ability of the somatic robot can be improved.

Description

Technical Field

[0001] This application relates to the field of intelligent embodied robot technology, and in particular to a method, system, device and storage medium for generating embodied robot motion based on structured motion representation. Background Technology

[0002] Actions in embodied robot maneuvers are often closely related to object pose, contact relationships, maneuver direction, or local geometry. Factors such as object placement, camera viewpoint, initial robot pose, and scene can change, leading to significant differences in action representations for similar maneuvers. Existing methods rely on policy models to implicitly learn these geometric changes, increasing the difficulty of action prediction and making action patterns difficult to reuse, thereby reducing the embodied robot's generalization ability and robustness in geometrically perturbed scenarios. Summary of the Invention

[0003] In view of the above, this application provides a method, system, device and storage medium for generating embodied robot actions based on structured motion representation, with the aim of solving the above-mentioned technical problems.

[0004] In a first aspect, this application provides a method for generating embodied robot actions based on structured action representation, the method comprising: The current observation information obtained by the embodied robot in the current scene is obtained at the current moment. The current observation information is input into the vision-language-action model to extract potential strategy features containing scene perception information and task execution intention. The latent strategy features are input into the multi-candidate coordinate system prediction branch in the lightweight action prediction head to predict multiple candidate local coordinate systems, predict the coordinate system confidence of each candidate local coordinate system, and determine the local coordinate system corresponding to the current moment from the multiple candidate local coordinate systems based on the coordinate system confidence. Each candidate local coordinate system corresponds to a candidate operation geometry, which refers to a potential spatial layout hypothesis related to the current task execution. Each candidate operation geometry corresponds to an object orientation, contact surface direction, or operation trajectory direction. The coordinate system confidence is used to characterize the degree of matching between the candidate local coordinate system and the task execution intention, the target object posture, and the end effector contact direction of the embodied robot. Based on the potential strategy features and the shared prototype action library, a local execution action corresponding to the current moment is generated in the local coordinate system; the shared prototype action library includes multiple prototype actions for implementing the embodied robot's operation tasks. Based on the mapping relationship between the local coordinate system and the world coordinate system, the local execution action is transformed into the initial execution action in the world coordinate system; Based on the initial execution action and supplementary control parameters, the final execution action of the embodied robot at the current moment is determined.

[0005] Secondly, this application provides a body robot motion generation system based on structured motion representation, the body robot motion generation system based on structured motion representation includes: The acquisition module is used to acquire the current observation information obtained by the embodied robot in the current scene at the current moment, input the current observation information into the vision-language-action model, and extract potential strategy features containing scene perception information and task execution intention; The first determining module is used to input the potential strategy features into the multi-candidate coordinate system prediction branch in the lightweight action prediction head, predict multiple candidate local coordinate systems, predict the coordinate system confidence of each candidate local coordinate system, and determine the local coordinate system corresponding to the current moment from the multiple candidate local coordinate systems based on the coordinate system confidence. Each candidate local coordinate system corresponds to a candidate operation geometry, which refers to a potential spatial layout hypothesis related to the current task execution. Each candidate operation geometry corresponds to an object orientation, contact surface direction, or operation trajectory direction. The coordinate system confidence is used to characterize the degree of matching between the candidate local coordinate system and the task execution intention, the target object posture, and the end effector contact direction of the embodied robot. A generation module is used to generate a local execution action corresponding to the current moment in the local coordinate system based on the potential strategy features and the shared prototype action library; the shared prototype action library includes multiple prototype actions for implementing the embodied robot's operation tasks. A conversion module is used to convert the local execution action to an initial execution action in the world coordinate system based on the mapping relationship between the local coordinate system and the world coordinate system. The second determining module is used to determine the final action of the embodied robot at the current moment based on the initial execution action and supplementary control parameters.

[0006] Thirdly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the embodied robot motion generation method based on structured motion representation as described in any embodiment of the first aspect.

[0007] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the embodied robot motion generation method based on structured motion representation as described in any embodiment of the first aspect.

[0008] The technical solutions provided in this application have the following advantages compared with the prior art: This method decouples action generation from the absolute world coordinate space by predicting a local coordinate system aligned with the operation geometry, significantly reducing the complexity of geometric changes that the model needs to implicitly learn. Simultaneously, by structurally combining local actions through a shared prototype action library, it effectively captures recurring operation motion patterns, ensuring that similar operations, even with different orientations in the world coordinate system, exhibit a highly consistent structural representation in the local coordinate system. Combined with joint prediction of auxiliary control parameters, this application improves the generalization ability and robustness of embodied robots in scenarios such as camera viewpoint shifts, object pose changes, and embodied robot initial pose perturbations, while maintaining the efficiency of end-to-end training and ensuring the learned action patterns have good reusability. This significantly reduces the amount of training data and training epochs required to achieve excellent operational performance. Attached Figure Description

[0009] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating a preferred embodiment of the embodied robot motion generation method based on structured motion representation in this application; Figure 2 This is an exemplary flowchart illustrating the determination of a local coordinate system according to some embodiments of this application; Figure 3 This is an exemplary flowchart illustrating the determination of local execution actions according to some embodiments of this application; Figure 4 This is an exemplary flowchart illustrating the determination of initial execution actions according to some embodiments of this application; Figure 5 This is an exemplary flowchart illustrating the determination of a lightweight motion prediction head according to some embodiments of this application; Figure 6This is a schematic diagram of a preferred embodiment of the embodied robot motion generation system based on structured motion representation in this application; Figure 7 This is a schematic diagram of a preferred embodiment of the electronic device of this application; The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0013] It should be noted that the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0014] Reference Figure 1 The diagram shown is a flowchart illustrating an embodiment of the embodied robot motion generation method based on structured motion representation according to this application. The method is executed by an electronic device (such as a processor of the electronic device), which can be implemented by a software system and / or a hardware system. This embodied robot motion generation method based on structured motion representation includes: Step 101: Obtain the current observation information obtained by the embodied robot in the current scene at the current moment, input the current observation information into the vision-language-action model, and extract the potential strategy features containing scene perception information and task execution intention.

[0015] Current observation information refers to the multimodal raw data acquired by the embodied robot at the current moment through its various onboard sensors to perceive its physical environment. For example, current observation information may include current scene images captured by cameras, scene depth maps acquired by depth cameras, body state data such as the angles of each joint of the embodied robot and the pose of the end effector, the opening and closing degree of the gripper, and natural language task commands issued by the user.

[0016] Latent policy features refer to the vector representation output by the backbone network of the vision-language-action model after the current observation information is input into the backbone network. This vector representation contains key information related to action decision-making, such as the position of objects in the current scene, the spatial layout, the state of the embodied robot itself, and the semantic intent of the task to be performed.

[0017] In some embodiments, the processor inputs the current scene image captured at the current moment into the visual encoder of the vision-language-action model. The visual encoder uses a visual transformer structure to perform block processing and self-attention encoding on the image, outputting a visual feature vector. Natural language task instructions are input into the language encoder, which uses a pre-trained language model to perform word segmentation and semantic encoding on the instructions, outputting a language feature vector. Embodied robot body state data is input into the state encoder, which uses a multilayer perceptron to encode joint angles and end-effector poses, outputting a state feature vector. The visual feature vector, language feature vector, and state feature vector are concatenated along the feature dimension, and then the concatenated features are input into the multilayer perceptron for fusion mapping, outputting a fixed-dimensional latent policy feature. Taking a clothes-folding scenario as an example, the processor inputs the image of the clothes laid out on the table captured by the current camera, the current state information of the gripper at the cuff position, and the user's language instruction to fold the sleeve towards the center line of the clothes into the model. After the above encoding and fusion processing, a high-dimensional vector is output as the latent policy feature for this scenario.

[0018] In some embodiments, the processor can also extract latent policy features using a cross-attention fusion approach: visual features are used as query vectors, and linguistic and state features are used as key-value pairs. Information interaction and deep fusion among the three are achieved through a cross-modal attention mechanism. The fused features are then processed through layer normalization and a feedforward network to output latent policy features.

[0019] Step 102: Based on the potential strategy features, determine the local coordinate system corresponding to the current moment in the current scene.

[0020] A local coordinate system is a three-dimensional reference coordinate system with a specific origin and coordinate axis orientation, dynamically predicted based on current observation information and task execution intent. Local coordinate systems are used to describe and execute actions within a local space, ensuring that the geometric description of the actions remains aligned with the scene layout and task orientation.

[0021] In some embodiments, the origin of the local coordinate system is located at the current position of the embodied robot's end effector. For example, the origin of the local coordinate system could be the center point of the embodied robot's end effector or another point. The three coordinate axes of the local coordinate system can be dynamically predicted based on latent policy features, rather than being fixed to the world coordinate system. Taking a folding clothes scenario as an example, in the local coordinate system predicted by the processor based on the current observation, the x-axis points in the direction of the sleeve folding towards the center line of the clothes, the y-axis extends along the edge of the clothes or the length of the clothes, and the z-axis is approximately perpendicular to the plane of the clothes for performing lifting or pressing actions.

[0022] In some embodiments, the processor determines the local coordinate system by: inputting potential policy features into a coordinate system prediction branch in a lightweight action prediction head, wherein the coordinate system prediction branch adopts a continuous six-dimensional rotation parameterization method, that is, predicting two orthogonal three-dimensional vectors, and then constructing the first two columns of a rotation matrix by performing Schmitt orthogonalization on these two vectors, and obtaining the third column by performing a cross product operation, thereby obtaining a complete rotation matrix; performing a rotation transformation on the world coordinate system based on the rotation matrix, and obtaining the local coordinate system corresponding to the current moment with the current position of the end effector as the origin.

[0023] In some embodiments, multiple candidate local coordinate systems are predicted based on latent policy features, each candidate local coordinate system corresponding to a candidate operational geometry. The processor further predicts the coordinate system confidence of each candidate local coordinate system and determines the local coordinate system corresponding to the current moment from the multiple candidate local coordinate systems based on the coordinate system confidence. The coordinate system confidence is used to characterize the degree of matching between the candidate local coordinate system and the current task execution intention, the target object pose, and the end effector contact direction.

[0024] A candidate local coordinate system refers to multiple possible local reference coordinate systems simultaneously predicted by the model based on the observation information at the current moment. Each candidate local coordinate system includes a rotation matrix, which defines the orientation of each coordinate axis of the candidate coordinate system in the world coordinate system.

[0025] In some embodiments, operable parts of a target object can be identified from current observation information, and at least one coordinate axis orientation of the local coordinate system can be determined based on the geometric orientation, contact normal, or motion constraint orientation of the operable parts.

[0026] Candidate operation geometry refers to the potential spatial layout assumptions related to the execution of the current task. Each candidate operation geometry corresponds to a possible object orientation, contact surface direction, or operation trajectory. For example, when faced with a drawer whose orientation is unclear, the model may simultaneously predict three candidate local coordinate systems. The first candidate local coordinate system assumes that the drawer is pulled forward, and its local X-axis points directly forward in the world coordinate system; the second candidate local coordinate system assumes that the drawer is pulled left, and its local X-axis points to the left of the world coordinate system; the third candidate local coordinate system assumes that the drawer is pulled right, and its local X-axis points to the right of the world coordinate system. These three candidates correspond to three possible operation geometries.

[0027] In some embodiments, the processor inputs latent policy features into a multi-candidate coordinate system prediction branch in the lightweight action prediction head. This branch outputs multiple consecutive rotation representations, each used to construct a rotation matrix for a candidate local coordinate system. Based on a continuous six-dimensional rotation parameterization method, each consecutive rotation representation is converted into a corresponding candidate rotation matrix, and multiple candidate local coordinate systems are determined based on these matrixes. Different candidate local coordinate systems can correspond to different candidate operation directions or candidate contact relationships, thus providing multiple optional local action representation benchmarks for action generation at the current moment.

[0028] Coordinate system confidence refers to the predicted confidence score for each candidate local coordinate system. This score characterizes the degree of match between the candidate local coordinate system and the task execution intent, the target object's pose, and the contact direction of the embodied robot's end effector. A higher confidence score indicates that the candidate coordinate system is more likely to accurately reflect the motion reference direction required for the current task.

[0029] In some embodiments, the processor may configure an additional confidence prediction subnetwork on top of the coordinate system prediction branch in the lightweight action prediction head. This subnetwork consists of a single fully connected layer with a single output dimension. Latent policy features are input into the confidence prediction subnetwork, which outputs the coordinate system confidence. Further explanation regarding the determination of the local coordinate system can be found later. Figure 2 And its detailed description.

[0030] Step 103: Based on the potential strategy features and the shared prototype action library, generate a local execution action corresponding to the current moment in the local coordinate system. The shared prototype action library includes multiple prototype actions for implementing embodied robot operation tasks.

[0031] Local execution actions refer to the motion commands of the end effector of a robot described in a local coordinate system, including local translational actions and local rotational actions.

[0032] A shared prototype action library refers to basic action templates that can be repeatedly invoked in different tasks and scenarios. Each prototype action represents a basic motion pattern direction. Locally executed actions are not directly predicted by model regression, but are generated by weighted combination of multiple prototype actions from the shared prototype action library, thereby reusing existing motion patterns.

[0033] In some embodiments, the current task stage is determined based on latent strategy features; multiple prototype actions are determined from a task stage prototype sub-library corresponding to the current task stage. Specifically, the local execution action is generated based on the multiple prototype actions. The shared prototype action library includes multiple task stage prototype sub-libraries, each corresponding to at least one of the following operation stages: approaching a target object, establishing contact, transporting a target object, fine alignment, and releasing a target object. The current task stage is determined based on latent strategy features, and translational and rotational motion prototypes are selected from the task stage prototype sub-library corresponding to the current task stage. Different task stage prototype sub-libraries correspond to different operation stages in the embodied robot's operation task. Operation stages may include at least one of approaching a target object, establishing contact, transporting a target object, fine alignment, and releasing a target object. The task stage prototype sub-library corresponding to approaching a target object may include prototype actions for moving the end effector toward the target object, adjusting the approach posture, or reducing the approach speed. The task stage prototype sub-library corresponding to establishing contact may include prototype actions for controlling the end effector to approach the contact surface, adjusting the contact angle, or controlling the gripper to close. The prototype sub-library for the task phase corresponding to moving the target object can include prototype actions for lifting, translating, obstacle avoidance, or maintaining the stability of the target object. The prototype sub-library for the task phase corresponding to fine alignment can include prototype actions for making minor corrections along holes, edges, rails, target placement areas, or target attitudes. The prototype sub-library for the task phase corresponding to releasing the target object can include prototype actions for reducing the speed of the end effector, opening the gripper, retracting the end effector, or maintaining a stable attitude after release.

[0034] Specifically, latent policy features can be input into the task stage prediction branch of the lightweight action prediction head, which outputs stage probabilities corresponding to multiple task stages. Stage probabilities characterize the likelihood that the current operation state belongs to the corresponding task stage. The task stage with the highest stage probability is determined as the current task stage. The processor selects translational and rotational action prototypes from the task stage prototype sub-library corresponding to the current task stage. Further, based on the latent policy features, multiple translational and rotational action prototypes associated with the current task execution intent are determined from the current task stage prototype sub-library, and the selected translational and rotational action prototypes are weighted, scaled, or combined to generate local execution actions in the local coordinate system.

[0035] In some embodiments, before generating a local execution action, safety constraint information can be determined based on current observation information. The activation weights of at least one prototype action in the shared prototype action library are then suppressed based on the safety constraint information. The local execution action is then generated based on the suppressed prototype action. The safety constraint information includes at least one of collision risk, joint limits, maximum end effector speed, and maximum contact force. The activation weights of each prototype action in the shared prototype action library are adjusted based on the safety constraint information to reduce the probability of prototype actions with safety risks being invoked.

[0036] Safety constraint information refers to the physical limitation parameters extracted or calculated from current observation information to define the safety boundaries of robot actions.

[0037] In some embodiments, the processor can extract depth images and robot body state data from the current observation information, construct a grid map or obstacle distance field of the current scene using the depth images, compare the current position and movement direction of the robot's end effector with the distance field, calculate the minimum distance between the end effector and the obstacle when it moves in each direction, and when the minimum distance is less than a preset safe distance threshold, assess the collision risk corresponding to that direction as high risk and record the constraint information.

[0038] In some embodiments, the processor reads the current angle values ​​of each joint of the robot from the current observation information, compares each joint angle with its mechanical limit angle, calculates the remaining range of motion angles of each joint in both positive and negative directions, and when the remaining range of motion of a certain joint is less than a preset angle threshold, the state of the joint is recorded as a limit approach state and the limit constraint information of the joint is recorded.

[0039] In some embodiments, the processor obtains the current motion speed of the end effector from the current observation information, compares it with a preset maximum permissible speed threshold, and sets the maximum speed of the end effector as speed constraint information. For the maximum contact force, the preset maximum permissible contact force threshold can be recorded as contact force constraint information based on the contact force value read in real time by the force sensor installed on the end effector.

[0040] Suppression processing refers to selectively reducing or attenuating the activation weights of the originally predicted prototype actions based on the various risk parameters in the safety constraint information, thereby reducing the contribution of prototype actions that may pose safety risks in the action synthesis process.

[0041] In some embodiments, the processor inputs safety constraint information as an additional modulation signal to the lightweight motion prediction head, enabling the lightweight motion prediction head to simultaneously consider safety factors when calculating weight allocation, thereby outputting activation weights after safety modulation. Taking collision risk constraint information as an example, if there is a nearby obstacle to the right of the end effector in the current scene, this suppression effect can significantly reduce the activation weight of the corresponding right-translating prototype action in the shared prototype action library, while keeping the weights of prototype actions in other directions unaffected or minimally affected.

[0042] In some embodiments, the processor may use the activation weights of each prototype action after suppression processing as input to replace the original activation weights of each prototype action, thereby generating local execution actions.

[0043] In some embodiments, the processor generates local execution actions in the following ways: predicting the weight distributions corresponding to translational actions and rotational actions based on latent policy features through the action prediction branch in the lightweight action prediction head; for the translation part, inputting the latent policy features into a translation gating network, which outputs a weight vector with a dimension equal to the number of translation prototypes; multiplying each weight in the weight vector with the corresponding translational action prototype, and summing all products to obtain the combined local translational action; for the rotation part, the same operation is used, outputting a rotation weight vector through a rotation gating network, and performing a weighted summation with the rotation action prototypes to obtain the local rotational action.

[0044] Further explanation regarding the determination of local execution actions can be found below. Figure 3 And its detailed description.

[0045] Step 104: Based on the mapping relationship between the local coordinate system and the world coordinate system, the local execution action is transformed into the initial execution action in the world coordinate system.

[0046] The initial execution action refers to the action representation obtained by mapping the local execution action described in the local coordinate system to the world coordinate system through coordinate system transformation. The initial execution action includes the initial translation action and the initial rotation action in the world coordinate system.

[0047] A mapping relationship refers to the rotation matrix of the local coordinate system relative to the world coordinate system. The rotation matrix can be constructed by representing the coordinate axes of the local coordinate system in the world coordinate system, and is used to transform vectors in the local coordinate system to be expressed in the world coordinate system.

[0048] In some embodiments, the specific method by which the processor performs coordinate system transformation includes: obtaining the rotation matrix predicted at the current moment from the coordinate system prediction branch, wherein each column of the rotation matrix represents the direction vector of the x-axis, y-axis, and z-axis of the local coordinate system in the world coordinate system; for local translation actions in local execution actions, directly multiplying the local translation action as a column vector with the rotation matrix on the left to obtain the initial translation action in the world coordinate system; for local rotation actions in local execution actions, multiplying the rotation vector represented by the local rotation action with the rotation matrix, that is, transforming the local rotation vector to the world coordinate system via the rotation matrix to obtain the initial rotation action in the world coordinate system; and determining the initial translation action and the initial rotation action together as the initial execution action.

[0049] Further explanation regarding determining the initial action can be found below. Figure 4 And its detailed description.

[0050] Step 105: Based on the initial execution action and supplementary control parameters, determine the final execution action of the embodied robot at the current moment.

[0051] Supplementary control parameters refer to other control quantities that need to be output simultaneously in the action space of the embodied robot, besides the end effector translational increment and end effector rotational increment. Supplementary control parameters include, but are not limited to, at least one of the following: opening and closing control quantities, opening and closing speed, grasping or releasing control quantities, and control force of the embodied robot's end effector.

[0052] The final executed action refers to the complete control signal that is finally output to each actuator of the embodied robot at the current moment. The final executed action can be obtained by splicing the initial executed action and supplementary control parameters in the action dimension.

[0053] In some embodiments, the processor determines the specific method of the final execution action by: inputting the potential policy features into the parameter prediction branch in the lightweight action prediction head, wherein the parameter prediction branch uses a multilayer perceptron structure to perform nonlinear mapping on the potential policy features and outputs a vector with the same dimension as the supplementary control parameters; concatenating the initial execution action and the supplementary control parameter vector in the dimension of the action space to form a complete action vector, which is the final execution action.

[0054] In some embodiments, the final execution action can be sent to the underlying controller of the embodied robot, which will parse it into torque commands or position commands for each joint and drive its execution.

[0055] In some embodiments, after the embodied robot completes the action execution at the current moment or in the current task phase, execution feedback information corresponding to the final executed action is obtained. If the execution feedback information meets a preset success condition, the partial executed action or the action feature corresponding to the partial executed action is written into the successful action memory. The call weight or scaling parameter of at least one prototype action in the shared prototype action library is updated based on the successful action memory. The execution feedback information includes at least one of the following: whether the task was successful, the change in the state of the target object, the contact state of the end effector, and the action deviation. When the execution feedback information meets the preset success condition, the corresponding partial executed action or its feature is written into the successful action memory. The prototype action weight, prototype action amplitude, or prototype action call priority in the shared prototype action library is updated based on the successful action memory.

[0056] In some embodiments, subsequent observation information after the final execution action can be obtained, and the execution deviation between the actual state change and the expected state change can be determined based on the subsequent observation information. When the execution deviation is greater than a preset deviation threshold, potential strategy features are re-extracted, and the local coordinate system is re-determined based on the re-extracted potential strategy features. A corrected execution action is then generated based on the re-determined local coordinate system.

[0057] In some embodiments, determining the final action of the embodied robot at the current moment based on the initial execution action and supplementary control parameters may include the following operations: S11, based on the potential policy features, predict supplementary control parameters for the initial execution action through the parameter prediction branch in the lightweight action prediction head.

[0058] The parameter prediction branch is a sub-network branch in the lightweight motion prediction head that is dedicated to predicting supplementary control parameters. This branch runs in parallel with the coordinate system prediction branch and the motion prediction branch, each handling different prediction tasks.

[0059] In some embodiments, the parameter prediction branch adopts a multilayer perceptron structure. The multilayer perceptron takes the latent policy features as input, performs nonlinear mapping through three fully connected layers, and each fully connected layer is followed by a batch normalization layer and a ReLU activation function. The last fully connected layer outputs a vector with a dimension equal to the total number of supplementary control parameters. Each element in the output vector corresponds to a different type of supplementary control parameter.

[0060] S12, the initial execution action and the supplementary control parameters are spliced ​​together in the action space of the embodied robot to obtain the final execution action at the current moment.

[0061] The motion space refers to the collection of all controllable dimensions of an embodied robot. It encompasses the position control dimension, attitude control dimension, and functional control dimension of the end effector itself.

[0062] Concatenation refers to joining the vector of the initial action and the vector of supplementary control parameters end to end in a dimension, merging them into a complete action vector with a longer dimension.

[0063] In this embodiment, supplementary control parameters are independently predicted by the parameter prediction branch, which decouples the position and attitude control of the end effector from its own opening, closing, grasping and other functional control in the network structure. This avoids interference between control quantities of different natures in the same regression task and improves the prediction accuracy of various control quantities. The supplementary control parameters cover multi-dimensional control information such as end effector opening and closing, speed, grasping and releasing and force, providing sufficient control expression capabilities for the embodied robot to perform complex operation tasks such as grasping, placing and folding.

[0064] In some embodiments of this application, by introducing a local coordinate system dynamically predicted by the model based on current observations, and generating actions within this local coordinate system using a shared prototype action library, and then converting them to the world coordinate system for execution, at least the following beneficial effects can be achieved: First, by aligning actions to the task-related local coordinate system first, the impact of geometric changes in the scene on action prediction is reduced, allowing similar operational behaviors to exhibit a consistent structure under different scene layouts; Second, by representing actions as a weighted combination of reusable prototypes through the shared prototype action library, it is beneficial to capture repetitive motion patterns and reduce the difficulty of action prediction; Third, the overall scheme still uses world coordinate system actions as the final supervision target, thus it can seamlessly integrate into the standard imitation learning training process and has wide applicability.

[0065] Figure 2 This is an exemplary flowchart illustrating the determination of a local coordinate system according to some embodiments of this application. In some embodiments, this process may be performed by an electronic device (such as its processor).

[0066] Step 201: Input the potential strategy features into the lightweight action prediction head, and use continuous 6D rotation parameterization through the coordinate system prediction branch in the lightweight action prediction head to predict the continuous six-dimensional rotation representation in the current scene corresponding to the current moment.

[0067] A lightweight action prediction head is a lightweight network module deployed on a vision-language-action model. In some embodiments, a lightweight action prediction head includes multiple parallel prediction branches, which are used to predict the local coordinate system, generate local actions, and output supplementary control parameters, respectively.

[0068] The coordinate system prediction branch is a sub-prediction branch in the lightweight motion prediction head used to predict local coordinate system parameters.

[0069] Continuous 6D rotation parameterization refers to using six consecutive values ​​to represent rotation in three-dimensional space. These six values ​​form two three-dimensional vectors, and through orthogonalization, a complete rotation matrix can be constructed.

[0070] The continuous six-dimensional rotation representation refers to the six values ​​output by the coordinate system prediction branch, which correspond to the representation of the three-dimensional rotation in a continuous six-dimensional space.

[0071] In some embodiments, the coordinate system prediction branch adopts a multilayer perceptron structure containing two fully connected layers and one output layer. The multilayer perceptron takes latent policy features as input, maps the dimensions of the latent policy features to an intermediate dimension through the first fully connected layer, introduces nonlinearity through the ReLU activation function, maps to the intermediate dimension through the second fully connected layer, and finally maps to a six-dimensional output space through the output layer, outputting six continuous values ​​as a continuous six-dimensional rotation representation.

[0072] In some embodiments, when predicting the six-dimensional rotation representation at the current moment, the coordinate system prediction branch can also use the six-dimensional rotation representation at the previous moment as a reference, predict the six-dimensional rotation residual at the current moment relative to the previous moment through the residual network module, and then add the six-dimensional rotation representation at the previous moment to the residual to obtain the continuous six-dimensional rotation representation at the current moment.

[0073] Step 202: Determine the rotation matrix based on the continuous six-dimensional rotation representation, and rotate the world coordinate system based on the rotation matrix to obtain the initial local coordinate system corresponding to the current time.

[0074] A rotation matrix is ​​used to represent the orientation of each coordinate axis of the local coordinate system in the world coordinate system. The rotation matrix can be a 3x3 orthogonal matrix, where each column represents the direction vector of the x-axis, y-axis, and z-axis of the local coordinate system in the world coordinate system. This matrix is ​​used to transform vectors in the local coordinate system to the world coordinate system, or to transform vectors in the world coordinate system to the local coordinate system.

[0075] The initial local coordinate system refers to the local reference coordinate system obtained by rotating the world coordinate system based on the rotation matrix predicted at the current moment.

[0076] In some embodiments, the specific method for determining the rotation matrix includes: resolving the continuous six-dimensional rotation representation into two three-dimensional vectors, using the first three-dimensional vector as the first column of the rotation matrix, and the second three-dimensional vector as the second column of the rotation matrix; orthogonalizing the first column vector and the second column vector, i.e., keeping the first column vector at unit length, subtracting the component of the second column vector that is parallel to the first column vector, and then normalizing the second column vector; performing a cross product operation between the first column vector and the orthogonalized second column vector to obtain a third column vector that is orthogonal to both; and arranging the first column vector, the second column vector, and the third column vector in sequence to form a 3x3 rotation matrix.

[0077] In some embodiments, the rotation matrix can also be determined by singular value decomposition: after parsing the continuous six-dimensional rotation representation into two three-dimensional vectors, these two vectors are used as two column vectors to form a 3x3 temporary matrix with other randomly initialized column vectors; singular value decomposition is performed on the temporary matrix to obtain three orthogonal matrices; and two of the orthogonal matrices are multiplied to obtain the legal rotation matrix that is closest to the original temporary matrix.

[0078] Step 203: Obtain the historical local coordinate system of the previous moment, and calculate the coordinate system change between the initial local coordinate system and the historical local coordinate system.

[0079] Historical local coordinate system refers to the local coordinate system that was determined and stored at the previous moment.

[0080] Coordinate system change refers to a quantitative indicator used to measure the degree of change in rotation between the initial local coordinate system and the historical local coordinate system. The change can be expressed as a measure of the difference between rotation matrices or the difference in rotation angles.

[0081] In some embodiments, the processor calculates the coordinate system change by performing matrix multiplication on the rotation matrix corresponding to the initial local coordinate system and the rotation matrix corresponding to the historical local coordinate system, that is, multiplying the initial rotation matrix by the transpose of the historical rotation matrix to obtain the relative rotation matrix between the two coordinate systems; extracting the corresponding rotation angle from the relative rotation matrix, specifically, calculating the rotation angle difference of the relative rotation matrix, which is the coordinate system change.

[0082] Step 204: Perform smoothing constraint processing on the initial local coordinate system based on the coordinate system change amount, and determine the initial local coordinate system after smoothing constraint processing as the local coordinate system corresponding to the current time.

[0083] Smoothing constraint processing refers to the process of adjusting or restricting the initial local coordinate system based on the magnitude of the coordinate system change.

[0084] In some embodiments, the processor performs smooth constraint processing in the following ways: pre-setting a rotation change threshold, which can be set according to the control frequency and operation accuracy requirements of the embodied robot; comparing the coordinate system change with the rotation change threshold; when the coordinate system change is less than or equal to the rotation change threshold, directly determining the initial local coordinate system as the local coordinate system at the current moment; when the coordinate system change is greater than the rotation change threshold, adjusting the initial local coordinate system, for example, interpolating and backtracking the initial local coordinate system towards the historical local coordinate system, so that the change between the adjusted coordinate system and the historical local coordinate system is equal to the rotation change threshold, and determining the adjusted coordinate system as the local coordinate system at the current moment.

[0085] In this embodiment, by using continuous six-dimensional rotation parameterization for coordinate system prediction and combining it with temporal smoothing constraint processing, the following beneficial effects can be achieved: First, continuous six-dimensional rotation representation can make coordinate system prediction more stable and easier to converge in neural network training; Second, by performing smoothing constraint processing on the coordinate system changes between adjacent time points, it is possible to effectively prevent abrupt changes in the local coordinate system in time, and ensure the smoothness and continuity of the actions of the end effector of the embodied robot.

[0086] Figure 3 This is an exemplary flowchart illustrating the determination of local execution actions according to some embodiments of this application. In some embodiments, this process may be executed by an electronic device (such as a processor).

[0087] Step 301: Using the motion prediction branch in the lightweight motion prediction head, based on the potential strategy features, determine from the shared prototype motion library multiple translational motion prototypes and multiple rotational motion prototypes associated with the task execution intention at the current moment.

[0088] Translational motion prototypes are vectors in a shared prototype motion library used to represent basic translational motion patterns. Each translational motion prototype represents a specific translational direction in the local coordinate system, such as a translation along the positive local x-axis.

[0089] A rotational motion prototype is a vector in a shared prototype motion library used to represent a basic rotational motion pattern. Each rotational motion prototype represents a specific rotational direction in a local coordinate system, such as rotation around the local z-axis.

[0090] In some embodiments, the action prediction branch employs a multi-head attention mechanism, using latent policy features as query vectors and all translational and rotational action prototypes in the shared prototype action library as key vectors. Attention scores are obtained by calculating the similarity between the query vector and each key vector. After normalizing the attention scores, attention weights are obtained, which reflect the degree of association between each action prototype and the current task execution intent. Based on the ranking of attention weights, multiple translational and rotational action prototypes with the highest rankings are selected.

[0091] Step 302: Based on the potential strategy features, perform adaptive adjustment of the multiple translational motion prototypes to obtain the local translational motion corresponding to the current time in the local coordinate system.

[0092] Action adaptive adjustment refers to the processor adjusting the amplitude and direction of multiple selected translational motion prototypes based on the current scene and task information implied by the potential policy features, so that each prototype action adapts to the specific operational requirements. Adaptive adjustment includes weighting, scaling, bias correction, or combination of prototype actions.

[0093] Local translational motion refers to the change in position of the end effector described in a local coordinate system. This change in position is represented by a translation vector, which indicates the distance and direction of movement of the embodied robot's end effector along each coordinate axis in the local coordinate system.

[0094] In some embodiments, in addition to predicting the weight vector and scaling vector, the processor can also predict an additional bias vector through the action prediction branch; the weighted sum of multiple translation action prototypes after weight and scaling adjustment is added to the bias vector to obtain the final local translation action.

[0095] In some embodiments, based on the potential strategy features, adaptive adjustment of the plurality of translational motion prototypes is performed to obtain the local translational motion corresponding to the current time in the local coordinate system, which may include the following operations: S21, for the i-th translation movement prototype among the plurality of translation movement prototypes, based on the potential strategy features, determine the translation weight vector and translation scaling vector corresponding to the i-th translation movement prototype; i is a positive integer less than or equal to the total number of the plurality of translation movement prototypes.

[0096] The translation weight vector is a weight allocation vector with a dimension equal to the total number of translation action prototypes, predicted based on the latent policy features. Each element in this vector corresponds to the activation weight of a translation action prototype, where the i-th element is the weight value corresponding to the i-th translation action prototype, and the sum of all elements after normalization is 1.

[0097] The translation scaling vector is an amplitude adjustment vector with a dimension equal to the total number of translation action prototypes, predicted based on the latent policy features. Each element in this vector corresponds to a scaling factor for a translation action prototype, where the i-th element is the scaling value corresponding to the i-th translation action prototype.

[0098] In some embodiments, the translation gating network in the action prediction branch adopts a dual-head output structure. The translation gating network takes latent policy features as input, extracts features through a shared multilayer perceptron, and then feeds them into two independent output heads. The first output head uses a fully connected layer to map the features to a weight space with a dimension equal to the total number of translation action prototypes, and outputs a translation weight vector, which is then normalized. The second output head uses a fully connected layer to map the features to a scaling space with a dimension equal to the total number of translation action prototypes, and outputs a translation scaling vector.

[0099] S22, based on the translation weight vector and translation scaling vector corresponding to the i-th translation movement prototype, adaptively adjust the i-th translation movement prototype to obtain the adjusted translation movement.

[0100] Adjusted translational motion refers to the translational motion vector obtained by applying weight and scaling values ​​to the original translational motion prototype. This vector represents the contribution of the i-th translational motion prototype in the final local translational motion.

[0101] In some embodiments, the processor extracts the i-th weight value from the translation weight vector and the i-th scaling value from the translation scaling vector; it takes the i-th translation movement prototype as the base vector, first multiplies it with the i-th scaling value to obtain the amplitude-adjusted prototype vector; then it multiplies the amplitude-adjusted prototype vector with the i-th weight value to obtain the adjusted translation movement.

[0102] S23, based on the adjustment translation movements corresponding to the plurality of translation movement prototypes respectively, determine the local translation movement corresponding to the current moment.

[0103] In some embodiments, the processor may determine the local translation movement using a weighted average method. The processor accumulates all adjusted translation movements and divides the result by the total number of translation movement prototypes to obtain a normalized local translation movement; alternatively, the processor performs a weighted average of all adjusted translation movements, with the weight values ​​being the corresponding elements in the translation weight vector. This method ensures that the amplitude of the local translation movement remains within a reasonable range, avoiding excessive movement amplitude due to the superposition of multiple prototypes.

[0104] In some embodiments, the local translational motion can be calculated based on the following formula (1): (1) in, Indicates the first Translation weight vector for each translation prototype Indicates the first The translational prototype is processed by the translation scaling vector. The adjusted translational motion is obtained after adjustment. This represents the time interval. It is understandable that the local translational motion at each time interval is obtained by a weighted sum of multiple translational motion prototypes.

[0105] In this embodiment, the weight vector and scaling vector control the participation level and motion amplitude of different prototypes in the combination, respectively, enabling the processor to flexibly integrate multiple basic motion modes to adapt to different operational needs. At the same time, by introducing the scaling value, the same translational motion prototype can output motions of different amplitudes in different scenarios, without having to learn a separate prototype for each amplitude, which greatly improves the reuse efficiency and data utilization of motion prototypes. In addition, the cumulative combination of multiple adjustable translational motions enables the final output motion to fully integrate multiple basic motion modes, and the generated translational motion has stronger expressive power and task adaptability.

[0106] Step 303: Based on the potential strategy features, perform adaptive adjustment of the multiple rotation action prototypes to obtain the local rotation action corresponding to the current time in the local coordinate system.

[0107] Local rotation refers to the change in attitude of the end effector described in a local coordinate system. This change in attitude is represented by a rotation vector, which indicates the direction and angle of rotation of the embodied robot end effector about each coordinate axis in the local coordinate system.

[0108] In some embodiments, the processor converts each rotation action prototype from rotation vector form into a quaternion representation; it then performs weighted normalization on each quaternion according to the rotation weight vector to obtain a weighted average quaternion; finally, it adjusts the magnitude of this weighted average quaternion according to the rotation scaling vector, that is, it interpolates in the quaternion space from the unit quaternion towards the weighted average quaternion, with the interpolation ratio determined by the scaling factor, ultimately obtaining the local rotation action. It is understood that in some embodiments, the local rotation action is obtained by weighted summation of rotation vectors; in other embodiments, the local rotation action is obtained by spherical weighted averaging of quaternions.

[0109] In some embodiments, the step of adaptively adjusting the plurality of rotational motion prototypes based on the latent strategy features to obtain the local rotational motion corresponding to the current moment in the local coordinate system may include the following operations: S31, for the j-th rotation action prototype among the plurality of rotation action prototypes, based on the strategy features, determine the rotation weight vector and rotation scaling vector corresponding to the j-th rotation action prototype; j is a positive integer less than or equal to the total number of the plurality of rotation action prototypes.

[0110] The rotation weight vector is a weight allocation vector with a dimension equal to the total number of rotation action prototypes, predicted based on the latent policy features. Each element in this vector corresponds to the activation weight of a rotation action prototype, where the j-th element is the weight value corresponding to the j-th rotation action prototype, and the sum of all elements after normalization is 1.

[0111] The rotation scaling vector is an amplitude adjustment vector with a dimension equal to the total number of rotation action prototypes, predicted based on the latent policy features. Each element in this vector corresponds to a scaling factor for a rotation action prototype, where the j-th element is the scaling value corresponding to the j-th rotation action prototype.

[0112] In some embodiments, the rotation gating network in the action prediction branch adopts a dual-head output structure symmetrical to the translation gating network. The rotation gating network takes latent policy features as input, extracts features through a shared multilayer perceptron, and feeds them into two independent output heads. The first output head uses a fully connected layer to map the features to a weight space with a dimension equal to the total number of rotation action prototypes, outputting a rotation weight vector, which is then normalized. The second output head uses a fully connected layer to map the features to a scaling space with a dimension equal to the total number of rotation action prototypes, outputting a rotation scaling vector.

[0113] S32, based on the rotation weight vector and rotation scaling vector corresponding to the j-th rotation action prototype, adaptively adjust the j-th rotation action prototype to obtain the adjusted rotation action.

[0114] Adjusting the rotational motion refers to the rotational motion vector obtained by applying both weight and scaling values ​​to the original rotational motion prototype. This vector represents the component contributed by the j-th rotational motion prototype in generating the final local rotational motion.

[0115] In some embodiments, the processor extracts the j-th weight value from the rotation weight vector and the j-th scaling value from the rotation scaling vector; it represents the j-th rotation action prototype as a rotation vector, where the direction of the rotation vector represents the rotation axis direction and the magnitude represents the rotation angle; it first uses the j-th scaling value to multiply the magnitude of the rotation vector, i.e., it scales the rotation angle to obtain the amplitude-adjusted rotation vector; then it uses the j-th weight value to multiply the amplitude-adjusted rotation vector to obtain the adjusted rotation action.

[0116] S33, based on the adjustment rotation actions corresponding to the multiple rotation action prototypes respectively, determine the local rotation action corresponding to the current moment.

[0117] In some embodiments, the processor converts each adjustment rotation action from rotation vector form to quaternion representation; it then aggregates all quaternions using a spherical weighted average method, that is, it performs a weighted average on a four-dimensional unit sphere for all weights, and the weight value is the rotation weight value corresponding to each rotation action prototype; the weighted averaged quaternion is then converted back to rotation vector form to obtain the local rotation action corresponding to the current moment.

[0118] In some embodiments, the local rotation action can be calculated based on the following formula (2): (2) in, Indicates the first The rotation weight vector of each rotation action prototype. Indicates the first A rotated prototype is processed by a rotation scaling vector. The resulting adjustment rotation motion This represents the time interval. It is understandable that the local translational motion at each time interval is obtained by a weighted sum of multiple rotational motion prototypes.

[0119] In this embodiment, by jointly adjusting the rotation weight vector and the rotation scaling vector, the processor can flexibly combine multiple basic rotation modes to adapt to different attitude adjustment requirements according to the current task intent. At the same time, the introduction of the scaling value allows the same rotation action prototype to output different rotation angles in different scenarios, eliminating the need to learn a prototype for each angle value, which significantly improves the reuse efficiency of the rotation action prototype. In addition, separating the rotation action from the translation action for independent adaptive adjustment and aggregation processing enables the processor to control the position and attitude of the end effector separately, improving the control accuracy of the action generation.

[0120] Step 304: Determine the local translation action and the local rotation action as the local execution actions corresponding to the current moment.

[0121] In some embodiments, the processor outputs local translation and local rotation actions as two independent vectors as local execution action outputs.

[0122] In this embodiment, by representing local actions as a weighted combination of shared prototypes instead of direct regression prediction, it is beneficial to capture common repetitive motion patterns between different tasks, thereby improving the accuracy and data efficiency of action prediction. At the same time, by assigning weights and scaling the selected action prototypes, it is possible to flexibly adapt to the specific needs of action amplitude and direction in different scenarios, and achieve adaptive adjustment in different operating environments.

[0123] Figure 4 This is an exemplary flowchart illustrating the determination of initial execution actions according to some embodiments of this application. In some embodiments, this process may be executed by an electronic device (such as a processor).

[0124] Step 401: Obtain the rotation matrix of the local coordinate system through the coordinate system transformation branch in the lightweight motion prediction head.

[0125] The rotation matrix is ​​used to characterize the representation of the orientation of each coordinate axis of the local coordinate system in the world coordinate system.

[0126] The coordinate system transformation branch refers to the sub-network branch in the lightweight motion prediction head used to perform coordinate system transformations.

[0127] For details on how to obtain the rotation matrix, please refer to the relevant description in step 202 above, which will not be repeated here.

[0128] Step 402: Based on the rotation matrix, perform a world coordinate system transformation on the local translation action to obtain the initial translation action in the world coordinate system.

[0129] The initial translational movement refers to the translational change of the end effector described in the local coordinate system after coordinate transformation using a rotation matrix, and its corresponding translational change in the world coordinate system. The initial translational movement can be represented as a three-dimensional vector, with its three components corresponding to the distances moved along the x, y, and z axes of the world coordinate system, respectively.

[0130] In some embodiments, the specific method by which the processor performs world coordinate system transformation includes: representing the local translation operation as a three-dimensional column vector, wherein the three elements of the three-dimensional column vector represent the amount of translation in the x-axis, y-axis and z-axis directions of the local coordinate system, respectively; and performing a left matrix multiplication operation between the rotation matrix and the three-dimensional column vector, that is, multiplying the local translation operation vector by the rotation matrix to obtain the initial translation operation.

[0131] Step 403: Based on the rotation matrix, perform a world coordinate system transformation on the local rotation action to obtain the initial rotation action in the world coordinate system.

[0132] In some embodiments, the specific method by which the processor performs world coordinate system transformation on the local rotation action includes: representing the local rotation action as a three-dimensional rotation vector, wherein the direction of the rotation vector represents the direction of the rotation axis in the local coordinate system, and the magnitude of the rotation vector represents the angle value of rotation around the axis; and performing a left multiplication operation between the three-dimensional rotation vector as a column vector and the rotation matrix, that is, multiplying the local rotation vector by the rotation matrix to obtain the initial rotation action.

[0133] Step 404: Determine the initial translation and initial rotation actions as the initial execution actions in the world coordinate system.

[0134] In some embodiments, the processor determines the initial execution action by concatenating the initial translational action and the initial rotational action in the action dimension to form a six-dimensional combined vector, which is the initial execution action. The three values ​​in the first preset dimension range of the six-dimensional combined vector correspond to the translation amounts in the x-axis, y-axis, and z-axis directions in the world coordinate system, and the three values ​​in the second preset dimension range correspond to the rotation amounts around the x-axis, y-axis, and z-axis directions in the world coordinate system.

[0135] In this embodiment, local translation and rotation actions are transformed into their respective coordinate systems using a rotation matrix. This ensures that actions with consistent geometry generated in the task-related local coordinate system can be correctly mapped to the world coordinate system for execution, guaranteeing both the generalization of the actions and the accuracy of the execution. At the same time, translation and rotation actions use independent transformation paths, ensuring that the two types of action components with different geometric properties can obtain accurate coordinate transformation results, avoiding transformation errors caused by confusing translation and rotation.

[0136] Figure 5 This is an exemplary flowchart illustrating the determination of a lightweight motion prediction head according to some embodiments of this application. In some embodiments, this process may be performed by an electronic device, such as a processor.

[0137] Step 501: Obtain the sample policy features obtained by observing the sample scene at the observation time, and input the sample policy features into the initial action prediction head.

[0138] The observation time refers to a specific point in time during the training data acquisition process. This point in time corresponds to the moment when a demonstration action data is recorded.

[0139] A sample scene refers to the physical environment configuration contained in the action demonstration data used for training. A sample scene includes scene images, embodied robot states, task instructions, and other information.

[0140] Sample strategy features refer to the feature vectors extracted after inputting the observation information in the sample scene into the backbone network of the vision-language-action model.

[0141] An initial motion prediction head refers to a lightweight motion prediction head that has not yet been trained or is in the process of training.

[0142] In some embodiments, the processor can read training samples in batches from a pre-collected demonstration dataset. Each training sample includes multimodal observation data at a specific observation time and the corresponding demonstration action label. The processor can input the multimodal observation data into the backbone network of the vision-language-action model. This backbone network maintains fixed parameters during the current training phase or is jointly trained with the initial action prediction head. After encoding and fusion by the backbone network, the sample policy features are output.

[0143] In some embodiments, the processor can use sample policy features as input to the initial action prediction head, and input them to the coordinate system prediction branch, action prediction branch and parameter prediction branch in the initial action prediction head, respectively.

[0144] Step 502: The sample rotation matrix is ​​predicted by the coordinate system prediction branch in the initial action prediction head, and the sample local coordinate system corresponding to the observation time in the sample scene is predicted based on the sample rotation matrix.

[0145] The sample rotation matrix refers to the rotation matrix predicted by the coordinate system prediction branch in the initial action prediction head based on the sample policy features.

[0146] The sample local coordinate system refers to the local reference coordinate system obtained by rotating the world coordinate system based on the sample rotation matrix.

[0147] In some embodiments, the coordinate system prediction branch may employ a continuous six-dimensional rotation parameterization method to output a continuous six-dimensional rotation representation based on sample policy features; orthogonalization processing is performed on the continuous six-dimensional rotation representation to construct a sample rotation matrix; the sample rotation matrix is ​​rotated and transformed with the world coordinate system, and the sample local coordinate system is obtained with the current position of the end effector as the origin. This prediction training process is consistent with steps 201 and 202 of the inference phase, ensuring that the training and inference behaviors are aligned.

[0148] Step 503: Through the action prediction branch in the initial action prediction head, a first execution action corresponding to the observation time is generated in the sample local coordinate system. Based on the mapping relationship between the sample local coordinate system and the world coordinate system, the first execution action is transformed into a second execution action in the world coordinate system.

[0149] The first execution action refers to the local execution action generated by the action prediction branch in the initial action prediction head in the sample local coordinate system, including sample local translation action and sample local rotation action in the sample local coordinate system.

[0150] The second execution action refers to the initial execution action obtained after mapping the first execution action from the sample local coordinate system to the world coordinate system.

[0151] In some embodiments, the processor can determine multiple translational action prototypes and multiple rotational action prototypes associated with the current sample task from a shared prototype action library based on sample policy features; adaptively adjust each translational action prototype and rotational action prototype by weighting and scaling, and combine them to generate a first execution action in the sample local coordinate system; the coordinate system transformation branch obtains the sample rotation matrix, and transforms the sample local translational action and sample local rotation action in the first execution action using the sample rotation matrix to obtain a second execution action in the world coordinate system. This process corresponds completely to the action generation and coordinate system transformation process in the aforementioned inference stage.

[0152] Step 504: Predict the sample control parameters of the second execution action through the parameter prediction branch in the initial action prediction head, and determine the third execution action at the observation time based on the second execution action and the sample control parameters.

[0153] Sample control parameters refer to the supplementary control parameters predicted and output by the parameter prediction branch in the initial motion prediction head based on the sample strategy characteristics. The type and meaning of these parameters are the same as those of the aforementioned supplementary control parameters, including at least one of the following: opening / closing control quantity of the end effector, opening / closing speed, gripping or releasing control quantity, and control force.

[0154] The third action refers to the complete action vector obtained by concatenating the second action with the sample control parameters in the action space. This action vector can have the same format and control information as the aforementioned final action, the difference being that the third action is the prediction result output by the initial action prediction head during training.

[0155] In some embodiments, the parameter prediction branch adopts a multilayer perceptron structure and outputs a sample control parameter vector based on the sample policy features; the second execution action is concatenated with the sample control parameter vector in the action dimension to obtain the third execution action.

[0156] Step 505: Based on the third execution action and the execution action label, train the initial action prediction head to obtain the lightweight action prediction head.

[0157] The execution action label refers to the actual action instruction in the demonstration dataset corresponding to the current observation time. This label serves as a supervisory signal to measure the prediction accuracy of the third execution action.

[0158] Training refers to the process of calculating the difference between the predicted third action and the action label, and iteratively updating the network parameters of the initial action prediction head using algorithms such as gradient backpropagation, so that the prediction result gradually approaches the true label.

[0159] In some embodiments, the processor can be trained in a phased manner. For example, in the first training phase, the processor trains the initial action prediction head using only the action supervision loss, enabling the network to quickly converge to the basic action prediction capability. In the second training phase, the processor gradually introduces prototype orthogonal constraint loss and temporal smoothing constraint loss on top of the action supervision loss to fine-tune the network. The phased training approach can avoid mutual interference between multiple loss functions in the early stages of training, which helps to achieve better convergence results.

[0160] In some embodiments, training the initial action prediction head based on the deviation between the third executed action and the executed action label to obtain the lightweight action prediction head may include the following operations: S41, calculate the action supervision loss between the third action and the action label.

[0161] Action supervision loss is a quantitative indicator used to measure the degree of difference between the third execution action predicted by the initial action prediction head and the actual execution action label recorded in the demonstration data. The smaller the loss value, the closer the predicted action is to the actual action.

[0162] In some embodiments, the processor can calculate the action supervision loss using the L1 loss function: calculate the L1 loss for each element-wise difference between the third action and the action label; use squared loss when the absolute value of the difference is less than a preset threshold, and use linear loss when the absolute value of the difference is greater than or equal to the preset threshold.

[0163] S42, apply prototype orthogonality constraints to the shared prototype action library, and apply temporal smoothing constraints to the sample rotation matrix and the rotation matrix at adjacent time points.

[0164] Prototype orthogonality constraint refers to the regularization restriction imposed on each action prototype in a shared prototype action library. This restriction ensures that different prototypes maintain a mutually orthogonal relationship, that is, the vector dot product of any two different prototypes approaches zero.

[0165] Temporal smoothing constraint refers to the regularization restriction imposed on the magnitude of change of the predicted sample rotation matrix between adjacent observation times, which makes the rotation matrix between adjacent times remain a small amount of change.

[0166] In some embodiments, the processor obtains the sample rotation matrix at the current observation time and the historical sample rotation matrix at the previous observation time; calculates the relative rotation matrix between the two rotation matrices, that is, multiplies the rotation matrix at the previous time (adjacent time) by the transpose of the rotation matrix at the current time; extracts the corresponding rotation angle from the relative rotation matrix, and uses the square of the rotation angle as the temporal smoothing constraint loss, or uses the square of the norm of the relative rotation matrix minus the identity matrix as the temporal smoothing constraint loss.

[0167] S43, weighted aggregation of action supervision loss, prototype orthogonality constraint loss and temporal smoothing constraint loss, to obtain the total loss function.

[0168] The total loss function is a single loss function that aggregates multiple different types of losses through a weighted summation. The total loss function comprehensively reflects the model's overall performance in three aspects: action prediction accuracy, prototype representation rationality, and temporal smoothness.

[0169] In some embodiments, the processor obtains a preset first weight coefficient, a second weight coefficient, and a third weight coefficient, wherein the first weight coefficient corresponds to the action supervision loss, the second weight coefficient corresponds to the prototype orthogonality constraint loss, and the third weight coefficient corresponds to the temporal smoothing constraint loss; the action supervision loss is multiplied by the first weight coefficient, the prototype orthogonality constraint loss is multiplied by the second weight coefficient, and the temporal smoothing constraint loss is multiplied by the third weight coefficient; the three product results are added together to obtain the total loss function.

[0170] S44, perform gradient backpropagation based on the total loss function to train the parameters of the initial action prediction head end-to-end, and obtain the lightweight action prediction head.

[0171] Gradient backpropagation refers to the process of propagating the error from the output layer to the input layer and updating the parameters layer by layer based on the gradient information of the total loss function with respect to each network parameter. This process uses the chain rule to calculate the partial derivative of the loss function with respect to the parameters of each layer and adjusts the parameter values ​​along the gradient descent direction to minimize the loss.

[0172] End-to-end training refers to jointly optimizing all branches in the initial action prediction head as a whole. The gradient of the total loss function is simultaneously backpropagated to the coordinate prediction branch, action prediction branch, and parameter prediction branch. The parameters of each branch are updated collaboratively during the same training process, rather than being trained independently.

[0173] In this embodiment, the action supervision loss directly uses the final predicted world coordinate system action as the supervision target, making the training process fully compatible with the standard imitation learning framework without the need for additional manual annotation or data preprocessing. At the same time, the prototype orthogonality constraint ensures that the prototypes in the shared prototype action library remain mutually orthogonal, effectively reducing functional redundancy between prototypes and making each prototype represent an independent and unique basic motion pattern, thus improving the representation efficiency of action combinations. The temporal smoothness constraint ensures that the learned local coordinate system changes smoothly between adjacent time steps, avoiding the problem of uneven and unstable actions caused by drastic coordinate system jitter. Finally, the end-to-end training method enables the coordinate system prediction branch, action prediction branch, and parameter prediction branch to optimize each other in a coordinated manner. The gradients of the three loss functions are synchronously backpropagated to each branch, jointly driving the model to converge toward the global optimum, effectively improving the accuracy and generalization ability of the final action prediction.

[0174] As illustrated in the above embodiments, the lightweight motion prediction head provided in this application does not directly generate end effector motion in the world coordinate system, but instead performs the following process: (1) Based on the current observation information, potential policy features are generated from the backbone network of the VLA model; (2) Based on the characteristics of the potential strategy, predict the local coordinate system corresponding to each time step in the current scene. At each time step t, a corresponding local coordinate system will be predicted. (3) In this local coordinate system, generate local execution actions in the local coordinate system at each time step. The local execution actions at each time step include local translation actions. Local rotational motion and remaining control dimensions ,Right now ; (4) Local translational and local rotational actions are obtained by combining them from a shared motion prototype library; (5) Map the local execution actions back to the world coordinate system using the predicted rotation matrix to obtain the target execution actions in the world coordinate system; (6) The final predicted action is composed of the target execution action and the remaining control dimensions predicted by the auxiliary head. .

[0175] The following is an exemplary embodiment of the method described above combined with a folding clothes scenario.

[0176] The embodied robot receives current observation information, including the current scene (e.g., the current camera image), the robot's own state, gripper state, and task commands. This current observation information is input into the backbone network f_backbone of the VLA model. This backbone network outputs latent policy features, which are key information extracted from the current observation information, such as the location of the target object, the task, and the robot's current pose. These latent policy features are used to subsequently predict the local coordinate system, motion prototype gating, latent scaling vector, and remaining motion control variables.

[0177] MCF-Proto (Lightweight Action Prediction Head) predicts a local coordinate system R_t associated with the current scene based on latent policy features. This local coordinate system includes a local x-axis, a local y-axis, and a local z-axis. Using folding clothes as an example: after seeing the current image and state, the model might learn: Local x-axis: points in the direction of "sleeve folding towards the center line of the garment"; Local y-axis: along the edge of the garment or the length of the garment body; Local z-axis: roughly perpendicular to the plane of the clothing, used for lifting / pressing down.

[0178] Because the model predicts the motion increment of the end effector, the local coordinate system can be approximated as "based on the current position of the end effector." For example, when folding clothes, the gripper is currently holding the cuff: Local origin: Current gripper / end effector position; Local x-axis: points to the center line of the clothing; Local y-axis: along the edge of the sleeve; Local z-axis: perpendicular to the desktop or used to lift and set as the reference origin, only changing the direction of the coordinate axis.

[0179] For example, a specific scenario could be: starting from the current gripper position, move 10cm along the local x-axis while simultaneously lifting 2cm along the local z-axis.

[0180] Further, based on latent policy features, local execution actions are generated at each time step in the local coordinate system. Each local execution action can include local translation, local rotation, and the remaining control dimensions. Within the defined local coordinate system, actions are not randomly predicted but are derived from a set of reusable "action prototypes." Action prototypes can be understood as basic action templates. For example, in the scenario of folding clothes, the local coordinate system has the following action prototypes: Prototype 1: Drag along the local x-direction; Prototype 2: Raise it slightly; Prototype 3: Smooth the fabric along the local y-direction; Prototype 4: The grippers rotate slightly; .....

[0181] Instead of generating actions from scratch each time, the model determines which action prototypes are needed to execute the target task based on the characteristics of the underlying policy, as well as how many of each action prototype to use and their magnitude. Action prototypes include translational and rotational action prototypes. At each time step, the model generates local translational and rotational actions in the local coordinate system for that time step.

[0182] Based on the characteristics of the potential strategy, determine the multiple translation movement prototypes required at each time step, and then generate the local translation movement at each time step based on the multiple translation movement prototypes at each time step.

[0183] The local translation motion at each time step is obtained by weighted combination of multiple translation motion prototypes. After softmax, all weights are summed to 1. For example, the local translation motion at time t can be obtained based on formula (1).

[0184] Similarly, based on the characteristics of the potential strategy, multiple rotational motion prototypes required at each time step are determined, and then local rotational motions at each time step are generated based on the multiple rotational motion prototypes at each time step.

[0185] The local rotation action at each time step is obtained by weighted combination of multiple rotation action prototypes. After softmax, all weights are added together to get 1. For example, the local rotation action at time t can be obtained based on formula (2).

[0186] Furthermore, the local execution action is transformed to the world coordinate system to obtain the target execution action in the world coordinate system.

[0187] The final predicted action is composed of the target execution action and the remaining control dimensions predicted by the auxiliary head. The remaining control dimensions are the quantities that need to be controlled in the action besides the translational increment Δx_t and rotational increment Δr_t of the end effector. These remaining control dimensions include, but are not limited to, gripper opening / closing control quantities, grasping / releasing control quantities, or other control quantities in the embodied robot's motion space besides the end effector's translational and rotational increments. Taking folding clothes as an example, the end effector's action is not just "where to move" and "how to turn," but may also include: how much the gripper opens, how much it closes, whether it maintains a grip, and whether it releases the clothes.

[0188] For example, when a robot folds its sleeves: Δx_t: The gripper moves from the cuff towards the center line of the garment; Δr_t: The gripper's posture rotates slightly with the edge of the sleeve; a_{t,rest}: The grippers remain closed and continue to hold the sleeve; After the sleeve is folded to the center line: Δx_t: The gripper retracts slightly; Δr_t: Gripper posture adjustment; a_{t,rest}: The gripper opens, releasing the sleeve.

[0189] Reference Figure 6 The diagram shown is a functional module schematic of the embodied robot motion generation system 100 based on structured motion representation of this application.

[0190] The embodied robot motion generation system 100 based on structured motion representation described in this application is installed in an electronic device. Depending on the functions implemented, the embodied robot motion generation system 100 includes an acquisition module 110, a first determination module 120, a generation module 130, a conversion module 140, and a second determination module 150. These modules can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, and are stored in the memory of the electronic device.

[0191] In this embodiment, the functions of each module / unit are as follows: The acquisition module 110 is used to acquire the current observation information obtained by the embodied robot in the current scene at the current moment, input the current observation information into the vision-language-action model, and extract potential strategy features containing scene perception information and task execution intention; The first determining module 120 is used to determine the local coordinate system corresponding to the current time in the current scene based on the potential strategy features; The generation module 130 is used to generate a local execution action corresponding to the current moment in the local coordinate system based on the potential strategy features and the shared prototype action library; the shared prototype action library includes multiple prototype actions for implementing the embodied robot operation tasks. The conversion module 140 is used to convert the local execution action to the initial execution action in the world coordinate system based on the mapping relationship between the local coordinate system and the world coordinate system. The second determining module 150 is used to determine the final action of the embodied robot at the current moment based on the initial execution action and supplementary control parameters.

[0192] The specific implementation of the embodied robot motion generation system based on structured motion representation in this application is largely the same as the specific implementation of the embodied robot motion generation method based on structured motion representation described above, and will not be repeated here.

[0193] Reference Figure 7The diagram shown is a schematic representation of a preferred embodiment of the electronic device of this application.

[0194] The electronic device includes a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. The memory 113 is used to store computer programs, such as a program for generating embodied robot motion based on structured motion representation; In some embodiments, the processor 111 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 111 is typically used to control the overall operation of the electronic device, such as performing data interaction or communication-related control and processing. In this embodiment, the processor 111 is used to run program code stored in the memory 113 or process data.

[0195] The communication interface 112 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The communication interface 112 may also be used to establish a communication connection between the electronic device and other electronic devices.

[0196] The memory 113 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 113 may be an internal storage unit of the electronic device, such as the hard disk or memory of the electronic device. In other embodiments, the memory 113 may also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. of the electronic device. Of course, the memory 113 may include both internal storage units and external storage devices of the electronic device. In this embodiment, the memory 113 is typically used to store the operating system and various computer programs installed on the electronic device, such as program code for a holographic robot motion generation program based on structured motion representation. In addition, the memory 113 can also be used to temporarily store various types of data that have been output or will be output.

[0197] Figure 7Only an electronic device having a processor 111, a communication interface 112, a memory 113 and a communication bus 114 is shown. However, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0198] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the embodied robot motion generation method based on structured motion representation provided in any of the foregoing method embodiments, including: The current observation information obtained by the embodied robot in the current scene is obtained at the current moment. The current observation information is input into the vision-language-action model to extract potential strategy features containing scene perception information and task execution intention. Based on the potential strategy features, determine the local coordinate system corresponding to the current moment in the current scene; Based on the potential strategy features and the shared prototype action library, a local execution action corresponding to the current moment is generated in the local coordinate system; the shared prototype action library includes multiple prototype actions for implementing the embodied robot's operation tasks. Based on the mapping relationship between the local coordinate system and the world coordinate system, the local execution action is transformed into the initial execution action in the world coordinate system; Based on the initial execution action and supplementary control parameters, the final execution action of the embodied robot at the current moment is determined.

[0199] For a detailed explanation of the above steps, please refer to the above. Figure 1 A flowchart illustrating an embodiment of a method for generating embodied robot motions based on structured motion representation.

[0200] Furthermore, this application also proposes a computer-readable storage medium that is both non-volatile and volatile. This computer-readable storage medium is any one or any combination of several of the following: hard disk, multimedia card, SD card, flash memory card, SMC, read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, etc. The computer-readable storage medium includes a data storage area and a program storage area. The program storage area stores a embodied robot motion generation program based on structured motion representation. When executed by a processor, the embodied robot motion generation program based on structured motion representation performs the following operations: The current observation information obtained by the embodied robot in the current scene is obtained at the current moment. The current observation information is input into the vision-language-action model to extract potential strategy features containing scene perception information and task execution intention. Based on the potential strategy features, determine the local coordinate system corresponding to the current moment in the current scene; Based on the potential strategy features and the shared prototype action library, a local execution action corresponding to the current moment is generated in the local coordinate system; the shared prototype action library includes multiple prototype actions for implementing the embodied robot's operation tasks. Based on the mapping relationship between the local coordinate system and the world coordinate system, the local execution action is transformed into the initial execution action in the world coordinate system; Based on the initial execution action and supplementary control parameters, the final execution action of the embodied robot at the current moment is determined.

[0201] The specific implementation of the computer-readable storage medium in this application is largely the same as the specific implementation of the above-described method for generating embodied robot actions based on structured action representation, and will not be repeated here.

[0202] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, system, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, system, article, or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, system, article, or method that includes that element.

[0203] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware simulation platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0204] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for generating embodied robot actions based on structured motion representation, characterized in that, The method includes: The current observation information obtained by the embodied robot in the current scene is obtained at the current moment. The current observation information is input into the vision-language-action model to extract potential strategy features containing scene perception information and task execution intention. The latent strategy features are input into the multi-candidate coordinate system prediction branch in the lightweight action prediction head to predict multiple candidate local coordinate systems, predict the coordinate system confidence of each candidate local coordinate system, and determine the local coordinate system corresponding to the current moment from the multiple candidate local coordinate systems based on the coordinate system confidence. Each candidate local coordinate system corresponds to a candidate operation geometry, which refers to a potential spatial layout hypothesis related to the current task execution. Each candidate operation geometry corresponds to an object orientation, contact surface direction, or operation trajectory direction. The coordinate system confidence is used to characterize the degree of matching between the candidate local coordinate system and the task execution intention, the target object posture, and the end effector contact direction of the embodied robot. Based on the potential strategy features and the shared prototype action library, a local execution action corresponding to the current moment is generated in the local coordinate system; the shared prototype action library includes multiple prototype actions for implementing the embodied robot's operation tasks. Based on the mapping relationship between the local coordinate system and the world coordinate system, the local execution action is transformed into the initial execution action in the world coordinate system; Based on the initial execution action and supplementary control parameters, the final execution action of the embodied robot at the current moment is determined, wherein the supplementary control parameters are predicted based on the latent policy features by the parameter prediction branch in the lightweight motion prediction head.

2. The method as described in claim 1, characterized in that, The step of generating a local execution action corresponding to the current moment in the local coordinate system based on the potential strategy features and the shared prototype action library includes: Using the motion prediction branch in the lightweight motion prediction head, based on the potential strategy features, multiple translational motion prototypes and multiple rotational motion prototypes associated with the task execution intention at the current moment are determined from the shared prototype motion library. Based on the potential strategy features, the multiple translational motion prototypes are adaptively adjusted to obtain the local translational motion corresponding to the current time in the local coordinate system. Based on the potential strategy features, the multiple rotation action prototypes are adaptively adjusted to obtain the local rotation action corresponding to the current moment in the local coordinate system. The local translation and local rotation actions are determined as the local execution actions corresponding to the current moment.

3. The method as described in claim 1, characterized in that, The step of generating a local execution action corresponding to the current moment in the local coordinate system based on the potential strategy features and the shared prototype action library includes: Safety constraint information is determined based on the current observation information; the safety constraint information includes at least one of collision risk, joint limit, end effector maximum speed, and maximum contact force. Based on the aforementioned safety constraint information, the activation weight of at least one prototype action in the shared prototype action library is suppressed. The local execution action is generated based on the prototype action after suppression processing.

4. The method as described in claim 2, characterized in that, The step of adaptively adjusting the actions of the multiple translational motion prototypes based on the potential strategy features to obtain the local translational motion corresponding to the current time in the local coordinate system includes: For the i-th translation movement prototype among the plurality of translation movement prototypes, based on the potential strategy features, determine the translation weight vector and translation scaling vector corresponding to the i-th translation movement prototype; i is a positive integer less than or equal to the total number of the plurality of translation movement prototypes; Based on the translation weight vector and translation scaling vector corresponding to the i-th translation movement prototype, the i-th translation movement prototype is adaptively adjusted to obtain the adjusted translation movement. Based on the adjustment translation movements corresponding to the multiple translation movement prototypes, the local translation movement corresponding to the current moment is determined.

5. The method as described in claim 2, characterized in that, The step of adaptively adjusting the multiple rotational motion prototypes based on the potential strategy features to obtain the local rotational motion corresponding to the current moment in the local coordinate system includes: For the j-th rotation action prototype among the plurality of rotation action prototypes, based on the latent strategy features, determine the rotation weight vector and rotation scaling vector corresponding to the j-th rotation action prototype; j is a positive integer less than or equal to the total number of the plurality of rotation action prototypes; Based on the rotation weight vector and rotation scaling vector corresponding to the j-th rotation action prototype, the j-th rotation action prototype is adaptively adjusted to obtain the adjusted rotation action; Based on the adjustment rotation actions corresponding to the multiple rotation action prototypes, the local rotation action corresponding to the current moment is determined.

6. The method as described in claim 2, characterized in that, The process of transforming the local execution action into an initial execution action in the world coordinate system based on the mapping relationship between the local coordinate system and the world coordinate system includes: The rotation matrix of the local coordinate system is obtained through the coordinate system transformation branch in the lightweight motion prediction head; the rotation matrix is ​​used to characterize the representation of the orientation of each coordinate axis of the local coordinate system in the world coordinate system; Based on the rotation matrix, the local translation action is transformed to the world coordinate system to obtain the initial translation action in the world coordinate system; Based on the rotation matrix, the local rotation action is transformed in the world coordinate system to obtain the initial rotation action in the world coordinate system. The initial translation and the initial rotation are determined as the initial execution actions in the world coordinate system.

7. The method as described in claim 1, characterized in that, The determination of the final action of the embodied robot at the current moment based on the initial execution action and supplementary control parameters includes: The initial execution action and the supplementary control parameters are spliced ​​together in the action space of the embodied robot to obtain the final execution action at the current moment; The supplementary control parameters include at least one of the following: opening and closing control amount, opening and closing speed, grasping / releasing control amount, and control force of the end effector of the embodied robot; the initial execution action includes at least one of the following: end translation increment and end rotation increment of the end effector of the embodied robot.

8. The method according to any one of claims 2-7, characterized in that, The method further includes: The sample policy features obtained from observing the sample scene at the observation time are acquired, and the sample policy features are input into the initial action prediction head; The sample rotation matrix is ​​predicted by the coordinate system prediction branch in the initial motion prediction head, and the sample local coordinate system corresponding to the observation time in the sample scene is predicted based on the sample rotation matrix. The first execution action corresponding to the observation time is generated in the sample local coordinate system through the action prediction branch in the initial action prediction head. Based on the mapping relationship between the sample local coordinate system and the world coordinate system, the first execution action is transformed into a second execution action in the world coordinate system. The sample control parameters of the second execution action are predicted through the parameter prediction branch in the initial action prediction head. Based on the second execution action and the sample control parameters, the third execution action is determined at the observation time. Based on the third action and the action label, the initial action prediction head is trained to obtain the lightweight action prediction head.

9. The method as described in claim 8, characterized in that, The process of training the initial action prediction head based on the deviation between the third executed action and the executed action label to obtain the lightweight action prediction head includes: Calculate the action supervision loss between the third action and the action label; The shared prototype action library is subject to prototype orthogonality constraints, and the sample rotation matrix and the rotation matrix at adjacent time points are subject to temporal smoothing constraints. The total loss function is obtained by weighted aggregation of the action supervision loss, prototype orthogonality constraint loss and temporal smoothing constraint loss. Gradient backpropagation is performed based on the total loss function to train the parameters of the initial action prediction head end-to-end, thereby obtaining the lightweight action prediction head.

10. A system for generating embodied robot motion based on structured motion representation, characterized in that, The system includes: The acquisition module is used to acquire the current observation information obtained by the embodied robot in the current scene at the current moment, input the current observation information into the vision-language-action model, and extract potential strategy features containing scene perception information and task execution intention; The first determining module is used to input the potential strategy features into the multi-candidate coordinate system prediction branch in the lightweight action prediction head, predict multiple candidate local coordinate systems, predict the coordinate system confidence of each candidate local coordinate system, and determine the local coordinate system corresponding to the current moment from the multiple candidate local coordinate systems based on the coordinate system confidence. Each candidate local coordinate system corresponds to a candidate operation geometry, which refers to a potential spatial layout hypothesis related to the current task execution. Each candidate operation geometry corresponds to an object orientation, contact surface direction, or operation trajectory direction. The coordinate system confidence is used to characterize the degree of matching between the candidate local coordinate system and the task execution intention, the target object posture, and the end effector contact direction of the embodied robot. A generation module is used to generate a local execution action corresponding to the current moment in the local coordinate system based on the potential strategy features and the shared prototype action library; the shared prototype action library includes multiple prototype actions for implementing the embodied robot's operation tasks. A conversion module is used to convert the local execution action to an initial execution action in the world coordinate system based on the mapping relationship between the local coordinate system and the world coordinate system. The second determining module is used to determine the final action of the embodied robot at the current moment based on the initial execution action and supplementary control parameters.

11. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor, when executing a program stored in memory, implements the embodied robot motion generation method based on structured motion representation as described in any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the embodied robot motion generation method based on structured motion representation as described in any one of claims 1 to 9.

Images