Automated task-independent data acquisition method and device for body agent, training method and device for action prediction model, equipment and storage medium

Abstract

The invention provides an automatic task-independent data acquisition method for a body-equipped agent, a training method and device for an action prediction model, equipment and a storage medium, and relates to the technical field of body-equipped agents. The task-independent data acquisition method comprises the following steps: defining a 3D position feasible range of an end effector as a bounded working space cube; training the reinforcement learning network through a near-end strategy optimization algorithm; randomly sampling a target point as a target point 3D position in the workspace cube, generating a corresponding joint position of the mechanical arm by using a reinforcement learning network, and constructing position mapping; the method comprises the following steps: driving an end effector of the mechanical arm to perform random sampling in an actual space, recording a sampling image obtained when the end effector reaches a 3D sampling position in the random sampling process of the end effector, and taking the sampling image and a corresponding joint position as task irrelevant data of the mechanical arm. The data collection process is simpler and is easy to expand, and cross-task multiplexing of data is supported.

Description

Technical Field

[0001] The present invention relates to the field of embodied intelligent body technology, and in particular to an automated task-independent data acquisition method for embodied intelligent bodies, a training method, device, equipment and storage medium for action prediction models. Background Art

[0002] Building general embodied agents capable of perceiving, reasoning, and acting in complex physical environments is a fundamental task. Recent vision-language-action (VLA) models, based on manually collected robotic task data, have advanced this goal by learning task-dependent action policies from visual observations and natural language instructions. These models have demonstrated promising performance on manipulation tasks, including object grasping, placement, and command following.

[0003] For example, the paper "Kim MJ, Pertsch K, Karamcheti S, et al. Openvla: An open-source vision-language-action model[J]. arXiv preprint arXiv:2406.09246,2024." is based on the Llama 2 language model, combined with the pre-trained visual features of DINOv2 and SigLIP, and fine-tuned on the Open X-Embodiment dataset containing 970,000 robot operation sequences to launch the open source VLA model OpenVLA.

[0004] Another example is the paper "Liu S, Wu L, Li B, et al. Rdt-1b: a diffusion foundationmodel for bimanual manipulation[J]. arXiv preprint arXiv:2410.07864, 2024." This paper uses a diffusion model to effectively represent multimodal action distributions, handles the heterogeneity of multimodal inputs with a scalable Transformer, and designs a physically interpretable unified action space to unify the action representations of different robots. The model was pre-trained on a large-scale multi-robot dataset and then fine-tuned on a self-collected dual-arm dataset. Ultimately, it demonstrated strong generalization and command-following capabilities in real-world robot experiments.

[0005] In addition, related technologies also explore methods to decouple upstream prediction and downstream execution, using visual generative models for future predictions and inverse dynamics models for action estimation and execution.

[0006] For example, the paper "Du Y, Yang S, Dai B, et al. Learning universal policies via text-guided video generation[J]. Advances in neural information processingsystems, 2023, 36: 9156-9172." uses a text-dependent video generation model, which takes the current image frame and the text segment describing the current target as input, generates a trajectory in the form of an image sequence, and then extracts the underlying action from the generated video through an inverse dynamics model and executes it.

[0007] A core bottleneck of existing approaches is the lack of scalable, diverse, and semantically aligned action data. Unlike vision and language, which boast internet-scale datasets, collecting robotics data is inherently expensive and time-consuming, and is often highly coupled to the robot hardware. In fact, most current datasets are task-specific, designed to help specific robots complete specific manipulation tasks, such as stacking blocks or opening doors. While these datasets can be effectively used to learn individual skills, they lack coverage across different robot modalities, lack diversity, and are rarely reusable across tasks or domains.

[0008] Among them, the papers "Kim MJ, Pertsch K, Karamcheti S, et al. OpenVLA: An open-source vision-language-action model[J]. arXiv preprint arXiv:2406.09246,2024." and "Liu S, Wu L, Li B, et al. Rdt-1b: a diffusion foundation model for bimanual manipulation[J]. arXiv preprint arXiv:2410.07864,2024." both used millions of robotics data for pre-training, reflecting the demand for large amounts of data. Indeed, VLA models have high-dimensional inputs and outputs, resulting in high training complexity and high action data requirements. However, even with such extensive data support, the generalization ability of these policies still has significant room for improvement. This further confirms the high demand for scalable, diverse, and semantically aligned action data in the field of embodied intelligence. At the same time, the generalization of the above two papers depends on the diversity of tasks in the dataset. The strategies and methods proposed are task-oriented and difficult to generalize to tasks that have never been seen before.

[0009] In addition, the inverse dynamics model implemented in the paper "Du Y, Yang S, Dai B, et al. Learning universal policies via text-guided video generation[J]. Advances in neural information processingsystems, 2023, 36: 9156-9172." only uses four convolutional layers, one pooling layer, and one MLP layer to predict the control values ​​of seven joint dimensions, which is not capable of high-precision tasks. At the same time, the data used are all trajectories of the robot performing fixed tasks, which does not achieve full coverage of the robot's accessible action space and executable tasks.

[0010] Ultimately, the aforementioned issues, such as motion data bottlenecks, heterogeneous robot bodies, and insufficient generalization, stem from the task-oriented design of strategies or methods. Specifically, the reliance of strategies and methods on task data creates the following three data challenges: (1) Data collection is cumbersome and difficult to scale, requiring specific tasks to be executed, and consuming manpower.

[0011] (2) Data and data-based training strategies are task-oriented and cannot be applied to multiple tasks.

[0012] (3) Data collection and labeling have high costs in money, manpower and time. Summary of the Invention

[0013] The present invention provides an automated task-independent data collection method for embodied intelligent bodies, a training method, apparatus, equipment and storage medium for action prediction models, to address the defects of the prior art in high data collection cost, low generalization, cumbersome data collection and difficulty in scalability, making the data collection process simpler and easier to expand, and enabling it to be achieved through autonomous exploration or simulation. It also supports cross-task reuse of data, enabling general models to be migrated between downstream tasks, and reducing human supervision and labeling costs.

[0014] In a first aspect, the present invention provides a method for automated task-independent data collection for an embodied intelligent agent, wherein the embodied intelligent agent comprises at least a robotic arm and an end effector located at the end of the robotic arm. The method for automated task-independent data collection for an embodied intelligent agent comprises the following steps: The feasible range of 3D positions of the end effector is defined as a bounded workspace cube; In a simulation environment, a reinforcement learning network is trained using a proximal policy optimization algorithm to predict, based on the trained reinforcement learning network, joint positions of the end effector when it reaches a target point 3D position in the workspace cube; wherein the proximal policy optimization algorithm is characterized as an algorithm that iteratively collects motion trajectory data of the end effector from an initial position to a target point 3D position based on a reward function, and iteratively updates parameters of the reinforcement learning network based on the motion trajectory data; the reward function is characterized as a function that drives the position reached by the end effector to approach the target point 3D position during each iterative training; Randomly sampling a target point in the workspace cube as the target point 3D position, generating a joint position of the manipulator corresponding to the target point 3D position using the trained reinforcement learning network, and constructing a position mapping between the joint position of the manipulator and the corresponding target point 3D position; In real space, driving the end effector of the robotic arm to perform random sampling, and recording the 3D sampling position reached by the end effector during the random sampling process, and recording the joint position of the robotic arm when the end effector reaches the 3D sampling position based on the 3D sampling position and the position mapping; the 3D sampling position is represented as an arbitrary position in the workspace cube; A sampling image is obtained when the end effector reaches the 3D sampling position, and the sampling image and the corresponding joint position are used as task-independent data of the robotic arm.

[0015] Preferably, according to the automated task-independent data collection method for an embodied intelligent agent provided by the present invention, the formula of the reward function is: r(a;w)=-||EEF_{pos}(a)-w||^2 Where r(a;w) represents the reward function value, EEF_{pos}(a) represents the 3D sampling position of the end effector in the workspace cube for calculating a given joint position a, and w represents the 3D position of the target point.

[0016] Preferably, according to the method for automated task-independent data collection for an embodied intelligent body provided by the present invention, before the step of driving the end effector of the robotic arm to randomly sample in the real space, the method includes: In the case where the embodied intelligence includes two robotic arms, a virtual boundary plane is set to divide the robotic arm sampling space to obtain a first sampling workspace of the first robotic arm and a second sampling workspace of the second robotic arm; wherein the first sampling workspace and the second sampling workspace have no intersection.

[0017] Preferably, according to the method for automated task-independent data collection for an embodied intelligent body provided by the present invention, in the process of driving the end effector of the robotic arm to randomly sample in real space, the method further comprises: The joint angles associated with the end direction of the end effector are randomly and uniformly sampled within the range of degrees of freedom to generate action samples of the end effector with various directions performing corresponding actions.

[0018] In a second aspect, the present invention further provides a method for training an action prediction model, wherein the action prediction model is trained based on task-independent data collected by the automated task-independent data collection method for an embodied intelligent body described in the first aspect, wherein the action prediction model includes at least an arm decoupling estimation module and a direction perception decoder module. The method for training the action prediction model includes: Inputting the task-independent data into an arm decoupling estimation module, so as to separate and process the sampled image through the arm decoupling estimation module to obtain a first robotic arm image and a second robotic arm image; Inputting the first robotic arm image and the second robotic arm image into an encoder to obtain pixel coding features; The pixel point encoding features are input into the direction perception decoder module, and the predicted action of the robotic arm is output. The loss function is calculated based on the comparison between the predicted action and the actual joint position to obtain a trained action prediction model.

[0019] Preferably, according to the training method of the motion prediction model provided by the second aspect of the present invention, the direction-aware decoder module comprises at least multi-scale dilated convolution, deformable convolution and angle-sensitive pooling components; Inputting the pixel point encoding features into the direction perception decoder module and outputting the predicted action of the robotic arm includes: Inputting the pixel point coded features into the multi-scale dilated convolution with a preset dilation rate for extraction processing, and outputting multi-scale directional features that capture spatial clues of multiple receptive fields; Inputting the multi-scale directional features into the deformable convolution to extract deformation features, and outputting deformation features that capture the motion deformation of the robotic arm during movement; Inputting the deformation feature into the angle-sensitive pooling component so that the deformation feature is rotated in multiple directions and then adaptively averaged pooled to output multiple features at different angles; The multiple different angle features are spliced ​​to obtain a spliced ​​feature, and the spliced ​​feature is mapped to a joint position space to determine a predicted action of the robotic arm, wherein the joint position space is obtained based on the joint position of the task-independent data.

[0020] In a third aspect, the present invention further provides an automated task-independent data collection device for an embodied intelligent body, wherein the embodied intelligent body comprises at least a robotic arm and an end effector located at the end of the robotic arm, the automated task-independent data collection device for an embodied intelligent body comprising: A definition module, configured to define a feasible range of 3D positions of the end effector as a bounded workspace cube; a training module for training a reinforcement learning network in a simulation environment using a proximal policy optimization algorithm to predict, based on the trained reinforcement learning network, joint positions when the end effector reaches a target point 3D position in the workspace cube; wherein the proximal policy optimization algorithm is characterized as an algorithm for iteratively collecting motion trajectory data of the end effector from an initial position to a target point 3D position based on a reward function, and iteratively updating parameters of the reinforcement learning network based on the motion trajectory data; and the reward function is characterized as a function that drives the position reached by the end effector to approach the target point 3D position during each iterative training; Constructing a position mapping module for randomly sampling a target point in the workspace cube as the 3D position of the target point, generating a joint position of the manipulator corresponding to the 3D position of the target point using the trained reinforcement learning network, and constructing a position mapping between the joint position of the manipulator and the corresponding 3D position of the target point; a random sampling module, configured to drive the end effector of the manipulator to perform random sampling in real space, record a 3D sampling position reached by the end effector during the random sampling process, and record a joint position of the manipulator when the end effector reaches the 3D sampling position, determined based on the 3D sampling position and the position mapping; the 3D sampling position is represented as an arbitrary position in the workspace cube; A task-independent data module is determined, which is used to obtain a sampling image when the end effector reaches the 3D sampling position, and use the sampling image and the corresponding joint position as task-independent data of the robotic arm.

[0021] In a fourth aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for collecting automated task-independent data for an embodied intelligent body as described in any one of the first aspects above or the method for training an action prediction model as described in any one of the second aspects above is implemented.

[0022] In a fifth aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the automated task-independent data acquisition method for an embodied intelligent body as described in any one of the first aspects above or the training method for an action prediction model as described in any one of the second aspects above.

[0023] In a sixth aspect, the present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the automated task-independent data acquisition method for an embodied intelligent body as described in any one of the first aspects above or the training method for an action prediction model as described in any one of the second aspects above.

[0024] The present invention provides an automated task-independent data collection method for an embodied intelligent body, a training method for an action prediction model, an apparatus, a device and a storage medium, which define the feasible range of the 3D position of the end effector as a bounded workspace cube; in a simulation environment, train a reinforcement learning network through a proximal policy optimization algorithm to predict the joint positions of the end effector when it reaches the 3D position of a target point in the workspace cube based on the trained reinforcement learning network; wherein the proximal policy optimization algorithm is characterized as an algorithm for iteratively collecting motion trajectory data of the end effector from an initial position to a 3D position of a target point based on a reward function, and iteratively updating the parameters of the reinforcement learning network based on the motion trajectory data; the reward function is characterized as a function that drives the position reached by the end effector to approach the 3D position of the target point in each iterative training; The method comprises the following steps: randomly sampling a target point in the workspace cube as the target point 3D position, using the trained reinforcement learning network to generate the joint position of the manipulator corresponding to the target point 3D position, and constructing a position mapping between the joint position of the manipulator and the corresponding 3D position of the target point; driving the end effector of the manipulator to randomly sample in real space, recording the 3D sampling position reached by the end effector during the random sampling process, and recording the joint position of the manipulator when the end effector reaches the 3D sampling position based on the 3D sampling position and the position mapping; the 3D sampling position is represented as an arbitrary position in the workspace cube; obtaining a sampling image when the end effector reaches the 3D sampling position, and using the sampling image and the corresponding joint position as the task-independent data of the manipulator. This method is used to address the defects of the prior art in high data collection cost, low generalization, cumbersome data collection, and difficulty in scalability, making the data collection process simpler and easier to expand, and can be implemented through autonomous exploration or simulation. It also supports cross-task data reuse, enabling the migration of general models between downstream tasks and reducing human supervision and annotation costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 This is one of the flow charts of the automated task-independent data collection method for embodied intelligent bodies provided by the present invention.

[0027] Figure 2 This is one of the flow charts of the training method of the action prediction model provided by the present invention.

[0028] Figure 3 It is a schematic diagram of the fusion of the automated task-independent data collection method for embodied intelligent bodies and the training method of the action prediction model provided by the present invention.

[0029] Figure 4 It is a schematic diagram comparing the automated task-independent data collection method and action prediction model training method for embodied intelligent bodies provided by the present invention with manually collected data.

[0030] Figure 5 It is a structural diagram of the automated task-independent data acquisition device for embodied intelligent bodies provided by the present invention.

[0031] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0032] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0033] The following combination Figures 1-6 The present invention describes an automated task-independent data collection method for embodied intelligent bodies, a training method for action prediction models, an apparatus, a device, and a storage medium, which are used to address the defects of the prior art in high data collection cost, low generalization, cumbersome data collection, and difficulty in scalability. The data collection process is simpler and easier to expand, can be achieved through autonomous exploration or simulation, and supports cross-task data reuse, enabling general models to be migrated between downstream tasks and reducing human supervision and labeling costs.

[0034] Figure 1 This is one of the flow charts of an automated task-independent data collection method for an embodied intelligent body provided by the present invention, such as Figure 1 As shown, the method may include but is not limited to steps S100 to S500: S100, defining a feasible range of 3D positions of the end effector as a bounded workspace cube; S200, in a simulation environment, training a reinforcement learning network using a proximal policy optimization algorithm to predict, based on the trained reinforcement learning network, joint positions of the end effector when it reaches a target point 3D position in the workspace cube; wherein the proximal policy optimization algorithm is characterized as an algorithm that iteratively collects motion trajectory data of the end effector from an initial position to a target point 3D position based on a reward function, and iteratively updates parameters of the reinforcement learning network based on the motion trajectory data; the reward function is characterized as a function that drives the position reached by the end effector to approach the target point 3D position during each iterative training; S300, randomly sampling a target point in the workspace cube as the target point 3D position, generating a joint position of the manipulator corresponding to the target point 3D position using the trained reinforcement learning network, and constructing a position mapping between the joint position of the manipulator and the corresponding target point 3D position; S400, in real space, driving the end effector of the robotic arm to perform random sampling, and recording a 3D sampling position reached by the end effector during the random sampling process, and recording a joint position of the robotic arm when the end effector reaches the 3D sampling position, determined based on the 3D sampling position and the position mapping; the 3D sampling position is represented as an arbitrary position in the workspace cube; S500 , obtaining a sampling image when the end effector reaches the 3D sampling position, and using the sampling image and the corresponding joint position as task-independent data of the robotic arm.

[0035] In step S100 of some embodiments, the feasible range of 3D positions of the end effector is defined as a bounded workspace cube.

[0036] It can be understood that the embodied intelligent body includes at least a robotic arm and an end effector located at the end of the robotic arm.

[0037] It's important to note that an embodied intelligence agent (AI) is a system with a physical form (i.e., a "body") that can achieve intelligent behavior through direct interaction with its environment (e.g., perception, action, and feedback). It emphasizes the close coupling of "body" and "environment" and represents a research direction at the intersection of artificial intelligence (AI), robotics, cognitive science, and other fields.

[0038] In the embodiment of the present invention, the embodied intelligent agent is described by taking a robot as an example. The robot has a robotic arm and an end effector located at the end of the robotic arm.

[0039] In some embodiments, the specific execution steps may be to first calculate the theoretical maximum reachable space of the end effector (usually a complex three-dimensional surface or polyhedron) based on parameters such as the joint type (rotation / movement), link length, and joint motion range of the robotic arm.

[0040] Then, according to the task requirements (such as the scope of the operating area) and safety requirements, a bounded cubic area is selected in the theoretical workspace.

[0041] Set the center position of the cube (usually aligned with the base of the robot) and the side lengths, for example: Center coordinates: (xc, yc, zc) (e.g., the center of the robot base). Side length: Lx × Ly × Lz (determined by the task scope, such as the space required for grasping operations).

[0042] Make sure the cube is completely contained within the actual reach of the robot arm, with a safety margin (such as a 10% indentation of the boundary).

[0043] Furthermore, the cube boundary is aligned with the coordinate system of the robot (such as the base coordinate system or the world coordinate system).

[0044] Set the cube bounding parameters (such as maximum / minimum X / Y / Z coordinates) in the robot controller.

[0045] Set up integrated safety logic to trigger speed limits or emergency stops when the end effector approaches the cube's boundaries. Test, through simulation or actual operation, whether the end effector can reach all vertices and corners of the cube. If unreachable areas exist, adjust the cube's boundaries or optimize the robot arm's joint parameters. This ensures the validity of the cube definition and prevents mission failures due to improper boundary settings.

[0046] Through the above steps, the feasible range of the 3D position of the end effector can be converted into a regular and safe cubic workspace, providing basic support for robot control and data acquisition.

[0047] When generating task-independent motion data (e.g., random sampling based on the PPO algorithm), constraining the end position w to within the cube avoids generating invalid or dangerous joint positions. Regular cube boundaries are easy to model, simplifying the training process of inverse dynamics models (such as the AnyPos model) and improving prediction accuracy. Furthermore, this prevents the robot arm from exceeding its physical workspace or entering collision zones (e.g., areas around surrounding equipment or personnel).

[0048] In step S200 of some embodiments, in a simulation environment, a reinforcement learning network is trained by a proximal policy optimization algorithm to predict the joint positions when the end effector reaches the 3D position of the target point in the workspace cube based on the trained reinforcement learning network.

[0049] Among them, the proximal policy optimization algorithm is characterized as an algorithm that iteratively collects the motion trajectory data of the end effector from the initial position to the 3D position of the target point based on a reward function, and iteratively updates the parameters of the reinforcement learning network according to the motion trajectory data; the reward function is characterized as a function that drives the position reached by the end effector to approach the 3D position of the target point in each iterative training.

[0050] It can be understood that the proximal policy optimization algorithm (PPO algorithm) is characterized as an algorithm that iteratively collects the motion trajectory data of the end effector from the initial position to the target point 3D position based on the reward function, and iteratively updates the parameters of the reinforcement learning network according to the motion trajectory data, so as to predict the joint position a when the end effector reaches the target point 3D position w based on the trained reinforcement learning network, and the reinforcement learning network is a=f(w).

[0051] In some embodiments, an environment is first constructed, a robotic arm model is loaded in a simulation environment (such as PyBullet, Mujoco, or a self-developed platform), and the initial joint positions and the end effector starting point are set.

[0052] The robot's initial state is when all joint angles are zero. In each reinforcement learning trajectory, the robot begins from this initial state and gradually attempts to approach the target 3D position w. With each action, the reinforcement learning network outputs a joint position a. For example, for a six-axis robotic arm, w is the coordinate of a 3D point in the robot's 3D workspace cube, and a is a 7-dimensional joint position vector, including the six-axis robotic arm's degrees of freedom plus one degree of freedom for the gripper.

[0053] In this embodiment of the present invention, a reward function r(a;w) is also designed to drive the position of the robot end effector to approach the target point 3D position w.

[0054] Among them, the formula of the reward function is: r(a;w)=-||EEF_{pos}(a)-w||^2 Where r(a;w) represents the reward function value, EEF_{pos}(a) represents the 3D sampling position of the end effector in the workspace cube for calculating a given joint position a, and w represents the 3D position of the target point.

[0055] Here, EEF_{pos}(a) is used to calculate the 3D sample position of the end effector in the workspace cube for a given joint position a. That is, the 3D coordinates of the robot end effector in the 3D workspace cube of the manipulator under the condition of joint position a. Therefore, ||EEF_{pos}(a)-w|| actually provides the distance between the robot end effector and the 3D position w of the target point.

[0056] This reward function gives a larger reward when the end effector successfully reaches the vicinity of the target point w; conversely, it gives a smaller reward when the robot deviates from the target.

[0057] During the training process, the proximal policy optimization algorithm (PPO algorithm) uses the policy gradient method to update the parameters of the reinforcement learning network through continuous iterative training.

[0058] Specifically, the Proximal Policy Optimization (PPO) algorithm collects a batch of trajectory data at each iteration. It then calculates the expected and immediate cumulative reward values ​​for each action (joint position and the actual 3D sampling position reached by the end effector). Using these calculated expected and immediate reward values, along with the probability distribution of the policy network output, it calculates the policy gradient and updates the reinforcement learning network parameters. Through continuous iterative training, the reinforcement learning network gradually learns how to output appropriate joint positions a based on different target point 3D positions w, thereby achieving precise control of the robot's end effector, ultimately yielding an effective mapping a = f(w).

[0059] Repeat trajectory collection and network update until the reinforcement learning network converges.

[0060] Training is stopped when the average reward (calculated using the expected cumulative reward function value and the immediate reward function value) stabilizes or reaches the maximum number of iterations. The model's success rate in reaching random target points, average step size, and action smoothness are tested in a simulation environment.

[0061] The technical benefits achieved by this embodiment include, but are not limited to, using a "clipping" mechanism in PPO to limit the policy update amplitude, avoiding policy crashes caused by excessive gradients and ensuring training stability. The reward function guides the policy to prioritize learning efficient paths (e.g., minimizing action energy consumption and time).

[0062] The trained reinforcement learning network can directly output a sequence of joint positions, solving the problem in existing technologies that inverse dynamics cannot be calculated using only 3D sampling positions in space. Traditional inverse dynamics calculations of the robotic arm body cannot calculate the sequence of joint positions when only the end effector position is given but not its orientation. Providing a random orientation will result in an unsolvable inverse dynamics algorithm (IK).

[0063] By training with diverse simulation environments (e.g., random target point distributions), the model can generalize to unseen targets.

[0064] This embodiment can also replace traditional teaching programming, reduce manual debugging costs, and is suitable for complex or dynamic environments.

[0065] This embodiment can be extended to scenarios such as multi-robot collaboration and dynamic obstacle avoidance (reward function and state space need to be adjusted).

[0066] In step S300 of some embodiments, a target point is randomly sampled in the workspace cube as the 3D position of the target point, and the trained reinforcement learning network is used to generate the joint position of the robotic arm corresponding to the 3D position of the target point, and a position mapping between the joint position of the robotic arm and the corresponding 3D position of the target point is constructed.

[0067] It can be understood that, based on the reinforcement learning network trained in step S200, a point w1 is randomly sampled in the workspace cube, and the corresponding joint position a1 is obtained using a = f(w). This is used to construct a mapping from the end effector's 3D position to the robot's actual action space (in our setting, the joint position space). This mapping covers all possible end effector 3D positions when the robot completes the real-world task.

[0068] Furthermore, a target point w1 is randomly generated within the workspace cube of the end effector of the robotic arm, which covers the diversity of the workspace, avoids the model from overfitting a specific area, and improves the generalization ability.

[0069] The target point w1 is input into a trained reinforcement learning network (such as the Actor-Critic model trained by the PPO algorithm), and the corresponding robotic arm joint position a1 is output. A data-driven strategy is used to directly map the target point to the joint position, replacing traditional inverse kinematics calculations and adapting to complex constraints (such as obstacles and joint restrictions).

[0070] Finally, the target point w1 is paired with the generated joint position a1 and stored to form a data set, and a lookup table of target points and joint positions is established for fast query, model verification or actual control.

[0071] The constructed mapping table can be used to quickly query joint positions in actual scenarios, reducing online computing overhead; the model can also be optimized by analyzing the mapping data (such as discovering areas where errors are concentrated).

[0072] By visualizing the motion trajectory of the robotic arm through mapping data, we can identify model defects (such as certain areas that cannot be reached) and make targeted adjustments to the reward function or training strategy.

[0073] In some embodiments of the present invention, in the process of driving the end effector of the robotic arm to randomly sample in real space, the method further includes: The joint angles associated with the end direction of the end effector are randomly and uniformly sampled within the range of degrees of freedom to generate action samples of the end effector with various directions performing corresponding actions.

[0074] It is understandable that in the actual space of the real world, in order to avoid collisions between the left and right robotic arms in the case of two arms, there is a range requirement for the y-coordinate of the end position of the robotic arm. We will sort the y-coordinate value of the target point w1 of the end effector from small to large in the mapping, randomly and uniformly sample the spatial position w1, and obtain the joint position a1 according to the position mapping set a=f(w). We also uniformly sample the joint angles related to the direction (for example, the roll / pitch / yaw of the wrist, randomly and uniformly sample within the minimum-maximum range of these three joints) for randomization of the joints to form enhanced actions, thereby generating physically valid and directional diverse action samples.

[0075] It should be noted that after the sampling space of the left and right robotic arms is divided, the working space of each robotic arm is sorted from small to large according to the Y coordinate of the end position w=(x, y, z).

[0076] Within the sorted Y coordinate range, random and uniform sampling of spatial positions w is performed to ensure that the entire subspace is covered and evenly distributed. This ensures that the sampling points are orderly and uniform in the Y-axis direction and avoids clustering or omissions.

[0077] For each sampled end position (3D sampling position) w, the corresponding joint position a is calculated by the mapping function a=f(w).

[0078] If a reinforcement learning network is used, directly input w and output the joint position a.

[0079] It should be noted that random sampling is used to ensure sufficient coverage of the workspace. After the 3D sampling position w is input, the output joint position a at least includes the joint angles that affect the end direction (such as the roll / pitch / yaw angles of the wrist).

[0080] Within the physical limits of each joint position a (e.g. ), generate random values.

[0081] Furthermore, it is necessary to check whether the generated joint position meets the conditions and whether it is a valid and legal action based on the joint limit and self-collision detection, as well as the end position allowable error.

[0082] Joint limits indicate that all joint angles are within physical limits. Self-collision detection ensures collision-free movement between the robot's links, using simulation or sensor data. End position error indicates that the deviation between the actual end position and the target point's 3D position is within the permitted range.

[0083] Pair valid joint positions with the end-point 3D sampling position w to form a set of motion samples. Redundant degrees of freedom (such as joints not related to the end-point orientation) are randomly and uniformly sampled to further expand sample diversity. This builds a high-coverage motion library to support model training or control strategy optimization.

[0084] This embodiment isolates the movement of both arms in terms of space and posture by dividing the Y-axis workspace and randomizing the direction, thereby eliminating the risk of physical collision.

[0085] In some embodiments of the present invention, before the step of driving the end effector of the robotic arm to randomly sample in the real space, the method includes: In the case where the embodied intelligence includes two robotic arms, a virtual boundary plane is set to divide the robotic arm sampling space to obtain a first sampling workspace of the first robotic arm and a second sampling workspace of the second robotic arm; wherein the first sampling workspace and the second sampling workspace have no intersection.

[0086] It's understandable that when collecting random data with both arms, a virtual random boundary plane (e.g., y∈[-0.15,0.15] (in meters)) is set to partition the workspace into non-intersecting left and right workspaces (i.e., the first sampling workspace for the first manipulator and the second sampling workspace for the second manipulator) to prevent collisions between the left and right manipulators. For each manipulator, the y range is set to 0.15 away from the virtual random boundary plane, resulting in a sampling space with a restricted y range. We then use the aforementioned random sampling method in real space to independently sample each arm and execute the corresponding joint angles to move the manipulator to the corresponding position.

[0087] It should be noted that the first robotic arm can be a left robotic arm or a right robotic arm, without specific limitation. The second robotic arm also has the same setting, but when the first robotic arm is a left robotic arm, the second robotic arm can only be a right robotic arm.

[0088] Based on the working range and physical structure of the robot arm, a virtual boundary plane (such as a plane perpendicular to the Y axis) is defined to divide the overall workspace into two independent subspaces: First sampling workspace: assigned to the first robotic arm (such as the left arm), limiting the Y coordinate range of its end position (for example ).

[0089] Second sampling workspace: assigned to the second robotic arm (such as the right arm), limiting the Y coordinate range of its end position (for example ), and satisfies .

[0090] Ensure that the two subspaces do not overlap in the Y-axis direction to prevent collisions between the arms in physical space. Hard constraints eliminate the risk of physical conflicts during arm movement, providing a secure foundation for subsequent sampling.

[0091] Through the above steps, the system can efficiently generate collision-free, highly diverse and physically feasible action samples, providing a reliable data foundation for embodied intelligence tasks (such as dual-arm collaboration and object handling).

[0092] In step S400 of some embodiments, in real space, the end effector of the robotic arm is driven to perform random sampling, and the 3D sampling position reached by the end effector during the random sampling process of the end effector is recorded, and the joint position of the robotic arm when the end effector reaches the 3D sampling position is recorded based on the 3D sampling position and the position mapping; the 3D sampling position is represented as an arbitrary position in the workspace cube.

[0093] It can be understood that the randomly generated 3D sampling position corresponding to the actual arrival point is input into the trained reinforcement learning network, and the corresponding joint position is obtained through the 3D sampling position and the position mapping.

[0094] In step S500 of some embodiments, a sampling image is obtained when the end effector reaches the 3D sampling position, and the sampling image and the corresponding joint position are used as task-independent data of the robotic arm.

[0095] It is understandable that a sampling image is acquired by a camera when the end effector reaches the 3D sampling position, and the sampling image and the corresponding joint position are used as task-independent data of the robotic arm.

[0096] Furthermore, we collect images and joint positions of the robotic arm at each moment to form <image, joint position (i.e., action)> pairs (i.e., task-independent data) as a "task-independent action" dataset. Because this randomized collection method does not depend on the task and is independent of the task, it can be applied to the learning of any robotic arm operation task.

[0097] The sampled image is a 640-degree image stitched from the high-position camera and the wrist cameras of the left and right robotic arms. The high-position camera captures the complete global information of both robotic arms using a 720-pixel resolution image. Joint positions are represented as 14-dimensional vectors. During the experiment, we successfully collected approximately 610,000 image-action pairs, totaling 638 trajectories, in a single run, taking 10 hours.

[0098] This design addresses the inefficient coverage of reachable states, redundant or degenerate motions (for example, the robot moves outside the camera's field of view), and frequent self-collisions that occur in fully randomized approaches. It also resolves the paradox that traditional inverse dynamics calculations for robot bodies cannot calculate joint positions when only the end effector position is given without its orientation, while providing a random orientation renders the inverse kinematics (IK) algorithm unsolvable.

[0099] The proposed method is fully defined by the robot's URDF and kinematic model and does not rely on intrinsic or extrinsic camera parameters. When the visual configuration changes, ATARA (Automated Task-Independent Data Acquisition for Embodied Agents) can regenerate new task-independent trajectories by repeating the workspace sampling process while maintaining the motion constraints of the specific robot. In this case, ATARA (Automated Task-Independent Data Acquisition for Embodied Agents) can provide scalable, collision-free motion sampling in diverse robot configurations. It achieves efficient task-independent data generation, fully preserves the robot's ontology information, and comprehensively covers the accessible workspace, which can be used as reusable prior knowledge for downstream policy learning.

[0100] Finally, ATARA (Automated Task-Independent Data Acquisition for Embodied Agents) will obtain <image, action> pairs (task-independent data) for training the AnyPos model (action prediction model) below.

[0101] Figure 2 This is one of the flow charts of the training method of the action prediction model provided by the present invention. Figure 2 As shown, the method may include but is not limited to steps S210 to S230: S210, inputting the task-independent data into an arm decoupling estimation module, so as to perform separation processing on the sampled image by the arm decoupling estimation module to obtain a first robotic arm image and a second robotic arm image; S220: Input the first robotic arm image and the second robotic arm image into an encoder to obtain pixel coding features; S230, input the pixel point coding features into the direction perception decoder module, output the predicted action of the robotic arm, and calculate the loss function based on the comparison between the predicted action and the actual joint position to obtain a trained action prediction model.

[0102] In step S210 of some embodiments, the task-independent data is input into an arm decoupling estimation module, so that the arm decoupling estimation module performs separation processing on the sampled image to obtain a first robotic arm image and a second robotic arm image.

[0103] It is understandable that task-independent data is input into the arm-decoupled estimation module. Specifically, arm-decoupled estimation (Arm-Decoupled Estimation): Since the images of the left and right robotic arms may have feature confusion (that is, the features of the left robotic arm will affect the prediction of the right robotic arm), thereby seriously reducing the accuracy of the predicted action, the arm-decoupled estimation will process the input image to obtain four decoupled images for subsequent action prediction.

[0104] Specifically, it initializes from the fixed base joint (such as the 0,1 joint of the robot arm, which is usually not affected by the upstream joint linkage and is a fixed pixel area in the high-position camera), and applies the flood fill algorithm (FloodFill) to find adjacent pixel points similar to the fixed base joint, thereby obtaining the pixel connected blocks of the left and right robot arms, separating the left and right arm images, and obtaining the first robot arm image and the second robot arm image.

[0105] Furthermore, if occlusion or overlap occurs, an alternate splitting line is used to ensure a clean separation between the left and right arm images. For example, the left robotic arm uses the left 3 / 5 splitting line, and the right robotic arm uses the left 2 / 5 splitting line. After arm decoupling estimation, the complete image in the dataset is divided into four images: the left robotic arm image, the right robotic arm image, the left gripper wrist camera image, and the right gripper wrist camera image.

[0106] It should be noted that in the high-position camera image, the fixed base joints of the robotic arm (such as joints 0 and 1) are located. These joints correspond to fixed pixel areas (such as the base or shoulder) in the image.

[0107] The pixel coordinates of these joints are recorded as the starting points for subsequent segmentation.

[0108] The specific operation of applying the flood fill algorithm to separate the left and right arms can be: Starting from the pixel coordinates of the fixed base joint, the flood fill algorithm (FloodFill) is applied to expand to the surrounding area to find pixels similar to the base joint (such as similar color and brightness) to form a connected area. Based on the connectivity, the image is divided into pixel blocks of the left and right robotic arms: Left robotic arm (first robotic arm): Flood filling starts from the left base joint and covers the left connected area.

[0109] Right robotic arm (second robotic arm): Flood filling starts from the right base joint and covers the right connected area.

[0110] Check whether there is any occlusion or overlap between the left and right arms (for example, the connected areas are mixed due to the crossing of the arms).

[0111] If occlusion or overlap is detected, use the backup split line: Left robotic arm: Use the vertical dividing line at the left 3 / 5 of the image (e.g., 3 / 5 of the image width) as the boundary.

[0112] Right robotic arm: Use the vertical dividing line at 2 / 5 of the left side of the image (e.g., 2 / 5 of the image width) as the boundary.

[0113] Crop the image along the dividing line, ensuring a clean separation between the left and right arm areas.

[0114] In some embodiments, the separated left robotic arm pixel blocks and right robotic arm pixel blocks are saved as independent images (eg, a left robotic arm image and a right robotic arm image).

[0115] Discard other irrelevant areas (such as the background or the gripper wrist camera area) and only keep the image of the target robot arm.

[0116] This embodiment physically separates the left and right arm images to avoid feature cross-interference and uses alternate segmentation lines to resolve occlusion or overlap issues, ensuring segmentation robustness.

[0117] In step S220 of some embodiments, the first robotic arm image and the second robotic arm image are input into an encoder to obtain pixel coding features.

[0118] It should be noted that each of the left robotic arm image (first robotic arm image), the right robotic arm image (second robotic arm image), the left gripper wrist camera image, and the right gripper wrist camera image is input into the visual encoder of the DINOv2 with register (DINOv2-Reg) model to obtain pixel encoding features.

[0119] The four independent images output by the arm decoupling estimation module are fed into the visual encoder separately: Left robotic arm image (first robotic arm image), right robotic arm image (second robotic arm image), left gripper wrist camera image, right gripper wrist camera image. Each image is processed separately to avoid feature confusion.

[0120] For each input image, the DINOv2 with register (DINOv2-Reg) model is used for feature extraction: The DINOv2-Reg model is based on the Vision Transformer (ViT) architecture, which divides the image into patches of fixed size (such as 16×16 pixels) and converts each patch into a vector through a linear embedding layer.

[0121] Through multi-layer self-attention mechanism and Transformer encoder, global and local feature dependencies are captured.

[0122] The registration mechanism (Register) aligns image features from different perspectives (such as a high-position camera and a wrist camera) to resolve spatial misalignment caused by perspective differences.

[0123] The model outputs pixel-level encoded features for each image (such as a two-dimensional feature map or a three-dimensional feature vector), preserving spatial and semantic information.

[0124] The encoded features of the four images (left robotic arm, right robotic arm, left gripper, right gripper) are stored or marked separately for joint processing by the subsequent decoder.

[0125] This embodiment uses the DINOv2-Reg model to process the high-position camera image (global robotic arm posture) and the wrist camera image (end gripper details) separately, retaining the semantic information of different perspectives.

[0126] Avoid feature confusion or information loss caused by directly stitching multi-view images.

[0127] This embodiment also aligns the features of images from different perspectives through the DINOv2-Reg registration mechanism (Register), for example, spatially aligning the global features of the high-position camera with the local features of the wrist camera to improve feature consistency.

[0128] Avoid feature misalignment caused by viewing angle differences (e.g., looking down vs. looking from the side).

[0129] In step S230 of some embodiments, the pixel point coding features are input into the direction perception decoder module, and the predicted action of the robotic arm is output to calculate the loss function based on the comparison between the predicted action and the actual joint position to obtain a trained action prediction model.

[0130] It will be appreciated that in some embodiments of the present invention, the direction-aware decoder module includes at least multi-scale dilated convolution, deformable convolution, and angle-sensitive pooling components; Inputting the pixel point encoding features into the direction perception decoder module and outputting the predicted action of the robotic arm includes: Inputting the pixel point coded features into the multi-scale dilated convolution with a preset dilation rate for extraction processing, and outputting multi-scale directional features that capture spatial clues of multiple receptive fields; Inputting the multi-scale directional features into the deformable convolution to extract deformation features, and outputting deformation features that capture the motion deformation of the robotic arm during movement; Inputting the deformation feature into the angle-sensitive pooling component so that the deformation feature is rotated in multiple directions and then adaptively averaged pooled to output multiple features at different angles; The multiple different angle features are spliced ​​to obtain a spliced ​​feature, and the spliced ​​feature is mapped to a joint position space to determine a predicted action of the robotic arm, wherein the joint position space is obtained based on the joint position of the task-independent data.

[0131] It is understandable that in some embodiments of the present invention, the Direction Aware Decoder (DINOv2-Reg) visual encoder only extracts pixel-related features. However, what we need are features related to the direction and angle of the robotic arm. Therefore, if we directly use the visual encoder plus the fully connected layer to predict the action, there will be accuracy issues.

[0132] Therefore, in an embodiment of the present invention, a direction-aware decoder is proposed, which introduces direction-related inductive bias into the model structure to capture the angular features and movements of the robotic arm with higher accuracy.

[0133] The features extracted by the DINOv2-Reg visual encoder are fed into the direction-aware decoder, which ultimately outputs the predicted action. Overall, the direction-aware decoder integrates three core components to improve directional sensitivity and spatial generalization, thereby meeting the model's high accuracy requirements (experiments have shown that this implementation uses an empirical accuracy of 0.06, which successfully completes the task at a scale of 3.0 units).

[0134] 1) Multi-scale Dilated Convolution: This method extracts multi-scale directional features through multi-scale dilated convolution with dilation rates of [1, 2, 3, 6]. The input is the pixel-encoded features extracted by the DINOv2-Reg visual encoder, and the output is multi-scale directional features that capture spatial cues from multiple receptive fields. In other words, the output features not only retain the detailed information of the input features but also incorporate multi-scale contextual information from local to global scales.

[0135] 2) Deformable Convolution 2D: This method is used to handle the motion of the robotic arm. It extracts deformation features from the multi-scale directional features obtained in the previous step. Specifically, Deformable Convolution dynamically adjusts the position of the convolution kernel by learning a set of offsets, enabling it to adaptively focus on more informative areas in the feature map. This allows the model to better capture deformations caused by posture changes during motion, thereby improving the understanding and modeling of the robotic arm's state.

[0136] 3) Angle-Sensitive Pooling: The deformation features obtained in the previous step are rotated in multiple directions (for example, [0°, 45°, 90°, 135°]). Then, adaptive average pooling (2D adaptive average pooling) is performed on each rotation. The resulting features at different angles are concatenated to generate multiple features at different angles, capturing features from different directions. Finally, these features at different angles are connected to a fully connected layer (MLP), which maps the concatenated feature vector to the joint position space to output the predicted action.

[0137] In some embodiments, the pixel encoding features output by the DINOv2-Reg visual encoder (such as the feature maps of the left robotic arm, the right robotic arm, the left gripper, and the right gripper) are used as input and passed to the direction perception decoder module.

[0138] The feature map preserves spatial structure information (such as resolution or feature map size), providing a basis for subsequent multi-scale and direction-sensitive operations.

[0139] Furthermore, the pixel-coded features are fed into a multi-scale dilated convolution. Using a convolution kernel with a preset dilation rate (e.g., [1, 2, 3, 6]), the multi-scale receptive field of the feature map is extracted: Dilation rate 1: Captures local details (such as joint edges or textures).

[0140] Expansion rate 2 / 3 / 6: Gradually expand the receptive field to capture global context information (such as the overall posture of the robotic arm).

[0141] Output: "Multi-scale directional features" that fuse multi-scale features, which not only retain the details of the input features but also combine local to global contextual information.

[0142] This step solves the problem that single-scale convolution cannot capture both details and global information at the same time, and enhances the modeling capability of the robot arm's directional sensitivity.

[0143] Furthermore, the multi-scale directional features are fed into the Deformable Convolution 2D. Deformable Convolution dynamically adjusts the position of the convolution kernel by learning a set of offsets, allowing it to adaptively focus on more informative areas in the feature map (such as the joints of a moving robotic arm or deformed areas).

[0144] Output: Capture the "deformation characteristics" of the robot arm during movement to improve the model's robustness to posture changes.

[0145] This step addresses deformation during robotic arm movement and avoids feature misalignment caused by the fixed receptive field of traditional convolution. Dynamically adjusting the convolution kernel position enhances modeling capabilities in key areas, such as joints.

[0146] Furthermore, the deformation features are input into the angle-sensitive pooling component (Angle-Sensitive Pooling).

[0147] The feature map is rotated in multiple preset directions (such as [0°, 45°, 90°, 135°]) to obtain feature representations at different angles.

[0148] Perform 2D Adaptive Average Pooling on each rotated feature map to extract features in different directions.

[0149] Output: multiple feature vectors at different angles (such as features in 4 directions).

[0150] This step captures spatial cues of the robot arm in different directions, resolving directional ambiguity. Multi-angle pooling enhances the model's generalization capabilities to changes in the robot arm's orientation.

[0151] Furthermore, the feature vectors from multiple angles are concatenated to form a comprehensive feature representation.

[0152] A fully connected layer (MLP) maps the concatenated features to the joint position space, outputting the predicted manipulator motion (e.g., 14-dimensional joint position vectors and end-point orientation). This step integrates multi-scale and multi-directional features to generate highly accurate motion predictions. Furthermore, based on the joint position space trained on task-independent data, the motion predictions ensure that they adhere to physical constraints.

[0153] Furthermore, the predicted actions are compared with the ground-truth labeled actions, and a loss function (such as mean squared error (MSE) or smoothed L1 loss) is calculated. The parameters of the direction-aware decoder are optimized through backpropagation, and training is iterated until the model converges.

[0154] Combined with the arm decoupling estimation module, the accuracy can be further improved (such as single arm for single use, dedicated arm for dedicated use).

[0155] This step enables the model to learn the mapping relationship from images to actions through end-to-end training. Combined with arm decoupling estimation, it eliminates feature confusion and meets the 0.06 accuracy threshold requirement.

[0156] Through the above steps, the direction-aware decoder can accurately extract direction-related inductive biases from pixel-level features, and ultimately output robotic arm movements that conform to physical constraints to meet the needs of complex operating tasks.

[0157] Combine Figure 3 and Figure 4 As can be seen, this invention first proposes a new concept: the task-independent action paradigm. Based on this concept, we further propose an automated task-independent data acquisition method (ATARA) for embodied agents. Using this method, we propose a motion prediction model (AnyPos)—a high-precision inverse dynamics motion prediction model trained on task-independent motion data. This invention can achieve at least the following technical effects: (1) Introducing Automated Task-Independent Random Actions (ATARA). This is a scalable data collection pipeline that uses scripted policies to uniformly explore the cubic workspace of a robotic manipulator, enabling efficient and repeatable collection of diverse task-independent action data without human intervention. This significantly reduces the cost and error of traditional human-demonstration-based datasets.

[0158] (2) Introducing AnyPos. This is a robot-specific image-to-action model trained entirely on task-independent trajectories. It improves data efficiency and model generalization by introducing structural priors and architectural biases. This model integrates two key techniques: (1) arm disentanglement estimation: used to isolate the region of each arm in a multi-arm setting and eliminate irrelevant joint dimensions, thereby reducing the effective hypothesis space and improving action prediction accuracy; (2) direction-aware decoder: a decoder that aligns visual features with possible motion directions, which is used to further inject physical priors (such as joint angles and link directions) to improve robustness to noise.

[0159] Compared with the prior art, the present invention adopts an approach that decouples high-level semantic understanding from low-level physical control to address the challenge of anchoring task semantics in action prediction while achieving generalization from task-irrelevant data.

[0160] The present invention adopts an automated task-independent data collection method, which has the following three advantages: (1) The data collection process is simpler and easier to expand, and can be achieved through autonomous exploration or simulation.

[0161] (2) Support cross-task data reuse, enabling general models to be migrated between downstream tasks.

[0162] (3) It reduces human supervision and annotation costs because it does not require success labels or target alignment.

[0163] The present invention proposes an inverse dynamics model training method based on task-independent data, which has the following two advantages: (1) Arm decoupling estimation is applied, based on flood filling and symmetry line segmentation algorithms to isolate the region of each arm and eliminate irrelevant joint dimensions, reducing the effective hypothesis space and improving the motion prediction accuracy by 30%+.

[0164] (2) We designed a direction-aware decoder that aligns visual features with possible motion directions, improving the accuracy of high-precision motion prediction and achieving an additional 20%+ improvement in the prediction accuracy of downstream tasks.

[0165] In summary, the present invention adopts a task-independent action paradigm, which has the following three advantages: (1) High data utilization and reusable motion data: The task-independent training method avoids expensive task-specific demonstrations and enables large-scale unsupervised data collection. The inverse dynamics model learns universal motion priors from task-independent motion data, serving as a shared motion library for diverse tasks.

[0166] (2) Zero-shot task generalization: The inverse dynamics model (IDM) models task-independent motion priors and generalizes to new tasks by only adapting the video generation model (e.g., through language cues) without retraining the IDM.

[0167] (3) Decoupled planning and low-level control: High-level planning (e.g., “opening a drawer”) is handled by the video generation model, while the IDM performs the visual trajectory. This modular approach simplifies policy design by framing the operation as an image-space prediction problem.

[0168] The present invention provides an automated task-independent data collection method for embodied intelligent agents, as well as a training method, apparatus, device, and storage medium for action prediction models. These methods address the shortcomings of existing technologies, such as high data collection costs, low generalization, cumbersome data collection, and difficulty in scalability. These methods simplify and expand the data collection process, enable autonomous exploration or simulation, support cross-task data reuse, enable the migration of general models between downstream tasks, and reduce human oversight and annotation costs.

[0169] The following describes the automated task-independent data collection device for an embodied intelligent body provided by the present invention. The automated task-independent data collection device for an embodied intelligent body described below and the automated task-independent data collection method for an embodied intelligent body described above can be referenced in conjunction with each other. A detailed description thereof will not be repeated here.

[0170] like Figure 5 1 is a schematic diagram of the structure of an automated task-independent data acquisition device for an embodied intelligent body provided by the present invention. The automated task-independent data acquisition device for an embodied intelligent body comprises at least a robotic arm and an end effector located at the end of the robotic arm. The automated task-independent data acquisition device for an embodied intelligent body comprises: A definition module 510 is configured to define a feasible range of 3D positions of the end effector as a bounded workspace cube; A training module 520 is configured to train a reinforcement learning network in a simulation environment using a proximal policy optimization algorithm to predict, based on the trained reinforcement learning network, joint positions when the end effector reaches a target point 3D position in the workspace cube; wherein the proximal policy optimization algorithm is characterized as an algorithm that iteratively collects motion trajectory data of the end effector from an initial position to a target point 3D position based on a reward function, and iteratively updates parameters of the reinforcement learning network based on the motion trajectory data; the reward function is characterized as a function that drives the position reached by the end effector to approach the target point 3D position during each iterative training; A position mapping module 530 is constructed, configured to randomly sample a target point in the workspace cube as the 3D position of the target point, generate a joint position of the manipulator corresponding to the 3D position of the target point using the trained reinforcement learning network, and construct a position mapping between the joint position of the manipulator and the corresponding 3D position of the target point; a random sampling module 540 for driving the end effector of the manipulator to perform random sampling in real space, and recording the 3D sampling position reached by the end effector during the random sampling process, and recording the joint position of the manipulator when the end effector reaches the 3D sampling position, determined based on the 3D sampling position and the position mapping; the 3D sampling position is represented as an arbitrary position in the workspace cube; The task-independent data determination module 550 is configured to obtain a sampling image when the end effector reaches the 3D sampling position, and use the sampling image and the corresponding joint position as task-independent data of the robotic arm.

[0171] The present invention provides an automated task-independent data collection method for embodied intelligent agents, as well as a training method, apparatus, device, and storage medium for action prediction models. These methods address the shortcomings of existing technologies, such as high data collection costs, low generalization, cumbersome data collection, and difficulty in scalability. These methods simplify and expand the data collection process, enable autonomous exploration or simulation, support cross-task data reuse, enable the migration of general models between downstream tasks, and reduce human oversight and annotation costs.

[0172] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call logic instructions in the memory 630 to execute an automated task-independent data collection method or an action prediction model training method for an embodied intelligent body.

[0173] Furthermore, the logic instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0174] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the automated task-independent data collection method for embodied intelligent bodies or the training method of action prediction models provided by the above methods.

[0175] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the automated task-independent data collection method for embodied intelligent bodies or the training method of action prediction models provided by the above-mentioned methods.

[0176] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0177] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for automated task-independent data collection for embodied agents, characterized in that: The embodied intelligent body includes at least a robotic arm and an end effector located at the end of the robotic arm. The automated task-independent data collection method for the embodied intelligent body includes: The feasible range of 3D positions of the end effector is defined as a bounded workspace cube; In a simulation environment, a reinforcement learning network is trained using a proximal policy optimization algorithm to predict, based on the trained reinforcement learning network, joint positions of the end effector when it reaches a target point 3D position in the workspace cube; wherein the proximal policy optimization algorithm is characterized as an algorithm that iteratively collects motion trajectory data of the end effector from an initial position to a target point 3D position based on a reward function, and iteratively updates parameters of the reinforcement learning network based on the motion trajectory data; the reward function is characterized as a function that drives the position reached by the end effector to approach the target point 3D position during each iterative training; Randomly sampling a target point in the workspace cube as the target point 3D position, generating a joint position of the manipulator corresponding to the target point 3D position using the trained reinforcement learning network, and constructing a position mapping between the joint position of the manipulator and the corresponding target point 3D position; In real space, driving the end effector of the robotic arm to perform random sampling, and recording the 3D sampling position reached by the end effector during the random sampling process, and recording the joint position of the robotic arm when the end effector reaches the 3D sampling position based on the 3D sampling position and the position mapping; the 3D sampling position is represented as an arbitrary position in the workspace cube; A sampling image is obtained when the end effector reaches the 3D sampling position, and the sampling image and the corresponding joint position are used as task-independent data of the robotic arm.

2. The automated task-independent data collection method for embodied intelligent agents according to claim 1, characterized in that: The formula of the reward function is: r(a;w)=-||EEF_{pos}(a)-w||^2 Where r(a;w) represents the reward function value, EEF_{pos}(a) represents the 3D sampling position of the end effector in the workspace cube for calculating a given joint position a, and w represents the 3D position of the target point.

3. The automated task-independent data collection method for embodied intelligent agents according to claim 1, wherein: Before the step of driving the end effector of the robotic arm to randomly sample in the real space, the method includes: In the case where the embodied intelligence includes two robotic arms, a virtual boundary plane is set to divide the robotic arm sampling space to obtain a first sampling workspace of the first robotic arm and a second sampling workspace of the second robotic arm; wherein the first sampling workspace and the second sampling workspace have no intersection.

4. The method for automated task-independent data collection for embodied intelligent agents according to claim 1, wherein: In the process of driving the end effector of the robotic arm to randomly sample in the real space, the method further comprises: The joint angles associated with the end direction of the end effector are randomly and uniformly sampled within the range of degrees of freedom to generate action samples of the end effector with various directions performing corresponding actions.

5. A method for training an action prediction model, characterized in that: Training an action prediction model based on task-independent data collected by the automated task-independent data collection method for embodied intelligence according to any one of claims 1 to 4, wherein the action prediction model comprises at least an arm decoupling estimation module and a direction perception decoder module, and the training method of the action prediction model comprises: Inputting the task-independent data into an arm decoupling estimation module, so as to separate and process the sampled image through the arm decoupling estimation module to obtain a first robotic arm image and a second robotic arm image; Inputting the first robotic arm image and the second robotic arm image into an encoder to obtain pixel coding features; The pixel point encoding features are input into the direction perception decoder module, and the predicted action of the robotic arm is output. The loss function is calculated based on the comparison between the predicted action and the actual joint position to obtain a trained action prediction model.

6. The method for training an action prediction model according to claim 5, wherein: The direction-aware decoder module includes at least multi-scale dilated convolution, deformable convolution and angle-sensitive pooling components; Inputting the pixel point coded features into the direction perception decoder module and outputting the predicted action of the robotic arm includes: Inputting the pixel point coded features into the multi-scale dilated convolution with a preset dilation rate for extraction processing, and outputting multi-scale directional features that capture spatial clues of multiple receptive fields; Inputting the multi-scale directional features into the deformable convolution to extract deformation features, and outputting deformation features that capture the motion deformation of the robotic arm during movement; Inputting the deformation feature into the angle-sensitive pooling component so that the deformation feature is rotated in multiple directions and then adaptively averaged pooled to output multiple features at different angles; The multiple different angle features are spliced ​​to obtain a spliced ​​feature, and the spliced ​​feature is mapped to a joint position space to determine a predicted action of the robotic arm, wherein the joint position space is obtained based on the joint position of the task-independent data.

7. An automated task-independent data acquisition device for an embodied intelligent agent, characterized in that: The embodied intelligent body comprises at least a robotic arm and an end effector located at the end of the robotic arm. The automated task-independent data acquisition device for the embodied intelligent body comprises: A definition module, configured to define a feasible range of 3D positions of the end effector as a bounded workspace cube; a training module for training a reinforcement learning network in a simulation environment using a proximal policy optimization algorithm to predict, based on the trained reinforcement learning network, joint positions when the end effector reaches a target point 3D position in the workspace cube; wherein the proximal policy optimization algorithm is characterized as an algorithm for iteratively collecting motion trajectory data of the end effector from an initial position to a target point 3D position based on a reward function, and iteratively updating parameters of the reinforcement learning network based on the motion trajectory data; and the reward function is characterized as a function that drives the position reached by the end effector to approach the target point 3D position during each iterative training; Constructing a position mapping module for randomly sampling a target point in the workspace cube as the 3D position of the target point, generating a joint position of the manipulator corresponding to the 3D position of the target point using the trained reinforcement learning network, and constructing a position mapping between the joint position of the manipulator and the corresponding 3D position of the target point; a random sampling module, configured to drive the end effector of the manipulator to perform random sampling in real space, record a 3D sampling position reached by the end effector during the random sampling process, and record a joint position of the manipulator when the end effector reaches the 3D sampling position, determined based on the 3D sampling position and the position mapping; the 3D sampling position is represented as an arbitrary position in the workspace cube; A task-independent data module is determined, which is used to obtain a sampling image when the end effector reaches the 3D sampling position, and use the sampling image and the corresponding joint position as task-independent data of the robotic arm.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, it implements the automated task-independent data acquisition method for an embodied intelligent body as described in any one of claims 1 to 4 or the training method of the action prediction model as described in claim 5 or 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the automated task-independent data acquisition method for an embodied intelligent body as described in any one of claims 1 to 4 or the training method of an action prediction model as described in claim 5 or 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the automated task-independent data acquisition method for an embodied intelligent body as described in any one of claims 1 to 4 or the training method of an action prediction model as described in claim 5 or 6.

Images