Body intelligence data synthesis and labeling method, device, medium and product

Abstract

This application relates to the field of information technology and discloses a method, device, medium, and product for synthesizing and annotating embodied intelligence data. The method includes: receiving an embodied intelligence task described in natural language, as well as an initial image and a depth image; decomposing the task into a semantic sequence of task stages with action type labels; performing a perturbation transformation on the initial image to generate a set of scene images with appearance differences from the initial image; generating a color video corresponding to the semantic sequence of the task stages based on the semantic sequence of the task stages and the set of scene images; performing frame-by-frame monocular depth estimation on the color video to generate a depth video aligned with the color video; deriving robot control trajectory data corresponding to the temporal sequence of the video through inverse dynamics; mapping the semantic sequence of the task stages to the timeline of the color video to generate a structured semantic annotation file; and outputting the above data; thereby generating embodied intelligence data with high diversity and physical and temporal consistency.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a method, device, medium and product for embodied intelligent data synthesis and annotation. Background Technology

[0002] Embodied AI is a cutting-edge field in artificial intelligence, aiming to endow intelligent agents with the ability to perceive, understand, and perform complex tasks in real or simulated environments. Training such agents demands extremely high-quality and large-scale data, primarily requiring massive, multimodal demonstration data, typically including visual information (RGB), language commands, and precise robot motion trajectories. However, the primary challenge currently facing technology is the high cost of data acquisition and annotation, especially when precise spatial motion localization, interaction phase segmentation, and fine-grained semantic annotation are required; traditional manual methods are inefficient and difficult to scale.

[0003] However, existing embodied intelligence data primarily relies on real-world data collection or traditional simulation generation. Real-world data collection is costly, inefficient, and struggles to cover diverse scenarios and extreme situations. While traditional simulators can generate large-scale data, the videos they produce often lack realism and diversity, and the consistency and physical reproducibility of control trajectory data with video temporal sequences are difficult to guarantee. This results in a significant "inter-domain gap" for intelligent agents when migrating from simulation to reality.

[0004] While existing methods for generating embodied data can generate motion videos and trajectories based on language or images, they still have the following shortcomings:

[0005] First, there are limitations imposed by the data modality. Most generation methods rely solely on RGB video as input or output, resulting in a lack of perception and constraint on scene depth information and 3D structure in the generated motion. Due to the lack of accurate spatial information, the motion trajectories derived from the model are often inaccurate in the actual physical world, making it difficult to guarantee the reliability of the operation.

[0006] Secondly, there is the issue of scene and environment uniformity. Existing methods, when generating videos and trajectories, are typically limited to a single or a few environmental samples, making it difficult to effectively cover the complex and varied lighting conditions and background changes in the real world. This lack of scene diversity severely limits the generalization ability and robustness of trained models when facing unknown environments.

[0007] Secondly, there is a lack of automated annotation. Existing data generation processes typically use video and control trajectories as the main outputs, lacking a mechanism for automatically and structurally aligning high-level task semantic sequences (such as task stages and action types) with the underlying timeline and control signals. This prevents the data from being directly used for training advanced embodied intelligence models that require fine semantic structures.

[0008] Finally, there is insufficient physical and temporal consistency. This is a critical flaw. Regardless of whether the data is obtained through simulation or generative models, there are often temporal discrepancies or physical mismatches between the output video and the control commands derived through inverse dynamics. This inconsistency makes the trajectory data lack physical reproducibility, making it difficult to use directly for high-performance simulations or robot control, and greatly limiting the practical value of the data.

[0009] Therefore, the industry urgently needs a new and efficient method that can utilize multimodal inputs to generate structured embodied intelligence data with high diversity, precise spatial constraints, and strict physical and temporal consistency between video and motion trajectories. Summary of the Invention

[0010] One objective of this application is to provide a method, apparatus, medium, and product for embodied intelligence data synthesis and annotation, at least to address the technical shortcomings of existing embodied intelligence data synthesis methods, such as single input modality, lack of diverse data scenarios, lack of automatic semantic annotation, and inconsistencies between video and action timing / physical sequences. To achieve the above objective, some embodiments of this application provide the following aspects:

[0011] This application provides a method for synthesizing and annotating embodied intelligence data, the method comprising:

[0012] The system receives an embodied intelligence task described in natural language, along with the corresponding initial and depth images. The first major model decomposes the embodied intelligence task into a semantic sequence of task stages with action type labels.

[0013] The initial image is perturbed and transformed, and a set of scene images with appearance differences from the initial image is generated by the second large model;

[0014] Based on the semantic sequence of the task stage and the set of scene images, a color video corresponding to the semantic sequence of the task stage is generated through the third major model;

[0015] Perform frame-by-frame monocular depth estimation on the color video to generate a depth video aligned with the color video;

[0016] By combining the color video and the corresponding depth video, the robot control trajectory data corresponding to the video time sequence is derived through inverse dynamics;

[0017] Based on the duration of each segment in the color video, the semantic sequence of the task stage is mapped to the timeline of the color video to generate a structured semantic annotation file;

[0018] Output the color video, the depth video, the robot control trajectory data, and the structured semantic annotation file.

[0019] Secondly, some embodiments of this application also provide an electronic device, the electronic device comprising: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method described above.

[0020] Thirdly, some embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the method described above.

[0021] Fourthly, some embodiments of this application also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0022] Compared with related technologies, the solution provided in this application significantly enhances the diversity and generalization ability of data by introducing a multi-model collaborative mechanism in the early stage of data generation. On the one hand, by utilizing the strong constraint semantic structuring capability of the first major model, complex natural language tasks are precisely decomposed into executable stage sequences, providing precise action guidance and a structured foundation for the entire data generation process. On the other hand, by adjusting the constrained appearance attributes of the initial scene images, a large-scale set of scenes is generated that maintains the original three-dimensional structure but has highly diverse appearances. This fundamentally solves the problem of single data scenes in traditional methods, ensuring that the trained model has strong environmental generalization ability. Furthermore, by using depth video that is strictly aligned with color video, robot control trajectory data corresponding to the video temporal sequence is derived through inverse dynamics, thus automatically outputting a complete four-modal data package containing color video, depth video, high-fidelity control trajectory, and structured semantic annotation files. This data achieves extremely high quality standards in terms of structure, physics, and temporal sequence, realizing a closed-loop and highly efficient embodied data generation process, greatly reducing the cost of manual collection and fine annotation, and providing key technical support for promoting the development of embodied intelligent models. Attached Figure Description

[0023] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0024] Figure 1 A flowchart illustrating an embodied intelligent data synthesis and annotation method provided as an exemplary embodiment of this disclosure;

[0025] Figure 2 An exemplary structural diagram of the electronic device provided for some embodiments of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] Figure 1 A flowchart of an embodied intelligence data synthesis and annotation method provided for an exemplary embodiment of this disclosure, the method comprising:

[0028] S101. Receive the embodied intelligence task described in natural language, as well as the corresponding initial image and depth image, and decompose the embodied intelligence task into a semantic sequence of task stages with action type labels through the first major model.

[0029] Specifically, the purpose of this step is to decompose complex embodied tasks described in natural language (e.g., "put the red water glass in the cupboard") into 2 to 6 executable stages, forming a structured semantic sequence. A primary model (e.g., Qwen2.5-72B) can be used as an embodied task planning assistant. After receiving the task description, this model can break it down into 2–6 executable stages based on the given Chinese task description, ensuring the continuity of the stages. The goal of this decomposition is to enable the robot or video generator to execute the task step by step, with each stage representing a clear action or phased objective. For example, "put the red water glass in the cupboard" can be broken down into: approaching the red water glass → grabbing the red water glass → moving the water glass to the front of the cupboard → placing the water glass in the cupboard.

[0030] S102. The initial image is perturbed and transformed, and a set of scene images with appearance differences from the initial image is generated by the second large model.

[0031] Specifically, this step involves perturbating the initial image (RGB image) to generate a set of enhanced scene images that differ in appearance from the original image. A second major model (such as a diffusion model like Stable Diffusion) can be used. By perturbing the image's lighting, color distribution, and environmental background with prompts, the core objective is to significantly increase the visual diversity of the dataset and enhance the model's generalization robustness while preserving the geometric structure and spatial relationships of objects in the initial scene.

[0032] S103. Based on the semantic sequence of the task stage and the set of scene images, a color video corresponding to the semantic sequence of the task stage is generated through the third model.

[0033] Specifically, this step, based on the semantic sequence of S101 and the scene image set of S102, uses a third major model to generate a color video corresponding to the task semantics. The third major model can be generated using a video world model (such as COSMOS), with the semantic sequence of the task stage as semantic constraints and the initial image as the starting point for appearance, independently generating a corresponding color video for each initial image.

[0034] S104. Perform frame-by-frame monocular depth estimation on the color video to generate a depth video aligned with the color video.

[0035] Specifically, this step is used to perform frame-by-frame depth prediction on the color video generated in S103, generating a depth video that is strictly aligned with the color video. A monocular depth estimation-based model (such as Depth Anything) can be used. The key is realism restoration: First, using the real depth maps corresponding to the predicted depth video and the initial image, pixels in the active object region are extracted as an alignment reference; then, a linear mapping relationship between the predicted depth and the real depth on the statistical characteristics (such as mean / variance) of the reference region is calculated; finally, this mapping relationship is applied to the entire predicted depth video to generate a depth video with true scale.

[0036] S105. Combining the color video and the corresponding depth video, the robot control trajectory data corresponding to the video time sequence is derived by inverse dynamics.

[0037] Specifically, by combining the color video from S103 and the depth video from S104, the robot control trajectory data corresponding to the video time sequence is derived using the inverse dynamics method. First, the color video and depth video are processed to accurately obtain the attitude changes of the target object and the end effector in the time dimension. The attitude changes are then input into the inverse dynamics network (IDM-A) for control parameter inference, thereby obtaining the robot control trajectory data corresponding to the video time sequence.

[0038] S106. Based on the duration of each stage segment in the color video, map the semantic sequence of the task stage to the timeline of the color video to generate a structured semantic annotation file.

[0039] Specifically, this step automatically aligns the stage semantic information obtained in step S101 with the video timeline generated in step S103 to form the final semantic annotation file. Since the first large model (e.g., Qwen2.5-72B) has already output skill sequence text containing clear stage and action descriptions in step S101, this step does not need to regenerate semantic content. Instead, it achieves accurate correspondence between semantics and video frames through time mapping and consistency correction. First, based on the actual number of frames or duration of each stage of the video, the start and end frame intervals [start_frame, end_frame] of each stage are calculated. Then, the stage names, descriptions, etc., are merged and integrated to generate a unified structured annotation.

[0040] S107. Output the color video, the depth video, the robot control trajectory data, and the structured semantic annotation file.

[0041] Specifically, the above data will be output in the final output, including: a color video of the task execution (MP4 format); a depth video strictly aligned with the color video (generated frame by frame by DepthAnything and saved in MKV format); inverse dynamics trajectory data (timestamps, joint / end sequence, etc., saved in .h5 format); and structured semantic annotations (actions / stages, bilingual descriptions in Chinese and English, saved in JSON format).

[0042] In this embodiment, by introducing a multi-model collaborative mechanism in the early stages of data generation, the diversity and generalization ability of the data are greatly enhanced. On the one hand, by utilizing the strong constraint semantic structuring capability of the first major model, complex natural language tasks are precisely decomposed into executable stage sequences, providing precise action guidance and a structured foundation for the entire data generation process. On the other hand, by adjusting the constrained appearance attributes of the initial scene images, a large-scale set of scenes is generated that maintains the original 3D structure but has highly diverse appearances. This fundamentally solves the problem of single data scenes in traditional methods, ensuring that the trained model has strong environmental generalization ability. Then, through depth video strictly aligned with color video, robot control trajectory data corresponding to the video temporal sequence is derived through inverse dynamics, thus ultimately enabling the automated output of a data package containing color video, depth video, high-fidelity control trajectory, and structured semantic annotation files. This data achieves extremely high quality standards in terms of structure, physics, and temporal sequence, realizing a closed-loop and highly efficient embodied data generation process, greatly reducing the cost of manual collection and fine annotation, and providing key technical support for promoting the development of embodied intelligent models.

[0043] In one embodiment, the step of decomposing the embodied intelligence task into a semantic sequence of task stages labeled with action types using the first major model specifically includes:

[0044] The first major model, based on the natural language description, decomposes the task into at least two executable stages;

[0045] The semantic sequence of task stages is composed of the decomposed executable stages arranged in chronological order;

[0046] The semantic sequence of the task phase is a strongly constrained structured format, which includes action type labels, native language instructions, and target language instructions.

[0047] Specifically, this embodiment aims to transform user-input natural language task descriptions (e.g., "Please move the green object on the table to the basket on the right") into strongly constrained structured data that can be recognized by machines and utilized by subsequent models. The first large model first decomposes the complex, continuous task into a series of logically clear and sequentially executable stages based on its understanding and reasoning of the natural language description. These decomposed stages, such as [reach_green_object], [move_to_basket], and [release], are arranged according to the temporal order of the task, collectively forming a semantic sequence of task stages.

[0048] The semantic sequence of task phases is constrained to a strongly constrained structured format (e.g., JSON or XML). This structured format ensures data rigor and the reliability of automated processing. In this structured format, each phase element contains at least three key pieces of information:

[0049] Action type label (skill): Selected from a predefined set of constraints (e.g., [reach, move, place, release]) to guide subsequent video generation models and trajectory analysis.

[0050] Native Language Instructions (instruction_zh): Task phase descriptions in the original language or the user's native language.

[0051] Target language instruction (instruction_en): The translated target language (usually English) task phase description.

[0052] Furthermore, in one embodiment, the task phase semantic sequence outputs only a JSON array and must not contain any explanations, comments, or additional text.

[0053] Each array element is a dictionary containing the following three fields:

[0054] "skill": Stage name, taken from the preset action type tag set;

[0055] "instruction_zh": Chinese step-by-step instructions for this stage;

[0056] "instruction_en": English instructions equivalent to Chinese content.

[0057] All fields above must be non-empty strings, and the output must be in strictly valid JSON format.

[0058] Furthermore, in one embodiment, the first major model decomposition is restricted to maintain task semantic consistency, and must not change the original task's object or target location, nor create new objects or new actions.

[0059] Furthermore, in one embodiment, the skill for each stage must be selected from the following set:

[0060] ["reach","grasp","move","place","align","insert","release","pick"].

[0061] For example: Input task: Put the red water glass in the cupboard.

[0062] Output example: [

[0064] {"skill":"reach","instruction_zh":"Approach the red cup", "instruction_en":"approach the red cup"}

[0065] {"skill":"grasp","instruction_zh":"Grasp the red cup", "instruction_en":"grasp the red cup"}

[0066] {"skill":"move","instruction_zh":"Move the cup to the cabinet", "instruction_en":"move the cup to the cabinet"}

[0067] {"skill":"place","instruction_zh":"Place the cup into the cabinet", "instruction_en":"Place the cup into the cabinet"} ]

[0069] In the above embodiments, by limiting the first major model to decompose the task based on a strongly constrained structured format, efficient, automated, and high-precision semantic annotation of embodied intelligence tasks is achieved. By decomposing the task into multiple stages arranged in a time sequence and forcing the output to be in a strict JSON format containing action type labels (skills), native language instructions, and target language instructions, the problems of high cost and low structure in semantic annotation in traditional methods are completely solved. In particular, the strict restrictions on the output format (e.g., only outputting JSON arrays, with non-empty fields) and the constraints on the semantic consistency of the task ensure that the semantic sequences relied upon by subsequent video generation and trajectory analysis have extremely high rigor, accuracy, and multilingual availability, significantly improving the automated processing efficiency of data packets and the training value of the semantic layer.

[0070] In one embodiment, the step of perturbating the initial image and generating a set of scene images with appearance differences from the initial image using a second large model specifically includes:

[0071] The second large model uses positive prompts to perform perturbation transformations on the lighting and color distribution of the initial image.

[0072] In one embodiment, negative prompts are used to ensure that the set of scene images generated by the second large model is consistent with the geometric structure and semantic information of the initial image.

[0073] Specifically, this step enhances the lighting and color of the task scene image without altering the original image's geometry (including object category, location, shape, size, and spatial relationships), thereby generating a set of scene images with slightly different appearances. This enhancement is used to improve the robustness of subsequent models to environmental changes. The enhancement process is based on a diffusion-based generative model (Stable Diffusion image-to-image mode img2img).

[0074] The generation direction of the second major model can be controlled by positive and negative prompts:

[0075] Positive prompts can be limited to adjusting only lighting, shadows, exposure, and color tone, while maintaining consistency with the original scene's composition, style, and geometry;

[0076] Negative hints can be used to suppress object deformation, relocation, style transfer, and material changes, preventing unintended geometric or semantic shifts.

[0077] By combining the above tips, natural lighting variations and color disturbances can be achieved while keeping the object and spatial layout unchanged. For example, each image remains consistent with the initial image in terms of object shape, position, and composition, with natural variations only in lighting, shadows, and color distribution.

[0078] In one embodiment, the generated set of scene images is verified by structural similarity and keypoint drift detection to ensure geometric consistency; if the offset exceeds the limit, it is automatically backtracked or resampled.

[0079] Specifically, the purpose of this embodiment is to ensure that although the scene image set differs from the initial image in appearance attributes (such as lighting and color), it remains precisely consistent in geometric structure and semantic information, preventing unwanted object deformation or positional shifts during the generation process. The verification process is as follows:

[0080] For each image in the generated scene image set, structural similarity (SSIM) verification and keypoint drift detection are performed. SSIM verification evaluates the overall similarity between the generated image and the initial image in terms of brightness, contrast, and structure. Keypoint drift detection identifies key feature points (such as object edges, corners, etc.) in the image and accurately calculates the spatial displacement (i.e., "drift") of these key points between the generated image and the initial image. The detection results are compared with a preset geometric consistency threshold. If the detected offset, especially the keypoint drift, exceeds the threshold limit, the image is determined to be non-compliant. In this case, an automatic backtracking or resampling mechanism is triggered, i.e., the non-compliant image is discarded, and the second large model is instructed to regenerate the image using a new random seed or slightly adjusting the prompt information, until the generated scene image set passes verification, ensuring that its geometric structure is precisely consistent with the initial image.

[0081] In the above embodiments, by introducing positive and negative prompts and verification mechanisms for structural similarity and keypoint drift detection, it is ensured that the enhanced scene image set obtained through the generative model, while possessing rich lighting and color variations, strictly maintains its original geometric structure and object spatial position. This effectively solves the problem of unexpected geometric deformation or positional shifts that are easily introduced when performing appearance perturbations in generative techniques such as diffusion models. By automatically detecting, backtracking, or resampling unqualified images, the geometric consistency and reliability of 3D constraints of all scene images are guaranteed, providing a high-quality and robust visual foundation for subsequent video generation and accurate robot pose extraction, thus ensuring the physical accuracy of the entire data packet from the source.

[0082] In one embodiment, the step of generating a color video corresponding to the task stage semantic sequence using a third model based on the task stage semantic sequence and the scene image set specifically includes:

[0083] The color video is composed of sequentially spliced ​​stage video segments, and the number of stage video segments is consistent with the number of stages in the semantic sequence of the task stages.

[0084] The third major model uses the semantic sequence of the task stage as the generation constraint and the image in the scene image set as the starting frame of the first stage video segment; subsequent stage video segments use the last frame of the previous stage video segment as their own starting frame to generate the color video.

[0085] Specifically, this embodiment aims to generate a complete and coherent long-time-series color video that precisely follows the task execution flow decomposed by the first major model. The color video is composed of multiple independent stage video segments sequentially spliced ​​together, and the number of stage video segments is strictly consistent with the number of stages decomposed from the semantic sequence of the task stages.

[0086] The generation process employs a third major model, where the semantic sequence of the task phase is used as the core generation constraint to ensure accurate matching between video content and semantic instructions. For example, the Video World Model (COSMOS) can be used for generation, with instruction_zh and instruction_en in the semantic sequence of the task phase as semantic constraints, and the initial image as the starting point for appearance.

[0087] The coherence of the video sequence is ensured by the following segment connection logic:

[0088] The starting frame of the first stage: When generating the video segment of the first stage, the third model uses an image from the set of scene images generated by the second model as the starting frame. This ensures that the starting scene of the video has the high diversity given in step S102.

[0089] Starting frame for subsequent stages: When generating video segments for subsequent stages, the model uses the final frame of the video segment from the previous stage as its starting frame.

[0090] In this way, the result of the action in the previous stage is directly used as the starting state for the next stage, ensuring a high degree of smoothness and coherence in the content, lighting, object position and action timing of the entire video sequence, effectively avoiding the abrupt jump phenomenon common in traditional segmented video generation.

[0091] Furthermore, in one embodiment, optical flow alignment and color matching are used to maintain a natural transition between consecutive frames, and a flow-guided crossfade is used at the boundaries of segments to smoothly connect them and avoid abrupt changes in the image.

[0092] Furthermore, in one embodiment, the negative cues consistent with step S102 are used throughout the color video generation process to suppress geometric and style drift, ensuring that objects, scenes, and actions remain visually and semantically consistent.

[0093] Specifically, to ensure that the complete color video, composed of multiple video segments, has high temporal smoothness and visual consistency, this embodiment introduces precise post-processing and continuous generation constraints:

[0094] First, regarding segment transition processing, this embodiment employs multiple techniques to maintain the naturalness of the transition between consecutive frames. Specifically, it utilizes optical flow alignment technology to analyze the pixel movement trends at the boundaries of adjacent segments and combines this with a color matching algorithm to eliminate tonal abrupt changes that may be caused by the randomness of the generation model. Furthermore, at segment boundaries, the system employs a flow-guided crossfade method, using optical flow information to guide the direction and intensity of the crossfade, thereby achieving a highly smooth and natural transition and avoiding screen jumps or flickering.

[0095] Secondly, regarding consistency control during the generation process, this embodiment applies the negative cue information used in step S102 throughout the entire color video generation process. The negative cue continuously acts on the third major model, aiming to suppress geometric and style drift. This continuous constraint ensures that objects, backgrounds, and the overall visual style in the scene maintain a high degree of visual and semantic consistency with the initial image and the enhanced scene set throughout the entire video sequence, preventing object deformation, relocation, or background distortion common in long-term generation.

[0096] In the above embodiments, through the above mechanism, this embodiment ensures that the final output of complete color video not only conforms to semantic instructions in terms of content, but also meets the quality standards that can be used for high-precision trajectory reverse engineering in terms of vision and timing.

[0097] In one embodiment, the step of combining the color video and the corresponding depth video to deduce the robot control trajectory data corresponding to the video time sequence through inverse dynamics specifically includes:

[0098] Based on the color video and depth video, the pose vectors of the target object and the end effector in each frame of the video are obtained to form a six-dimensional pose sequence.

[0099] Using an inverse dynamics network with the attitude vectors of adjacent frames in the six-dimensional attitude sequence as input, the attitude difference components are calculated and features are extracted to predict the control increment parameters that describe the robot's action command at that time step.

[0100] The control increment parameters are integrated with the corresponding timestamps and end-effector pose data to form the robot control trajectory data.

[0101] Specifically, firstly, multimodal processing is performed on the color video and the corresponding depth video. Through methods such as visual feature recognition and localization, and 3D reconstruction combined with depth information, the six-dimensional pose vectors of the target object and end effector (e.g., a robotic arm) involved in the task are accurately obtained for each frame of the video. These pose vectors are then arranged in chronological order to form a full-time continuous six-dimensional pose sequence s containing all frames. t =[x,y,z,roll,pitch,yaw]. Here, x, y, and z represent the spatial position. These three parameters define the absolute three-dimensional coordinates of the object in a global coordinate system (e.g., the robot's base or world coordinate system). The x-axis, y-axis, and z-axis represent the object's displacement along these three orthogonal directions, precisely determining the object's position in space. roll, pitch, and yaw represent the spatial attitude. These three parameters, usually expressed as Euler angles, define the object's rotation angles around its own coordinate axes, i.e., its orientation. Roll is typically the rotation around the x-axis, pitch is the rotation around the y-axis, and yaw is the rotation around the z-axis. These three angles combined completely describe all possible orientation states of the object in space.

[0102] After obtaining the complete attitude time series, it is input into the Inverse Dynamics-A (IDM-A) network for control inference. The IDM-A model uses the attitude vectors of adjacent frames ( As input, the difference components of attitude change are first calculated. Motion features are extracted through an encoding network; then, incremental control parameters are predicted using a multilayer perceptron. This is used to describe the robot's action commands at that time step, such as the change in joint angle Δq or the increment of end-effector pose Δx.

[0103] Finally, the predicted control increment parameters, along with the corresponding timestamp information and the original extracted end-effector pose data, are integrated and encapsulated to form the robot control trajectory data. This trajectory data serves as the input basis for subsequent physical calibration and post-processing.

[0104] Furthermore, in one embodiment, after acquiring the robot control trajectory data, the method further includes performing consistency optimization on the robot control trajectory data, wherein the consistency optimization specifically includes:

[0105] The control parameters predicted by the inverse dynamics are input into the differentiable physics simulation engine to generate a simulation video. By calculating the difference between the simulation video and the real video, the trajectory optimization error is determined, and the optimization error is backpropagated to the inverse dynamics network to correct the control parameters.

[0106] Specifically, this embodiment introduces a bidirectional consistency optimization mechanism based on the Brax differentiable physics engine and optical flow estimation to perform closed-loop calibration on the robot control trajectory data predicted by the inverse dynamics network (IDM-A).

[0107] In the optimization process, the individual action command predicted by the IDM-A at each time step is called a control increment parameter. All control increment parameters continuously predicted by the IDM-A over the entire task timeline are arranged chronologically to form a sequence of control signals used as input to the simulation engine. This mechanism aims to ensure that the final trajectory data meets strict physical feasibility and temporal consistency requirements.

[0108] This embodiment employs forward consistency optimization, inputting the control signal sequence output by IDM-A into the Brax simulation environment of the differentiable physics engine for execution. The simulation engine rigorously calculates the motion of each joint of the robot, the end effector trajectory, and the corresponding physical feedback according to the control signals, generating virtual simulation video frames. The simulated video is compared frame by frame with real video (including color and depth video), and the consistency between the simulated and real actions is measured by calculating pixel-level differences and structural similarity assessment (SSIM). If the deviation exceeds a preset threshold, the error is backpropagated to the IDM-A network using the differentiable nature of Brax. This mechanism automatically corrects the network parameters to ensure that the control signals output by IDM-A in the next inference can drive the simulation results to more accurately match the real video actions. For example, if the position of the robotic arm grasping the water cup in the simulation deviates from the real video by 3cm, the system will adjust the joint angular velocity components in reverse for that stage until the grasping position in the simulation coincides with the video image, thus ensuring the physical feasibility of the trajectory.

[0109] In the above embodiments, by introducing a bidirectional consistency optimization mechanism based on a differentiable physics engine and optical flow estimation, the problem of physical and temporal inconsistencies between video observations and control signals in existing trajectory derivation methods is fundamentally solved. Forward optimization utilizes the differentiable physics engine to perform closed-loop verification and reverse correction of the control signals, ensuring the physical feasibility and dynamic constraints of the final trajectory. Reverse optimization, through optical flow analysis, transforms visually observed motion into control signals, guaranteeing strict alignment between the trajectory and the video in amplitude and timing. This bidirectional calibration mechanism eliminates accumulated errors and uncertainties in the inverse dynamics reasoning process, resulting in highly physical reproducibility and practical executability of the final robot control trajectory data, thereby significantly enhancing the training value of the data.

[0110] Furthermore, in one embodiment, the pixel motion between adjacent frames of the video is analyzed based on optical flow estimation, and combined with depth information, it is converted into spatial motion quantities to form a visual back-inference control signal; the inverse dynamics prediction control parameters are corrected by comparing the difference between the visual back-inference control signal and the inverse dynamics prediction control parameters.

[0111] Specifically, this embodiment employs reverse consistency optimization, which focuses on ensuring alignment of timing and amplitude. Based on optical flow estimation algorithms, it analyzes pixel motion between adjacent video frames, extracts the displacement and orientation changes of the end effector in pixel space, and uses depth information to convert these into three-dimensional spatial motion quantities, forming a visual reverse control signal. This visual reverse control signal is then numerically compared and directionally corrected with the control increment parameters output by the IDM-A, aiming to maintain a high degree of consistency between the two in amplitude, orientation, and timing. For example, when optical flow estimation shows that the robotic arm end effector moves forward 5mm within 0.1s, while the IDM-A predicts an end effector displacement of 7mm, the system will automatically adjust the network output or scaling factor to ensure consistency in subsequent iterations, guaranteeing complete alignment between the control trajectory and the motion rhythm observed in the video.

[0112] Furthermore, in one embodiment, depth data is used to assist in outlier culling and accurate spatial location recovery.

[0113] Specifically, the depth video is used to assist the state estimation module in performing high-precision tracking of target objects and end effectors involved in the task. Specifically, the application of depth data is reflected in the following two key aspects:

[0114] Outlier Removal: When performing feature point matching and tracking based on color video, erroneous matching points are inevitable. Depth information provides a reliable 3D geometric constraint, allowing for the identification and removal of unreasonable 2D feature points or incorrect 3D reprojection points by using the difference between the depth value corresponding to an outlier and its surrounding points. This significantly improves the robustness and accuracy of pose estimation.

[0115] Precise spatial location recovery: Combining depth information with camera intrinsic parameters is key to achieving accurate backprojection from 2D image coordinates to 3D spatial coordinates. By using scale-calibrated real depth data, the recovered pose information can be ensured to have extremely high 3D spatial accuracy, thus effectively solving the problem of accumulated errors caused by relying solely on vision or single-modal estimation.

[0116] Through the above application, this embodiment ensures that the final six-dimensional attitude sequence is stable, continuous and highly accurate.

[0117] Furthermore, in one embodiment, the generated robot control trajectory data undergoes velocity, acceleration, and torque limiting processing to ensure that the signal is physically feasible within the robot's execution range;

[0118] Furthermore, in one embodiment, the trajectory is smoothed using spline curves or quadratic programming algorithms to ensure continuous, jitter-free motion that conforms to dynamic constraints.

[0119] Specifically, the robot control trajectory data (including parameters such as position, velocity, acceleration, and torque) needs to undergo velocity limiting, acceleration limiting, and torque limiting processing. This processing ensures that the amplitude of the trajectory signal does not exceed the maximum execution range allowed by the actual robot system hardware and dynamic constraints, thereby guaranteeing the physical feasibility of the signal within the robot's execution environment.

[0120] After amplitude limiting, the robot control trajectory data needs further smoothing using spline curves or quadratic programming algorithms. This smoothing process eliminates high-frequency noise and jitter that may occur during inverse dynamics reasoning and optimization, ensuring that the final robot trajectory exhibits continuous, jitter-free motion in time and conforms to the robot's dynamic constraints.

[0121] In this embodiment, a rigorous quality assurance mechanism is established at the end of the trajectory data generation process. By utilizing depth information for outlier removal and precise spatial position recovery during pose extraction, this method significantly improves the three-dimensional spatial accuracy and stability of the six-dimensional pose sequence, overcoming the cumulative error problem easily introduced by relying solely on color vision, and providing a reliable input foundation for high-precision inverse dynamics derivation. Simultaneously, the generated control trajectory undergoes velocity / acceleration / torque limiting and spline curve / quadratic programming smoothing, fundamentally ensuring that the trajectory data conforms to the robot's physical constraints in terms of signal amplitude and is continuous and jitter-free in terms of motion. This combined mechanism guarantees that the final output control trajectory data has extremely high physical reproducibility and practical executability, and can be directly used to drive real robots or high-performance simulators.

[0122] In one embodiment, the step of obtaining the pose vectors of the target object and the end effector in each frame of the video based on the color video and the depth video to form a six-dimensional pose sequence specifically includes:

[0123] The two-dimensional image coordinates of the target object and the end effector are determined in the color video using a visual feature recognition model.

[0124] By combining the corresponding depth video and camera model parameters, the two-dimensional image coordinates are converted into three-dimensional spatial position information;

[0125] The relative kinematic parameters of the object and the end effector are calculated by feature correlation between time frames.

[0126] The relative kinematic parameters are temporally smoothed to obtain the six-dimensional attitude sequence.

[0127] Specifically, a visual feature recognition model (e.g., object detection and keypoint recognition model) is used to analyze the color video frame by frame to determine the two-dimensional image coordinates (i.e., pixel positions) of the target object and the end effector in each frame. Then, combining the depth video corresponding to the color video with pre-calibrated camera model parameters (e.g., camera intrinsics), the two-dimensional image coordinates are back-projected to transform the two-dimensional pixel positions into three-dimensional spatial position information. This process realizes the transformation from two-dimensional image observation to three-dimensional coordinates with real-scale constraints. After obtaining continuous three-dimensional spatial position information, temporal frame-to-frame feature association techniques (e.g., consecutive frame matching or feature point tracking) are used to analyze the motion of the object and the actuator. By comparing and calculating the motion states between adjacent frames, the relative kinematic parameters of the object and the end effector, i.e., all six degrees of freedom parameters including displacement and rotation, are deduced.

[0128] Finally, the calculated relative kinematic parameters are time-series smoothed (e.g., by incorporating inertial constraints or employing filtering algorithms). This smoothing process aims to eliminate measurement noise and transient jitter during the tracking process.

[0129] In this embodiment, the above steps ensure that the final obtained six-dimensional pose sequence is continuous, stable and highly accurate, and guarantee that the pose data extracted from the video can be used as reliable and accurate kinematic input for subsequent inverse dynamics network inference.

[0130] In one embodiment, the step of performing frame-by-frame monocular depth estimation on the color video to generate a depth video aligned with the color video specifically includes:

[0131] Perform frame-by-frame monocular depth estimation on the color video to obtain the predicted depth video;

[0132] From the depth image corresponding to the initial image, extract the pixels corresponding to the active object region as the depth reference;

[0133] Based on the statistical characteristics of the predicted depth video and the depth benchmark in the region of the moving object, a linear scale mapping relationship is calculated.

[0134] The linear scale mapping relationship is applied to the predicted depth video to generate a depth video aligned with the color video.

[0135] Specifically, this embodiment aims to generate depth information that is strictly aligned temporally and spatially with the color video and possesses a real-world scale. Frame-by-frame monocular depth estimation (e.g., using the DepthAnything model) is performed on the color video to obtain a predicted depth video with only relative depth information. To convert the relative depth to a real-world scale, pixels corresponding to task-related moving object regions (such as the regions where the target object and end effector are located) are first extracted from the real depth image corresponding to the initial image, and the real depth values ​​of these pixels are used as a depth reference. Based on the depth value distribution of moving object regions in the predicted depth video and the statistical characteristics (e.g., mean, variance, median, etc.) of the depth reference, a linear scale mapping relationship (i.e., a scaling factor α and an offset β) is calculated. This mapping relationship reflects a fixed proportional deviation between the predicted depth and the real depth. The calculated linear scale mapping relationship is applied to all pixel values ​​of the predicted depth video.

[0136] In this embodiment, this global linear calibration operation ultimately generates a depth video with real-world scale that is strictly aligned temporally and spatially with the color video. This real-scale depth data ensures the accuracy of subsequent pose estimation and trajectory inversion.

[0137] Furthermore, in one embodiment, the depth video is stored in a single-channel high bit-depth format to ensure the accuracy and precision of the depth information.

[0138] Specifically, the depth video is output in MKV format and encoded using single-channel 16-bit or float16 precision. This storage format avoids the quantization errors caused by insufficient bit depth in common 8-bit image formats (such as RGB images), thus accurately recording scale-calibrated real-world depth values. By employing a high bit-depth format, the accuracy and precision of the depth data are ensured to remain intact during storage, providing a reliable data foundation for subsequent high-precision pose estimation and inverse dynamics analysis.

[0139] Through the above embodiments, this application utilizes a large model for structured semantic decomposition of the task, providing precise action guidance and semantic constraints for the entire data generation process. Simultaneously, constrained image enhancement techniques are employed to significantly increase the appearance and environmental diversity of the data without altering the scene's geometry, effectively enhancing the environmental generalization ability of the embodied intelligent model trained on this data. A seamless transition strategy using the final frame of the previous stage as the starting frame for subsequent stages ensures the visual coherence of long-term action videos. More importantly, by calibrating and aligning the monocular depth estimation results to a real-scale scale, reliable 3D spatial constraints are provided for subsequent precise pose extraction, overcoming the deficiency of lacking accurate depth information. A bidirectional consistency optimization mechanism, introducing a differentiable physics engine for forward physical calibration and optical flow estimation for reverse temporal correction, solves the problems of physical mismatch and temporal deviation between video observation and control trajectory, ensuring that the final trajectory data has extremely high physical reproducibility and strict temporal alignment. Finally, by automatically and precisely aligning the task semantics with the video timeline, this method achieves integrated and automated output of color video, depth video, control trajectory, and structured semantic annotation files. This greatly reduces the cost of manual data collection and detailed annotation, providing crucial technical support for training robust embodied intelligent models.

[0140] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0141] The electronic device includes: one or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the steps of the methods provided in any one or more of the above embodiments. Figure 2An exemplary structural diagram of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0142] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103 and output device 1104 may be connected by a bus or other means, as shown in the figure, which is connected by a bus.

[0143] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display, a light-emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0144] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0145] In this embodiment, a computer-readable medium stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0146] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0147] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0148] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0149] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0150] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet through an Internet service provider).

[0151] In the above embodiments, all or part of the implementation can be achieved through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the above steps or functions. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0152] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0153] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0154] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used only for distinguishing descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0155] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A method for embodied intelligent data synthesis and annotation, characterized in that, The method includes: The system receives an embodied intelligence task described in natural language, along with the corresponding initial and depth images. The first major model decomposes the embodied intelligence task into a semantic sequence of task stages with action type labels. The initial image is perturbed and transformed, and a set of scene images with appearance differences from the initial image is generated by the second large model; Based on the semantic sequence of the task stage and the set of scene images, a color video corresponding to the semantic sequence of the task stage is generated through the third major model; Perform frame-by-frame monocular depth estimation on the color video to generate a depth video aligned with the color video; The step of performing frame-by-frame monocular depth estimation on the color video to generate a depth video aligned with the color video specifically includes: Perform frame-by-frame monocular depth estimation on the color video to obtain the predicted depth video; From the depth image corresponding to the initial image, extract the pixels corresponding to the active object region as the depth reference; Based on the statistical characteristics of the predicted depth video and the depth benchmark in the region of the moving object, a linear scale mapping relationship is calculated. The linear scale mapping relationship is applied to the predicted depth video to generate a depth video aligned with the color video; By combining the color video and the corresponding depth video, the robot control trajectory data corresponding to the video time sequence is derived through inverse dynamics; Based on the duration of each segment in the color video, the semantic sequence of the task stage is mapped to the timeline of the color video to generate a structured semantic annotation file; Output the color video, the depth video, the robot control trajectory data, and the structured semantic annotation file.

2. The method according to claim 1, characterized in that, The process of decomposing the embodied intelligence task into a semantic sequence of task stages labeled with action types using the first major model specifically includes: The first major model, based on the natural language description, decomposes the task into at least two executable stages; The semantic sequence of task stages is composed of the decomposed executable stages arranged in chronological order; The semantic sequence of the task phase is a strongly constrained structured format, which includes action type labels, native language instructions, and target language instructions.

3. The method according to claim 1, characterized in that, The step of performing a perturbation transformation on the initial image and generating a set of scene images with appearance differences from the initial image through the second large model specifically includes: The second large model is prompted to perform perturbation transformation on the lighting and color distribution of the initial image by positive prompts; And / or through negative prompts, make the set of scene images generated by the second large model consistent with the geometric structure and semantic information of the initial image.

4. The method according to claim 1, characterized in that, The step of generating a color video corresponding to the semantic sequence of the task stage using a third model based on the semantic sequence of the task stage and the scene image set specifically includes: The color video is composed of sequentially spliced ​​stage video segments, and the number of stage video segments is consistent with the number of stages in the semantic sequence of the task stages. The third major model uses the semantic sequence of the task stage as the generation constraint and the image in the scene image set as the starting frame of the first stage video segment; subsequent stage video segments use the last frame of the previous stage video segment as their own starting frame to generate the color video.

5. The method according to claim 1, characterized in that, The step of combining the color video and the corresponding depth video to deduce the robot control trajectory data corresponding to the video time sequence through inverse dynamics specifically includes: Based on the color video and depth video, the pose vectors of the target object and the end effector in each frame of the video are obtained to form a six-dimensional pose sequence. Using an inverse dynamics network with the attitude vectors of adjacent frames in the six-dimensional attitude sequence as input, the attitude difference components are calculated and features are extracted to predict the control increment parameters that describe the robot's action command at that time step. The control increment parameters are integrated with the corresponding timestamps and end-effector pose data to form the robot control trajectory data.

6. The method according to claim 5, characterized in that, The step of obtaining the pose vectors of the target object and the end effector in each frame of the video based on the color video and the depth video, and forming a six-dimensional pose sequence, specifically includes: The two-dimensional image coordinates of the target object and the end effector are determined in the color video using a visual feature recognition model. By combining the corresponding depth video and camera model parameters, the two-dimensional image coordinates are converted into three-dimensional spatial position information; The relative kinematic parameters of the object and the end effector are calculated by feature correlation between time frames. The relative kinematic parameters are temporally smoothed to obtain the six-dimensional attitude sequence.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method as described in any one of claims 1 to 6.

8. A computer-readable medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 6.

Images