A method for generating embodied data for category generalization operation

Abstract

This invention discloses an embodied data generation method for category generalization operations, belonging to the field of embodied operation technology. The method includes: constructing a shape library containing warp mappings between templates and target shape spaces; acquiring the 3D model, pose trajectory, and function point annotations of source demonstration data; generating a new shape model through the shape library mapping; calculating the geometric deviation of key points; solving for the optimal alignment transformation sequence and correcting the robotic arm's motion trajectory; integrating the new model into the scene point cloud; and driving the virtual robotic arm to synthesize a new sequence of observation action pairs. This invention achieves function-aware nonlinear warp mapping, solves the function alignment problem under complex geometric deformations, and improves the generalization ability of embodied operations; it adopts a reality-to-reality paradigm, completing efficient data augmentation with a small number of annotations, significantly reducing data collection costs.

Description

Technical Field

[0001] This invention relates to the field of embodied operation technology, and more particularly to an embodied data generation method for category-generalized operations. Background Technology

[0002] Embodied manipulation, as a core technology for robots to achieve autonomous operation in uncontrolled scenarios, is widely used in industrial assembly and home service fields. With the development of visual motion strategies, related technologies have constructed a complete system from data acquisition to enhancement through collaborative operations of 3D reconstruction, task planning, and cross-object correspondence extraction. Specifically, this system covers the entire process from texture appearance transformation to spatial configuration adjustment, including key links such as point cloud editing, trajectory replay, and semantic feature mapping, aiming to improve the geometric diversity of training data.

[0003] However, existing class-level data generation methods directly employ simple axial scaling or mechanical trajectory replay without establishing a deep geometric mapping based on functional availability. This may result in the generated motion trajectory failing to accurately align with the functional components of the new object, or in physically infeasible interactions due to a lack of nonlinear deformation modeling. Consequently, these methods severely restrict the robot's intra-class generalization ability and deployment efficiency in fine manipulation tasks such as picking and dumping. Summary of the Invention

[0004] The main objective of this invention is to provide a method for generating embodied data oriented towards category generalization operations.

[0005] Another object of the present invention is to provide an embodied data generation apparatus for category generalization operations.

[0006] The third objective of this invention is to provide an electronic device.

[0007] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0008] To achieve the above objectives, a first aspect of the present invention proposes a method for generating embodied data oriented to category generalization operations, comprising:

[0009] S1, construct a shape library containing the spatial warping mapping relationship between the template shape and the target shape, and obtain the original object 3D model, object pose trajectory and key function point annotation information from the source demonstration data; S2, based on the spatial warping mapping relationship in the shape library, maps the original object's 3D model to a new shape 3D model under the target category, and calculates the geometric deviation of the key points on the surface of the new shape 3D model relative to the corresponding key points on the surface of the original object; S3, based on geometric deviation and key functional point annotation information, calculate the optimal alignment transformation sequence that keeps the functional parts of the new shape 3D model spatially aligned with the original demonstration scene, and generate the corrected robotic arm motion trajectory based on the optimal alignment transformation sequence. S4 uses the corrected robotic arm motion trajectory to drive the virtual robotic arm model, and places the new shape 3D model into the scene point cloud after background separation processing to synthesize a new sequence of observation action pairs containing the new shape object and the corrected motion trajectory.

[0010] Optionally, construct a shape library containing spatial warp mappings between template shapes and target shapes, including: Select one shape from objects of the same category as the template shape, and collect multiple target shapes with different geometric forms within that category; For each target shape, a spatial warp mapping neural network is constructed with the template shape as input and the target shape as output. The spatial warp mapping neural network is trained end-to-end using a signed distance function as a supervision signal, so that the signed distance function value of any point on the surface and inside of the template shape after spatial warp mapping is consistent with the signed distance function value of the corresponding position of the point on the target shape. The spatial warp mapping relationship from the template shape to each target shape is obtained and assembled into a plug-and-play shape library.

[0011] Optionally, calculate the geometric deviation of key points on the surface of the new shape 3D model relative to the corresponding key points on the original object's 3D surface, including: In the data generation stage, a scanned model of the manipulated actual object is obtained as the source shape, and the spatial warping mapping relationship from the source shape to the template shape is trained. By combining the spatial warping mapping relationship from the source shape to the template shape and the spatial warping mapping relationship from the template shape to the target shape, a composite spatial warping mapping relationship from the source shape directly to the target shape is obtained; The points on the original object's 3D model are input into the composite space warping mapping relationship to obtain the initial mapping points. The point with the smallest Euclidean distance to the initial mapping points is then searched on the surface of the target shape's 3D model as the optimized mapping point, thereby reducing the error caused by imperfect warping. The optimized mapping point is determined by the following formula:

[0012] in, For the optimized mapping point, Let S be the original input point and S be the constraint set. Let S be the candidate output points in set S. Indicates that when the condition is met All Find the one that minimizes the objective function. ; The final composite space warp mapping is expressed as:

[0013] in, For the final mapping, To start from the initial domain The mapping function that maps to intermediate layer 0. For the intermediate layer constraint set, This is the final set of constraints.

[0014] Optionally, based on geometric deviations and key functional point annotation information, calculate the optimal alignment transformation sequence that maintains spatial alignment between the functional parts of the new shape 3D model and the original demonstration scene, including: Obtain annotation information provided by the annotator, including timestamps, ordered point sets fixed on task-related objects, and alignment cost functions, where the ordered point sets identify the spatial locations of key functional parts on the objects; Based on the spatial warping mapping relationship, the functional points on the new shape 3D model corresponding to the key points of the original object 3D model are determined. An optimization problem is constructed with the goal of minimizing the alignment cost function. The rigid body transformation matrix that best matches the functional points on the new shape 3D model with the corresponding positions in the original demonstration scene is solved as the optimal alignment transformation. The optimal alignment transformation at each time step in the time series is interpolated to obtain a continuous and smooth optimal alignment transformation sequence.

[0015] Optionally, the corrected robotic arm motion trajectory is generated based on the optimal alignment transformation sequence, including: Identify the gripper key points that represent the position of the gripped object in the gripper, and use the original object pose trajectory and the expected object pose trajectory of the new 3D model to calculate the trajectory difference of the gripper key points and their corresponding points in the camera space. Based on the trajectory differences, a transformation sequence containing only translation components is calculated to maintain visual fidelity and reduce the negative impact of inaccurate hand-eye calibration. Obtain the transformation matrix from the end effector to the robot base and the calibration transformation matrix from the camera to the base in the original motion. Use the transformation sequence containing only translation components to correct the original motion, generating the corrected robotic arm motion trajectory. The formula for calculating the corrected motion is as follows:

[0016] in, This represents a transform sequence that contains only translation components. This represents the calibration transformation matrix from the camera to the base. Indicates the original action.

[0017] Optionally, the modified robotic arm motion trajectory is used to drive a virtual robotic arm model, and the new-shape 3D model is placed into the scene point cloud after background separation processing to synthesize a new sequence of observation action pairs containing the new-shape object and the modified motion trajectory, including: The original observed image is processed using a video segmentation model to obtain the object mask, and a color-depth image of the empty workbench is captured as background data. The point cloud data of the robot arm is segmented from the original scene point cloud by depth value comparison, and the point cloud data of the robot arm is spatially transformed using the transformation relationship corresponding to the corrected robot arm motion trajectory. The new shape 3D model is placed in the correct position in the background data according to the optimal alignment transformation sequence, and then fused with the transformed robot arm point cloud data. A camera-aware 3D post-processing method is used to sample point clouds from the fused synthetic scene to generate a new sequence of observation action pairs as policy input.

[0018] To achieve the above objectives, a second aspect of the present invention provides an embodied data generation apparatus for category generalization operations, comprising: The building module is used to construct a shape library containing the spatial warping mapping relationship between the template shape and the target shape, and to obtain the original object 3D model, object pose trajectory and key function point annotation information from the source demonstration data; The mapping module is used to map the original object's 3D model to a new shape 3D model under the target category based on the spatial warping mapping relationship in the shape library, and to calculate the geometric deviation of the key points on the surface of the new shape 3D model relative to the corresponding key points on the surface of the original object 3D model. The calculation module calculates the optimal alignment transformation sequence that keeps the functional parts of the new shape 3D model spatially aligned with the original demonstration scene based on the geometric deviation and key functional point annotation information, and generates the corrected robotic arm motion trajectory based on the optimal alignment transformation sequence. The synthesis module is used to drive a virtual robotic arm model with the corrected robotic arm motion trajectory and place the new shape 3D model into the scene point cloud after background separation processing to synthesize a new sequence of observation action pairs containing the new shape object and the corrected motion trajectory.

[0019] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0020] To achieve the above objectives, a third aspect of this application provides an electronic device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, for implementing the embodied data generation method for category-oriented generalization operations as described in the first aspect embodiment.

[0021] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the embodied data generation method for category-oriented generalization operations as described in the first aspect embodiment.

[0022] The embodiments of the present invention have the following beneficial effects: This invention can establish a function-aware nonlinear spatial warp mapping, solving the problem of function alignment for fine manipulation tasks under complex geometric deformations, and significantly improving the intra-class generalization ability of embodied manipulation strategies. At the same time, through a reality-to-reality generation paradigm that does not require simulators, it achieves efficient and physically reasonable data expansion with minimal manual annotation, greatly reducing data collection costs. Attached Figure Description

[0023] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating an embodiment data generation method for category generalization operations provided in this invention; Figure 2 An overview diagram of the planning and use of the shape library provided for embodiments of the present invention; Figure 3 This invention provides a data generation pipeline diagram based on a shape library for embodiments of the invention. Figure 4 This is a data result diagram provided for an embodiment of the present invention; Figure 5 This is a structural diagram of an embodied data generation device for category generalization operations provided in an embodiment of the present invention. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] The following description, with reference to the accompanying drawings, describes an embodiment of the present invention for generating embodied data oriented towards category generalization operations.

[0027] Example 1 This invention provides a method for generating embodied data oriented to category generalization operations, such as... Figure 1 As shown, the method includes the following steps: S1. Construct a shape library containing the spatial warping mapping relationship between the template shape and the target shape, and obtain the original object 3D model, object pose trajectory and key function point annotation information from the source demonstration data.

[0028] To lay the foundation for subsequent shape replacement and data synthesis, this application constructs a function-aware shape library and obtains source demonstration-related data and lightweight annotations to support shape mapping and alignment.

[0029] In this embodiment, a calibrated RGB-D (color-depth) camera is used to collect raw observation data. At the same time, a visual motion strategy with colorless point clouds as observation input is trained. The core purpose is to solve the technical problems of high data collection cost and insufficient shape diversity in existing strategy training through automated data generation.

[0030] In this application embodiment, in order to achieve efficient generation of diverse training data, this application first plans and constructs a function-aware shape library for each task-related object category. This shape library not only contains three-dimensional models of various geometric shapes under the category, but also stores the spatial warping mapping relationship from general template shapes to various target shapes in the library. This warping mapping can achieve accurate correspondence of functional points between different shapes, providing core technical support for subsequent object shape replacement and action correction.

[0031] In the embodiments of this application, given a set of original observation-action pair sequences, this application only requires the annotator to perform a very simple manual operation, namely, to identify the key points on the actual object that are related to the task and to specify the method of cross-shape alignment. Then, this application will use the established shape library to automatically calculate the best alignment method after shape replacement and the corrected robotic arm action, without the need for manual intervention in a large amount of data annotation and action adjustment, which greatly improves the data generation efficiency.

[0032] In this application embodiment, the core technical concepts involved in this application are first defined to clarify the core connotation of the visual motion strategy and the target requirements for data generation. In this application embodiment, the visual motion strategy is essentially a mapping from the observation space to the action space. : O→A, where O represents the observation space and A represents the action space. The core function of this strategy is to output corresponding action instructions based on the acquired observation data in order to complete a specific task.

[0033] In this embodiment of the application, in order to train a visual motion strategy that can stably complete a specific task, it is usually necessary to prepare a dataset containing multiple demonstration data. Each demonstration data point D is an observation-action pair sequence. In each sequence These represent the observation data at time t and the corresponding action command, respectively.

[0034] In the embodiments of this application, it should be noted that the generalization ability of visual motion strategies directly depends on the diversity of the training dataset. In the prior art, to obtain diverse training data, a large amount of manpower is often required to collect demonstration data of objects with different shapes, and diverse shape assets also need to be prepared, which is extremely costly and inefficient. Based on this, this application aims to enhance the shape diversity of task-related objects through automated data generation, thereby effectively reducing the manpower required in the data collection process, reducing the dependence on diverse shape assets, and solving the pain points of the prior art.

[0035] In this application embodiment, the core idea is to provide a source demo with minimal human annotations. Based on the core principles of physical rationality and functional correctness, the source demonstration is automatically displayed. The task-related objects in the training data are replaced with objects of the same category but different geometric shapes (hereinafter referred to as "shapes"), thereby generating a large amount of new training data with diverse shapes, providing sufficient support for the generalization training of visual motion strategies.

[0036] In this application embodiment, the planning of the shape library is the core foundation for the automated data generation of this application. Its core objective is to develop an automated mechanism that, given an actual object and its associated key points appearing in a source demonstration, can find the functional points corresponding to the key points of the actual object on each shape in the shape library, thereby realizing point-based alignment operations in the generation pipeline and ensuring the correctness of the task function after shape replacement.

[0037] In this embodiment, the correspondence between key points must possess "function-aware" characteristics. That is, a key point with a specific function on the original object, after being mapped onto the target shape, must still be located at a position with the same function on the target shape. For example, given a mug and a key point on the inside of its handle for gripping, after mapping, this key point should also be located on the inside of the handle on the target mug to ensure the effectiveness of subsequent robotic arm gripping actions. An overview of the shape library's planning and use is as follows: Figure 2 As shown.

[0038] In this embodiment of the application, in order to realize the above-mentioned function-aware correspondence, this application applies to any two 3D shapes of the same category. and The aim is to establish a dense correspondence between functional perceptions, that is, to construct a spatial warping mapping between the two. The mapping :R3→R3 can achieve From any point above Precise mapping of corresponding functional points. In the embodiments of this application, it has been found through research that among shapes of the same category with significant differences, functional correspondence usually originates from geometric correspondence. This is because the functions of an object and its specific parts are derived from the object's geometric shape. Based on this key observation, this application proposes to construct a spatial warping mapping through geometric clues. In this embodiment, since the correspondence between different shapes is often complex, it is difficult to achieve accurate mapping by relying solely on traditional geometric calculations. Therefore, this application employs a neural network to fit the spatial warping mapping. And only pure geometric information is used as the supervision signal to ensure the geometric rationality and functional consistency of the mapping.

[0039] In this embodiment, specifically, the signed distance function (SDF) is used as the core supervision signal to train the spatial warping map. In this embodiment of the application, a 3D shape S and a point are given. The absolute value of the signed distance function SDF(x, S) represents the distance from point x to the surface of shape S, while the sign indicates whether point x is inside or outside shape S. A positive sign means point x is outside shape S, a negative sign means point x is inside shape S, and a sign of 0 means point x is on the surface of shape S. In the embodiments of this application, to establish... and The functional perception correspondence between them, this application hopes to provide for any point near the surface After spatial warping mapping The result after mapping It can satisfy:

[0040] This constraint ensures that when When =0, =0, that is Points on the surface, after being mapped, still fall on... On the surface, the mapping constraints of non-surface points can also provide more comprehensive geometric supervision, further improving the mapping accuracy.

[0041] In this embodiment of the application, spatial warp mapping is trained in an end-to-end manner. This application is primarily for the shape By fitting a neural SDF model, a spatially differentiable SDF function is obtained, denoted as . The neural SDF model was then fixed. Focusing on training space warp mapping In the embodiments of this application, training The main training loss used is:

[0042] in, For shape The corresponding neural SDF function. In this embodiment, for ease of gradient calculation, let... , and use Representation space warping mapping Given the parameters, the gradient of the training loss can be calculated as follows:

[0043] because It is in the form of a neural network, and its spatial derivative is... It can be easily calculated through backpropagation of the neural network, ensuring the smooth progress of the training process.

[0044] In this embodiment, to avoid excessive spatial warping during training, which could lead to functional failure after shape replacement, this application, referencing previous research, incorporates point-wise regularization loss and point-pair regularization loss into the training loss to constrain the warping mapping. In this embodiment, the point-wise regularization loss is as follows:

[0045] in It is the Huber kernel, used to suppress the impact of outliers on training. The reference point is set as the preset point; point-to-point regularization loss:

[0046] The superscripts (1) and (2) represent different samples in the batch. , The preset threshold is used to limit the change in the distance ratio between point pairs before and after warping, so as to avoid excessive deformation.

[0047] In this embodiment of the application, the final training loss is a weighted sum of the losses of each component:

[0048] in, and These are the weight coefficients for pointwise regularization loss and point-to-point regularization loss, respectively, which can be adjusted according to the actual training effect.

[0049] In the embodiments of this application, after training the spatial warp mapping between various shapes, this application assembles different three-dimensional shapes and their warp functions into a plug-and-play shape library, which facilitates rapid calling and expansion in the subsequent data generation process.

[0050] In this application embodiment, for each fixed object category, a general template shape is selected. And train from that template shape Spatial warp mapping to all other target shapes within the library This design makes the time complexity of adding a new shape to the shape library O(1), requiring only training the new shape and the template shape. The warp mapping between shapes can be used, significantly reducing the cost of expanding the shape library. In this embodiment, the shape library also has good compatibility and can be "inserted" into any shape Sa. Specifically, it can be trained from shapes... To template shape Spatial warp mapping Then and By combining them, you can seamlessly obtain shapes To the target shape Spatial warp mapping No separate training required The mapping between each target shape further enhances the practicality and efficiency of the shape library.

[0051] In this embodiment, the function-aware generation section will detail the data generation pipeline based on the aforementioned shape library. The core of this pipeline is to automatically replace and correct the shapes of objects in the source demonstration, generating novel observation-action pair sequences with diverse shapes. An overview of the process is as follows: Figure 3As shown. In this embodiment of the application, for the sake of simplicity, it is assumed that the source demonstration contains only one task-related object, the shape of which is... The objective of this application is to replace the object with another object of the same category but different shape from the shape library. This leads to the generation of entirely new training data.

[0052] In this embodiment of the application, a shape library with spatial warp mapping is constructed and source demonstration data and minimalist annotations are collected to provide basic support for subsequent shape replacement, alignment correction and data synthesis.

[0053] S2, based on the spatial warping mapping relationship in the shape library, maps the original object's 3D model to a new shape 3D model under the target category, and calculates the geometric deviation of the key points on the surface of the new shape 3D model relative to the corresponding key points on the original object's 3D surface.

[0054] In this embodiment of the application, before running the data generation pipeline, the application first performs a 3D scan on the object actually being manipulated in the source demonstration to obtain a precise 3D model of the object. This ensures the accuracy of subsequent shape mapping and alignment. In this embodiment, the application then starts from the raw observation data acquired by the calibrated RGB-D camera and the robotic arm motion data, and runs an existing model-based object tracking method to continuously track the object. The pose changes throughout the entire source demonstration process, thereby acquiring the object. Complete pose trajectory ,in SE(3) represents the three-dimensional Euclidean transformation space. It contains the position and orientation information of the object at time t, providing a basis for pose alignment after subsequent shape replacement.

[0055] In this embodiment of the application, for the source demonstration data, only minimal manual annotation is required from the annotator. The annotation content is as follows: ,in It is fixed to the task-related object. An ordered set of key points on the surface, which is closely related to task execution, is the core to ensure the correctness of function after shape replacement; It is the corresponding timestamp, used to identify the specific moment of the key point in the presentation sequence; It is the alignment cost function, used to calculate the optimal alignment transformation after shape replacement.

[0056] In this embodiment, the core purpose of annotation is to guide subsequent shape alignment steps, ensuring that the new shape maintains functional consistency with the original demonstration scene. In this embodiment, the "hanging a mug" task is used as an example. During the hanging process, the upper corner of the mug handle must be in a precise spatial position to successfully insert it into the holder and complete the hanging action. Therefore, the annotator can... Set the mount start time point, Let's define a single-element point set at the top corner of the cup handle, and simultaneously... Set a simple squared error function to constrain the top corner of the handle of the new shaped mug after replacement, keeping it consistent with the original position.

[0057] In addition to the above, this application also requires the annotator to identify a key point of the gripper in the embodiments of this application. This key point represents the position of the gripped object within the robotic arm's gripper, used for subsequent correction of the robotic arm's motion trajectory to ensure that the robotic arm can accurately grasp the new shape object after shape replacement. In this embodiment, it is important to emphasize that these annotation operations only need to be performed once for each source demonstration, taking approximately one minute, significantly reducing manual annotation costs and solving the problem of time-consuming and labor-intensive data annotation in existing technologies.

[0058] In this embodiment of the application, a fine optimization step is deliberately added when calculating the composite warp function (i.e., the spatial mapping function used to realize the correspondence between different shapes). Its core purpose is to optimize the output results of the warp mapping from the template shape to the target shape and the warp mapping from the original object to the template shape by finding the nearest corresponding point on the shape surface. This effectively reduces the error caused by the imperfect warp mapping, ensures the accuracy of the mapping between different shapes, and thus ensures the rationality of subsequent data generation and action adjustment.

[0059] Specifically, the refined optimization function used in this application is:

[0060] The core function of this feature is to find the closest point y on a specified shape surface S to any input point x, using this as the optimized mapping point to avoid functional misalignment caused by mapping deviations. Based on this, the composite warping function used in this application... (That is, the spatial mapping function used to map the original object shape to the target shape), its specific calculation method is as follows:

[0061] in, For template shape, For the target shape, this formula means that it first passes through Map points on the original object to the template shape. The corresponding points on the graph, after fine-tuning (refine step), are then... Map this point to the target shape Finally, through a final refinement optimization process, it is ensured that the point falls within the target shape. The surface and position are accurate.

[0062] It should be noted that the optimization problem involved in this fine-grained optimization can be easily approximated by using a trained SDF network (i.e., a neural network model for calculating the distance between shape surfaces) and spatial gradient calculation. This eliminates the need for complex manual calculations, ensuring both optimization efficiency and accuracy of the optimization results. It further enhances the rationality and precision of spatial mapping, providing a reliable mapping basis for subsequent shape replacement and motion correction, and ensuring the scientific rigor of the entire data generation process.

[0063] In this embodiment, by adding a fine optimization step to the calculation of the composite warp function and combining the SDF network with spatial gradient to solve the approximate solution, the error caused by imperfect warp is reduced, the accuracy of shape mapping is ensured, and reliable support is provided for subsequent data generation and motion correction.

[0064] S3, based on geometric deviations and key functional point annotation information, calculates the optimal alignment transformation sequence that keeps the functional parts of the new shape 3D model spatially aligned with the original demonstration scene, and generates the corrected robotic arm motion trajectory based on the optimal alignment transformation sequence.

[0065] In order to achieve accurate alignment between the new shape and the original scene and to meet the grasping requirements, this application calculates the optimal alignment transformation and correction actions to ensure the correctness of the task function and the fidelity of the visuals.

[0066] In this embodiment, based on the manually annotated information obtained in the aforementioned steps, the optimal alignment transformation is obtained by solving a preset optimization formula. The specific calculation formula is as follows:

[0067] in, This represents the ability to shape the target. The transformation matrix is ​​used to achieve optimal spatial alignment between the key points on the object and the corresponding key points on the original object Sa. In this embodiment, the alignment transformation obtained by this optimization method can ensure that the spatial position of the key functional parts of the object after shape replacement remains consistent with the source demonstration, thereby maintaining the functional correctness and physical rationality of task execution.

[0068] In this embodiment of the application, the optimal alignment transformation at key time nodes is obtained. Subsequently, this application generates a complete alignment transformation sequence (T~0,...,T~L) using temporal interpolation. 1) This ensures the continuity and smoothness of changes at each moment, avoiding abrupt attitude shifts or spatial misalignment. Based on this, this application combines the original object... pose trajectory The target shape is further calculated. Desired object pose trajectory:

[0069] This trajectory can be directly used for subsequent 3D scene synthesis and motion planning, ensuring that the motion logic of the new shape object in the scene remains consistent with the original demonstration.

[0070] In this application's embodiments, for complex tasks such as pouring water, this application not only needs to achieve pose alignment between the entire object, but also needs to satisfy alignment constraints of key points relative to another variable shape, such as maintaining a stable relative position between key points on a mug and the edge of the mug's rim. For such scenarios, this application calculates an alignment sequence in the camera coordinate system that is consistent with the above logic. This allows for the coordination and alignment of multiple objects and key points, thereby improving the applicability and generalization ability of the data generation solution.

[0071] In this embodiment, considering the differences in gripping position, size, and geometry between objects of different shapes, the original robotic arm movements cannot be directly adapted to the new shape after replacement. Therefore, the gripper movements need to be modified accordingly. This application is based on the original object's pose trajectory. With the target shape and pose trajectory The key points of the clamp were calculated. The motion trajectory of the corresponding point on the target shape in camera space is calculated, and a transformation sequence containing only translation components is obtained accordingly. This allows the gripper to be positioned in a suitable location for grasping new shapes.

[0072] In this embodiment of the application, the transformation of the original robotic arm end effector relative to the base is assumed to be... It belongs to the SE(3) transformation, and is combined with the pre-calibrated coordinate transformation between the camera and the robot base. This application obtains the corrected robotic arm motion through coordinate transformation, and the specific calculation formula is as follows:

[0073] In the embodiments of this application, this application is limited to It only includes translation transformation, which aims to maintain the visual fidelity of the synthetic observation data. Under small translation conditions, there is no need to re-render the gripper. At the same time, it can reduce the adverse effects of hand-eye calibration errors on motion accuracy and improve the stability and realism of data generation.

[0074] In this embodiment, by solving the optimal alignment transformation, generating the pose trajectory, and correcting the robotic arm's movements, the precise alignment and adaptive grasping of the new-shaped object is achieved, while ensuring visual fidelity and motion reliability. This lays the foundation for the subsequent synthesis of a new observation action pair sequence containing the new-shaped object and the corrected motion trajectory.

[0075] S4 uses the corrected robotic arm motion trajectory to drive the virtual robotic arm model, and places the new shape 3D model into the scene point cloud after background separation processing to synthesize a new sequence of observation action pairs containing the new shape object and the corrected motion trajectory.

[0076] In order to generate diverse training observation data that are adapted to the policy input, this application synthesizes 3D scenes based on pose trajectories and samples point clouds to obtain observation results that meet the requirements.

[0077] In this embodiment of the application, the desired pose trajectory of the target shape is completed. and the clamp translation transformation sequence After calculation, based on the aforementioned trajectory and transformation parameters, this application synthesizes new observation data in three-dimensional space, thereby obtaining standardized observation samples that can be used for visual motion strategy training. In the embodiments of this application, to achieve accurate scene synthesis, this application first uses the video segmentation model SAM2 to process the original observation data, segmenting and extracting the mask information of task-related objects to distinguish object regions from background regions and avoid mutual interference between scene elements. Simultaneously, this application pre-captures RGB-D images in an empty workbench state as the background substrate for the synthesized scene, ensuring the realism and consistency of the scene background.

[0078] In this embodiment, to complete the spatial reconstruction of the robot arm and the target object, the original scene is segmented into point clouds based on the difference in depth values. The point cloud data corresponding to the robot arm is then separated, and a transformation sequence is then used. Direct spatial transformation is performed on the point cloud of the robot arm to adapt its pose and position to the grasping requirements of objects with new shapes. Simultaneously, this application utilizes the desired pose trajectory. Target 3D model from shape library Place the objects in their corresponding spatial positions in the synthetic scene according to their temporal posture, ensuring that their motion logic remains consistent with that of the objects in the original demonstration, thus achieving a reasonable layout of the new-shaped objects in the scene.

[0079] In this embodiment, since the visual motion strategy trained in this application uses colorless point clouds as observation input, after scene synthesis is completed, this application directly adopts a camera-aware 3D post-processing method to sample and generate point cloud data from the constructed virtual synthetic scene, ensuring a complete match between the synthetic observation and the strategy input format. In this embodiment, the final data result generated after the above process is as follows: Figure 4 As shown, this data possesses both the diversity of object shapes and the rationality of scene structure and action logic, and can be directly used to improve the generalization ability of visual motion strategies.

[0080] In this embodiment, by segmenting the scene, synthesizing 3D space, and sampling point clouds, new observation data adapted to the strategy input is generated, providing effective samples for visual motion training.

[0081] Example 2 This invention provides an embodiment data generation apparatus for category generalization operations, such as... Figure 5 As shown, the device includes: Module 100 is used to build a shape library containing the spatial warping mapping relationship between the template shape and the target shape, and to obtain the original object 3D model, object pose trajectory and key function point annotation information from the source demonstration data; The mapping module 200 is used to map the original object 3D model to a new shape 3D model under the target category based on the spatial warping mapping relationship in the shape library, and to calculate the geometric deviation of the key points on the surface of the new shape 3D model relative to the corresponding key points on the 3D surface of the original object. The calculation module 300 calculates the optimal alignment transformation sequence that keeps the functional parts of the new shape 3D model spatially aligned with the original demonstration scene based on the geometric deviation and key functional point annotation information, and generates the corrected robotic arm motion trajectory based on the optimal alignment transformation sequence. The synthesis module 400 is used to drive a virtual robotic arm model with the corrected robotic arm motion trajectory and place the new shape 3D model into the scene point cloud after background separation processing to synthesize a new sequence of observation action pairs containing the new shape object and the corrected motion trajectory.

[0082] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0083] Example 3 To implement the methods of the above embodiments, the present invention also provides an electronic device, which includes a memory and a processor; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the various steps of the methods described above.

[0084] Example 4 To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0085] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0086] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0087] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for generating embodied data oriented towards category generalization operations, characterized in that, include: S1, construct a shape library containing the spatial warping mapping relationship between the template shape and the target shape, and obtain the original object 3D model, object pose trajectory and key function point annotation information from the source demonstration data; S2, based on the spatial warping mapping relationship in the shape library, maps the original object's 3D model to a new shape 3D model under the target category, and calculates the geometric deviation of the key points on the surface of the new shape 3D model relative to the corresponding key points on the surface of the original object; S3, based on geometric deviation and key functional point annotation information, calculate the optimal alignment transformation sequence that keeps the functional parts of the new shape 3D model spatially aligned with the original demonstration scene, and generate the corrected robotic arm motion trajectory based on the optimal alignment transformation sequence. S4 uses the corrected robotic arm motion trajectory to drive the virtual robotic arm model, and places the new shape 3D model into the scene point cloud after background separation processing to synthesize a new sequence of observation action pairs containing the new shape object and the corrected motion trajectory.

2. The method according to claim 1, characterized in that, The construction of the shape library, which includes the spatial warp mapping relationship between the template shape and the target shape, includes: Select one shape from objects of the same category as the template shape, and collect multiple target shapes with different geometric forms within that category; For each target shape, a spatial warp mapping neural network is constructed with the template shape as input and the target shape as output. The spatial warp mapping neural network is trained end-to-end using a signed distance function as a supervision signal, so that the signed distance function value of any point on the surface and inside of the template shape after spatial warp mapping is consistent with the signed distance function value of the corresponding position of the point on the target shape. The spatial warp mapping relationship from the template shape to each target shape is obtained and assembled into a plug-and-play shape library.

3. The method according to claim 1, characterized in that, The calculation of the geometric deviation of key points on the surface of the new 3D model relative to the corresponding key points on the 3D surface of the original object includes: In the data generation stage, a scanned model of the manipulated actual object is obtained as the source shape, and the spatial warping mapping relationship from the source shape to the template shape is trained. By combining the spatial warping mapping relationship from the source shape to the template shape and the spatial warping mapping relationship from the template shape to the target shape, a composite spatial warping mapping relationship from the source shape directly to the target shape is obtained; The points on the original object's 3D model are input into the composite space warping mapping relationship to obtain the initial mapping points. The point with the smallest Euclidean distance to the initial mapping points is then searched on the surface of the target shape's 3D model as the optimized mapping point, thereby reducing the error caused by imperfect warping. The optimized mapping point is determined by the following formula: in, For the optimized mapping point, Let S be the original input point and S be the constraint set. Let S be the candidate output points in set S. Indicates that when the condition is met All Find the one that minimizes the objective function. ; The final composite space warp mapping is expressed as: in, For the final mapping, To start from the initial domain The mapping function that maps to intermediate layer 0. For the intermediate layer constraint set, This is the final set of constraints.

4. The method according to claim 1, characterized in that, The step of calculating the optimal alignment transformation sequence for maintaining spatial alignment between the functional parts of the new 3D model and the original demonstration scene based on geometric deviations and key functional point annotation information includes: Obtain annotation information provided by the annotator, including timestamps, ordered point sets fixed on task-related objects, and alignment cost functions, where the ordered point sets identify the spatial locations of key functional parts on the objects; Based on the spatial warping mapping relationship, the functional points on the new shape 3D model corresponding to the key points of the original object 3D model are determined. An optimization problem is constructed with the goal of minimizing the alignment cost function. The rigid body transformation matrix that best matches the functional points on the new shape 3D model with the corresponding positions in the original demonstration scene is solved as the optimal alignment transformation. The optimal alignment transformation at each time step in the time series is interpolated to obtain a continuous and smooth optimal alignment transformation sequence.

5. The method according to claim 1, characterized in that, The generation of the corrected robotic arm motion trajectory based on the optimal alignment transformation sequence includes: Identify the gripper key points that represent the position of the gripped object in the gripper, and use the original object pose trajectory and the expected object pose trajectory of the new 3D model to calculate the trajectory difference of the gripper key points and their corresponding points in the camera space. Based on the trajectory differences, a transformation sequence containing only translation components is calculated to maintain visual fidelity and reduce the negative impact of inaccurate hand-eye calibration. Obtain the transformation matrix from the end effector to the robot base and the calibration transformation matrix from the camera to the base in the original motion. Use the transformation sequence containing only translation components to correct the original motion, generating the corrected robotic arm motion trajectory. The formula for calculating the corrected motion is as follows: in, This represents a transform sequence that contains only translation components. This represents the calibration transformation matrix from the camera to the base. Indicates the original action.

6. The method according to claim 1, characterized in that, The process involves using the corrected robotic arm motion trajectory to drive a virtual robotic arm model, and placing the new-shape 3D model into the scene point cloud after background separation processing to synthesize a new sequence of observation action pairs containing the new-shape object and the corrected motion trajectory, including: The original observed image is processed using a video segmentation model to obtain the object mask, and a color-depth image of the empty workbench is captured as background data. The point cloud data of the robot arm is segmented from the original scene point cloud by depth value comparison, and the point cloud data of the robot arm is spatially transformed using the transformation relationship corresponding to the corrected robot arm motion trajectory. The new shape 3D model is placed in the correct position in the background data according to the optimal alignment transformation sequence, and then fused with the transformed robot arm point cloud data. A camera-aware 3D post-processing method is used to sample point clouds from the fused synthetic scene to generate a new sequence of observation action pairs as policy input.

7. An embodied data generation device for category generalization operations, characterized in that, include: The building module is used to construct a shape library containing the spatial warping mapping relationship between the template shape and the target shape, and to obtain the original object 3D model, object pose trajectory and key function point annotation information from the source demonstration data; The mapping module is used to map the original object's 3D model to a new shape 3D model under the target category based on the spatial warping mapping relationship in the shape library, and to calculate the geometric deviation of the key points on the surface of the new shape 3D model relative to the corresponding key points on the surface of the original object 3D model. The calculation module calculates the optimal alignment transformation sequence that keeps the functional parts of the new shape 3D model spatially aligned with the original demonstration scene based on the geometric deviation and key functional point annotation information, and generates the corrected robotic arm motion trajectory based on the optimal alignment transformation sequence. The synthesis module is used to drive a virtual robotic arm model with the corrected robotic arm motion trajectory and place the new shape 3D model into the scene point cloud after background separation processing to synthesize a new sequence of observation action pairs containing the new shape object and the corrected motion trajectory.

8. An electronic device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images