Embodied intelligence-based training data generation method and device, equipment and medium

Abstract

The application belongs to the field of industrial data processing, and particularly relates to a training data generation method and device based on embodied intelligence, equipment and medium, comprising: determining object pose data of a target object in a mechanical arm coordinate system; controlling an image acquisition device of the mechanical arm to move to each image acquisition pose in sequence according to a pre-generated image acquisition pose sequence to acquire candidate image information of the target object; determining target image information according to the candidate image information, the target image information being determined based on an area ratio of a mask and a bounding box of the target object in the image; generating training data according to the target image information and the object pose data; and the training data being used for training an object pose estimation model. The method significantly reduces manual labeling cost and time, improves data generation efficiency, effectively covers full view angle changes in complex industrial scenes, overcomes interference such as light and shielding, and guarantees consistency and quality of labeled data.

Description

Technical Field

[0001] This application belongs to the field of industrial data processing, and specifically relates to a method, apparatus, device and medium for generating training data based on embodied intelligence. Background Technology

[0002] In the fields of intelligent manufacturing and industrial automation, embodied intelligence technology is gradually becoming a core driving force for improving the autonomous operation capabilities of robots. Taking robotic arm operations in industrial scenarios as an example, robots need to accurately perceive the three-dimensional spatial position of target objects (6D pose, i.e., x, y, z coordinates and rotation angles rx, ry, rz around the three axes) to complete complex tasks such as grasping, assembly, and sorting. These scenarios place extremely high demands on the accuracy and robustness of 6D pose estimation models.

[0003] However, in existing technologies, the training of 6D pose estimation models relies on a large amount of high-quality labeled data, and the data collection, cleaning, and labeling processes face significant bottlenecks: on the one hand, manual labeling is costly and inefficient, with labeling a single image potentially taking several minutes; on the other hand, complex lighting, occlusion, and material reflections in industrial scenarios make it difficult for traditional data acquisition methods to cover the entire field of view, and the consistency of labeled data is poor. Furthermore, traditional methods rely on motion capture systems or manual labeling tools, which are costly, complex to deploy, and difficult to adapt to dynamically changing industrial environments.

[0004] Therefore, there is an urgent need for efficient and low-cost automated data generation methods to avoid data bottlenecks in the training of 6D pose estimation models. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a method, apparatus, device, and medium for generating training data based on embodied intelligence.

[0006] In a first aspect, this application provides a method for generating training data based on embodied intelligence, the method comprising:

[0007] Determine the object pose data of the target object in the robot arm coordinate system;

[0008] Based on the pre-generated image acquisition pose sequence, the image acquisition device of the robotic arm is controlled to move sequentially to each image acquisition pose to acquire candidate image information of the target object;

[0009] Based on the candidate image information, target image information is determined, which is determined based on the area ratio of the mask of the target object in the image to the bounding box.

[0010] Training data is generated based on the target image information and the object pose data; the training data is used to train the object pose estimation model.

[0011] In one possible implementation, before the image acquisition device of the controlled robotic arm sequentially moves to each image acquisition pose to acquire candidate image information of the target object, it further includes:

[0012] Based on the position information of the robotic arm and the worktable, the boundary of the planar area where the image acquisition device of the robotic arm moves above the target object is determined;

[0013] Determine multiple coordinate points within the boundary of the planar region;

[0014] Determine at least one candidate image acquisition pose corresponding to each coordinate point, wherein the candidate image acquisition pose includes rotation angle and / or perturbation data;

[0015] Based on at least one candidate image acquisition pose corresponding to each coordinate point, the image acquisition pose sequence is generated. The image acquisition pose sequence includes multiple coordinate points sorted by distance, and at least one candidate image acquisition pose corresponding to each coordinate point.

[0016] In one possible implementation, the candidate image information includes multiple candidate images; determining the target image information based on the candidate image information includes:

[0017] For each candidate image, determine the bounding box corresponding to the target object, and determine multiple segmentation masks within the bounding box;

[0018] Based on the multiple segmentation masks in the bounding box, determine the mask area corresponding to the candidate image;

[0019] If the ratio of the mask area to the bounding box is greater than a preset ratio, then the candidate image is determined to be the target image;

[0020] The target image information is obtained based on the determination result of each candidate image.

[0021] In one possible implementation, determining the bounding box corresponding to the target object and determining multiple segmentation masks within the bounding box includes:

[0022] The candidate image is detected using a pre-trained YOLO model to obtain the bounding box corresponding to the target object in the candidate image;

[0023] The bounding box corresponding to the target object is used as a cue input into the SAM model to obtain multiple segmentation masks in the bounding box output by the SAM model.

[0024] In one possible implementation, determining the object pose data of the target object in the robot arm coordinate system includes:

[0025] Establish an object coordinate system, wherein the directions of each coordinate axis of the object coordinate system are the same as the directions of each coordinate axis of the robotic arm coordinate system;

[0026] The robotic arm is controlled to mark the origin on the worktable, and the end of the robotic arm is controlled to draw lines based on the origin to obtain a visualized planar coordinate system;

[0027] With the target object already placed at the origin of the worktable, the object pose data of the target object in the robotic arm coordinate system is obtained based on the coordinate position of the origin in the robotic arm coordinate system; wherein the X and Y axis directions of the object coordinate system are the same as the X and Y axis directions of the planar coordinate system.

[0028] In one possible implementation, determining multiple coordinate points in the boundary of the planar region includes:

[0029] Within the boundary of the planar region, a grid is divided according to a preset sampling interval;

[0030] Based on the positions of the divided grid nodes, multiple coordinate points are generated.

[0031] In one possible implementation,

[0032] The target image information includes at least one of the following: RGB image, depth image, and mask data of the target object, bounding box of the target object, and category of the target object.

[0033] The training data includes multiple sets of sample data, each set of sample data including target image information and corresponding object pose data.

[0034] Secondly, this application provides a training data generation device based on embodied intelligence, the device comprising:

[0035] The analysis module is used to determine the object pose data of the target object in the robot arm coordinate system;

[0036] The control module is used to control the image acquisition device of the robotic arm to move sequentially to each image acquisition pose to acquire candidate image information of the target object according to the pre-generated image acquisition pose sequence.

[0037] The processing module is used to determine target image information based on the candidate image information, wherein the target image information is determined based on the area ratio of the mask of the target object in the image to the bounding box.

[0038] The generation module is used to generate training data based on the target image information and the object pose data; the training data is used to train the object pose estimation model.

[0039] In one possible implementation, the control module is also used for:

[0040] Based on the position information of the robotic arm and the worktable, the boundary of the planar area where the image acquisition device of the robotic arm moves above the target object is determined;

[0041] Determine multiple coordinate points within the boundary of the planar region;

[0042] Determine at least one candidate image acquisition pose corresponding to each coordinate point, wherein the candidate image acquisition pose includes rotation angle and / or perturbation data;

[0043] Based on at least one candidate image acquisition pose corresponding to each coordinate point, the image acquisition pose sequence is generated. The image acquisition pose sequence includes multiple coordinate points sorted by distance, and at least one candidate image acquisition pose corresponding to each coordinate point.

[0044] In one possible implementation, the candidate image information includes multiple candidate images; the processing module is specifically used for:

[0045] For each candidate image, determine the bounding box corresponding to the target object, and determine multiple segmentation masks within the bounding box;

[0046] Based on the multiple segmentation masks in the bounding box, determine the mask area corresponding to the candidate image;

[0047] If the ratio of the mask area to the bounding box is greater than a preset ratio, then the candidate image is determined to be the target image;

[0048] The target image information is obtained based on the determination result of each candidate image.

[0049] In one possible implementation, the processing module is specifically used for:

[0050] The candidate image is detected using a pre-trained YOLO model to obtain the bounding box corresponding to the target object in the candidate image;

[0051] The bounding box corresponding to the target object is used as a cue input into the SAM model to obtain multiple segmentation masks in the bounding box output by the SAM model.

[0052] In one possible implementation, the analysis module is specifically used for:

[0053] Establish an object coordinate system, wherein the directions of each coordinate axis of the object coordinate system are the same as the directions of each coordinate axis of the robotic arm coordinate system;

[0054] The robotic arm is controlled to mark the origin on the worktable, and the end of the robotic arm is controlled to draw lines based on the origin to obtain a visualized planar coordinate system;

[0055] With the target object already placed at the origin of the worktable, the object pose data of the target object in the robotic arm coordinate system is obtained based on the coordinate position of the origin in the robotic arm coordinate system; wherein the X and Y axis directions of the object coordinate system are the same as the X and Y axis directions of the planar coordinate system.

[0056] In one possible implementation, the analysis module is specifically used for:

[0057] Within the boundary of the planar region, a grid is divided according to a preset sampling interval;

[0058] Based on the positions of the divided grid nodes, multiple coordinate points are generated.

[0059] In one possible implementation, within the generation module:

[0060] The target image information includes at least one of the following: RGB image, depth image, and mask data of the target object, bounding box of the target object, and category of the target object.

[0061] The training data includes multiple sets of sample data, each set of sample data including target image information and corresponding object pose data.

[0062] Thirdly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of the first aspects.

[0063] Fourthly, this application provides an electronic device, comprising: at least one processor and a memory; wherein,

[0064] The memory stores computer-executed instructions;

[0065] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any of the first aspects.

[0066] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps of the method as described in any of the first aspects.

[0067] This application provides a training data generation method, apparatus, device, and medium based on embodied intelligence. It determines the precise pose data of a target object using a robotic arm coordinate system; based on a pre-planned image acquisition pose sequence, the robotic arm carrying an image acquisition device sequentially moves to each pose point to acquire candidate images of the target object; based on the area ratio of the target object mask to the bounding box in the candidate images, it filters out target image information that meets the requirements; finally, by combining the filtered target image information with the object pose data, it automatically generates training data for training an object pose estimation model. This method significantly reduces the cost and time of manual annotation and improves data generation efficiency through automated image acquisition and filtering by a robotic arm. Simultaneously, the robotic arm can precisely control the camera angle, effectively covering all-angle changes in complex industrial scenarios and overcoming interference from lighting, occlusion, etc., ensuring the consistency and quality of the labeled data. Furthermore, this method reduces reliance on high-cost motion capture systems and complex deployments, making it more flexible and adaptable to dynamic industrial environments, and providing efficient and reliable data support for object pose estimation models. Attached Figure Description

[0068] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0069] Figure 1 The flowchart of the training data generation method based on embodied intelligence provided in the embodiments of this application Figure 1 ;

[0070] Figure 2 The flowchart of the training data generation method based on embodied intelligence provided in the embodiments of this application Figure 2 ;

[0071] Figure 3 The flowchart of the training data generation method based on embodied intelligence provided in the embodiments of this application Figure 3 ;

[0072] Figure 4 A diagram of a training data generation device based on embodied intelligence provided in an embodiment of the present invention;

[0073] Figure 5 This is a hardware schematic diagram of an electronic device provided in an embodiment of the present invention.

[0074] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0076] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein.

[0077] In this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0078] In existing technologies, training data for 6D pose estimation models typically relies on manual annotation or high-cost motion capture systems. Manual annotation methods involve manually labeling the bounding boxes and keypoints of objects in 2D images, then combining this with depth information to infer the 3D pose. However, this process is not only time-consuming but also susceptible to subjective factors, making it difficult to guarantee annotation accuracy and consistency. While motion capture systems (such as optical capture or inertial sensors) can provide high-precision 6D pose data, their equipment is expensive, deployment is complex, and they are only suitable for laboratory environments, making them unsuitable for the dynamic and complex real-world needs of industrial scenarios. Furthermore, traditional data acquisition methods often use fixed-view cameras, failing to systematically cover the full field of view of the target object, resulting in insufficient diversity in training data and limiting the model's generalization ability. In the data cleaning stage, existing solutions typically rely on manual screening of invalid images or semi-automatic tools to generate segmentation masks, which is inefficient and yields highly subjective results.

[0079] To address the problems in existing technologies, this application provides a training data generation method based on embodied intelligence. The method involves determining the precise pose data of a target object using a robotic arm coordinate system; controlling the robotic arm to sequentially move the image acquisition device to each pose point according to a pre-planned image acquisition pose sequence, acquiring candidate images of the target object; filtering out qualified target image information based on the area ratio of the target object mask to the bounding box in the candidate images; and finally, automatically generating training data for training an object pose estimation model by combining the filtered target image information with the object pose data.

[0080] This method automates image acquisition and screening using a robotic arm, significantly reducing the cost and time of manual annotation and improving data generation efficiency. Simultaneously, the robotic arm can precisely control the camera's viewing angle, effectively covering all perspective changes in complex industrial scenarios and overcoming interference from lighting and occlusion, ensuring the consistency and quality of the labeled data. Furthermore, this method reduces reliance on high-cost motion capture systems and complex deployments, making it more flexible and adaptable to dynamic industrial environments, and providing efficient and reliable data support for object pose estimation models.

[0081] The technical solutions of this application and how they solve the aforementioned technical problems are described in detail below with specific embodiments. These specific embodiments may exist independently or in combination with each other. Identical or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0082] This embodiment provides a training data generation method based on embodied intelligence. Figure 1 The flowchart of the training data generation method based on embodied intelligence provided in the embodiments of this application Figure 1 The method includes:

[0083] S101. Determine the object pose data of the target object in the robot arm coordinate system.

[0084] In this step, the target object refers to a specific industrial part or object that needs to be identified and located. The robotic arm coordinate system is a three-dimensional spatial reference system established with the robotic arm base or a fixed joint as the origin, serving as the benchmark for spatial measurement of the entire system. The object pose data refers to the 6D pose of the target object in the robotic arm coordinate system, namely, 3D position (X, Y, Z) and 3D rotation, which fully describes the object's position and orientation in space.

[0085] For example, a target object can be grasped through teaching or visual guidance, and its pose data can be determined sequentially. During the grasping process, force / torque sensors are used to ensure a stable grasp, and the pose of the robotic arm's end effector at this point (which can be accurately calculated through forward kinematics) is used as an approximation of the object's pose. Alternatively, after grasping, the robotic arm can be controlled to move the object into the field of view of a fixed, high-precision sensor (such as a fixedly mounted industrial camera), and the object's pose data can be verified and corrected through a single, high-precision measurement.

[0086] For example, determining the object pose data of the target object in the robot arm coordinate system includes:

[0087] Establish an object coordinate system, with the coordinate axes of the object coordinate system having the same orientation as the coordinate axes of the robotic arm coordinate system;

[0088] The robotic arm is controlled to mark the origin on the worktable, and the end effector of the robotic arm is controlled to draw lines based on the origin to obtain a visualized planar coordinate system;

[0089] With the target object already placed at the origin of the worktable, the object pose data in the robot arm coordinate system is obtained based on the coordinate position of the origin in the robot arm coordinate system; where the X and Y axes of the object coordinate system are the same as the X and Y axes of the planar coordinate system.

[0090] In this example, the object coordinate system refers to a local coordinate system fixed to the target object, which can be established in 3D modeling software. Its origin is located at a specific point on the target object (such as the geometric center or feature point), and its three axes can be preset to be consistent with the directions of the robotic arm coordinate system.

[0091] Furthermore, the origin and coordinate axes can be physically calibrated on the worktable using the robotic arm's own movement, establishing a visual planar coordinate system aligned with the robotic arm's coordinate system. Specifically, a calibration tool (such as a scribing pen) is installed at the end of the robotic arm. By controlling the robotic arm to mark the origin and draw X and Y axis lines on the worktable, a reference planar coordinate system is formed. The X and Y axes of this planar coordinate system are the same as the X and Y axes of the object's coordinate system. When the target object is precisely placed at this origin, since the coordinate axes of the object's coordinate system and the robotic arm's coordinate system are aligned, the object's pose data can be directly derived from the coordinate position of the origin in the robotic arm's coordinate system. Rotation is assumed to be unit rotation (i.e., no relative rotation).

[0092] It should be noted that, to ensure that the target object is accurately placed on the origin marked by the robotic arm in the workbench, a vision camera or force sensor can be integrated into the end of the robotic arm. The offset between the object's feature points and the origin mark can be detected by visual recognition algorithms, or the contact position can be sensed by force feedback, thereby calibrating the object's position in real time. If a deviation is detected, the robotic arm can be controlled to make fine adjustments or prompt manual adjustments until the object's coordinate system is aligned with the origin of the plane coordinate system, thereby confirming the accuracy of the placement.

[0093] This example significantly reduces the reliance on expensive measuring equipment and complex calibration processes found in traditional methods. By autonomously creating a visual reference frame through a robotic arm, operators can quickly deploy and verify object positions, enhancing process repeatability and ease of operation. Furthermore, this method is highly adaptable and can flexibly respond to environmental changes in industrial settings.

[0094] S102. Based on the pre-generated image acquisition pose sequence, control the image acquisition device of the robotic arm to move sequentially to each image acquisition pose to acquire candidate image information of the target object.

[0095] In this step, the image acquisition pose sequence refers to a pre-planned set of positions and orientations defined in the robotic arm coordinate system, which the image acquisition device needs to reach, aiming to cover the target object from different perspectives. The image acquisition device can be an RGB-D camera or an industrial camera mounted on the end effector of the robotic arm, used to synchronously acquire images of the object.

[0096] For example, a virtual sphere can be assumed to surround the target object. On this sphere, a series of uniformly distributed sampling points are generated using a Fibonacci spiral or other uniform sampling algorithm. Each sampling point corresponds to an image acquisition pose, with its orientation (camera optical axis) pointing towards the center of the sphere (object center), and a random rotation around the optical axis can be added to increase the diversity of viewpoints. The robotic arm moves sequentially to these poses to capture images.

[0097] For example, a 3D model of the target object can be loaded first. Using next-best-perspective or visibility analysis algorithms from computer graphics, the image acquisition pose sequence is actively planned. The algorithm prioritizes perspectives that allow observation of more un-captured surfaces and richer features of the object, automatically avoiding locations with kinematic limitations of the robotic arm or potential collisions.

[0098] S103. Based on the candidate image information, determine the target image information. The target image information is determined based on the ratio of the area of ​​the mask of the target object in the image to the area of ​​the bounding box.

[0099] In this step, the mask refers to a binary image where white pixels (value 1) represent the region containing the target object, and black pixels (value 0) represent the background. The bounding box is the smallest outer rectangle that completely encloses the masked region of the target object in the image.

[0100] For example, for each candidate image, a mask can be quickly generated by first performing depth map thresholding or foreground / background rendering using a known object model and camera pose. Then, the minimum bounding rectangle of the mask and its area ratio are calculated. An empirical threshold (e.g., 0.6) is preset, and images with ratios below this threshold are discarded. These discarded images typically correspond to invalid viewpoints where the object is severely occluded, too far away to be clearly visible, or mostly outside the field of view.

[0101] For example, the mask-boundary area ratio can also be calculated for each image, but instead of simply discarding images using a fixed threshold, all candidate images are sorted from highest to lowest according to this ratio. Simultaneously, other quality metrics (such as image sharpness, illumination uniformity, and feature point richness) can be combined for a comprehensive score. Finally, based on the size of the required training dataset, a predetermined number of images are selected from the top-ranked images as the target image information.

[0102] S104. Generate training data based on the target image information and object pose data; the training data is used to train the object pose estimation model.

[0103] In this step, the object pose estimation model refers to a deep learning model whose input is one or more images, and whose output is the 6D pose of the target object in the image relative to the camera. Training data refers to standardized data pairs used to train the machine learning model. For 6D pose estimation, this can include, but is not limited to: input (RGB image, depth image, etc.) and ground truth labels (the 6D pose of the object relative to the camera).

[0104] For example, the object pose data determined in the robotic arm coordinate system in S101 can be used as ground truth labels and directly paired with the target images selected in S103 to form image-ground truth pose data pairs. During this process, the generated data can also be associated with and contain other key information, such as camera intrinsics, the robotic arm end-effector pose at the time of acquisition, and the object mask obtained through rendering or segmentation. Subsequently, this raw data undergoes a series of standardization processes, including image resizing, depth map alignment, color normalization, and data augmentation to enhance the model's generalization ability (such as adding noise, simulating occlusion, and background replacement), ultimately forming a uniformly formatted dataset that can be directly used for model training.

[0105] For example, the target image information includes at least one of RGB image, depth image, and mask data of the target object, bounding box of the target object, and category of the target object;

[0106] The training data includes multiple sets of sample data, each set of sample data including target image information and corresponding object pose data.

[0107] In this example, the RGB image provides information about the object's color and texture, serving as the primary input for feature extraction and initial recognition. The depth image encodes the object's 3D geometry and spatial distance information, helping the model understand the object's spatial structure. The target object's mask data is used to accurately segment foreground objects in the image, eliminating complex background interference and ensuring the model focuses on learning the object's own features. Bounding boxes provide a rough location of the object in the image, used for initialization or to assist detection. Object category information can be used for identification and differentiation in multi-object scenes.

[0108] The embodied intelligence-based training data generation method provided in this embodiment determines the precise pose data of the target object through a robotic arm coordinate system. Based on a pre-planned image acquisition pose sequence, the robotic arm carrying the image acquisition device moves sequentially to each pose point to acquire candidate images of the target object. Based on the area ratio of the target object mask to the bounding box in the candidate images, the method filters out the target image information that meets the requirements. Finally, by combining the filtered target image information with the object pose data, training data for training the object pose estimation model is automatically generated. This method significantly reduces the cost and time of manual annotation and improves data generation efficiency through automated image acquisition and filtering by a robotic arm. Simultaneously, the robotic arm can precisely control the camera angle, effectively covering all-angle changes in complex industrial scenarios and overcoming interference from lighting, occlusion, etc., ensuring the consistency and quality of the labeled data. Furthermore, this method reduces reliance on high-cost motion capture systems and complex deployments, making it more flexible and adaptable to dynamic industrial environments, and providing efficient and reliable data support for the object pose estimation model.

[0109] This embodiment provides a training data generation method based on embodied intelligence. Figure 2 The flowchart of the training data generation method based on embodied intelligence provided in the embodiments of this application Figure 2 .like Figure 2 As shown, in this embodiment... Figure 1 Based on the embodiments, the process of generating the image acquisition pose sequence is described in detail. This method includes:

[0110] S201. Based on the position information of the robotic arm and the worktable, determine the boundary of the planar area where the image acquisition device of the robotic arm moves above the target object.

[0111] In this step, for example, a fixed rectangular boundary can be manually defined based on the center of the workbench or the object's placement point, according to the maximum reach of the robotic arm, the physical dimensions of the workbench, and a pre-set safety margin. For instance, it could be set as an area extending 20 centimeters around the object. Alternatively, the 3D models of the robotic arm, workbench, target object, and known obstacles can be imported into the robot simulation environment. Using motion planning algorithms, all possible positions in 3D space where the robotic arm's end-effector camera can safely reach and observe the object are calculated. These positions are then projected onto a horizontal plane, and their convex hull or minimum bounding rectangle is automatically calculated as the boundary of the planar region.

[0112] For example, based on the relative positions of the robotic arm and the worktable, as well as the installation pose of the image acquisition device at the end effector, the image acquisition device is first adjusted to an angle approximately perpendicular to the material surface. Subsequently, based on the approximate distribution area of ​​the target object in the field of view of the image acquisition device (such as upper left, lower right, etc.), the movement boundaries of the robotic arm end effector in the X and Y axes of the robot coordinate system are determined, denoted as X∈[X_min, X_max], Y∈[Y_min, Y_max].

[0113] S202. Determine multiple coordinate points in the boundary of the planar region.

[0114] For example, determining multiple coordinate points in the boundary of a planar region includes:

[0115] Within the boundaries of the planar region, a grid is divided according to a preset sampling interval;

[0116] Based on the positions of the grid nodes after division, multiple coordinate points are generated.

[0117] For example, within a defined X and Y axis movement range (i.e., the planar region boundary), a uniform grid is created according to a set sampling interval (e.g., 5 mm). Based on the positions of the grid nodes, a series of target points in the XY plane are generated for the robotic arm end effector. For example, dividing a 20 cm × 20 cm area at 5 mm intervals will yield 400 different planar sampling positions.

[0118] This example ensures uniform coverage of the horizontal area, with regular and easy-to-generate sampling points, and can systematically obtain the head-up view of the top surface and various sides of the object.

[0119] Alternatively, coordinate points can be selected selectively based on the target object's 3D shape and feature distribution. For example, first obtain a rough 3D outline of the object, then increase the density of sampling points above the object's outline edges, corners, or areas with rich features; and reduce the number of sampling points above flat or featureless areas on the top of the object.

[0120] S203. Determine at least one candidate image acquisition pose corresponding to each coordinate point, wherein the candidate image acquisition pose includes rotation angle and / or perturbation data.

[0121] In this step, the rotation angle refers to the orientation of the image acquisition device after reaching a certain horizontal coordinate point, allowing the device to view the target object from different angles. The perturbation data refers to small, random variations added to the preset standard image acquisition pose (e.g., position offset ±5mm, angle offset ±3 degrees). The purpose is to introduce subtle differences in perspective at identical logical points, thereby increasing the diversity of the acquired image data, simulating minor vibrations or positioning errors in the real environment, and helping to improve the robustness of the model.

[0122] For example, a set of fixed viewing angles can be preset for each horizontal coordinate point. For instance, three standard pitch angles can be set (e.g., 30 degrees, 60 degrees, and 90 degrees overhead), each paired with 2-3 different yaw angles. For each such standard pose, a small random perturbation conforming to a normal distribution is added to its position and angle before execution. For example, the 360° omnidirectional angle can be divided into six evenly distributed yaw angles at fixed intervals (e.g., 60°). At each yaw angle, a small random perturbation is applied to the six degrees of freedom pose (including X, Y, Z, Rx, Ry, and Rz) at the end effector of the robotic arm; for example, eight random perturbations are generated based on each fixed pose. Based on this, all photographic poses can be automatically planned. For example, 400 planar positions multiplied by 6 rotation angles and then multiplied by 8 random perturbations can ultimately generate 19,200 photographic poses.

[0123] For example, the algorithm can also take a 3D model of the target object as input, and search within a spherical space above the vertical line corresponding to the coordinate point to evaluate the quality of object features (such as visible surface area, contour complexity, and number of feature points) seen by the image acquisition device at different rotation angles. The N angles with the highest evaluation scores (such as the top 3) are selected as the main image acquisition poses for the coordinate point, and the optimal pose can be slightly perturbed to generate a copy.

[0124] S204. Based on at least one candidate image acquisition pose corresponding to each coordinate point, generate an image acquisition pose sequence. The image acquisition pose sequence includes multiple coordinate points sorted by distance, and at least one candidate image acquisition pose corresponding to each coordinate point.

[0125] In this step, for example, starting from the initial position of the robotic arm (or the first acquisition point), among all unsorted coordinate points, the point with the closest Euclidean distance to the current point can be selected as the next target point, and all its corresponding image acquisition poses can be added to the sequence. This process is repeated until all points have been traversed. Alternatively, an optimization algorithm can be used to find the shortest path that traverses all coordinate points and returns to the starting point.

[0126] S205. Based on the image acquisition pose sequence, control the image acquisition device of the robotic arm to move sequentially to each image acquisition pose to acquire candidate image information of the target object.

[0127] This step is the same as S102, and will not be repeated here.

[0128] The embodied intelligence-based training data generation method provided in this embodiment can significantly optimize the motion path of the robotic arm, greatly shorten the total data acquisition time, and improve overall efficiency by pre-defining the boundaries of the planar region and generating image acquisition pose sequences based on distance sorting. At the same time, by systematically selecting multiple coordinate points within the boundary and configuring image acquisition poses containing different rotation angles and perturbation data for each point, it can ensure the acquisition of high-quality images that cover the full field of view of the target object and have rich diversity, thereby effectively improving the robustness and generalization ability of the pose estimation model obtained from subsequent training.

[0129] This embodiment provides a training data generation method based on embodied intelligence. Figure 3 The flowchart of the training data generation method based on embodied intelligence provided in the embodiments of this application Figure 3 .like Figure 3 As shown, in this embodiment... Figure 2 Based on the embodiments, the process of determining target image information is described in detail. This method includes:

[0130] S301. Determine the object pose data of the target object in the coordinate system of the robotic arm, and control the image acquisition device of the robotic arm to move sequentially to each image acquisition pose to acquire candidate image information of the target object; the candidate image information includes multiple candidate images.

[0131] This step is the same as S101 and S102, and will not be repeated here.

[0132] S302. For each candidate image, determine the bounding box corresponding to the target object, and determine multiple segmentation masks within the bounding box.

[0133] For example, determining the bounding box corresponding to the target object, and determining multiple segmentation masks within the bounding box, including:

[0134] The candidate image is detected by a pre-trained YOLO model to obtain the bounding box corresponding to the target object in the candidate image;

[0135] The bounding box corresponding to the target object is used as a cue input into the SAM model to obtain multiple segmentation masks in the bounding box output by the SAM model.

[0136] In this example, the YOLO model refers to a single-stage object detection model that treats object detection as a regression problem, directly predicting the bounding boxes and categories of objects in an image in a single forward propagation. In this example, an open-source, general-purpose pre-trained YOLO model can be used, or it can be fine-tuned based on a domain-adaptive approach. Specifically, based on the aforementioned general-purpose pre-trained model, transfer learning can be performed using a small number of self-collected labeled images containing target objects (only bounding boxes and categories are needed). Fine-tuning allows the model to focus on recognizing the appearance features of specific industrial objects.

[0137] The SAM (Segment Anything Model) is a general-purpose visual segmentation model. Its core feature is its ability to perform high-quality segmentation of corresponding regions in an image based on input cues (such as points, boxes, or text), strong zero-shot capability, and the ability to generate fine-grained and diverse segmentation masks. In this example, the bounding boxes detected by YOLO are used as spatial cues, telling the SAM model to segment the objects within those boxes.

[0138] This example eliminates the need to train a segmentation model for specific objects, significantly reducing the technical threshold and deployment costs. At the same time, by leveraging YOLO's localization capabilities to guide SAM, the targeting and accuracy of mask generation are ensured, thus providing reliable and precise input for subsequent image screening steps based on mask area. Overall, this significantly improves the automation level and quality of training data generation.

[0139] For example, the known pose of the object in the robotic arm coordinate system in S301, and the precise pose of the image acquisition device when acquiring the current image, can be used to obtain the 6D pose of the object relative to the image acquisition device through coordinate system transformation. Then, using computer graphics technology, the 3D model of the object is rendered into the virtual camera based on this relative pose, thereby directly generating a precise and ideal object mask that perfectly matches the current image viewpoint. Simultaneously, this rendering process can naturally provide a minimum bounding rectangle that perfectly matches the mask and tightly encloses the object as a bounding box.

[0140] S303. Determine the mask area corresponding to the candidate image based on the multiple segmentation masks in the bounding box.

[0141] In this step, for example, when S302 generates multiple segmentation mask candidates (e.g., due to the model predicting multiple segments, or the object being broken due to occlusion), the pixel area of ​​each individual mask is first calculated. Then, assuming the effective object region is the largest continuous part, all independent mask regions are identified through a connected component analysis algorithm, and the connected component with the largest pixel area is directly selected as the final effective mask. Its area is then determined as the mask area of ​​the image.

[0142] Alternatively, when multiple mask candidates exist, instead of directly discarding small-area masks, the spatial distribution of these masks is first analyzed. If they are adjacent to each other and may belong to different parts of the same object (e.g., segmentation caused by reflection or slight occlusion), morphological operations (such as closing operations) or distance-based clustering methods are used to fuse these fragmented masks into a single, unified mask. Subsequently, the total pixel area of ​​this fused mask is calculated as the mask area.

[0143] S304. Determine whether the ratio of the mask area to the bounding box is greater than a preset ratio.

[0144] S305. If not, the candidate image is determined to be an invalid image.

[0145] S306. If so, then the candidate image is determined as the target image.

[0146] S307. Based on the determination result of each candidate image, obtain the target image information.

[0147] S308. Generate training data based on the target image information and object pose data; the training data is used to train the object pose estimation model.

[0148] For example, a globally applicable fixed threshold (e.g., 0.8) can be set in advance based on experience or experimentation. Alternatively, a dynamic threshold can be used, where the preset ratio is not fixed but dynamically adjusted based on the object's known physical dimensions and the estimated theoretical size of the object in the image from the current camera distance, allowing the judgment standard to adapt to different shooting distances. Furthermore, a phased judgment approach can be adopted: first, a low ratio (e.g., 0.2) is set for initial screening, excluding only completely invalid images; then, the images that pass the initial screening are comprehensively scored and ranked based on other quality indicators (e.g., mask shape compactness, image sharpness); finally, a certain percentage of the top-ranked images are selected as valid target images.

[0149] During the generation of training data, the object pose data, target image, mask, bounding box and other data obtained above can be precisely aligned and integrated to ensure that each sample contains complete six types of data elements: RGB image, depth image, target object mask data, target object bounding box and target object category.

[0150] Furthermore, standardized data storage structures and naming rules can be established for the aforementioned data elements, specifying the file format, storage path, and metadata recording method for each data element to ensure that all training samples adhere to unified organizational standards. Then, an automated script batch-packages the aligned multimodal data according to the established specifications, generating a training dataset directly usable for model training. During the packaging process, data integrity and consistency checks are performed to ensure that each training sample contains all necessary data elements in the correct format. The final output is a training dataset with a unified structure and controllable quality, fully compatible with the input requirements of mainstream pose estimation models, providing high-quality, standardized data support for subsequent model training.

[0151] The training data generation method based on embodied intelligence provided in this embodiment avoids the reliance on expensive manual annotation and high-cost motion capture systems, greatly reducing the cost and time of data acquisition. It can also generate a large-scale training dataset with comprehensive viewpoint coverage and consistent and accurate annotation, thereby effectively solving the annotation problem caused by lighting, occlusion and other factors in industrial scenarios, and laying a solid foundation for training a high-precision and robust object pose estimation model.

[0152] This embodiment also provides a training data generation device based on embodied intelligence. Figure 4 A diagram of a training data generation device based on embodied intelligence provided in an embodiment of the present invention is shown, as follows: Figure 4 As shown, the training data generation device 40 includes:

[0153] Analysis module 401 is used to determine the object pose data of the target object in the robot arm coordinate system;

[0154] The control module 402 is used to control the image acquisition device of the robotic arm to move sequentially to each image acquisition pose to acquire candidate image information of the target object according to the pre-generated image acquisition pose sequence.

[0155] The processing module 403 is used to determine the target image information based on the candidate image information. The target image information is determined based on the ratio of the area of ​​the mask of the target object in the image to the area of ​​the bounding box.

[0156] The generation module 404 is used to generate training data based on the target image information and object pose data; the training data is used to train the object pose estimation model.

[0157] In one possible implementation, the control module 402 is also used for:

[0158] Based on the position information of the robotic arm and the worktable, determine the boundary of the planar area where the image acquisition device of the robotic arm moves above the target object;

[0159] Determine multiple coordinate points within the boundary of a planar region;

[0160] Determine at least one candidate image acquisition pose for each coordinate point, the candidate image acquisition pose including rotation angle and / or perturbation data;

[0161] Based on at least one candidate image acquisition pose corresponding to each coordinate point, an image acquisition pose sequence is generated. The image acquisition pose sequence includes multiple coordinate points sorted by distance, and at least one candidate image acquisition pose corresponding to each coordinate point.

[0162] In one possible implementation, the candidate image information includes multiple candidate images; the processing module 403 is specifically used for:

[0163] For each candidate image, determine the bounding box corresponding to the target object, and determine multiple segmentation masks within the bounding box;

[0164] Based on multiple segmentation masks within the bounding box, determine the mask area corresponding to the candidate image;

[0165] If the ratio of the mask area to the bounding box is greater than a preset ratio, then the candidate image is determined to be the target image;

[0166] Based on the determination result of each candidate image, the target image information is obtained.

[0167] In one possible implementation, the processing module 403 is specifically used for:

[0168] The candidate image is detected by a pre-trained YOLO model to obtain the bounding box corresponding to the target object in the candidate image;

[0169] The bounding box corresponding to the target object is used as a cue input into the SAM model to obtain multiple segmentation masks in the bounding box output by the SAM model.

[0170] In one possible implementation, the analysis module 401 is specifically used for:

[0171] Establish an object coordinate system, with the coordinate axes of the object coordinate system having the same orientation as the coordinate axes of the robotic arm coordinate system;

[0172] The robotic arm is controlled to mark the origin on the worktable, and the end effector of the robotic arm is controlled to draw lines based on the origin to obtain a visualized planar coordinate system;

[0173] With the target object already placed at the origin of the worktable, the object pose data in the robot arm coordinate system is obtained based on the coordinate position of the origin in the robot arm coordinate system; where the X and Y axes of the object coordinate system are the same as the X and Y axes of the planar coordinate system.

[0174] In one possible implementation, the analysis module 401 is specifically used for:

[0175] Within the boundaries of the planar region, a grid is divided according to a preset sampling interval;

[0176] Based on the positions of the grid nodes after division, multiple coordinate points are generated.

[0177] In one possible implementation, in generation module 404:

[0178] The target image information includes at least one of the following: RGB image, depth image, and mask data of the target object, bounding box of the target object, and category of the target object.

[0179] The training data includes multiple sets of sample data, each set of sample data including target image information and corresponding object pose data.

[0180] This embodiment provides a training data generation device based on embodied intelligence, which can execute the training data generation method based on embodied intelligence provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0181] Figure 5 This is a hardware schematic diagram of an electronic device provided in an embodiment of the present invention. For example... Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. The device 50 also includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0182] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above method.

[0183] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0184] In the above Figure 5In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0185] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0186] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0187] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described above.

[0188] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0189] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0190] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0191] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0192] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0193] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0194] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0195] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating training data based on embodied intelligence, characterized in that, The method includes: Establish an object coordinate system, wherein the directions of each coordinate axis of the object coordinate system are the same as the directions of each coordinate axis of the robotic arm coordinate system; The robotic arm is controlled to mark the origin on the worktable, and the end of the robotic arm is controlled to draw lines based on the origin to obtain a visualized planar coordinate system; With the target object already placed at the origin of the worktable, the object pose data of the target object in the robotic arm coordinate system is obtained based on the coordinate position of the origin in the robotic arm coordinate system; wherein, the X and Y axis directions of the object coordinate system are the same as the X and Y axis directions of the planar coordinate system. Based on the pre-generated image acquisition pose sequence, the image acquisition device of the robotic arm is controlled to move sequentially to each image acquisition pose to acquire candidate image information of the target object; Based on the candidate image information, target image information is determined, which is determined based on the area ratio of the mask of the target object in the image to the bounding box. Training data is generated based on the target image information and the object pose data; the training data is used to train the object pose estimation model. Before the image acquisition device controlling the robotic arm moves sequentially to each image acquisition pose to acquire candidate image information of the target object, it further includes: Based on the position information of the robotic arm and the worktable, the boundary of the planar area where the image acquisition device of the robotic arm moves above the target object is determined; Determine multiple coordinate points within the boundary of the planar region; Determine at least one candidate image acquisition pose corresponding to each coordinate point, wherein the candidate image acquisition pose includes rotation angle and / or perturbation data; Based on at least one candidate image acquisition pose corresponding to each coordinate point, the image acquisition pose sequence is generated. The image acquisition pose sequence includes multiple coordinate points sorted by distance, and at least one candidate image acquisition pose corresponding to each coordinate point.

2. The method according to claim 1, characterized in that, The candidate image information includes multiple candidate images; determining the target image information based on the candidate image information includes: For each candidate image, determine the bounding box corresponding to the target object, and determine multiple segmentation masks within the bounding box; Based on the multiple segmentation masks in the bounding box, determine the mask area corresponding to the candidate image; If the ratio of the mask area to the bounding box is greater than a preset ratio, then the candidate image is determined to be the target image; The target image information is obtained based on the determination result of each candidate image.

3. The method according to claim 2, characterized in that, The step of determining the bounding box corresponding to the target object and determining multiple segmentation masks within the bounding box includes: The candidate image is detected using a pre-trained YOLO model to obtain the bounding box corresponding to the target object in the candidate image; The bounding box corresponding to the target object is used as a cue input into the SAM model to obtain multiple segmentation masks in the bounding box output by the SAM model.

4. The method according to claim 1, characterized in that, Determining multiple coordinate points in the boundary of the planar region includes: Within the boundary of the planar region, a grid is divided according to a preset sampling interval; Based on the positions of the divided grid nodes, multiple coordinate points are generated.

5. The method according to claim 1, characterized in that, The target image information includes at least one of the following: RGB image, depth image, and mask data of the target object, bounding box of the target object, and category of the target object. The training data includes multiple sets of sample data, each set of sample data including target image information and corresponding object pose data.

6. A training data generation device based on embodied intelligence, characterized in that, The device includes: The analysis module is used to establish an object coordinate system, wherein the coordinate axes of the object coordinate system are in the same direction as the coordinate axes of the robotic arm coordinate system; it controls the robotic arm to calibrate the origin on the worktable and controls the end effector of the robotic arm to draw lines based on the origin, thereby obtaining a visualized planar coordinate system; with the target object already placed at the origin on the worktable, it obtains the object pose data of the target object in the robotic arm coordinate system based on the coordinate position of the origin in the robotic arm coordinate system; wherein the X and Y axes of the object coordinate system are in the same direction as the X and Y axes of the planar coordinate system. The control module is used to control the image acquisition device of the robotic arm to move sequentially to each image acquisition pose to acquire candidate image information of the target object according to the pre-generated image acquisition pose sequence. The processing module is used to determine target image information based on the candidate image information, wherein the target image information is determined based on the area ratio of the mask of the target object in the image to the bounding box. The generation module is used to generate training data based on the target image information and the object pose data; the training data is used to train the object pose estimation model. The generation module is further configured to, before the image acquisition device of the controlled robotic arm moves sequentially to each image acquisition pose to acquire candidate image information of the target object, determine the boundary of the planar region where the image acquisition device of the robotic arm moves above the target object based on the position information of the robotic arm and the worktable; determine multiple coordinate points in the planar region boundary; determine at least one candidate image acquisition pose corresponding to each coordinate point, the candidate image acquisition pose including rotation angle and / or perturbation data; and generate the image acquisition pose sequence based on the at least one candidate image acquisition pose corresponding to each coordinate point, the image acquisition pose sequence including multiple coordinate points sorted by distance, and at least one candidate image acquisition pose corresponding to each coordinate point.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5.

8. An electronic device, characterized in that, include: At least one processor and memory; wherein, The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1-5.

Images