A model training data generation method and system

Abstract

The application provides a model training data generation method and system, which comprises the following steps: determining a static scene based on a static scene dataset; sampling a target region in the static scene to determine a target position; the target position is a collision-free motion trajectory planning end point; determining a free space position by using an initial escape mechanism according to a starting position in the static scene; the free space position is a collision-free motion trajectory planning start point of the escape starting position; and performing active obstacle avoidance planning according to the starting position, the free space position and the target position to generate model training motion planning trajectory data. The application can quickly and efficiently generate high-quality motion planning trajectory data, and provide rich spatial information and operation constraints for the model. This not only reduces the data acquisition cost, but also improves the speed and diversity of data generation, and helps to improve the spatial trajectory understanding ability and task execution efficiency of the robot in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for generating model training data. Background Technology

[0002] In the fields of embodied intelligence and robotics, robots need to understand complex spatial commands and generate corresponding motion trajectories. For robots to accurately complete tasks, their underlying models require strong spatial trajectory understanding capabilities. This not only requires the model to understand the semantic information in commands but also to translate this semantic information into concrete, physically-compliant motion trajectories. Trajectory training data is a key factor in training the spatial trajectory understanding capabilities of robot models. High-quality trajectory data provides the model with rich spatial information and operational constraints, helping it learn how to safely and efficiently generate operating trajectories in complex environments. However, existing model training datasets rely on real robot data collection or handwritten simulation environments, which are costly, slow, and lack diversity. Summary of the Invention

[0003] This invention provides a method and system for generating model training data, addressing the shortcomings of existing datasets in embodied intelligence and robotics fields, namely high cost, slow speed, and poor diversity. This invention can quickly and efficiently generate high-quality motion planning trajectory data, providing models with rich spatial information and operational constraints. This not only reduces data acquisition costs but also improves the speed and diversity of data generation, contributing to enhanced spatial trajectory understanding and task execution efficiency of robots in complex environments.

[0004] This invention provides a method for generating model training data, comprising: determining a static scene based on a static scene dataset; sampling a target region in the static scene to determine a target position; the target position being the endpoint of a collision-free motion trajectory planning; determining a free space position based on an initial escape mechanism according to a starting position in the static scene; the free space position being the starting point of a collision-free motion trajectory planning that escapes from the starting position; and performing active obstacle avoidance planning based on the starting position, the free space position, and the target position to generate model training motion planning trajectory data.

[0005] According to a model training data generation method provided by the present invention, after determining the static scene based on the static scene dataset, the method further includes: initializing the static scene; the scene initialization method includes gravity alignment, bounding box generation, and object role assignment.

[0006] According to a model training data generation method provided by the present invention, the step of sampling the target region in the static scene to determine the target position includes: determining the target region based on a reference object in the static scene; determining multiple candidate points based on the target region using an inside-out polar coordinate sampling strategy; performing static collision detection on each candidate point, and selecting a collision-free point as the target position.

[0007] According to a model training data generation method provided by the present invention, the step of determining the free space position based on the initial position in the static scene using an initial escape mechanism includes: calculating an initial escape vector based on the initial position in the static scene using visual opening analysis or geometric push-back; at the initial position, the moving object is in a collision state; the initial escape vector is the minimum translation vector that removes the moving object from the collision; determining the free space position based on the initial position and the initial escape vector; wherein, the visual opening analysis is an analysis and calculation method for escaping along the direction of the visual opening in free space, and the geometric push-back is a calculation method for escaping by retreating based on the geometric penetration depth.

[0008] According to a model training data generation method provided by the present invention, the step of actively planning obstacle avoidance based on the starting position, the free space position, and the target position to generate model training motion planning trajectory data includes: planning an escape path using a first path planning algorithm based on the starting position and the free space position; the first path planning algorithm is used to plan a path from the starting position to the free space position based on the initial escape vector; planning a motion path using a second path planning algorithm based on the free space position and the target position; the second path planning algorithm is used to determine the path positions of the motion and plan a path from the free space position through the path positions to the target position; and generating the model training motion planning trajectory data based on the escape path and the motion path.

[0009] According to a model training data generation method provided by the present invention, the motion path is constrained by a cost function during planning; the cost function is a function determined by weighted summation of total path length, corner penalty, reverse movement penalty, and lateral offset penalty.

[0010] According to a model training data generation method provided by the present invention, the method further includes: determining original semantic instructions based on the model training motion planning trajectory data; performing retrospective semantic discovery based on the model training motion planning trajectory data to determine instruction enhancement elements; the instruction enhancement elements include potential path locations, motion modes, units, and relative running directions; and adding the instruction enhancement elements to the original semantic instructions to obtain semantically enhanced instructions.

[0011] The method for generating model training data according to the present invention further includes: correcting the model training motion planning trajectory data; the correction methods include geometric smoothing, sparsification, physical landing correction and visual alignment correction.

[0012] According to a model training data generation method provided by the present invention, the method further includes: determining the three-dimensional coordinates required for the robot to perform the task based on the model training motion planning trajectory data; obtaining the depth information of the grasping position based on the three-dimensional coordinates; and generating model training object grasping task data according to the three-dimensional coordinates and the depth information.

[0013] According to a model training data generation method provided by the present invention, the step of generating model training object grasping task data based on three-dimensional coordinates and depth information includes: updating the three-dimensional coordinates according to the depth information; determining the grasping posture of the robot to perform the task based on the updated three-dimensional coordinates; and generating the model training object grasping task data based on the grasping posture and the updated three-dimensional coordinates.

[0014] This invention also provides a model training data generation system, comprising: a scene determination module for determining a static scene based on a static scene dataset; a target position determination module for sampling a target region in the static scene to determine a target position; the target position being the endpoint of a collision-free motion trajectory planning; a free space position determination module for determining a free space position based on an initial escape mechanism according to a starting position in the static scene; the free space position being the starting point of a collision-free motion trajectory planning that escapes from the starting position; and a data generation module for performing active obstacle avoidance planning based on the starting position, the free space position, and the target position to generate model training motion planning trajectory data.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the model training data generation method as described above.

[0016] This invention provides a method and system for generating model training data, which can quickly and efficiently generate high-quality motion planning trajectory data, providing the model with rich spatial information and operational constraints. This not only reduces data acquisition costs but also improves the speed and diversity of data generation, helping to enhance the robot's spatial trajectory understanding and task execution efficiency in complex environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a method for generating model training data provided by the present invention.

[0019] Figure 2 This is a schematic diagram illustrating the principle of a model training data generation method provided by the present invention.

[0020] Figure 3 This is a schematic diagram illustrating the specific process of a model training data generation method provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the structure of a model training data generation system provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Embodied intelligence is a cutting-edge field at the intersection of artificial intelligence and robotics. It emphasizes that intelligent agents achieve autonomous learning and evolution through dynamic interaction between their bodies and the environment, with its core being the deep integration of perception, action, and cognition. Currently, visual language models (VLMs) are typically used to enable embodied robots to interact with the three-dimensional physical world.

[0025] In the fields of embodied intelligence and robotic manipulation, robots need to understand complex spatial instructions (such as "move the cup 20 centimeters to the right and avoid the vase") and generate corresponding motion trajectories. Currently, the data sources for training such models mainly rely on real robot data collection, which, while containing realistic trajectories, is extremely costly. Alternatively, handwritten simulation environments can be used, but these lack diversity.

[0026] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a method for generating model training data provided by the present invention.

[0027] This invention provides a method for generating model training data, comprising: 101: Determine the static scene based on the static scene dataset; 102: Sample the target area in a static scene to determine the target location; the target location is the endpoint of the collision-free motion trajectory planning; 103: Based on the initial position in the static scene, the free space position is determined using an initial escape mechanism; the free space position is the starting point for planning the collision-free motion trajectory from the escape initiation position. 104: Based on the starting position, free space position, and target position, perform active obstacle avoidance planning to generate model training motion planning trajectory data.

[0028] This invention provides a method for generating model training data. This method combines static 3D scanning datasets and advanced motion planning techniques to generate high-quality motion planning trajectory data, which is used to train the spatial trajectory understanding ability of robot models.

[0029] Specifically, a suitable scene is first selected from a static 3D scanning dataset. These datasets typically contain detailed 3D geometric information of indoor environments, such as the ScanNet or CA-1M datasets. These datasets provide point clouds, meshes, and object annotation information for the scene. For example, a living room scene containing a table, chairs, and cups can be selected from the ScanNet dataset. The annotation information can be the precise coordinates of the objects and their specific dimensions (length, width, and height). Exemplarily, annotation can be performed using existing automatic annotation tools or manually; this invention does not limit the specific annotation method.

[0030] Within a selected static scene, a target region is defined, which is the final target location for the robot's operations. A collision-free point within the target region is selected as the target location using random sampling or a rule-based method. For example, the target region could be the surface of a table, and the target location could be the final placement of a cup. During sampling, ensure that the target location maintains a safe distance from surrounding objects to avoid collisions.

[0031] Starting from a position in a static scene, an initial escape mechanism is used to determine the free space position. This mechanism analyzes the environment around the starting position to find a collision-free free space location as the starting point for trajectory planning. For example, using visual ray casting, rays are emitted from the starting position in multiple directions, and the intersections of these rays with surrounding objects are detected. The direction with the furthest intersection is selected as the escape direction, and the starting position is moved to a safe location in that direction.

[0032] It should be noted that the steps of determining the target location and determining the free space location can be performed simultaneously or sequentially. This invention does not impose any particular limitation on the order of these steps.

[0033] Active obstacle avoidance planning is performed by combining the starting position, free space position, and target position. A sampling-based path planning algorithm (such as...) is used. The process generates collision-free motion trajectories. During planning, cost functions are introduced to optimize the smoothness and naturalness of the trajectory. For example, cost functions may include total path length, corner penalties, backtracking penalties, and lateral offset penalties. Through these cost functions, the generated trajectory is not only collision-free but also conforms to human operating habits.

[0034] Ultimately, the generated motion planning trajectory data includes a series of three-dimensional coordinate points that describe a collision-free path from the starting position to the target position. This data can be used to train robot models, enabling them to understand and generate complex motion trajectories. For example, the generated trajectory data can be used to train a visual language model (VLM), allowing it to generate corresponding motion trajectories based on natural language instructions.

[0035] This invention utilizes a massive static indoor 3D scan dataset to automatically generate millions of labeled dynamic trajectories. This achieves "transforming static into dynamic," obtaining massive training data without the need for physical hardware investment.

[0036] The method of this invention is also applicable to scenarios based on NeRF (Neural Radiation Field) or 3D Gaussian Splatting reconstruction. Simply replace explicit AABB / OBB detection with implicit field density-based collision detection in the collision detection stage.

[0037] Please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the principle of a model training data generation method provided by the present invention.

[0038] Please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the specific process of a model training data generation method provided by the present invention.

[0039] As a preferred embodiment, after determining the static scene based on the static scene dataset, the method further includes: scene initialization of the static scene; the scene initialization method includes gravity alignment, bounding box generation, and object role assignment.

[0040] In this embodiment, scene initialization is required after the static scene is determined.

[0041] Using gravity alignment matrix Standardize the coordinates of all objects in the scene to ensure that the "up" direction aligns with physical intuition. Specifically, calculate the global gravity direction vector of the scene and construct a gravity alignment matrix. This matrix transforms the coordinates of all objects in the scene into a coordinate system aligned with the global gravity direction. Applications Perform coordinate transformation on each object in the scene to ensure that all objects are in the same "up" direction.

[0042] Of course, it also filters out objects with degenerate bounding boxes or containing anomalous values.

[0043] Considering that 3D datasets like ScanNet often lack 2D bounding box annotations, in order for VLM to understand the image, the 3D bounding box must be accurately projected back into 2D. This solves the problem of inaccurate occlusion handling in 3D scan data from a 2D perspective, leading to excessively large bounding boxes or the inclusion of invisible areas. Specifically, a point sampling strategy is adopted, instead of directly projecting the eight corner points of the 3D bounding box (because the object is non-convex), a large number of points (e.g., 5000 points) are uniformly sampled on the surface of the 3D bounding box. Furthermore, depth consistency filtering is performed; after projecting the 3D points onto the 2D image plane, the depth of the projected points is compared. Depth of the corresponding pixel in the depth map Only when Only when the point is selected is it preserved, thus eliminating occluded points. A compact 2D bounding box is generated based on the minimum / maximum coordinates of the preserved valid projected points.

[0044] To generate clear instructions, the objects in the scene are divided into four categories (object role assignment): Moving object (O) src : Select from the "high-quality object subset" (objects with rich VLM generation descriptions) as the objects to be operated on.

[0045] Reference object (O) ref : Used to define a reference point for the target location (e.g., the table "placed on the table").

[0046] Obstacles (O) obs : Any object in the scene that may block the path.

[0047] Passing objects (O)via ): The object that needs to be bypassed when planning an obstacle avoidance task.

[0048] As a preferred embodiment, target region sampling is performed in a static scene to determine the target location, including: determining the target region based on reference objects in the static scene; determining multiple candidate points based on the target region using an inside-out polar coordinate sampling strategy; performing static collision detection on each candidate point and selecting a collision-free point as the target location.

[0049] To find a safe placement location in a cluttered scene, in this embodiment, a target area is determined based on a reference object within a selected static scene. The target area can be the reference object (O). ref The area near the reference object (e.g., centered on the reference object) is used to place the target object.

[0050] Based on the target region, a polar coordinate sampling strategy from the inside out is adopted to determine multiple candidate points. Specifically, the geometric center of the target region is selected as the origin of the polar coordinate sampling. A series of concentric circle radii (e.g., 0.03m, 0.06m, etc.) and angular step sizes are set. Candidate points are iteratively sampled from the inner circle to the outer circle. The height is adjusted according to the height of the target object. Static collision detection is performed on each candidate point, and the first collision-free point is selected as the target location. Prioritize locations close to the center and away from the edges where there is a risk of falling, to improve placement stability.

[0051] As a preferred embodiment, the free space position is determined based on the initial position in the static scene using an initial escape mechanism, including: calculating the initial escape vector based on the initial position in the static scene using visual opening analysis or geometric push-back; at the initial position, the moving object is in a collision state; the initial escape vector is the minimum translation vector that removes the moving object from the collision; the free space position is determined based on the initial position and the initial escape vector; wherein, visual opening analysis is an analysis and calculation method for escaping along the direction of the visual opening in free space, and geometric push-back is a calculation method for escaping by retreating based on the geometric penetration depth.

[0052] Considering the mesh noise that often exists in 3D scanning reconstruction, causing slight "soft collisions" (interpenetration) between objects and the environment at the initial position, this embodiment first "escapes" the object into free space and then plans it through visual ray detection and geometric push calculation. This makes motion planning possible in a cluttered, low-quality scan mesh, significantly improving the success rate (availability) of data generation.

[0053] Specifically, when a moving object is detected at the starting position In a collision state (cannot run directly) When visual opening analysis occurs, the initial escape mechanism is initiated. Visual opening analysis is a visual information-based escape method that calculates the initial escape vector by analyzing the direction of visual openings in free space. Rays are emitted from the starting position along six reference directions (up, down, left, right, front, and back), the depth map buffer is analyzed, and the "free space score" for each direction is calculated. Directions with longer free rays are prioritized for movement. Geometric pushback is a geometric information-based escape method. If visual analysis is ineffective (e.g., the camera is occluded), the penetration depth of the object's axis-aligned bounding box and overlapping obstacles is calculated, and the minimum translation vector is calculated to resolve the collision.

[0054] This bimodal escape mechanism uses visual rays and geometric push. It can use a physics engine-based "force-guided escape," where a repulsive force is applied to the object, and the physics engine automatically calculates the bounce direction. Alternatively, a nearby collision-free pose can be directly sampled as the escape point.

[0055] In a preferred embodiment, active obstacle avoidance planning is performed based on the starting position, free space position, and target position to generate model training motion planning trajectory data. This includes: planning an escape path using a first path planning algorithm based on the starting position and free space position; the first path planning algorithm is used to plan a path from the starting position to the free space position based on an initial escape vector; planning a motion path using a second path planning algorithm based on the free space position and target position; the second path planning algorithm is used to determine the path points of the motion and plan a path from the free space position to the target position; and generating model training motion planning trajectory data based on the escape path and motion path.

[0056] In this embodiment, an escape path is planned using a first path planning algorithm based on the initial position and the free space position. The first path planning algorithm moves the object along the calculated initial escape vector until it reaches the free space position. .Will The escape path is spliced ​​to the front end of the subsequent planned path.

[0057] To generate complex "bypass" data (such as "around the water cup from the left"), rather than simple straight-line movement, a second path planning algorithm is used to plan the motion path based on the free-space position and the target position. The second path planning algorithm first attempts to generate a straight-line path; if obstacles exist on the path, they are marked as passing objects. Calculate the center of the candidate path area in six directions (up, down, left, right, front, and back) of the object being passed. The formula takes into account the safety margin: .in, The center of the obstacle, For direction The normal vector, Let the radius of the obstacle be . Let be the radius of the moving object. For safety margin.

[0058] The escape path and the movement path are combined to form a complete motion planning trajectory.

[0059] The path planning algorithm of this invention can be adopted The (Fast Expanding Random Tree Star) algorithm can also be used, or PRM (Probabilistic Roadmaps) can be used. Bi-RRT (bidirectional RRT) or optimization-based planning algorithms (such as CHOMP, TrajOpt) can be used as alternatives. Generate the path.

[0060] As a preferred embodiment, the motion path is constrained by a cost function during planning; the cost function is a function determined by weighted summation of the total path length, corner penalty, reverse movement penalty, and lateral offset penalty.

[0061] Considering that existing technologies typically only pursue geometric shortest paths, resulting in abrupt paths, frequent turns, or paths that hug obstacles, this embodiment designs a cost function. The most natural detour direction is chosen. Through specific mathematical constraints, it is ensured that the generated obstacle avoidance path is not only collision-free but also conforms to human operational intuition, avoiding strange paths generated by machines.

[0062] in, Given the total path length (find the shortest path); To penalize corners (cosine similarity), avoid sharp turns and make the trajectory more like human hand operation; A penalty for "returning" (reverse movement penalty) (with a very high weight, such as 2.0) is set to strictly prohibit reverse movement; For lateral offset penalty; , , and These are the corresponding weights used to balance the contributions of different penalty terms, and can be dynamically adjusted according to the scenario. Alternatively, new constraints can be introduced, such as "minimizing energy consumption", "robotic arm joint limit penalty" or "avoidance penalty for specific semantic regions (such as avoiding hazardous material areas)".

[0063] The cost function designed in this invention forces the planner to generate smooth, natural trajectories without sharp turns that conform to human operating habits, thereby enhancing the value of data for imitation learning algorithms.

[0064] As a preferred embodiment, the method further includes: determining the original semantic instruction based on the model training motion planning trajectory data; performing retrospective semantic discovery based on the model training motion planning trajectory data to determine instruction enhancement elements; the instruction enhancement elements include potential path locations, motion modes, units, and relative running directions; and adding the instruction enhancement elements to the original semantic instruction to obtain the semantically enhanced instruction.

[0065] In this embodiment, the original semantic instructions can be determined based on the generated motion planning trajectory data. The original semantic instructions are basic instructions describing the movement of an object from its starting position to its target position. For example, assuming the moving object is a cup and the target position is a plate, the original semantic instruction could be: "Put the cup on the plate." Retrospective semantic discovery is performed based on motion planning trajectory data to identify instruction enhancement elements. These elements include potential path locations, motion modes, units, and relative directions of movement.

[0066] Even in a simple "move from A to B" task, the generated trajectory may accidentally pass through object C. Therefore, in this embodiment, the semantic level of the data can be enriched as follows: Proximity analysis for the generated trajectory Calculate the minimum Euclidean distance between it and all high-quality objects in the scene.

[0067] Potential waypoint determination, if the distance from the surface of an object is... If so, mark it as a "potential transit object".

[0068] The classification system for operation primitives, in addition to regular movement, explicitly lists five specific atomic operations that the system supports generating: Place Relative, Directional Move, Stacking, and ActiveBypass. Place (bypassing obstacles) and Active Bypass Stack (obstacle-free stacking).

[0069] Relative placement moves the source object to the vicinity of a reference object in a specific spatial orientation (such as left, right, front, or back). This is the most basic spatial relationship task. Instruction template: "Place the {source_obj} to the {endpoint_direction} of the {reference_obj}." Directional movement is independent of a reference object; it moves the source object a specific distance along an absolute or relative direction. Command template: "Move the {source_obj} toward the {endpoint_direction}." Stacking places the source object on top of the reference object. (The area of ​​the top surface, Arearef, must be greater than or equal to Areasrc.) Instruction template: "Place the {source_obj} on top of the {reference_obj}." Obstacle avoidance placement is a high-level primitive. The planner detects an obstacle on the straight line from the start point to the end point, and therefore explicitly generates a detour trajectory. Instruction template: "Move the {source_obj} around the {via_obj} on its {via_direction} side, then place it to the {endpoint_direction} of the {reference_obj}." Obstacle avoidance stacking combines obstacle avoidance and stacking operations, making it the most complex task type in spatial reasoning. Instruction template: "Move the {source_obj} around the {via_obj} on its {via_direction}side, then place it on top of the {reference_obj}." The dynamic unit selection mechanism ensures the model doesn't just learn "meters." The system randomly selects units based on the numerical value. Values ​​less than 1 meter are highly likely to be converted to "centimeters" or "inches," while values ​​larger than 1 meter are highly likely to be converted to "meters" or "feet." By mixing different units (meters / centimeters) in the instructions and associating them with specific numerical values, the model is forced to align visual features with physical scales, solving the problem that traditional VLMs can only understand relative relationships (large / small) and cannot understand absolute scales (20 centimeters).

[0070] Relative direction classification involves calculating the direction vector of the object's center relative to the trajectory tangent to determine the orientation relationship (e.g., "passing from the left").

[0071] To enrich the spatial semantics of standard tasks, instruction enhancement elements are added to the original semantic instructions to obtain semantically enhanced instructions. For example, a semantically enhanced instruction could be: "Move the glass past the right side of the apple and place it on the plate." This invention not only supports complex obstacle avoidance and stacking, but also uncovers implicit geometric relationships within the trajectory (such as "passing the right side of the vase") through backtracking analysis. This allows a trajectory to provide multi-dimensional spatial logical supervision signals, enabling the trained model to understand not only the endpoint but also process constraints.

[0072] This invention can also introduce large language models (such as GPT-4) to rewrite the template-generated structured instructions into more diverse and colloquial natural language instructions, thereby further improving the generalization ability of VLM.

[0073] As a preferred embodiment, it further includes: correcting the model training motion planning trajectory data; the correction methods include geometric smoothing, sparsification, physical landing correction and visual alignment correction.

[0074] Considering that the generated original trajectory (model training motion planning trajectory data) is a geometric path, in this embodiment, it needs to be converted into a natural visual trajectory.

[0075] Specifically, geometric smoothing: the path is smoothed using Catmull-Rom spline interpolation, and keypoint sparsification is performed using the RDP algorithm.

[0076] Physical landing correction: targeting To address the potential issue of the endpoint being suspended in mid-air, the endpoint is projected onto the image plane, and the Ground-Truth depth map is consulted. The endpoint is then lowered along the direction of gravity until it contacts the physical surface. This physical landing correction prevents the object from being suspended, ensuring the physical realism of the simulation data and facilitating the transfer from simulation to reality.

[0077] Visual alignment correction: 3D planning uses the geometric center of the object, but in a 2D image, the projection of the geometric center may not be within the object mask (e.g., a "C"-shaped object). Correction method: Obtain the 2D segmentation mask of the source object and calculate the visual centroid of the mask. This method forcibly replaces the 2D coordinates of the trajectory's starting point with the visual centroid while preserving 3D depth information. This ensures the accuracy of visual representation.

[0078] To address the problem of how to automatically generate object manipulation trajectories that conform to physical laws, are collision-free, and have rich spatial semantics (such as obstacle avoidance and stacking) in static 3D scanning scenes lacking real dynamic data.

[0079] This invention initializes the scene through gravity alignment and role assignment; secondly, it designs a visual-geometric bimodal escape mechanism, enabling objects to move from an initial position with mesh interlacing to free space; thirdly, it employs a method based on... The system actively plans and generates complex paths around obstacles. Finally, through a backtracking semantic discovery mechanism, it detects whether the generated trajectory has passed near a specific object and adds it back to the language instructions.

[0080] This invention achieves low-cost conversion from static 3D scenes to dynamic operational data; solves the planning failure problem caused by initial collisions in common scan data; and generates high-quality training data containing accurate measurement information (such as how many meters to move) and rich spatial semantics (such as passing through a certain object), significantly improving the spatial reasoning ability of robot models.

[0081] What needs to be explained is: 3D Spatial Trace / Trajectory: refers to a sequence of three-dimensional coordinates formed over time in three-dimensional space, usually containing position coordinates (x,y,z) or (x,y,d) (d is the depth).

[0082] VLM (Vision-Language Model): A visual-language model, an artificial intelligence model capable of processing both image and text input simultaneously.

[0083] (Rapidly-exploring Random Tree Star): An asymptotically optimal sampling-based path planning algorithm for finding collision-free paths in high-dimensional space.

[0084] AABB (Axis-Aligned Bounding Box): A cube bounding box with each face parallel to the coordinate axis, used for fast collision detection.

[0085] OBB (Oriented Bounding Box): An oriented bounding box is a cube that is oriented as the object rotates, and it is more tightly bound than AABB.

[0086] As a preferred embodiment, the method further includes: determining the three-dimensional coordinates required for the robot to perform the task based on the model training motion planning trajectory data; obtaining depth information of the grasping position based on the three-dimensional coordinates; and generating model training object grasping task data based on the three-dimensional coordinates and the depth information.

[0087] In this embodiment, the spatial trajectory consists of a series of three-dimensional coordinates arranged in chronological order. By sequentially executing these three-dimensional coordinates, the robot can move along the predetermined spatial trajectory. By combining two-dimensional pixel coordinates and depth information, a complete three-dimensional coordinate system can be constructed. In the spatial trajectory, to ensure that the movement path is both short and simple, the trajectory is usually represented as a smooth curve. Therefore, the spatial trajectory necessarily includes a starting position, an ending position, and an intermediate position connecting the two. In this embodiment of the invention, for the convenience of describing the processing procedures corresponding to the three-dimensional coordinates of the starting position, ending position, and path position, the first three-dimensional coordinate, the second three-dimensional coordinate, and the third three-dimensional coordinate are used to distinguish them, which does not represent the order or priority relationship between the starting position, ending position, and path position.

[0088] When a robot performs a grasping task, it must first ensure that it can successfully grasp the target object. Only on the basis of successful grasping does the predicted spatial trajectory have practical significance. If the robot fails to grasp the object, no matter how accurate the predicted spatial trajectory is, it is worthless. Therefore, in this embodiment of the invention, the depths of the starting position, free space position, and target position have been optimized and adjusted.

[0089] The spatial trajectory output by the machine model includes a predicted depth, which is the depth predicted by the machine based on the target image. However, this depth usually differs from the actual depth. Therefore, in this embodiment of the invention, the original first and second three-dimensional coordinates are updated by using the first true depth at the starting point and the second true depth at the ending point to obtain three-dimensional coordinates with true depth, thereby ensuring that the machine can successfully grasp the object.

[0090] Sensors are mounted at specific locations on the robot to detect depth information. These sensors can be conventional types, such as ultrasonic distance sensors, laser distance sensors, or infrared distance sensors. The true depth information obtained by these sensors has higher accuracy compared to the depth information predicted by a spatial trajectory model based on a target image. This invention utilizes the true depth information obtained by the sensors to update the three-dimensional coordinates, thereby significantly improving the robot's accuracy in grasping target objects in the depth direction and ensuring the success of the grasping task.

[0091] In a preferred embodiment, generating model training object grasping task data based on the three-dimensional coordinates and the depth information includes: updating the three-dimensional coordinates based on the depth information; determining the grasping posture of the robot to perform the task based on the updated three-dimensional coordinates; and generating the model training object grasping task data based on the grasping posture and the updated three-dimensional coordinates.

[0092] While spatial trajectories can provide the three-dimensional coordinates required for a robot to perform a task, they are insufficient for object grasping. The object's grasping posture must also be determined—that is, at what angle the robotic arm needs to approach the object to successfully grasp it.

[0093] For example, if you want a robot to grasp a banana, it can only grasp it from the circumference, not from either end. Therefore, you need to first determine the fourth 3D coordinates and shape of the target object from the target image. Specifically, the fourth 3D coordinates can be input into the existing GraspNet open-source framework. The open-source algorithm in GraspNet will calculate the position and pose based on the shape of the target object and output the optimal grasping pose.

[0094] It is understandable that the posture when grasping the target object is the first grasping posture, but the robotic arm can adjust its posture during movement after grasping the target object. For example, the posture of the robotic arm during movement after grasping the target object is the second grasping posture.

[0095] The model training data generation system provided by the present invention is described below. The model training data generation system described below can be referred to in correspondence with the model training data generation method described above.

[0096] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of a model training data generation system provided by the present invention.

[0097] The present invention also provides a model training data generation system, comprising: a scene determination module 401, used to determine a static scene based on a static scene dataset; a target position determination module 402, used to sample a target region in the static scene to determine a target position; the target position is the endpoint of a collision-free motion trajectory planning; a free space position determination module 403, used to determine a free space position based on the starting position in the static scene using an initial escape mechanism; the free space position is the starting point of a collision-free motion trajectory planning from the escape starting position; and a data generation module 404, used to perform active obstacle avoidance planning based on the starting position, free space position, and target position to generate model training motion planning trajectory data.

[0098] Figure 5 An example is a schematic diagram of the structure of an electronic device, such as... Figure 5As shown, the electronic device may include: a processor 501, a communication interface 502, a memory 503, and a communication bus 504. The processor 501, communication interface 502, and memory 503 communicate with each other via the communication bus 504. The processor 501 can call logical instructions in the memory 503 to execute a model training data generation method. This method includes: determining a static scene based on a static scene dataset; sampling a target region in the static scene to determine the target position; the target position is the endpoint of the collision-free motion trajectory planning; determining the free space position using an initial escape mechanism based on the starting position in the static scene; the free space position is the starting point of the collision-free motion trajectory planning from the escape starting position; and performing active obstacle avoidance planning based on the starting position, free space position, and target position to generate model training motion planning trajectory data.

[0099] Furthermore, the logical instructions in the aforementioned memory 503 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the model training data generation method provided by the above methods. The method includes: determining a static scene based on a static scene dataset; sampling a target region in the static scene to determine a target position; the target position is the endpoint of a collision-free motion trajectory planning; determining a free space position based on an initial escape mechanism according to the starting position in the static scene; the free space position is the starting point of a collision-free motion trajectory planning from the escape starting position; and performing active obstacle avoidance planning based on the starting position, the free space position, and the target position to generate model training motion planning trajectory data.

[0101] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for generating model training data provided by the methods described above. This method includes: determining a static scene based on a static scene dataset; sampling a target region in the static scene to determine a target position; the target position being the endpoint of a collision-free motion trajectory plan; determining a free space position using an initial escape mechanism based on the starting position in the static scene; the free space position being the starting point of a collision-free motion trajectory plan from the escape starting position; and performing active obstacle avoidance planning based on the starting position, the free space position, and the target position to generate model training motion planning trajectory data.

[0102] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating model training data, characterized in that, include: Based on the static scene dataset, determine the static scene; In the static scene, target area sampling is performed to determine the target location; The target location is the endpoint of the collision-free motion trajectory planning; Based on the initial position in the static scene, an initial escape mechanism is used to determine the free space position; the free space position is the starting point for planning a collision-free motion trajectory to escape from the initial position. Based on the starting position, the free space position, and the target position, active obstacle avoidance planning is performed to generate model training motion planning trajectory data.

2. The model training data generation method according to claim 1, characterized in that, After determining the static scene based on the static scene dataset, the process also includes: The static scene is initialized; the scene initialization methods include gravity alignment, bounding box generation, and object role assignment.

3. The model training data generation method according to claim 1, characterized in that, The step of sampling the target region in the static scene to determine the target location includes: The target area is determined based on reference objects in the static scene; Based on the target region, a polar coordinate sampling strategy from the inside out is used to determine multiple candidate points; Static collision detection is performed on each candidate point, and a collision-free point is selected as the target location.

4. The model training data generation method according to claim 1, characterized in that, The step of determining the free space position based on the initial position in the static scene using an initial escape mechanism includes: Based on the initial position in the static scene, the initial escape vector is calculated using visual opening analysis or geometric push-back method; at the initial position, the moving object is in a collision state; the initial escape vector is the minimum translation vector that removes the moving object from the collision. The free space position is determined based on the starting position and the initial escape vector; The visual opening analysis is an analysis and calculation method for escaping along the direction of the visual opening in free space, while the geometric push-back is a calculation method for retreating and escaping based on the geometric penetration depth.

5. The model training data generation method according to claim 4, characterized in that, The step of performing active obstacle avoidance planning based on the starting position, the free space position, and the target position to generate model training motion planning trajectory data includes: Based on the starting position and the free space position, an escape path is planned using a first path planning algorithm; the first path planning algorithm is used to plan a path from the starting position to the free space position based on the initial escape vector; Based on the free space position and the target position, a second path planning algorithm is used to plan the motion path; the second path planning algorithm is used to determine the path locations of the motion and plan the path from the free space position through the path locations to the target position; Based on the escape path and the motion path, the model training motion planning trajectory data is generated.

6. The model training data generation method according to claim 5, characterized in that, The motion path is constrained by a cost function during planning; the cost function is a function determined by weighted summation of the total path length, corner penalty, reverse movement penalty, and lateral offset penalty.

7. The model training data generation method according to claim 1, characterized in that, Also includes: Based on the model, train motion planning trajectory data to determine the original semantic instructions; Based on the model-trained motion planning trajectory data, retrospective semantic discovery is performed to determine instruction enhancement elements; The instruction enhancement elements include potential path locations, movement modes, units, and relative directions of movement; The instruction enhancement element is added to the original semantic instruction to obtain the semantically enhanced instruction.

8. The method for generating model training data according to any one of claims 1 to 7, characterized in that, Also includes: The motion planning trajectory data trained on the model is corrected; The correction methods include geometric smoothing, sparsification, physical landing correction, and visual alignment correction.

9. The model training data generation method according to claim 1, characterized in that, Also includes: Based on the motion planning trajectory data trained by the model, the three-dimensional coordinates required for the robot to perform the task are determined. Based on the three-dimensional coordinates, the depth information of the grab position is obtained; Based on the three-dimensional coordinates and the depth information, model training data for object grasping tasks is generated.

10. The model training data generation method according to claim 9, characterized in that, The step of generating object grasping task data for model training based on the three-dimensional coordinates and the depth information includes: The depth of the three-dimensional coordinates is updated based on the depth information; Based on the updated 3D coordinates, determine the robot's grasping posture for performing the task; Based on the grasping posture and the updated 3D coordinates, the model training object grasping task data is generated.

11. A model training data generation system, characterized in that, include: The scene determination module is used to determine static scenes based on a static scene dataset; The target location determination module is used to sample the target area in the static scene and determine the target location; the target location is the endpoint of the collision-free motion trajectory planning. The free space position determination module is used to determine the free space position based on the initial position in the static scene using an initial escape mechanism; the free space position is the starting point for planning a collision-free motion trajectory to escape from the initial position. The data generation module is used to perform active obstacle avoidance planning based on the starting position, the free space position, and the target position, and generate model training motion planning trajectory data.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the model training data generation method as described in any one of claims 1 to 10.

Images