Object grabbing method and device
By updating the depth of the three-dimensional coordinates of the starting and ending points in the robot's spatial trajectory, the spatial trajectory predicted by the model is corrected, solving the problem of robot grasping task failure and achieving accurate object grasping.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING ACAD OF ARTIFICIAL INTELLLIGENCE
- Filing Date
- 2026-02-10
- Publication Date
- 2026-04-21
AI Technical Summary
In existing technologies, robots that directly grasp objects based on the spatial trajectory output by the model are prone to failure in grasping tasks, especially when they miss.
By updating the three-dimensional coordinates of the starting and ending points in the spatial trajectory, the spatial trajectory predicted by the model is corrected, ensuring the absolute accuracy of the starting and ending points, and an object grasping task is generated based on the corrected coordinates.
This effectively avoids situations where the robot misses its target, ensuring that it can accurately complete the object-grabbing task.
Smart Images
Figure CN121893231A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for grasping objects. Background Technology
[0002] Embodied intelligence is a cutting-edge field at the intersection of artificial intelligence and robotics. It emphasizes that intelligent agents achieve autonomous learning and evolution through dynamic interaction between their bodies and the environment. Its core lies in the deep integration of perception, action, and cognition.
[0003] Currently, visual language models (VLMs) are typically used to enable robots to interact with the three-dimensional physical world. In existing object grasping methods for robots, the robot directly grasps objects based on the acquired spatial trajectory. However, since the spatial trajectory is usually predicted by a large model trained by the machine, its accuracy depends entirely on the training precision of the model. Therefore, if the robot directly grasps objects according to the spatial trajectory, it is easy to miss, causing the robot system to be unable to perform the corresponding object grasping task. Summary of the Invention
[0004] This invention provides an object grasping method and apparatus to address the problem in existing technologies where robots directly grasp objects based on spatial trajectories output by models, often leading to grasping task failures. By updating the depth of the three-dimensional coordinates of the starting and ending positions in the spatial trajectory, the model-predicted spatial trajectory can be corrected, ensuring the absolute accuracy of the starting and ending positions. Furthermore, the object grasping task is generated based on the corrected starting and ending positions, as well as the three-dimensional coordinates of the locations along the path in the spatial trajectory. This allows for mid-air movement during grasping, avoiding missed grasps, and completing the grasping movement according to the model-predicted spatial trajectory.
[0005] This invention provides an object grasping method, comprising the following steps: acquiring a spatial trajectory; wherein the spatial trajectory includes multiple three-dimensional coordinates arranged in chronological order; determining from the spatial trajectory a first three-dimensional coordinate corresponding to a starting position, a second three-dimensional coordinate corresponding to an ending position, and a third three-dimensional coordinate corresponding to a path position located between the starting position and the ending position; acquiring a first true depth of the starting position and a second true depth of the ending position based on the first three-dimensional coordinates and the second three-dimensional coordinates, and updating the first three-dimensional coordinates and the second three-dimensional coordinates according to the first true depth and the second true depth, respectively; generating an object grasping task based on the third three-dimensional coordinates, the updated first three-dimensional coordinates, and the updated second three-dimensional coordinates, so that the robot can grasp an object based on the object grasping task.
[0006] Optionally, the three-dimensional coordinates include: predicted depth; the step of obtaining the first true depth of the starting position and the second true depth of the ending position based on the first three-dimensional coordinates and the second three-dimensional coordinates respectively, and updating the first three-dimensional coordinates and the second three-dimensional coordinates according to the first true depth and the second true depth respectively, includes: using a sensor to detect the first three-dimensional coordinates to obtain the first true depth of the starting position, and using a sensor to detect the second three-dimensional coordinates to obtain the second true depth of the ending position; replacing the first predicted depth in the first three-dimensional coordinates with the first true depth to update the first three-dimensional coordinates; and replacing the second predicted depth in the second three-dimensional coordinates with the second true depth to update the second three-dimensional coordinates.
[0007] Optionally, the three-dimensional coordinates further include pixel coordinates; generating an object grasping task based on the third three-dimensional coordinates, the updated first three-dimensional coordinates, and the updated second three-dimensional coordinates includes: performing coordinate restoration on the third three-dimensional coordinates, the updated first three-dimensional coordinates, and the updated second three-dimensional coordinates to generate intermediate third three-dimensional coordinates, intermediate first three-dimensional coordinates, and intermediate second three-dimensional coordinates based on the camera coordinate system in the robot system; performing calibration transformation on the intermediate third three-dimensional coordinates, intermediate first three-dimensional coordinates, and intermediate second three-dimensional coordinates according to the correspondence between the camera coordinate system and the base coordinate system in the robot system to obtain target third three-dimensional coordinates, target first three-dimensional coordinates, and target second three-dimensional coordinates; and generating an object grasping task based on the target third three-dimensional coordinates, target first three-dimensional coordinates, and target second three-dimensional coordinates.
[0008] Optionally, generating the object grasping task based on the target's third three-dimensional coordinates, the target's first three-dimensional coordinates, and the target's second three-dimensional coordinates includes: determining the robot system's first object grasping posture; and generating the object grasping task based on the first grasping posture, the target's third three-dimensional coordinates, the target's first three-dimensional coordinates, and the target's second three-dimensional coordinates.
[0009] Optionally, determining the first object grasping posture of the robot system includes: acquiring a target image corresponding to the spatial trajectory; determining the fourth three-dimensional coordinates corresponding to the target object in the target image; and determining the first grasping posture of the robot system when grasping the target object based on the fourth three-dimensional coordinates.
[0010] Optionally, the method further includes: determining a second grasping posture of the robot system at the path location; and generating an object grasping task based on the first grasping posture, the second grasping posture, the third three-dimensional coordinates of the target, the first three-dimensional coordinates of the target, and the second three-dimensional coordinates of the target.
[0011] Optionally, the method further includes: performing the step of acquiring spatial trajectory according to a preset period, and regenerating the object grasping task based on the new fourth three-dimensional coordinates in response to a change in the fourth three-dimensional coordinates.
[0012] Optionally, the spatial trajectory is obtained based on the output of a pre-trained spatial trajectory model; wherein the pre-trained spatial trajectory model is trained through the following process: acquiring a target image, target annotation data corresponding to the target image, and target instructions; training the original spatial trajectory model based on the target image, the target instructions, and the target annotation data to obtain a first spatial trajectory model; and training the first spatial trajectory model based on the target image and the target instructions to obtain the pre-trained spatial trajectory model.
[0013] Optionally, training the original spatial trajectory model based on the target image, the target instruction, and the target annotation data to obtain the first spatial trajectory model includes: obtaining a first spatial trajectory and scale information based on the target image, the target instruction, and the original spatial trajectory model; wherein the scale information indicates the scaling degree between the actual size and the predicted size of the target object in the target image; the first spatial trajectory is a three-dimensional trajectory; and performing supervised fine-tuning on the original spatial trajectory model based on the scale information, the first spatial trajectory, and the target annotation data to obtain the first spatial trajectory model.
[0014] The present invention also provides an object grasping device, comprising the following modules: An acquisition module is used to acquire a spatial trajectory; wherein the spatial trajectory includes multiple three-dimensional coordinates arranged in chronological order; The determining module is used to determine, from the spatial trajectory, the first three-dimensional coordinates corresponding to the starting position, the second three-dimensional coordinates corresponding to the ending position, and the third three-dimensional coordinates corresponding to the path position located between the starting position and the ending position; The update module is used to obtain the first true depth of the starting position and the second true depth of the ending position based on the first three-dimensional coordinates and the second three-dimensional coordinates, respectively, and update the first three-dimensional coordinates and the second three-dimensional coordinates according to the first true depth and the second true depth, respectively. The generation module is used to generate an object grasping task based on the third three-dimensional coordinates, the updated first three-dimensional coordinates, and the updated second three-dimensional coordinates, so that the robot can grasp the object based on the object grasping task.
[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the object grasping method as described above.
[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the object grasping method as described above.
[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the object grasping method as described above.
[0018] The object grasping method and apparatus provided by this invention can correct the spatial trajectory predicted by the model by updating the three-dimensional coordinates of the starting and ending positions in the spatial trajectory to ensure the absolute accuracy of the starting and ending positions. Furthermore, by generating an object grasping task based on the corrected starting and ending positions and the three-dimensional coordinates of the positions along the path in the spatial trajectory, the object grasping task can be completed according to the spatial trajectory predicted by the model, avoiding missed grasps. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0020] Figure 1 This is one of the flowcharts illustrating the object grasping method provided by the present invention.
[0021] Figure 2 This is a schematic diagram of a specific process for depth updating of the first three-dimensional coordinates and the second three-dimensional coordinates provided by the present invention.
[0022] Figure 3 This is a schematic diagram illustrating the specific process of generating an object grasping task provided by the present invention.
[0023] Figure 4 This is a schematic diagram illustrating the specific process of determining the first grasping posture provided by the present invention.
[0024] Figure 5 This is a schematic diagram of the specific process of step 306 provided by the present invention.
[0025] Figure 6 This is a schematic diagram of the object grasping device provided by the present invention.
[0026] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0028] Figure 1 This is one of the flowcharts illustrating the object grasping method provided by the present invention, such as... Figure 1 As shown, the method includes the following: Step 101: Obtain the spatial trajectory; wherein, the spatial trajectory includes multiple three-dimensional coordinates arranged in chronological order; In this embodiment of the invention, spatial trajectory refers to the spatial movement trajectory of an object in a target image when executing a target instruction. It is mostly generated by a pre-trained spatial trajectory model after inputting the target instruction and the target image. For example, if the robot captures a scene with a banana and a plate on a table, the input target image is the image captured by the camera showing the banana and plate on the table. The target instruction can be the user's voice input recognized by the robot through sound recognition, or the user's text input obtained through a touchscreen. For example, if the target instruction is to place the banana on the plate, then after the robot acquires the target image and the target instruction, the spatial trajectory model in the robot system will generate a movement trajectory (i.e., spatial trajectory) to move the banana from its current position to the plate. Finally, based on the spatial trajectory, a grasping task is generated to achieve the complete action of the robot grasping the object according to the user's target instruction. Among them, the training accuracy and precision of the spatial trajectory model directly affect whether the output spatial trajectory can be directly used for machine execution. In the fields of embodied intelligence and robot operation, robots need to understand complex spatial instructions and execute corresponding motion trajectories to accurately complete spatial tasks. This means that the model behind it needs to have strong spatial trajectory understanding ability and achieve a certain level of accuracy. Therefore, the spatial trajectory in the embodiments of this invention can be understood as the output of a trained spatial trajectory model with a certain level of accuracy and spatial trajectory understanding ability.
[0029] In one optional embodiment, the spatial trajectory model for outputting the spatial trajectory can be trained through a supervised fine-tuning stage and a reinforcement fine-tuning stage. Specifically, the training process includes: acquiring a target image, target annotation data corresponding to the target image, and target instructions; training the original spatial trajectory model based on the target image, the target instructions, and the target annotation data to obtain a first spatial trajectory model; and training the first spatial trajectory model based on the target image and the target instructions to obtain the pre-trained spatial trajectory model.
[0030] The training process for the original spatial trajectory model is supervised fine-tuning training. The target data is the training dataset used for model training. In this embodiment, the target data may include multiple target images, and each target image corresponds to target annotation data and target instructions. Specifically, a target image refers to an image containing at least one target object, such as an image of an apple (target object) on a table, or an image of multiple flowerpots (target objects) on a balcony. When using different data types as the training dataset, the target data can also be point cloud data or video stream data. Target instructions are natural language descriptions of spatial constraints corresponding to the target image, such as "water the multiple flowerpots on the balcony from left to right, with the watering can hovering 1-5 cm above the flowers." Target annotation data is the annotation data for the precise coordinates and specific dimensions (length, width, and height) of the objects in the target image. Exemplarily, annotation can be performed using existing automatic annotation tools or manually; this invention does not limit the specific annotation method.
[0031] In a further optional embodiment, training the original spatial trajectory model based on the target image, the target instruction, and the target annotation data to obtain a first spatial trajectory model includes: obtaining a first spatial trajectory and scale information based on the target image, the target instruction, and the original spatial trajectory model; wherein the scale information indicates the scaling degree between the actual size and the predicted size of the target object in the target image; the first spatial trajectory is a three-dimensional trajectory; and performing supervised fine-tuning on the original spatial trajectory model based on the scale information, the first spatial trajectory, and the target annotation data to obtain the first spatial trajectory model.
[0032] As can be seen, the above model training process employs two adjustment phases: supervised fine-tuning and reinforcement fine-tuning. In supervised fine-tuning, by adding scale information during model training and performing supervised fine-tuning based on this scale information, the model's understanding of the actual scale of objects can be enhanced. In reinforcement fine-tuning, reinforcement learning algorithms effectively supervise the key processes at each step of spatial trajectory generation, ensuring the rationality of reasoning between multiple steps.
[0033] Specifically, the spatial trajectory consists of a series of three-dimensional coordinates arranged in chronological order. That is, by executing the three-dimensional coordinates corresponding to multiple moments in chronological order, movement along the spatial trajectory can be achieved. Here, three-dimensional coordinates refer to coordinates containing three dimensions. For example, they can include pixel coordinates and predicted depth, represented by (u, v, d). Here, u and v are the coordinates of a pixel in a two-dimensional plane; u is the horizontal coordinate of the pixel in the image, v is the vertical coordinate of the pixel in the image, and d is the distance (depth) between the coordinate point and the robot. The three-dimensional coordinates are obtained by combining the pixel coordinates of the two-dimensional plane with the depth dimension.
[0034] Step 102: Determine the first three-dimensional coordinates corresponding to the starting position, the second three-dimensional coordinates corresponding to the ending position, and the third three-dimensional coordinates corresponding to the path position between the starting position and the ending position from the spatial trajectory; For a spatial trajectory, in order to ensure that the movement distance is relatively short and the trajectory is relatively simple, it is usually represented as a smooth curve. Therefore, there must be a starting position, an ending position, and a path position connecting the starting position and the ending position in the spatial trajectory. In order to facilitate the description of the processing process corresponding to the three-dimensional coordinates of the starting position, the ending position, and the path position, the embodiments of the present invention use the first three-dimensional coordinate, the second three-dimensional coordinate, and the third three-dimensional coordinate to distinguish them. This does not represent the order or priority relationship between the starting position, the ending position, and the path position.
[0035] Step 103: Based on the first three-dimensional coordinates and the second three-dimensional coordinates, obtain the first true depth of the starting position and the second true depth of the ending position, and update the first three-dimensional coordinates and the second three-dimensional coordinates according to the first true depth and the second true depth, respectively. It should be noted that when the robot performs a grasping task, it must first ensure that it can successfully grasp the object. Only on this basis does the predicted spatial trajectory have meaning. If it fails to grasp the object, no matter how accurate the predicted spatial trajectory is, it is meaningless. Therefore, in this embodiment of the invention, the depths of the starting and ending positions are rigidly adjusted.
[0036] Specifically, the spatial trajectory output by the machine model includes a predicted depth, which is the depth predicted by the machine based on the target image. However, this depth usually differs from the actual depth. Therefore, in this embodiment of the invention, the original first and second three-dimensional coordinates are updated by using the first true depth at the starting point and the second true depth at the ending point to obtain three-dimensional coordinates with true depth, thereby ensuring that the machine can successfully grasp the object.
[0037] In one optional embodiment, the specific process of updating the depth of the first three-dimensional coordinates and the second three-dimensional coordinates can be as follows: Figure 2 As shown, it includes: Step 201: Use a sensor to detect the first three-dimensional coordinates to obtain the first true depth of the starting position, and use a sensor to detect the second three-dimensional coordinates to obtain the second true depth of the ending position. Step 202: Replace the first predicted depth in the first three-dimensional coordinates with the first true depth to update the first three-dimensional coordinates; Step 203: Replace the second predicted depth in the second three-dimensional coordinates with the second true depth to update the second three-dimensional coordinates.
[0038] The sensors are distance sensors installed at specific locations on the robot, and can include any of the conventional distance sensors such as ultrasonic distance sensors, laser distance sensors, and infrared distance sensors. By detecting depth using these sensors, the true depth of the starting and ending points in a real-world scene can be obtained. This accuracy differs significantly from the predicted depth obtained by a spatial trajectory model based on a target image. This embodiment of the invention utilizes the true depth obtained by the sensors to update the three-dimensional coordinates, effectively improving the depth accuracy for grasping target objects and ensuring the successful completion of the grasping task.
[0039] The order of steps 202 and 203 can be interchanged. This invention does not impose a specific limitation on the update order of the first three-dimensional coordinates and the second three-dimensional coordinates. They can be updated simultaneously or sequentially.
[0040] Step 104: Generate an object grasping task based on the third 3D coordinates, the updated first 3D coordinates, and the updated second 3D coordinates, so that the robot can grasp objects based on the object grasping task.
[0041] It should be noted that since the path locations in the entire spatial trajectory are predicted by the spatial trajectory model and do not actually exist in the real scene, the third three-dimensional coordinates of the path locations need to be completely trusted to the output results of the spatial trajectory model. Therefore, in this embodiment of the invention, the third three-dimensional coordinates are not updated, only the first three-dimensional coordinates and the second three-dimensional coordinates are updated, and the object grasping task is generated based on the updated coordinates.
[0042] As can be seen, this embodiment of the invention performs depth updates using the three-dimensional coordinates of the starting and ending points in the spatial trajectory, which corrects the spatial trajectory predicted by the model, ensuring the absolute accuracy of the starting and ending points. Furthermore, by generating an object grasping task based on the corrected starting and ending points and the three-dimensional coordinates of the points along the path in the spatial trajectory, the object grasping task can be completed according to the spatial trajectory predicted by the model, avoiding missed grasps.
[0043] Furthermore, it is necessary to ensure that the coordinate system used in the object grasping task is the same as the coordinate system corresponding to the robot's base (e.g., the base of the robotic arm). Since different coordinate systems correspond to different positions, and for the robotic arm, specific control can only be achieved using its own base as the zero point during the grasping task execution, therefore, in a further optional embodiment, the object grasping method provided by this invention is as follows: Figure 3 As shown, it specifically includes: Step 301: Obtain the spatial trajectory; wherein, the spatial trajectory includes multiple three-dimensional coordinates arranged in time order; the three-dimensional coordinates include the predicted depth and pixel coordinates; Step 302: Determine the first three-dimensional coordinates corresponding to the starting position, the second three-dimensional coordinates corresponding to the ending position, and the third three-dimensional coordinates corresponding to the path position between the starting position and the ending position from the spatial trajectory; Step 303: Based on the first three-dimensional coordinates and the second three-dimensional coordinates, obtain the first true depth of the starting position and the second true depth of the ending position, and update the first three-dimensional coordinates and the second three-dimensional coordinates according to the first true depth and the second true depth, respectively. Step 304: Restore the coordinates of the third 3D coordinates, the updated first 3D coordinates, and the updated second 3D coordinates to generate intermediate third 3D coordinates, intermediate first 3D coordinates, and intermediate second 3D coordinates based on the camera coordinate system in the robot system. Step 305: Based on the correspondence between the camera coordinate system and the base coordinate system in the robot system, perform calibration transformation on the intermediate third 3D coordinate, intermediate first 3D coordinate, and intermediate second 3D coordinate to obtain the target third 3D coordinate, target first 3D coordinate, and target second 3D coordinate; Step 306: Generate an object grabbing task based on the target's third 3D coordinates, the target's first 3D coordinates, and the target's second 3D coordinates.
[0044] In step 304, coordinate restoration refers to restoring the aforementioned three-dimensional coordinates (u, v, d) including two-dimensional pixels into three-dimensional coordinates (x, y, z) in the camera coordinate system based on the camera's internal parameters, so as to represent the position of the three-dimensional coordinates relative to the camera origin in the coordinate system with the camera as the origin.
[0045] For example, the z value can be directly equal to the depth value d, the x value can be obtained by subtracting the difference between the pixel u value and the x-coordinate of the image center, multiplying by the depth d, and then dividing by the focal length fx, and the y value can be obtained by subtracting the difference between the pixel v value and the y-coordinate of the image center, multiplying by the depth d, and then dividing by the focal length fy. The focal lengths fx and fy can be directly obtained from the camera's internal parameters.
[0046] Furthermore, in step 305, the correspondence between the camera coordinate system and the base coordinate system in the robot system is set through hand-eye calibration. Specifically, the conversion between the camera coordinate system and the base coordinate system can be achieved through a calibration matrix (usually a 4x4 matrix), that is, from the three-dimensional coordinates (x, y, z) in the camera coordinate system to the three-dimensional coordinates (x1, y1, z1) in the base coordinate system.
[0047] For example, the calibration matrix includes two types of information: rotation information (used to indicate how many degrees the camera is tilted relative to the base) and translation information (used to indicate the distance between the center of the camera and the center of the base). The specific calculation logic is to first rotate the three-dimensional coordinates in the camera coordinate system by angle and then translate them to obtain the three-dimensional coordinates in the base coordinate system.
[0048] Through the above process, the three-dimensional coordinates that the robot can execute are obtained based on the spatial trajectory. However, in the object grasping process, obtaining only the three-dimensional coordinates is not enough; it is also necessary to determine the object grasping posture, that is, at what angle the robotic arm can grasp the object. Therefore, in an optional embodiment of the present invention, step 104 may specifically include: determining the first object grasping posture of the robot system; generating an object grasping task based on the first grasping posture, the third three-dimensional coordinates of the target, the first three-dimensional coordinates of the target, and the second three-dimensional coordinates of the target.
[0049] Specifically, the process of determining the first object's grasping posture is as follows: Figure 4 As shown, it includes: Step 401: Obtain the target image corresponding to the spatial trajectory; Step 402: Determine the fourth three-dimensional coordinates of the target object in the target image and the shape of the target object; Step 403: Determine the first grasping posture of the robot system when grasping the target object based on the fourth three-dimensional coordinates and the shape of the target object.
[0050] Using the aforementioned target instruction as an example, if the robot wants to grasp a banana, it can only grasp it from the circumference, not from either end. Therefore, it's necessary to first determine the fourth 3D coordinates and shape of the target object from the target image. Specifically, the fourth 3D coordinates can be input into the existing GraspNet open-source framework. The open-source algorithm in GraspNet will calculate the position and pose based on the shape of the target object and output the optimal first grasping pose.
[0051] It is understood that the posture when grasping the target object is the first grasping posture, but during the movement after grasping the target object, the robotic arm can adjust its posture. That is, in an optional embodiment, the process of step 306 can be as follows: Figure 5 As shown, it includes: Step 501: Determine the first object grasping posture of the robot system; Step 502: Determine the second grasping posture of the robot system at the path location; Step 503: Generate an object grasping task based on the first grasping posture, the second grasping posture, the target's third three-dimensional coordinates, the target's first three-dimensional coordinates, and the target's second three-dimensional coordinates.
[0052] The second grasping posture refers to the posture of the robotic arm during the movement process after grasping the target object. For example, a fixed second grasping posture can be preset. After the robot system grasps the object in the first object grasping posture, it is first adjusted to the fixed second grasping posture, and then the robotic arm moves until it reaches the three-dimensional coordinates corresponding to the endpoint.
[0053] In an optional embodiment, the step of acquiring the spatial trajectory can also be executed according to a preset cycle. In response to a change in the fourth three-dimensional coordinates, the object grasping task is regenerated based on the new fourth three-dimensional coordinates. Through the above process, it is possible to detect in a timely manner whether the target object has moved, so as to avoid the situation where the machine system performs an empty task because the target object is no longer at the starting position of the original spatial trajectory.
[0054] In summary, the object grasping method provided in this embodiment of the invention can correct the spatial trajectory predicted by the model by updating the three-dimensional coordinates of the starting and ending positions in the spatial trajectory, thereby ensuring the absolute accuracy of the starting and ending positions. Furthermore, by generating an object grasping task based on the corrected three-dimensional coordinates of the starting and ending positions and the positions along the path in the spatial trajectory, the object grasping task can be completed in mid-air according to the spatial trajectory predicted by the model, while avoiding the occurrence of missed grasping.
[0055] The object grasping device provided by the present invention is described below. The object grasping device described below can be referred to in correspondence with the object grasping method described above.
[0056] like Figure 6 As shown, the object grasping device 600 provided by the present invention includes: Acquisition module 601 is used to acquire a spatial trajectory; wherein, the spatial trajectory includes multiple three-dimensional coordinates arranged in chronological order; The determining module 602 is used to determine from the spatial trajectory the first three-dimensional coordinates corresponding to the starting position, the second three-dimensional coordinates corresponding to the ending position, and the third three-dimensional coordinates corresponding to the path position located between the starting position and the ending position; The update module 603 is used to obtain the first true depth of the starting position and the second true depth of the ending position based on the first three-dimensional coordinates and the second three-dimensional coordinates, respectively, and update the first three-dimensional coordinates and the second three-dimensional coordinates according to the first true depth and the second true depth, respectively. The generation module 604 is used to generate an object grasping task based on the third three-dimensional coordinates, the updated first three-dimensional coordinates, and the updated second three-dimensional coordinates, so that the robot can grasp objects based on the object grasping task.
[0057] In an optional embodiment of the present invention, the three-dimensional coordinates include: a predicted depth; the update module 603 is further configured to: use a sensor to detect the first three-dimensional coordinates to obtain a first true depth at the starting point position, and use a sensor to detect the second three-dimensional coordinates to obtain a second true depth at the ending point position; replace the first predicted depth in the first three-dimensional coordinates with the first true depth to update the first three-dimensional coordinates; and replace the second predicted depth in the second three-dimensional coordinates with the second true depth to update the second three-dimensional coordinates.
[0058] In an optional embodiment of the present invention, the three-dimensional coordinates further include pixel coordinates; the generation module 604 is further configured to: restore the coordinates of the third three-dimensional coordinates, the updated first three-dimensional coordinates, and the updated second three-dimensional coordinates to generate intermediate third three-dimensional coordinates, intermediate first three-dimensional coordinates, and intermediate second three-dimensional coordinates based on the camera coordinate system in the robot system; perform calibration transformation on the intermediate third three-dimensional coordinates, intermediate first three-dimensional coordinates, and intermediate second three-dimensional coordinates according to the correspondence between the camera coordinate system and the base coordinate system in the robot system to obtain target third three-dimensional coordinates, target first three-dimensional coordinates, and target second three-dimensional coordinates; and generate an object grasping task based on the target third three-dimensional coordinates, target first three-dimensional coordinates, and target second three-dimensional coordinates.
[0059] In an optional embodiment of the present invention, the generation module 604 is further configured to determine a first object grasping posture of the robot system; and generate an object grasping task based on the first grasping posture, the third three-dimensional coordinates of the target, the first three-dimensional coordinates of the target, and the second three-dimensional coordinates of the target.
[0060] In an optional embodiment of the present invention, the generation module 604 is further configured to: acquire a target image corresponding to the spatial trajectory; determine the fourth three-dimensional coordinates of the target object in the target image and the shape of the target object; and determine the first grasping posture of the robot system when grasping the target object based on the fourth three-dimensional coordinates and the shape of the target object.
[0061] In an optional embodiment of the present invention, the generation module 604 is further configured to determine the second grasping posture of the robot system at the path position; and generate an object grasping task based on the first grasping posture, the second grasping posture, the third three-dimensional coordinates of the target, the first three-dimensional coordinates of the target, and the second three-dimensional coordinates of the target.
[0062] In an optional embodiment of the present invention, the generation module 604 is further configured to perform the step of acquiring spatial trajectory according to a preset period, and in response to a change in the fourth three-dimensional coordinates, regenerate the object grasping task according to the new fourth three-dimensional coordinates.
[0063] In an optional embodiment of the present invention, the spatial trajectory is obtained based on the output of a pre-trained spatial trajectory model; wherein the pre-trained spatial trajectory model is trained through the following process: acquiring a target image, target annotation data corresponding to the target image, and target instructions; training an original spatial trajectory model based on the target image, the target instructions, and the target instructions to obtain a first spatial trajectory model; and training the first spatial trajectory model based on the target image and the target instructions to obtain the pre-trained spatial trajectory model.
[0064] In an optional embodiment of the present invention, training the original spatial trajectory model based on the target image, the target instruction, and the target instruction to obtain a first spatial trajectory model includes: obtaining a first spatial trajectory and scale information based on the target image, the target instruction, and the original spatial trajectory model; wherein the scale information indicates the scaling degree between the actual size and the predicted size of the target object in the target image; the first spatial trajectory is a three-dimensional trajectory; and performing supervised fine-tuning on the original spatial trajectory model based on the scale information, the first spatial trajectory, and the target annotation data to obtain the first spatial trajectory model.
[0065] In summary, the object grasping device provided in this embodiment of the invention can correct the spatial trajectory predicted by the model by updating the three-dimensional coordinates of the starting and ending positions in the spatial trajectory, thereby ensuring the absolute accuracy of the starting and ending positions. Furthermore, by generating an object grasping task based on the corrected starting and ending positions and the three-dimensional coordinates of the positions along the path in the spatial trajectory, the device can complete the movement in mid-air according to the spatial trajectory predicted by the model, while avoiding the occurrence of missed grasps.
[0066] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute an object grasping method. This method includes: acquiring a spatial trajectory; wherein the spatial trajectory includes multiple three-dimensional coordinates arranged in chronological order; determining from the spatial trajectory a first three-dimensional coordinate corresponding to a starting position, a second three-dimensional coordinate corresponding to an ending position, and a third three-dimensional coordinate corresponding to a path position between the starting position and the ending position; acquiring a first true depth of the starting position and a second true depth of the ending position based on the first and second three-dimensional coordinates, and updating the first and second three-dimensional coordinates according to the first and second true depths respectively; generating an object grasping task based on the third three-dimensional coordinate, the updated first three-dimensional coordinate, and the updated second three-dimensional coordinate, so that the robot can grasp an object based on the object grasping task.
[0067] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0068] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the object grasping method provided by the above methods. The method includes: acquiring a spatial trajectory; wherein the spatial trajectory includes a plurality of three-dimensional coordinates arranged in chronological order; determining a first three-dimensional coordinate corresponding to a starting position, a second three-dimensional coordinate corresponding to an ending position, and a third three-dimensional coordinate corresponding to a path position located between the starting position and the ending position from the spatial trajectory; acquiring a first true depth of the starting position and a second true depth of the ending position based on the first three-dimensional coordinate and the second three-dimensional coordinate, and updating the first three-dimensional coordinate and the second three-dimensional coordinate according to the first true depth and the second true depth, respectively; generating an object grasping task based on the third three-dimensional coordinate, the updated first three-dimensional coordinate, and the updated second three-dimensional coordinate, so that the robot can grasp an object based on the object grasping task.
[0069] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the object grasping method provided by the above methods. The method includes: acquiring a spatial trajectory; wherein the spatial trajectory includes a plurality of three-dimensional coordinates arranged in chronological order; determining from the spatial trajectory a first three-dimensional coordinate corresponding to a starting position, a second three-dimensional coordinate corresponding to an ending position, and a third three-dimensional coordinate corresponding to a path position located between the starting position and the ending position; acquiring a first true depth of the starting position and a second true depth of the ending position based on the first three-dimensional coordinate and the second three-dimensional coordinate, respectively, and updating the first three-dimensional coordinate and the second three-dimensional coordinate according to the first true depth and the second true depth, respectively; generating an object grasping task based on the third three-dimensional coordinate, the updated first three-dimensional coordinate, and the updated second three-dimensional coordinate, so that the robot performs object grasping based on the object grasping task.
[0070] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for grasping an object, characterized in that, Applied to robotic systems, including: Acquire a spatial trajectory; wherein the spatial trajectory includes multiple three-dimensional coordinates arranged in chronological order; From the spatial trajectory, determine the first three-dimensional coordinates corresponding to the starting position, the second three-dimensional coordinates corresponding to the ending position, and the third three-dimensional coordinates corresponding to the path position between the starting position and the ending position; Based on the first three-dimensional coordinates and the second three-dimensional coordinates, the first true depth of the starting position and the second true depth of the ending position are obtained respectively, and the first three-dimensional coordinates and the second three-dimensional coordinates are updated according to the first true depth and the second true depth respectively; Based on the third 3D coordinates, the updated first 3D coordinates, and the updated second 3D coordinates, an object grasping task is generated so that the robot can grasp the object based on the object grasping task.
2. The object grasping method according to claim 1, characterized in that, The three-dimensional coordinates include: predicted depth; the step of obtaining the first true depth of the starting position and the second true depth of the ending position based on the first three-dimensional coordinates and the second three-dimensional coordinates respectively, and updating the first three-dimensional coordinates and the second three-dimensional coordinates according to the first true depth and the second true depth respectively, includes: The first true depth of the starting point is obtained by using a sensor to detect the first three-dimensional coordinates, and the second true depth of the ending point is obtained by using a sensor to detect the second three-dimensional coordinates. The first predicted depth in the first three-dimensional coordinates is replaced with the first true depth to update the first three-dimensional coordinates; And the second predicted depth in the second three-dimensional coordinates is replaced with the second true depth to update the second three-dimensional coordinates.
3. The object grasping method according to claim 2, characterized in that, The three-dimensional coordinates also include pixel coordinates; the step of generating an object grasping task based on the third three-dimensional coordinates, the updated first three-dimensional coordinates, and the updated second three-dimensional coordinates includes: The third 3D coordinates, the updated first 3D coordinates, and the updated second 3D coordinates are restored to generate intermediate third 3D coordinates, intermediate first 3D coordinates, and intermediate second 3D coordinates based on the camera coordinate system in the robot system. Based on the correspondence between the camera coordinate system and the base coordinate system in the robot system, the intermediate third three-dimensional coordinates, intermediate first three-dimensional coordinates, and intermediate second three-dimensional coordinates are calibrated and transformed to obtain the target third three-dimensional coordinates, target first three-dimensional coordinates, and target second three-dimensional coordinates. An object grabbing task is generated based on the target's third three-dimensional coordinates, the target's first three-dimensional coordinates, and the target's second three-dimensional coordinates.
4. The object grasping method according to claim 3, characterized in that, The process of generating an object grasping task based on the target's third three-dimensional coordinates, the target's first three-dimensional coordinates, and the target's second three-dimensional coordinates includes: Determine the first object grasping posture of the robot system; An object grasping task is generated based on the first grasping posture, the target's third three-dimensional coordinates, the target's first three-dimensional coordinates, and the target's second three-dimensional coordinates.
5. The object grasping method according to claim 4, characterized in that, Determining the first object grasping posture of the robot system includes: Obtain the target image corresponding to the spatial trajectory; Determine the fourth three-dimensional coordinates of the target object in the target image and the shape of the target object; The first grasping posture of the robot system when grasping the target object is determined based on the fourth three-dimensional coordinates and the shape of the target object.
6. The object grasping method according to claim 4, characterized in that, Also includes: Determine the second grasping posture of the robot system at the path location; An object grasping task is generated based on the first grasping posture, the second grasping posture, the target's third three-dimensional coordinates, the target's first three-dimensional coordinates, and the target's second three-dimensional coordinates.
7. The object grasping method according to claim 5, characterized in that, Also includes: The steps of acquiring spatial trajectory are executed according to a preset cycle. In response to a change in the fourth three-dimensional coordinates, the object grasping task is regenerated based on the new fourth three-dimensional coordinates.
8. The object grasping method according to claim 1, characterized in that, The spatial trajectory is obtained based on the output of a pre-trained spatial trajectory model; wherein, the pre-trained spatial trajectory model is trained through the following process: Acquire the target image, the target annotation data corresponding to the target image, and the target instructions; The original spatial trajectory model is trained based on the target image, the target instruction, and the target annotation data to obtain a first spatial trajectory model; The first spatial trajectory model is trained based on the target image and the target instruction to obtain the pre-trained spatial trajectory model.
9. The object grasping method according to claim 8, characterized in that, The step of training the original spatial trajectory model based on the target image, the target instruction, and the target annotation data to obtain the first spatial trajectory model includes: A first spatial trajectory and scale information are obtained based on the target image, the target command, and the original spatial trajectory model; wherein, the scale information indicates the scaling degree between the actual size and the predicted size of the target object in the target image; the first spatial trajectory is a three-dimensional trajectory; Based on the scale information, the first spatial trajectory, and the target annotation data, the original spatial trajectory model is subjected to supervised fine-tuning to obtain the first spatial trajectory model.
10. An object grasping device, characterized in that, include: An acquisition module is used to acquire a spatial trajectory; wherein the spatial trajectory includes multiple three-dimensional coordinates arranged in chronological order; The determining module is used to determine, from the spatial trajectory, the first three-dimensional coordinates corresponding to the starting position, the second three-dimensional coordinates corresponding to the ending position, and the third three-dimensional coordinates corresponding to the path position located between the starting position and the ending position; The update module is used to obtain the first true depth of the starting position and the second true depth of the ending position based on the first three-dimensional coordinates and the second three-dimensional coordinates, respectively, and update the first three-dimensional coordinates and the second three-dimensional coordinates according to the first true depth and the second true depth, respectively. The generation module is used to generate an object grasping task based on the third three-dimensional coordinates, the updated first three-dimensional coordinates, and the updated second three-dimensional coordinates, so that the robot can grasp the object based on the object grasping task.
11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the object grasping method as described in any one of claims 1 to 9.
12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the object grasping method as described in any one of claims 1 to 9.
13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the object grasping method as described in any one of claims 1 to 9.