Robot grasping method and system with dynamic spatial relationship graph

Abstract

This invention provides a robot grasping method and system that incorporates a dynamic spatial relationship graph to enhance the model, belonging to the field of robotics technology. The method includes: determining the point cloud data and geometric attribute data of each object in the current grasping scene based on a color image and point cloud data of the current grasping scene; constructing a dynamic spatial relationship graph based on the point cloud data and geometric attribute data of each object; generating structured text prompts based on the dynamic spatial relationship graph and the grasping priority score of each object in the dynamic spatial relationship graph; inputting the color image of the current grasping scene, the input language command, and the structured text prompts into a vision-language-action model to obtain the grasping pose output by the vision-language-action model; and driving the robotic arm to perform grasping according to the grasping pose. The construction of the dynamic spatial relationship graph allows for the prediction of the grasping pose applicable to the scene, thereby improving the success rate of robot grasping.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a robot grasping method and system that incorporates dynamic spatial relationship graphs to enhance the model. Background Technology

[0002] In the relevant technical solutions, before the robot performs the grasping operation, it is necessary to establish a geometric model of the object to be grasped, and then select the grasping pose based on the geometric model before performing the grasping operation.

[0003] Stacked scenes are those where objects have supporting, occluding, or adjacency relationships.

[0004] In a stacked scenario, different objects can influence each other. If the robot still performs grasping operations based on the geometric model of the objects, it will be affected by interference from other objects during the grasping process, thus affecting the success rate of grasping.

[0005] Clearly, ensuring the success rate of robot grasping in stacked scenarios is a pressing issue that needs to be addressed. Summary of the Invention

[0006] This invention provides a robot grasping method and system that introduces a dynamic spatial relationship graph to enhance the model, in order to solve the urgent problem of ensuring the success rate of robot grasping in stacked scenarios.

[0007] This invention provides a robot grasping method, comprising the following steps: Based on the color image of the current capture scene and the point cloud data of the current capture scene, determine the point cloud data and geometric attribute data of each object in the current capture scene. A dynamic spatial relationship graph is constructed based on the point cloud data and geometric attribute data of each of the objects. Based on the dynamic spatial relationship graph and the grasping priority score of each object in the dynamic spatial relationship graph, a structured text prompt is generated; The color image of the current grasping scene, the input language command, and the structured text prompt are input into the vision-language-action model to obtain the grasping pose output by the vision-language-action model; The robotic arm is driven to perform the grasping action according to the grasping posture.

[0008] The robot grasping method provided by this invention, wherein determining the point cloud data and geometric attribute data of each object in the current grasping scene based on a color image of the current grasping scene and point cloud data of the current grasping scene, includes: The instance segmentation model is invoked to segment the color image of the current capture scene to obtain the two-dimensional contour of each object in the current capture scene; Determine the two-dimensional mask corresponding to each object based on the two-dimensional contour; The point cloud data of the current capture scene is extracted using the two-dimensional mask to obtain the point cloud data of each object in the current capture scene; The geometric attribute data of each object is determined based on the point cloud data of each object in the current grasping scene.

[0009] The robot grasping method provided by the present invention includes a dynamic spatial relationship graph comprising multiple nodes and relational edges connecting different nodes. The relational edges include one or more of a first type of edge, a second type of edge, and a third type of edge. The first type of edge is used to indicate that a first object is supported by a second object, the second type of edge is used to indicate that a first object occludes a second object, and the third type of edge is used to indicate that a first object is adjacent to a second object. The multiple nodes include a first node and a second node. The first node is a node defined for the first object in the dynamic spatial relationship graph, and the second node is a node defined for the second object in the dynamic spatial relationship graph. The construction of a dynamic spatial relationship graph based on the point cloud data and geometric attribute data of each object includes: A first subset is determined in the point cloud data of the first object, wherein the z-axis coordinate value of the point cloud located in the first subset is less than the z-axis coordinate value of the point cloud located in a first other subset, and the first other subset is a subset of the point cloud data of the first object that is outside the first subset. A second subset is determined in the point cloud data of the second object, wherein the z-axis coordinate value of the point cloud located in the second subset is greater than the z-axis coordinate value of the point cloud located in the other second subset, and the other second subset is a subset of the point cloud data of the second object that is not in the second subset. Based on the fact that the minimum vertical distance between the point cloud data corresponding to the first subset and the point cloud data corresponding to the second subset is less than or equal to a preset threshold, the first type of edge is constructed from the first node to the second node; The point cloud data of the first object is projected onto the two-dimensional plane of the camera to obtain a first depth map, and the point cloud data of the second object is projected onto the two-dimensional plane of the camera to obtain a second depth map. The overlapping area of ​​the masks is determined based on the two-dimensional mask corresponding to the first object and the two-dimensional mask corresponding to the second object. Based on the premise that the first region has no valid depth value or is empty, and the second region has a valid depth value, a second type of edge is constructed from the first node to the second node. The first region is the region in the first depth map that corresponds to the overlapping region of the mask, and the second region is the region in the second depth map that corresponds to the overlapping region of the mask. Obtain the shortest Euclidean distance between the first bounding box and the second bounding box, where the first bounding box is the bounding box corresponding to the point cloud data of the first object, and the second bounding box is the bounding box corresponding to the point cloud data of the second object. Based on the fact that the shortest Euclidean distance is less than a preset distance value, the third type of edge is constructed from the first node to the second node.

[0010] The robot grasping method provided by the present invention further includes: Based on the dynamic spatial relationship diagram, the top sharpness score, stability impact score, and accessibility score corresponding to each object are determined. The top sharpness score, stability impact score, and accessibility score corresponding to each object are weighted and summed to obtain the grab priority score for each object.

[0011] The robot grasping method provided by this invention includes the following step: weighted summation of the top sharpness score, the stability impact score, and the reachability score corresponding to each object to obtain a grasping priority score for each object, including: Obtain a first weight value, a second weight value, and a third weight value, wherein the first weight value is greater than the second weight value, and the third weight value is greater than the first weight value; Calculate the first product of the top sharpness score and the first weight value; Calculate the second product of the stability impact score and the second weight value; Calculate the third product of the reachability score and the third weight value; The sum of the first product value, the second product value, and the third product value is calculated to obtain the capture priority score.

[0012] The robot grasping method provided by this invention, wherein the determination of the top sharpness score, stability impact score, and reachability score for each object based on the dynamic spatial relationship graph includes: Obtain the number of relational edges of the first node, wherein the number of relational edges includes a first number of first-type edges, a second number of second-type edges, and a third number of third-type edges; If there is no first-type edge in the first node, the top sharpness score is 1; if there is a first-type edge in the first node, the top sharpness score is 0. The stability impact fraction is exp(-α×N_sup), where α is the first attenuation coefficient and N_sup is the second quantity; The reachability score is exp(-β×N_occ), where β is the second attenuation coefficient and N_occ is the third quantity.

[0013] This invention provides a robot grasping device, comprising the following modules: The determination module is used to determine the point cloud data and geometric attribute data of each object in the current capture scene based on the color image of the current capture scene and the point cloud data of the current capture scene. A construction module is used to construct a dynamic spatial relationship graph based on the point cloud data and geometric attribute data of each of the objects. The processing module is used to generate structured text prompts based on the dynamic spatial relationship graph and the grasping priority score of each object in the dynamic spatial relationship graph; and The color image of the current grasping scene, the input language command, and the structured text prompt are input into the vision-language-action model to obtain the grasping pose output by the vision-language-action model; The grasping module is used to drive the robotic arm to perform grasping according to the grasping posture.

[0014] The present invention also provides a robot grasping system, comprising: 3D camera; robotic arm; A controller, connected to the 3D camera and the robotic arm, is used to perform any of the robot grasping methods described above.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot grasping method as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot grasping method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the robot grasping method as described above.

[0018] The present invention provides a robot grasping method and system that incorporates a dynamic spatial relationship graph to enhance the model. The dynamic spatial relationship graph can describe the geometric attribute data of different objects, as well as the spatial relationships between different objects. Therefore, the structured text prompts generated by using the dynamic spatial relationship graph and the grasping priority score of each object in the dynamic spatial relationship graph can provide a basis for judging the stability of the current grasping scene structure, so that the grasping pose inferred by the vision-language-action model can be applied to the grasping requirements of the scene, thereby improving the success rate of robot grasping. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is one of the flowcharts illustrating the robot grasping method provided by the present invention; Figure 2 This is a schematic block diagram of the multimodal perception and instance segmentation module, dynamic spatial relationship graph construction module, VLA-driven graph augmentation reasoning and action generation module, and action execution and closed-loop feedback module provided by the present invention. Figure 3 This is the second flowchart of the data processing method provided by the present invention; Figure 4 This is a schematic block diagram of the robot grasping device provided by the present invention; Figure 5 This is a schematic diagram of the robot provided by the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] It should be noted that in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, as a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection through an intermediate medium; or a connection within two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0023] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0024] The following explanations are provided for the terms used in this application.

[0025] 1. A Vision-Language-Action Model (VLA) is a multimodal artificial intelligence model that can receive visual (images / videos) and linguistic (text commands) inputs and directly output a series of action commands (such as the pose of a robot's end effector) to complete a specific task. For example, Google's RT-2 model is a good example of a Vision-Language-Action Model.

[0026] 2. Segment Anything Model (SAM) is a basic vision model with powerful zero-shot instance segmentation capabilities, enabling accurate contour segmentation of previously unseen objects in an image.

[0027] 3. A 3D camera, also known as an RGB-D camera, is a camera capable of simultaneously acquiring color images (RGB) and depth information. Depth information typically exists in the form of a depth map or point cloud, recording the distance from the camera to various points in the scene.

[0028] 4. Dynamic Spatial Relation Graph (DSRG) is a data structure that uses a graph to model objects and their physical and spatial relationships (such as support, occlusion, and proximity) in a robot's working scene. The dynamic spatial relation graph is dynamically updated according to changes in the scene.

[0029] 5. End-to-End: In the field of machine learning, this refers to a system that directly maps raw input (such as an image) to the final output (such as an action instruction), without relying on human-designed, structured intermediate representations.

[0030] 6. DoF pose (6-Degrees-of-Freedom Pose) describes the position and orientation of an object in three-dimensional space. It is usually represented by three translational components (x, y, z) and three rotational components (roll, pitch, yaw) or a quaternion.

[0031] 7. Point cloud: A collection of points in a three-dimensional coordinate system. These points are usually acquired by 3D scanning devices (such as RGB-D cameras or LiDAR) and can be used to represent the three-dimensional surface shape of an object.

[0032] The following is combined with Figures 1-6 The present invention describes a robot grasping method and system that introduces dynamic spatial relationship graphs to enhance the model, aiming to solve the urgent problem of ensuring the success rate of robot grasping in stacked scenarios.

[0033] Figure 1 This is one of the flowcharts illustrating the robot grasping method provided by the present invention, such as... Figure 1 As shown, including but not limited to the following steps: Step 101: Based on the color image of the current capture scene and the point cloud data of the current capture scene, determine the point cloud data and geometric attribute data of each object in the current capture scene.

[0034] The current grasping scenario can be a scenario determined by the robot when performing the task, based on the analysis of the environment in which the object being grasped is located.

[0035] For example, if the robot is performing a task and there is only one object to grasp, then the current grasping scenario is a single-object grasping scenario.

[0036] For example, if a robot is performing a task and there are multiple objects that need to be grasped, but these objects are located in different positions and there is obviously no spatial interference between them, then this scenario is a single object grasping scenario with multiple superimposed objects.

[0037] For example, if there are multiple objects to be grasped when the robot is performing a task, and the multiple objects to be grasped are stacked, then the current grasping scenario is a stacking scenario.

[0038] In some embodiments, the color image of the current grasping scene can be obtained using a color camera deployed on the robot; similarly, the point cloud data of the current grasping scene can be obtained using a depth camera deployed on the robot.

[0039] In some embodiments, a 3D camera is deployed on the robot, so that the color image of the current grasping scene and the point cloud data of the current grasping scene can be obtained by the 3D camera.

[0040] Step 102: Construct a dynamic spatial relationship graph based on the point cloud data and geometric attribute data of each object.

[0041] In some embodiments, the geometric attribute data includes at least one of the following: object identification code, 3D point cloud, center point coordinates, 3D bounding box, and average color.

[0042] Step 103: Generate structured text prompts based on the dynamic spatial relationship graph and the grasping priority score of each object in the dynamic spatial relationship graph.

[0043] In this process, the grasping priority score is used to represent the priority of the object being grasped. When generating structured text prompts, the grasping priority score of each object is taken into account. This allows the vision-language-action model to prioritize grasping the top object in the stacked scene when it gives the grasping pose, thereby reducing physical conflicts and safety hazards while increasing the success rate of grasping.

[0044] Step 104: Input the color image of the current grasping scene, the input language command, and the structured text prompts into the vision-language-action model to obtain the grasping pose output by the vision-language-action model.

[0045] In some embodiments, the input language commands may be audio captured using a microphone.

[0046] For example, if the audio captured by the microphone is "grab the ball on top", then the input language command is "grab the ball on top".

[0047] In some embodiments, the input language command may be a command obtained by analyzing the audio captured by the microphone.

[0048] For example, if the audio captured by the microphone is "remove the long planks from that pile and take out the ball at the bottom", then the input language commands are "remove the long planks from that pile" and "take out the ball at the bottom".

[0049] Step 105: Drive the robotic arm to perform the grasping action according to the grasping posture.

[0050] In this embodiment, after acquiring the color image and point cloud data of the current grasping scene, the point cloud data and corresponding geometric attribute data of each object in the current scene can be determined based on these data. Then, a dynamic spatial relationship graph is constructed based on the point cloud data and corresponding geometric attribute data of each object. This dynamic spatial relationship graph can describe not only the geometric attribute data of different objects but also the spatial relationships between them. Therefore, the structured text prompts generated using the dynamic spatial relationship graph and the grasping priority score of each object in the graph can provide a basis for judging the structural stability of the current grasping scene. This allows the grasping pose inferred using the vision-language-action model to be applicable to the grasping requirements of the scene, thereby increasing the success rate of robot grasping.

[0051] In some embodiments, based on a color image of the current grasping scene and point cloud data of the current grasping scene, the point cloud data and geometric attribute data of each object in the current grasping scene are determined, including: The instance segmentation model is invoked to segment the color image of the current capture scene, and the two-dimensional contour of each object in the current capture scene is obtained. Determine the two-dimensional mask corresponding to each object based on its two-dimensional contour. Two-dimensional masks are used to extract point cloud data of the current capture scene to obtain point cloud data of each object in the current capture scene; The geometric attribute data of each object is determined based on the point cloud data of each object in the current grasping scene.

[0052] In this embodiment, the instance segmentation model is used to segment the color image of the current capture scene, which can accurately distinguish different objects in the current capture scene and avoid recognition confusion caused by mutual occlusion between objects or background interference.

[0053] For example, in a scene with multiple stacked parts or tools, the 2D contours obtained through segmentation can clearly define the boundaries of each object. A corresponding 2D mask is then generated, which acts as a "filtering template" for binarization, accurately filtering background point cloud data. This process efficiently extracts point cloud data from individual objects, avoiding the mixing of point clouds from different objects and providing accurate target data for subsequent targeted capture.

[0054] Point cloud data of a single object can accurately calculate key geometric attributes such as volume, surface area, curvature, and center point coordinates. For example, when grasping fragile items, geometric attributes can determine the object's center of gravity, allowing the gripping device to precisely target the center of gravity area and prevent slippage or damage. When grasping irregular parts, curvature data can help determine the surface unevenness and select a flat gripping contact surface. This data directly provides quantitative basis for gripping strategy formulation, significantly reducing gripping deviation.

[0055] In some embodiments, the instance segmentation model can be SAM, or other advanced instance segmentation models such as Mask R-CNN or YOLACT can be used.

[0056] Mask R-CNN or YOLACT are existing instance segmentation models that can be fine-tuned on datasets specific to particular application scenarios to achieve higher segmentation accuracy. For simple scenarios, traditional segmentation methods based on point cloud clustering (such as Euclidean clustering) can also be used.

[0057] In some embodiments, the dynamic spatial relationship graph includes multiple nodes and relationship edges connecting different nodes. The relationship edges include one or more of the first type of edge, the second type of edge, and the third type of edge. The first type of edge is used to indicate that the first object is supported by the second object, the second type of edge is used to indicate that the first object occludes the second object, and the third type of edge is used to indicate that the first object is adjacent to the second object. The multiple nodes include a first node and a second node. The first node is a node defined for the first object in the dynamic spatial relationship graph, and the second node is a node defined for the second object in the dynamic spatial relationship graph. A dynamic spatial relationship graph is constructed based on the point cloud data and geometric attribute data of each object, including: A first subset is determined in the point cloud data of the first object, wherein the z-axis coordinate value of the point cloud located in the first subset is less than the z-axis coordinate value of the point cloud located in the first other subset, and the first other subset is the subset of the point cloud data of the first object that is not in the first subset. A second subset is determined in the point cloud data of the second object, wherein the z-axis coordinate value of the point cloud located in the second subset is greater than the z-axis coordinate value of the point cloud located in other subsets of the second subset, and other subsets of the second subset are subsets in the point cloud data of the second object that are not in the second subset. Based on the minimum vertical distance between the point cloud data corresponding to the first subset and the point cloud data corresponding to the second subset being less than or equal to a preset threshold, construct a first-class edge pointing from the first node to the second node. The point cloud data of the first object is projected onto the two-dimensional plane of the camera to obtain the first depth map, and the point cloud data of the second object is projected onto the two-dimensional plane of the camera to obtain the second depth map. The overlapping area of ​​the masks is determined based on the two-dimensional mask corresponding to the first object and the two-dimensional mask corresponding to the second object. Based on the premise that the first region has no valid depth value or is empty, and the second region has a valid depth value, a second type of edge is constructed from the first node to the second node. The first region is the region in the first depth map that corresponds to the mask overlapping region, and the second region is the region in the second depth map that corresponds to the mask overlapping region. Obtain the shortest Euclidean distance between the first bounding box and the second bounding box. The first bounding box is the bounding box corresponding to the point cloud data of the first object, and the second bounding box is the bounding box corresponding to the point cloud data of the second object. Based on the premise that the shortest Euclidean distance is less than a preset distance value, a third type of edge is constructed pointing from the first node to the second node.

[0058] In this embodiment, the first type of edge, the second type of edge, and the third type of edge are all defined by calculation using objective data, rather than relying on fuzzy visual perception. Therefore, spatial relationships can be accurately quantified, avoiding subjective judgment bias.

[0059] By defining three types of relationship edges—that the first object is supported by the second object, the first object occludes the second object, and the first object is adjacent to the second object—the dynamic spatial relationship graph can cover the spatial relationship between two objects in the space, achieving comprehensive coverage of the core spatial relationships in the grasping scene. The three types of relationship edges cover the core spatial constraints of the multi-object grasping scene, and no additional relationship dimensions are needed to provide complete spatial logic support for grasping strategy formulation.

[0060] In the above embodiments, a relationship graph is constructed through standardized steps such as "subset partitioning," "distance calculation," and "depth verification," without relying on specific object types, shapes, or scene layouts. Whether it is a regular geometry (such as a box), an irregular part, or a stacked or scattered scene, spatial relationships can be calculated through unified logic.

[0061] When the position or orientation of objects in the scene changes, such as when objects move or their stacking state changes during the grasping process, the relationships can be recalculated based on the updated point cloud data and geometric attributes, and the spatial relationship graph can be dynamically updated to adapt to the dynamic changes in the scene during the grasping process.

[0062] In some embodiments, the robot grasping method further includes: Based on the dynamic spatial relationship graph, the top sharpness score, stability impact score, and accessibility score corresponding to each object are determined. The top sharpness score, stability impact score, and accessibility score corresponding to each object are weighted and summed to obtain the grab priority score for each object.

[0063] In this embodiment, a top sharpness score, a stability impact score, and an accessibility score are calculated for each object. The top sharpness score is related to whether the grab point is easy to locate; if there is no top obstruction, the score is high. The stability impact score is related to whether the object and its surroundings are easy to tip over after grabbing; if supported by multiple objects, the score is high. The accessibility score is related to whether the grab path is unobstructed; if there is no obstruction and little nearby interference, the score is high.

[0064] In this process, "quantitative scores" are used to express three types of vague "qualitative judgments": whether the object is easy to locate, whether the object and its surroundings are easy to tip over after being grasped, and whether the grasping path is unobstructed. A clear priority score is obtained by weighted summation, avoiding priority confusion caused by relying on experience or subjective visual judgment.

[0065] Since the top sharpness score, stability impact score, and accessibility score are calculated based on a dynamic spatial relationship graph, the sorting is kept separate from the actual scene. For example, objects that are severely occluded or unsupported will not be given high priority, ensuring that the grabbing decision fits the physical scene constraints.

[0066] In some embodiments, the top sharpness score, stability impact score, and accessibility score corresponding to each object are weighted and summed to obtain the grab priority score for each object, including: Obtain the first weight value, the second weight value, and the third weight value. The first weight value is greater than the second weight value, and the third weight value is greater than the third weight value. Calculate the first product of the top sharpness score and the first weight value; Calculate the second product of the stability impact score and the second weight value; Calculate the third product of the accessibility score and the third weight value; Calculate the sum of the first, second, and third product values ​​to obtain the capture priority score.

[0067] In this embodiment, the first weight value is greater than the second weight value, and the first weight value is greater than the third weight value, so that the capture priority score is more inclined to the top clarity score, focusing on the top clarity. In this process, while highlighting the core position of top clarity, the role of the second weight (stability influence) and the third weight (accessibility) is retained. The sorting logic of prioritizing core needs and supplementing secondary needs is realized by multiplying and summing.

[0068] It avoids the limitations of single-dimensional sorting, such as only looking at the top clarity which may ignore the risk of the object tipping over after being grasped. The priority score focuses on both grasping feasibility and grasping safety and efficiency, making it more in line with actual application scenarios.

[0069] In some embodiments, based on a dynamic spatial relationship graph, the top sharpness score, stability impact score, and accessibility score for each object are determined, including: Get the number of relation edges of the first node. The number of relation edges includes the first number of first-type edges, the second number of second-type edges, and the third number of third-type edges. If there is no first-type edge in the first node, the top sharpness score is 1; if there is a first-type edge in the first node, the top sharpness score is 0. The stability impact fraction is exp(-α×N_sup), where α is the first attenuation coefficient and N_sup is the second quantity; The reachability score is exp(-β×N_occ), where β is the second attenuation coefficient and N_occ is the third quantity.

[0070] In this embodiment, the top sharpness score is determined using 0-1 binarization, and the value can be directly obtained simply by whether there is a first-class edge, without the need for complex gradient calculation or feature fusion.

[0071] Among them, α and β are adjustable parameters that can optimize the score sensitivity according to the characteristics of the scene. For example, in industrial sorting scenarios, β can be increased to make the accessibility score more sensitive to nearby objects and prioritize grasping objects without interference. When grasping heavy objects, α can be increased to make the stability score more sensitive to occlusion and prioritize grasping objects with less occlusion and less likely to tip over.

[0072] In this process, the binarization rule, the exponential decay formula, and the variable definitions (N_sup corresponds to the number of edges of the second type, and N_occ corresponds to the number of edges of the third type) are all clearly quantifiable.

[0073] Regardless of the type or quantity of objects in the scene, scores can be calculated according to uniform rules, avoiding calculation deviations caused by subjective judgment or ambiguous parameters, and ensuring the comparability of scores and the reliability of ranking in different scenes.

[0074] In some embodiments, the overall process of the robot grasping method can be divided into four main steps, which correspond to four modules, such as... Figure 2 As shown, the modules are: multimodal perception and instance segmentation module 201, dynamic spatial relationship graph construction module 202, VLA-driven graph augmentation reasoning and action generation module 203, and action execution and closed-loop feedback module 204.

[0075] like Figure 3As shown, the robot grasping method includes the following steps: Step 301: Data Acquisition.

[0076] Specifically, the target scene is perceived by an RGB-D camera, and color images and corresponding point cloud data are collected to provide multimodal input information for subsequent instance segmentation and 3D structure reasoning.

[0077] Step 302: Instance splitting.

[0078] Specifically, an instance segmentation model (such as the Segment Anything Model, SAM) is invoked to segment the RGB image, identify and label the two-dimensional region contours of each independent object in the scene, and generate a preliminary 2D instance mask.

[0079] Step 303: Extraction of 3D objects.

[0080] Specifically, the 2D mask is aligned with the point cloud data, 3D point cloud fragments of each object are extracted, and their geometric properties are calculated.

[0081] Step 304: Construct a directed graph: nodes represent object instances, and edges represent the relationships between objects.

[0082] Specifically, based on the 3D point cloud and attributes of all objects, a dynamic spatial relationship graph (DSRG) is constructed to establish the support, occlusion, and proximity relationships between objects.

[0083] Step 305: Calculate the overall crawling priority score.

[0084] Specifically, in the DSRG graph structure, the feasibility of crawling each target object node is analyzed and ranked. Combining the structural dependencies of adjacent edges in the graph, the crawling priority score of each object is calculated.

[0085] Step 306: Graph information serialization.

[0086] Specifically, the structure and priority analysis results of DSRG are serialized into structured text prompts.

[0087] Step 307: Multimodal fusion input.

[0088] Specifically, the original RGB image, user language commands, and structured text prompts are input into the VLA model.

[0089] Step 308: VLA Reasoning and Decision Making.

[0090] Specifically, the VLA model performs multimodal reasoning to generate physically plausible 6-DoF grasping poses.

[0091] Step 309: Action planning and execution.

[0092] Specifically, the robot motion planning module plans a collision-free path and drives the robotic arm to perform grasping.

[0093] Step 310: Status update, task completed. If the judgment result is negative, return to step 1 and enter the next round of perception-reasoning-action loop; if the judgment result is positive, end.

[0094] The multimodal perception and instance segmentation module is responsible for receiving sensor data and segmenting object instances in the scene, corresponding to steps 301 to 303 above; the dynamic spatial relationship graph construction module is responsible for constructing and analyzing the physical relationship graph of the scene based on the object instances, corresponding to steps 304 and 305 above; the VLA-driven graph augmentation reasoning and action generation module is responsible for fusing multimodal information and making the final grasping decision, corresponding to steps 306 to 308 above; and the action execution and closed-loop feedback module is responsible for executing the grasping action and updating the system state, corresponding to steps 309 and 310 above.

[0095] The data is transmitted through the following flow: Multimodal perception and instance segmentation module → Dynamic spatial relationship graph construction module → VLA-driven graph augmentation reasoning and action generation module → Action execution and closed-loop feedback module → Multimodal perception and instance segmentation module, thus forming a closed loop.

[0096] Specifically, the multimodal perception and instance segmentation module mainly includes the following operations: The data acquisition corresponding to step 301 is as follows: After the robot starts, the 3D camera, i.e., the RGB-D camera (such as the Intel RealSense D435), is controlled to capture the color image I_rgb and the depth map I_depth of the current working scene. Among them, the depth map I_depth is converted into a 3D point cloud P_scene, which serves as the point cloud data of the current grasping scene.

[0097] The instance segmentation corresponding to step 302 involves inputting the color image I_rgb into a pre-trained instance segmentation base model, such as SAM. SAM processes the image and outputs a set of binary masks {M_1, M_2, ..., M_n}, where each mask M_i corresponds to an independent object instance in the scene.

[0098] The 3D object extraction corresponding to step 303 is as follows: For each mask M_i, the system uses the camera's intrinsic and extrinsic parameters to project it onto the 3D point cloud P_scene, thereby extracting the point cloud subset P_i belonging to that object. For each point cloud subset P_i, its geometric attribute data is calculated, including: object ID, 3D point cloud, center point coordinates, 3D bounding box, average color, etc.

[0099] Specifically, in the dynamic spatial relationship graph construction module, a directed graph G=(V,E), namely the dynamic spatial relationship graph (DSRG), is constructed based on all object instances {P_1, P_2, ..., P_n} extracted by the multimodal perception and instance segmentation module. This mainly includes the following operations: 1. Node (V) definition: Each node v_i in the graph represents an object instance, and its attributes store the information calculated in step 303.

[0100] 2. The definition and calculation of the edge (E) corresponding to step 304, i.e., the relationship change in this invention, wherein the edge represents the key physical relationship between objects, and is determined by the following algorithm: The first type of edge SUPPORTED_BY(v_i, v_j): indicates that object i is supported by object j, and the algorithm for determining this is as follows: a represents the subset P_i_bottom with the lowest Z coordinate in the point cloud P_i of object i.

[0101] b, extract the subset P_j_top with the highest Z coordinate in the point cloud P_j of object j.

[0102] c. Regarding the contact condition in the Z direction, that is, calculate the minimum vertical distance d_z = min_(p_i∈P_i_bottom, p_j∈P_j_top)|z(p_i)-z(p_j)| between the bottom point cloud subset P_i_bottom of object i and the top point cloud subset P_j_top of object j.

[0103] When d_z≤δ_z, it is determined that there is real contact between the two objects in the Z-axis direction. δ_z is a threshold set according to the accuracy of the depth sensor, which is also the preset threshold in this invention. A SUPPORTED_BY edge from v_i to v_j is established. If d_z>δ_z, it is determined that there is a gap or suspension between the two objects, and no SUPPORTED_BY edge is established.

[0104] The preset threshold value ranges from 0.002m to 0.003m.

[0105] In some embodiments, the algorithm for determining the first type of edge SUPPORTED_BY further includes: c. Check if a sufficient proportion of the points in P_i_bottom are projected onto the XY plane and fall within the convex hull of the projection of P_j_top onto the XY plane.

[0106] d, calculate whether the projection of the centroid of v_i onto the XY plane also falls within the supporting convex hull of v_j.

[0107] If the judgment conditions c and d are satisfied in addition to satisfying the judgment algorithm for the first type of edge SUPPORTED_BY, the stability of the first type of edge SUPPORTED_BY can be ensured.

[0108] Regarding the second type of edge OCCLUDES(v_i, v_j), it indicates that from the camera's perspective, object i occludes object j. The algorithm for determining this is as follows: The point clouds P_i and P_j of v_i and v_j are reprojected onto the two-dimensional image plane of the camera to generate depth maps D_i(u,v) and D_j(u,v) respectively, and the corresponding two-dimensional masks M_i and M_j are obtained from the instance segmentation module.

[0109] Calculate the intersection region M_ov=M_i∩M_j of the mask, which represents the overlapping portion of the projections of the two objects on the image plane.

[0110] Within the M_ov region, if D_j(u, v) does not exist (invalid depth or null value), while D_i(u, v) has a valid depth at that location and is continuous with its surrounding pixels, then v_j is considered to be completely occluded by v_i in that viewing direction. In this case, an OCCLUDES(v_i, v_j) will be established pointing from v_i to v_j.

[0111] Regarding the third type of edge ADJACENT_TO(v_i, v_j), which indicates that object i is adjacent to object j, the algorithm for determining this proximity is as follows: Calculate the shortest Euclidean distance between the 3D bounding boxes of v_i and v_j.

[0112] If the distance is less than the preset threshold d_adj, which is the preset distance value in this invention, then an ADJACENT_TO edge is established from v_i to v_j.

[0113] In some embodiments, the preset distance value is 2 centimeters.

[0114] 3. The dynamic crawling priority analysis corresponding to step 305 is used to calculate a comprehensive crawling priority score for each node v_i on DSRG. Specifically: Top_Clearance_Score: If node v_i has no incoming edges SUPPORTED_BY(v_j, v_i), meaning no other object is supported by v_i, its score is 1; otherwise, it is 0. This is a primary condition for crawling.

[0115] Stability Score: This score is inversely proportional to the number of objects supported by v_i. If the number of incoming edges SUPPORTED_BY(v_j, v_i) of v_i is N_sup, then the score is exp(-α×N_sup), where α is the first decay coefficient. The more objects supported, the lower the score.

[0116] Accessibility Score: This score is inversely proportional to the degree to which v_i is occluded. If the number of incoming edges OCCLUDES(v_j, v_i) of v_i is N_occ, then the score is exp(-β×N_occ), where β is the second attenuation coefficient.

[0117] The overall priority (Priority_Score) is a crawling priority score calculated for each node v_i, expressed as: Priority_Score(v_i) = w1×Top_Clearance_Score+w2×Stability_Score+w3×Accessibility_Score.

[0118] Where w1, w2, and w3 are the first, second, and third weight values, respectively, and w1 >> w2 and w1 >> w3, to ensure that the top-level object is captured first.

[0119] Specifically, the VLA-driven graph augmentation reasoning and action generation module mainly includes the following operations: 1. Graph information serialization corresponding to step 306: Convert the structure and analysis results of DSRG into structured text prompts that the VLA model can understand, with the following format: text Scene Description: -Object 1 (blue cube) at [0.5, 0.1, 0.8]. Priority_Score: 0.95.Graspability: High (Top-level, stable to pick). -Object 2 (red cylinder) at [0.5, 0.1, 0.7]. Priority_Score: 0.20.Graspability: Low (Supporting Object 1). -Object 3 (green sphere) at [0.6, 0.2, 0.7]. Priority_Score: 0.65.Graspability: Medium (Partially occluded by Object 1). Physical Relationships: - Object 1 SUPPORTED_BY Object 2. - Object 1 OCCLUDES Object 3." 2. The multimodal fusion input corresponding to step 307, that is, the following three parts of information are provided as unified input to the VLA model: Visual input: the original RGB image I_rgb.

[0120] Language commands: Task commands given by the user, such as the specific command "pick up the blue cube" or the vague command "clear the table".

[0121] Structured context: Serialized text of the graph information generated in the previous step.

[0122] 3. The VLA reasoning and decision-making corresponding to step 308, that is, the attention mechanism inside the VLA model will process these three parts of information simultaneously.

[0123] Specifically, when it receives the instruction "pick up the blue cube", VLA uses visual-linguistic alignment to associate "blue cube" in the instruction with the blue cube in the image and "Object 1" in the text. Then, it reads "Priority_Score: 0.95" and "Graspability: High", which greatly enhances its confidence in selecting Object1.

[0124] When VLA receives the instruction "clear the table", it does not specify a target. Instead, it analyzes the priority of all objects in the text and automatically selects "Object1", which has the highest Priority_Score, as the first target to grab.

[0125] Even if the VLA's visual part might think that "Object2" (the red cylinder) is more regular in shape and easier to grasp, the text prompt "Supporting Object1" will act as a "veto," preventing the model from generating the action to grasp Object2, thereby avoiding scene collapse.

[0126] The VLA ultimately outputs a specific grasping action of the robot's end effector, which is in the format of an 8-dimensional vector representing the 6-DoF pose and gripper opening / closing amount: [x, y, z, qx, qy, qz, qw, d].

[0127] Where [x, y, z, qx, qy, qz] are 6-DoF poses, and d is the gripper opening / closing amount.

[0128] Specifically, the action execution and closed-loop feedback module mainly includes the following operations: 1. Path planning and execution corresponding to step 309: This refers to the motion planning module (such as MoveIt!) in the robot control system receiving the target grasping pose generated by VLA. This module performs collision detection, plans a collision-free path from the current state to the target pose, and drives the robotic arm to execute it precisely.

[0129] 2. State Update and Looping Corresponding to Step 310: After a successful grasping action (confirmed again via gripper sensor or vision), the grasped object has been removed from the scene. The system determines whether the task is complete (e.g., the specified object has been grasped, or there are no more objects on the table). If the task is not complete, the system loops back to step 301 and performs scene perception again. At this time, the newly generated DSRG will no longer contain the grasped object node, and the relationships and priority scores of the objects previously supported or occluded by it will be automatically updated, enabling intelligent execution of the next operation.

[0130] In this embodiment, a method is proposed to serialize structured graph information (DSRG) into natural language or structured text and use it as additional input context (Prompt) for the VLA model. This "graph augmentation hint" technique "injects" explicit physical constraints and common sense into the VLA decision-making process, significantly improving its reasoning accuracy and robustness in physically complex scenarios.

[0131] Furthermore, the system re-perceives and remodels the scene after each action. This enables the system not only to perform single-step grasping but also to handle continuous, complex tasks that change the environmental state and adapt in real time to scene changes caused by its own actions or external interference.

[0132] It should be noted that the robot grasping device provided by the present invention can execute the robot grasping method of any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0133] like Figure 4 As shown, the robot grasping device provided by the present invention includes: The determination module 401 is used to determine the point cloud data and geometric attribute data of each object in the current capture scene based on the color image of the current capture scene and the point cloud data of the current capture scene. Module 402 is used to construct a dynamic spatial relationship graph based on the point cloud data and geometric attribute data of each object. Processing module 403 is used to generate structured text prompts based on the dynamic spatial relationship graph and the grasping priority score of each object in the dynamic spatial relationship graph; and The color image of the current grasping scene, the input language command, and the structured text prompt are input into the vision-language-action model to obtain the grasping pose output by the vision-language-action model; The gripping module 404 is used to drive the robotic arm to perform gripping according to the gripping posture.

[0134] In this embodiment, the dynamic spatial relationship graph can describe the geometric attribute data of different objects, as well as the spatial relationship between different objects. Therefore, the structured text prompts generated by the dynamic spatial relationship graph and the grasping priority score of each object in the dynamic spatial relationship graph can provide a basis for judging the stability of the current grasping scene structure, so that the grasping pose inferred by the vision-language-action model can be applied to the grasping requirements of the scene, thereby improving the success rate of robot grasping.

[0135] In some embodiments, a robotic grasping system is proposed, comprising: 3D camera; robotic arm; The controller, connected to the 3D camera and robotic arm, is used to perform any of the robotic grasping methods described above.

[0136] For example, such as Figure 5 As shown, the 3D camera 501 is mounted on the robotic arm 502. The robotic arm 502 achieves flexible movement through a multi-joint design, providing pose adjustment function for the end effector.

[0137] The end effector is equipped with grippers, which rely on the opening and closing action of the mechanical structure to complete operations such as grasping and releasing the target object.

[0138] Among them, the 3D camera 501 and the robotic arm 502 are connected to the central control platform 503 via cables. The central control platform can realize the deployment of control systems and data interaction for the robotic arm, camera, and gripper.

[0139] In some embodiments, the controller is deployed on a central control host platform.

[0140] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a robot grasping method, which includes: determining the point cloud data and geometric attribute data of each object in the current grasping scene based on the color image and point cloud data of the current grasping scene; constructing a dynamic spatial relationship graph based on the point cloud data and geometric attribute data of each object; generating structured text prompts based on the dynamic spatial relationship graph and the grasping priority score of each object in the dynamic spatial relationship graph; inputting the color image of the current grasping scene, the input language command, and the structured text prompts into the vision-language-action model to obtain the grasping pose output by the vision-language-action model; and driving the robotic arm to perform grasping according to the grasping pose.

[0141] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by the computer, the computer is able to execute the robot grasping method provided in the above embodiments. The method includes: determining the point cloud data and geometric attribute data of each object in the current grasping scene based on the color image and point cloud data of the current grasping scene; constructing a dynamic spatial relationship graph based on the point cloud data and geometric attribute data of each object; generating a structured text prompt based on the dynamic spatial relationship graph and the grasping priority score of each object in the dynamic spatial relationship graph; inputting the color image of the current grasping scene, the input language command, and the structured text prompt into the vision-language-action model to obtain the grasping pose output by the vision-language-action model; and driving the robotic arm to perform grasping according to the grasping pose.

[0143] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the robot grasping method provided in the above embodiments. The method includes: determining the point cloud data and geometric attribute data of each object in the current grasping scene based on a color image of the current grasping scene and point cloud data of the current grasping scene; constructing a dynamic spatial relationship graph based on the point cloud data and geometric attribute data of each object; generating a structured text prompt based on the dynamic spatial relationship graph and the grasping priority score of each object in the dynamic spatial relationship graph; inputting the color image of the current grasping scene, the input language command, and the structured text prompt into a vision-language-action model to obtain a grasping pose output by the vision-language-action model; and driving the robotic arm to perform grasping according to the grasping pose.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot grasping method, characterized in that, include: Based on the color image of the current capture scene and the point cloud data of the current capture scene, determine the point cloud data and geometric attribute data of each object in the current capture scene. A dynamic spatial relationship graph is constructed based on the point cloud data and geometric attribute data of each of the objects. Based on the dynamic spatial relationship graph and the grasping priority score of each object in the dynamic spatial relationship graph, a structured text prompt is generated; The color image of the current grasping scene, the input language command, and the structured text prompt are input into the vision-language-action model to obtain the grasping pose output by the vision-language-action model; The robotic arm is driven to perform the grasping action according to the grasping posture.

2. The robot grasping method according to claim 1, characterized in that, The step of determining the point cloud data and geometric attribute data of each object in the current capture scene based on the color image and point cloud data of the current capture scene includes: The instance segmentation model is invoked to segment the color image of the current capture scene to obtain the two-dimensional contour of each object in the current capture scene; Determine the two-dimensional mask corresponding to each object based on the two-dimensional contour; The point cloud data of the current capture scene is extracted using the two-dimensional mask to obtain the point cloud data of each object in the current capture scene; The geometric attribute data of each object is determined based on the point cloud data of each object in the current grasping scene.

3. The robot grasping method according to claim 1, characterized in that, The dynamic spatial relationship graph includes multiple nodes and relationship edges connecting different nodes. The relationship edges include one or more of the first type of edge, the second type of edge, and the third type of edge. The first type of edge is used to indicate that the first object is supported by the second object. The second type of edge is used to indicate that the first object occludes the second object. The third type of edge is used to indicate that the first object is adjacent to the second object. The multiple nodes include a first node and a second node. The first node is a node defined for the first object in the dynamic spatial relationship graph, and the second node is a node defined for the second object in the dynamic spatial relationship graph. The construction of a dynamic spatial relationship graph based on the point cloud data and geometric attribute data of each object includes: A first subset is determined in the point cloud data of the first object, wherein the z-axis coordinate value of the point cloud located in the first subset is less than the z-axis coordinate value of the point cloud located in a first other subset, and the first other subset is a subset of the point cloud data of the first object that is outside the first subset. A second subset is determined in the point cloud data of the second object, wherein the z-axis coordinate value of the point cloud located in the second subset is greater than the z-axis coordinate value of the point cloud located in the other second subset, and the other second subset is a subset of the point cloud data of the second object that is not in the second subset. Based on the fact that the minimum vertical distance between the point cloud data corresponding to the first subset and the point cloud data corresponding to the second subset is less than or equal to a preset threshold, the first type of edge is constructed from the first node to the second node; The point cloud data of the first object is projected onto the two-dimensional plane of the camera to obtain a first depth map, and the point cloud data of the second object is projected onto the two-dimensional plane of the camera to obtain a second depth map. The overlapping area of ​​the masks is determined based on the two-dimensional mask corresponding to the first object and the two-dimensional mask corresponding to the second object. Based on the premise that the first region has no valid depth value or is empty, and the second region has a valid depth value, a second type of edge is constructed from the first node to the second node. The first region is the region in the first depth map that corresponds to the overlapping region of the mask, and the second region is the region in the second depth map that corresponds to the overlapping region of the mask. Obtain the shortest Euclidean distance between the first bounding box and the second bounding box, where the first bounding box is the bounding box corresponding to the point cloud data of the first object, and the second bounding box is the bounding box corresponding to the point cloud data of the second object. Based on the fact that the shortest Euclidean distance is less than a preset distance value, the third type of edge is constructed from the first node to the second node.

4. The robot grasping method according to any one of claims 1 to 3, characterized in that, The robot grasping method also includes: Based on the dynamic spatial relationship diagram, the top sharpness score, stability impact score, and accessibility score corresponding to each object are determined. The top sharpness score, stability impact score, and accessibility score corresponding to each object are weighted and summed to obtain the grab priority score for each object.

5. The robot grasping method according to claim 4, characterized in that, The top sharpness score, stability impact score, and accessibility score corresponding to each object are weighted and summed to obtain the grabbing priority score for each object, including: Obtain a first weight value, a second weight value, and a third weight value, wherein the first weight value is greater than the second weight value, and the third weight value is greater than the first weight value; Calculate the first product of the top sharpness score and the first weight value; Calculate the second product of the stability impact score and the second weight value; Calculate the third product of the reachability score and the third weight value; The sum of the first product value, the second product value, and the third product value is calculated to obtain the capture priority score.

6. The robot grasping method according to claim 4, characterized in that, The determination of the top sharpness score, stability impact score, and accessibility score for each object based on the dynamic spatial relationship graph includes: Obtain the number of relational edges of the first node, wherein the number of relational edges includes a first number of first-type edges, a second number of second-type edges, and a third number of third-type edges; If there is no first-type edge in the first node, the top sharpness score is 1; if there is a first-type edge in the first node, the top sharpness score is 0. The stability impact fraction is exp(-α×N_sup), where α is the first attenuation coefficient and N_sup is the second quantity; The reachability score is exp(-β×N_occ), where β is the second attenuation coefficient and N_occ is the third quantity.

7. A robotic grasping device, characterized in that, include: The determination module is used to determine the point cloud data and geometric attribute data of each object in the current capture scene based on the color image of the current capture scene and the point cloud data of the current capture scene. A construction module is used to construct a dynamic spatial relationship graph based on the point cloud data and geometric attribute data of each of the objects. The processing module is used to generate structured text prompts based on the dynamic spatial relationship graph and the grasping priority score of each object in the dynamic spatial relationship graph; as well as The color image of the current grasping scene, the input language command, and the structured text prompt are input into the vision-language-action model to obtain the grasping pose output by the vision-language-action model; The grasping module is used to drive the robotic arm to perform grasping according to the grasping posture.

8. A robotic grasping system, characterized in that, include: 3D camera; robotic arm; A controller, connected to the 3D camera and the robotic arm, is used to perform the robot grasping method as described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the robot grasping method as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot grasping method as described in any one of claims 1 to 6.

Images