A grasping method based on TSDF and semantic constraint

Abstract

This invention discloses a grasping method based on TSDF and semantic constraints, belonging to the field of robot vision perception and intelligent operation technology. The method includes the following steps: S1: Scene observation and point cloud construction; S2: Scene distance field construction; S3: Target region extraction; S4: Candidate grasping posture generation; S5: Execution volume construction; S6: Minimum clearance calculation; S7: Geometric feasibility screening and optimal grasping posture determination; S8: Execution mapping. This invention can simultaneously consider the rationality of local contact and the overall execution feasibility under target semantic constraints, improving the effectiveness and stability of grasping posture generation in desktop grasping scenarios.

Description

Technical Field

[0001] This invention relates to the field of robot vision perception and intelligent operation technology, specifically to a grasping method based on TSDF and semantic constraints. The method constructs a unified scene representation of TSDF / ESDF based on multi-view RGB-D observation data, generates candidate grasping postures under target semantic constraints, uses ESDF to perform geometric feasibility screening of the approach phase, closing phase and gripper body entry phase, and then outputs an executable grasping posture. Background Technology

[0002] With the widespread application of robotics technology in intelligent manufacturing, warehousing and sorting, service robots, and human-robot collaboration, autonomous grasping has become a crucial foundational capability for achieving intelligent robot operation. For target grasping tasks in desktop scenarios, robots typically need to perform continuous operations based on visual perception, including target localization, grasping posture analysis, end effector approach, gripper closure, and stable lifting. Therefore, grasping methods must consider not only the local contact relationships on the target surface but also the spatial accessibility and environmental constraints during the grasping process.

[0003] Among existing grasping methods, one type primarily constructs the grasping posture based on the local geometric features of the target surface. These methods can typically generate candidate grasping postures quickly on the target's local surface. However, for a two-finger parallel gripper, seemingly reasonable contact point pairs on the local surface do not necessarily mean that the grasping posture is executable in a real-world scenario. The gripper must also go through stages such as approach, closing, and body entry during grasping. If any of these stages interferes with the tabletop, background, or nearby obstacles, the corresponding grasping posture cannot be considered a valid grasping solution.

[0004] Another type of method attempts to directly predict or regress the grasping pose using point cloud or depth observation data. However, these methods are often sensitive to the completeness of the input observation data, noise levels, and scene changes. When there are occlusions, interference objects, or limited space around the target, generating grasping results based solely on locally visible areas can easily lead to problems where the target appears graspable in some areas but cannot be safely approached or closed during actual execution.

[0005] Furthermore, when performing grabbing analysis directly on the entire scene surface, the desktop, background, and non-target objects may all enter the candidate grabbing generation range. This not only expands the search space but also weakens the effectiveness of grabbing analysis for the current task objective. For desktop single-target grabbing tasks, if the task objective is not defined before grabbing analysis, the subsequently generated candidate grabbing postures are prone to mixing into non-target areas, increasing the risk of invalid candidates and false grabbing.

[0006] Therefore, how to conduct a unified analysis of the executability of candidate grabbing postures in the approach, closure, and ontology entry phases under the constraint of target semantics and by combining continuous spatial distance representation, so as to ensure both the rationality of local contact and the feasibility of overall execution, is a technical problem that urgently needs to be solved in desktop grabbing tasks. Summary of the Invention

[0007] The technical problem to be solved by this invention is: how to solve the problem that existing grasping methods generate grasping postures based only on the local contact relationship of the target surface, and it is difficult to simultaneously take into account the semantic constraints of the target, free space constraints and the execution mapping of the robotic arm. This invention provides a grasping method based on TSDF and semantic constraints, so that the generated grasping postures not only meet the rationality of local contact, but also meet the overall execution feasibility of the gripper during the approach, closing and body entry processes.

[0008] The present invention solves the above-mentioned technical problems through the following technical solution, and the present invention includes the following steps:

[0009] S1: Scene Observation and Point Cloud Construction

[0010] The RGB-D camera is used to acquire multi-view RGB images, depth images, and camera pose information of the target scene, and the effective depth observation data from each viewpoint is used to reconstruct the scene point cloud in a unified reference coordinate system. ;

[0011] S2: Scene Distance Field Construction

[0012] Based on scene point cloud Construct a truncated symbolic distance field (TSDF) for scene voxels, and generate an Euclidean symbolic distance field (ESDF) from the TSDF;

[0013] S3: Target Region Extraction

[0014] Based on the semantic region corresponding to the target category, from the scene point cloud Extract the target point cloud corresponding to the current task objective. For the target point cloud The purification process is performed to obtain the purified target point cloud. ;

[0015] S4: Candidate Grasping Pose Generation

[0016] Based on target point cloud Generate candidate grasping poses, which include at least the grasping center, closing direction, approach direction, grasping pose, pre-grab pose, and gripper opening width;

[0017] S5: Perform volume building

[0018] For each candidate grasping posture, construct the proximity channel volume, the closed corridor volume, and the gripper body volume respectively;

[0019] S6: Minimum Clearance Calculation

[0020] ESDF is used to calculate the minimum clearance corresponding to the volume of the approach channel, the volume of the closed corridor, and the volume of the gripper body, respectively.

[0021] S7: Geometric Feasibility Screening and Determination of Optimal Grasping Posture

[0022] Based on the minimum clearance, the candidate grasping postures are geometrically screened to obtain a set of feasible candidate grasping postures. The system performs redundancy suppression and comprehensive scoring on the feasible candidate grasping set to determine the optimal grasping posture. ;

[0023] S8: Execute mapping

[0024] Optimal grasping posture The mapping is a sequence of pre-grasping, approaching, closing, and lifting phased motions that can be executed by the end-effector of the robotic arm.

[0025] Furthermore, in step S1, the specific processing procedure is as follows:

[0026] S11: Based on depth image and camera intrinsics The effective depth pixels under each viewpoint are back-projected into three-dimensional points in the camera coordinate system;

[0027] S12: Based on the extrinsic transformation matrix from the camera coordinate system to the robot base coordinate system Map 3D points from various perspectives to the robot's base coordinate system;

[0028] S13: Unify and fuse 3D points from multiple perspectives to obtain scene point clouds. ,in, Indicates the first The set of spatial points corresponding to the frame, and has .

[0029] Furthermore, in step S2, the specific processing procedure is as follows:

[0030] S21: Scene point cloud under a unified reference coordinate system Perform voxel fusion, update the truncated symbolic distance value and cumulative weight of each voxel in a recursive manner, and construct the truncated symbolic distance field (TSDF) of scene voxels;

[0031] S22: Restore the scene surface based on TSDF;

[0032] S23: Based on the restored scene surface and the occupied voxels and free space boundaries, generate an ESDF through Euclidean distance propagation or neighborhood update to obtain any query location in the scene. Euclidean distance to the nearest obstacle .

[0033] Furthermore, in step S3, the specific processing procedure is as follows:

[0034] S31: Based on target category Semantic regions corresponding to each perspective Constrained recovery is performed on multi-view effective depth observation data to obtain a preliminary target point cloud. ;

[0035] S32: Initial target point cloud Perform at least one of the following processes: connected component consistency processing, outlier removal, and voxel downsampling, to obtain the purified target point cloud. .

[0036] Furthermore, in step S4, the specific processing procedure is as follows:

[0037] S41: Based on target point cloud Left and right contact points in and Determine the contact center Closing direction and gripper opening width ;

[0038] S42: Based on left and right contact normals and Constructing approach direction And establish a local coordinate system for the candidate grasping postures;

[0039] S43: Determine the grasping pose based on the local coordinate system. and pre-grab pose ;

[0040] S44: Based on the gripper opening width range, left and right contact normal matching relationship, and local contact quality threshold, preliminary screening of candidate gripping postures is performed, eliminating candidate gripping postures that do not meet the gripper opening range constraint or local contact quality requirements, forming an original candidate gripping posture set. .

[0041] Furthermore, in step S5, the specific processing procedure is as follows:

[0042] S51: Construct a volume close to the channel volume , used to describe the sweep volume when the gripper advances from the pre-grasping pose to the grasping pose;

[0043] S52: Construct the volume of a closed corridor This describes the unobstructed space required for the gripper fingers to complete the closing action;

[0044] S53: Construct the gripper body volume It is used to describe the overall space occupied by the palm and base of the fingers of the gripper in the grasping position.

[0045] 7. The crawling method based on TSDF and semantic constraints according to claim 6, characterized in that, in step S6, the specific processing procedure is as follows:

[0046] S61: Calculate the volume of the proximity channel Corresponding minimum clearance ;

[0047] S62: Calculate the volume of a closed corridor Corresponding minimum clearance ;

[0048] S63: Calculate the volume of the gripper body Corresponding minimum clearance .

[0049] Furthermore, in step S7, the geometric feasibility screening process includes the following steps:

[0050] S71: Minimum clearance when approaching the channel volume Not less than the safety margin threshold of the approach stage Minimum clearance for the volume of a closed corridor Not less than the safety margin threshold during the closing phase And the minimum clearance of the gripper body volume Not less than the security margin threshold of the main body occupation phase When the time is right, the corresponding candidate grasping posture is determined to be a feasible candidate grasping posture;

[0051] S72: Minimum clearance margin based on feasible candidate grasping postures during the approach, closing, and body occupancy phases. Form a set of feasible candidate grasping postures .

[0052] Furthermore, in step S7, the redundancy suppression and comprehensive scoring process includes the following steps:

[0053] S73: Set of feasible candidate grasping postures For any two candidate grasping poses, the difference in translation between their grasping centers is considered. and posture differences A similarity judgment is performed. When both the translation difference and the pose difference are less than the corresponding threshold, redundant candidate grasping poses are deleted to obtain a set of feasible candidate grasping poses after redundancy removal.

[0054] S74: Based on at least two of the following indices—normal consistency, contact symmetry, contact balance, minimum clearance margin, gripping consistency, and attitude stability—comprehensively score the candidate gripping postures in the set of feasible candidate gripping postures after redundancy removal, and determine the candidate gripping posture with the highest score as the optimal gripping posture. .

[0055] Furthermore, in step S8, the specific processing procedure is as follows:

[0056] S81: Optimal grasping posture Capture pose in camera coordinate system Mapped to the end-effector target pose in the robot base coordinate system That is, to obtain the grasping pose. ;

[0057] S82: Retreat a predetermined distance in the opposite direction from the approach direction of the optimal grasping posture. Constructing pre-grasp pose ;

[0058] S83: Based on pre-captured pose and grasping pose The robot arm generates a sequence of actions that sequentially complete the four stages of pre-grabbing, approaching, closing, and lifting.

[0059] The present invention has the following advantages over the prior art:

[0060] 1. This invention constructs a unified TSDF / ESDF scene representation using multi-view RGB-D observation data, which can simultaneously provide continuous and stable geometric support for target surface restoration, candidate grabbing construction, and free space distance query, thereby improving the integrity and consistency of grabbing analysis.

[0061] 2. By introducing target semantic region extraction, this invention narrows the scope of grasping analysis from the entire scene to the vicinity of the current task target, reducing the interference of desktop, background and non-target objects on the generation of candidate grasping postures, thereby improving the correlation between candidate grasping postures and task targets.

[0062] 3. This invention constructs the approach channel volume, the closed corridor volume, and the gripper body volume separately, and uses ESDF to calculate the corresponding minimum clearance. It performs unified constraint analysis on the candidate gripping in the three execution stages of approach, closure, and body entry, so that the selected gripping posture not only meets the local contact conditions, but also meets the overall execution feasibility.

[0063] 4. By performing redundancy suppression and comprehensive scoring on feasible candidate grasping postures, this invention can output a better grasping posture based on consideration of contact quality, safety margin and task preference, and further map it into a sequence of pre-grasping, approaching, closing and lifting actions that can be directly executed by the end of the robotic arm, thereby improving the effectiveness and stability of grasping posture generation in desktop grasping scenarios. Attached Figure Description

[0064] Figure 1 This is the overall flowchart of the crawling method based on TSDF and semantic constraints in this embodiment of the invention;

[0065] Figure 2 This is a diagram showing the coordinate system transformation and capture execution mapping relationship in an embodiment of the present invention;

[0066] Figure 3 This is a multi-view depth observation and back-projection point cloud result image in an embodiment of the present invention;

[0067] Figure 4 This is a flowchart illustrating the TSDF / ESDF mapping principle in an embodiment of the present invention;

[0068] Figure 5 This is a diagram showing the results of scene voxel reconstruction and surface restoration in an embodiment of the present invention;

[0069] Figure 6 These are images showing the target region extraction results before and after semantic filtering in this embodiment of the invention;

[0070] Figure 7 These are images showing the results before and after candidate capture and filtering in an embodiment of the present invention;

[0071] Figure 8 This is a schematic diagram of the capture and execution process in an embodiment of the present invention. Detailed Implementation

[0072] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0073] like Figure 1As shown, this embodiment provides a technical solution: a grasping method based on TSDF and semantic constraints, including the following steps: scene observation and point cloud construction, scene distance field construction, target region extraction, candidate grasping posture generation, execution volume construction, minimum clearance calculation, geometric feasibility screening and optimal grasping determination, and execution mapping. This method does not simplify the grasping problem to local geometric matching of the target surface, but rather organizes target semantic constraints, local surface geometric analysis, environmental free space constraints, and robotic arm execution mapping into a complete grasping decision chain. For ease of explanation, this embodiment adopts the following coordinate system definition: the world coordinate system is denoted as... The robot's base coordinate system is denoted as The target coordinate system is denoted as The gripper coordinate system is denoted as The camera coordinate system is denoted as .in, This represents the homogeneous pose transformation matrix of the robot's base coordinate system relative to the world coordinate system. This represents the homogeneous pose transformation matrix of the target coordinate system relative to the world coordinate system. This represents the homogeneous pose transformation matrix of the gripper coordinate system relative to the robot base coordinate system. This represents the homogeneous pose transformation matrix of the camera coordinate system relative to the gripper coordinate system.

[0074] Step 1: Scene Observation and Point Cloud Construction

[0075] like Figure 3 As shown, firstly, multi-view RGB images, depth images, and camera pose information of the target scene are acquired. Based on the RGB-D imaging model and the depth back-projection relationship, the effective depth observation data (the effective depth observation data refers to the set of depth measurement values ​​corresponding to the effective depth pixels after removing invalid values ​​in the depth image) under each viewpoint are restored to three-dimensional points in the camera coordinate system, and further processed through camera extrinsic parameters. Unified mapping to the robot's base coordinate system, thus obtaining the first... The set of spatial points corresponding to a frame When the specific observation perspective is not distinguished, the camera coordinate system is denoted as {C}, and... Figure 2 The camera coordinate system is consistent throughout; when distinguishing the k-th observation viewpoint, the corresponding camera coordinate system is denoted as... After unifying and fusing the spatial point sets from each frame, the scene point cloud is obtained:

[0076]

[0077] in, This represents the scene point cloud under a unified reference coordinate system. After unifying multi-view observations, locally visible surfaces from different perspectives in the scene can be merged within the same spatial framework, thereby expanding the spatial coverage of the target and environment surfaces and providing more complete observational support for subsequent voxel mapping.

[0078] Step 2: Constructing the Scene Distance Field

[0079] like Figure 4 and Figure 5 As shown, after obtaining the scene point cloud in a unified reference coordinate system, a single-frame signed distance is defined for each voxel center relative to the current observation surface, and a truncation normalization method is used to form single-frame TSDF observation values. For the voxel centers... In terms of its first The signed distance within a frame is denoted as:

[0080]

[0081] in, Voxel centers In the The projection position on the frame image plane, This represents the corresponding observed depth value. Voxel centers The theoretical depth along the current line of sight.

[0082] Furthermore, a truncation-normalization method is used to define the single-frame TSDF observations:

[0083]

[0084] in, For validity indicator functions, A truncation distance threshold is used. Then, the TSDF values ​​and cumulative weights of each voxel are updated recursively to construct the scene TSDF. After recursive fusion of multiple frames, the scene surface can be recovered from the TSDF zero-level set. Based on this, using the scene surface recovered by the TSDF and the occupied voxels as a foundation, an ESDF is further generated through Euclidean distance propagation or neighborhood update to provide any query location in the scene. Euclidean distance to the nearest obstacle TSDF is used for scene surface restoration and target surface geometry description, while ESDF is used for free space distance query and security analysis. Together, they constitute the unified spatial representation upon which subsequent geometry crawling and solving depend.

[0085] Step 3: Target Region Extraction

[0086] like Figure 6As shown, after completing the full-scene TSDF / ESDF modeling, the crawling analysis still cannot be directly performed on the entire scene surface. This is because the full scene surface simultaneously includes the desktop, background, and non-target objects. Without pre-defining the current task object, it would not only expand the candidate crawling search space but also weaken the specificity of subsequent geometric analysis. Therefore, this embodiment introduces semantic constraints, utilizing target masks or target category priors to extract the target local region corresponding to the current task from the full scene. Let the... Target category in frame The corresponding semantic region is The target point cloud can then be jointly reconstructed from effective depth observation data under multi-view semantic constraints as follows:

[0087]

[0088] in, This represents the target point cloud set. Considering that the original target point cloud may still contain issues such as connected component missegmentation, outliers, and uneven sampling density, further connected component consistency processing, outlier removal, and voxel downsampling are performed to obtain the purified target point cloud. After the above processing, the subsequent capture and analysis range can be stably limited to the vicinity of the current target surface, while preserving the information complementarity between multi-view observations.

[0089] Step 4: Candidate Grabbing Pose Generation

[0090] Obtaining the target local point cloud Subsequently, solving the grasping posture is transformed into constructing candidate graspers that satisfy the gripper structure constraints on the target surface. For a two-finger parallel gripper, the grasping posture is no longer represented as a single point or a single posture on the surface, but as a structured object composed of contact point pairs, closing direction, opening width, and grasping pose. Let the first... The left and right contact points corresponding to the candidate grasping postures are respectively and Then its contact center, closing direction and opening width are defined as:

[0091]

[0092] in, For the contact center, The direction of gripper closure. For the required opening width, Width margin is introduced to compensate for surface noise and execution redundancy. To ensure that candidate grasping postures can be accurately realized by the gripper, structural constraints are further imposed on the opening width:

[0093]

[0094] Based on the contact point pairs, it is also necessary to construct approach directions consistent with the local surface geometry and to grasp the local coordinate system. First, define the average direction of the left and right contact normals. Then project it onto the closed direction. Within the orthogonal plane, the approach direction of the candidate grasping posture is obtained. And obtain the auxiliary direction from the cross product relationship. Based on this, the grasping pose of candidate grasping postures in the robot's base coordinate system can be constructed. and pre-grab pose Then, based on the gripper opening width range and local contact quality threshold, the candidate gripping postures are initially screened. Specifically, when the gripper opening width corresponding to a candidate gripping posture exceeds the allowable range of the gripper, or the left-right contact normal relationship does not meet the requirements for facing contact, or the local contact quality is lower than a preset threshold, the corresponding candidate gripping posture is removed, forming the original candidate gripping posture set. After this step, the candidate grasping pose is no longer a simple pair of surface points, but a structured object that simultaneously contains grasping pose, pre-grab pose, opening width, and local contact quality information.

[0095] Step 5: Perform volume building

[0096] To map candidate grasping poses to the actual execution process, this embodiment constructs three types of execution volumes for each candidate grasping pose, namely, proximity channel volumes. Volume of a closed corridor and the volume of the gripper body Among them, the approach channel volume is the sweep volume formed by the gripper advancing from the pre-grasping pose to the grasping pose; the closing corridor volume describes the unobstructed space required for finger closure; and the gripper body volume describes the overall space occupied by the palm and finger roots in the current grasping pose. These three types of execution volumes together constitute the analysis object for subsequent ESDF clearance queries. For the closure phase and the body occupancy phase, the corresponding volumes can be expressed as follows:

[0097]

[0098] The approach phase is clearly a process, therefore it is described using swept volume:

[0099]

[0100] in, This represents the continuous trajectory of a candidate grasping posture as it progresses from the pre-grasping pose to the final grasping pose. Thus, the spatial occupancy of the candidate grasping posture is uniformly expressed in the three stages of approach, closure, and body entry.

[0101] Step Six: Minimum Clearance Calculation

[0102] After obtaining the three types of execution volumes, the minimum clearance for candidate grasping postures is further calculated using the ESDF distance field. For the first... The minimum clearance for each of the candidate grasping postures, the approach phase, the closing phase, and the body occupation phase is denoted as follows: , and Its expression is:

[0103]

[0104] in, For position The Euclidean distance to the nearest obstacle. These three factors together constitute the fundamental measure of safety throughout the entire candidate grasping posture process.

[0105] Step 7: Geometric Feasibility Screening and Optimal Crawling Determination

[0106] like Figure 7 As shown, the original candidate grasping pose set This only reflects the possibility of forming a gripping contact on a localized part of the target surface, and does not mean that all these candidate gripping postures can be executed in the current scenario. For the actual gripping process, the gripper must also go through stages such as approach, closing, and body entry. If environmental interference occurs in any of these stages, the corresponding candidate gripping posture cannot be considered a valid solution. Based on this, this embodiment adopts a two-stage strategy of "hard constraint screening—soft scoring ranking." First, candidates that meet the three-stage safety requirements are defined as the feasible set:

[0107]

[0108] in, , , These represent the safety margin thresholds for the approach phase, closure phase, and body occupancy phase, respectively. Furthermore, to characterize the safety redundancy of feasible candidate grasping postures, the minimum clearance margin is defined as:

[0109]

[0110] Then, since the set of feasible candidate grasping postures usually still contains a large number of redundant grasping postures with very similar spatial positions and postures, a similarity judgment is further performed based on the differences in grasping center translation and posture among the candidate grasping postures to refine the set of feasible candidate grasping postures. For any two candidate grasping poses, the difference in translation between their grasping centers is considered. and posture differences A similarity judgment is performed. When both the translation difference and the pose difference are less than the corresponding threshold, redundant candidate grasping poses are deleted to obtain a set of feasible candidate grasping poses after redundancy removal.

[0111] After completing geometric feasibility screening and redundancy suppression, local contact quality, safety margin, and task preference are further incorporated into the comprehensive scoring framework. The comprehensive score can be based on at least the normal consistency index. Contact symmetry index Contact balance index Minimum clearance margin index , grasp the consistency index and attitude stability index It consists of two or more of the above indicators. Through the joint evaluation of the above indicators, a unified balance is achieved between local contact stability, global execution security, and scenario task preference, and the highest score is selected as the optimal grasping posture. Among them, the normal consistency index is used to characterize the degree of matching between the contact normal and the gripper closing direction; the contact symmetry index is used to characterize the opposition relationship between the left and right contact normals; the contact balance index is used to characterize the degree of balance between the contact center and the left and right contact points; the minimum clearance margin index is used to characterize the minimum safety margin retained in the execution phase with the smallest clearance margin value among the candidate gripping posture in the approach phase, closing phase and body entry phase; the downward gripping consistency index is used to characterize the degree of consistency between the approach direction and the table normal direction; and the posture stability index is used to characterize the degree of tilt of the gripping posture relative to the table normal.

[0112] It should be noted that, in this embodiment, the comprehensive score is obtained by normalizing at least two of the following indicators: normal consistency index, contact symmetry index, contact balance index, minimum clearance margin index, gripping consistency index, and attitude stability index, and then weighting and summing them according to preset weights.

[0113] Step 8: Perform mapping

[0114] like Figure 8 As shown, in obtaining the optimal grasping posture Next, it needs to be converted from a grasping geometry description into a control target that can be executed by the robotic arm's end effector. Let the optimal grasping posture in the camera coordinate system correspond to the grasping pose as... The optimal grasping posture is the end effector target pose in the robot base coordinate system. It can be represented as:

[0115]

[0116] in, This represents the gripper coordinate system corresponding to the optimal gripping posture. Relative to camera coordinate system The homogeneous pose transformation matrix; Representing the camera coordinate system Relative to the robot base coordinate system The homogeneous pose transformation matrix.

[0117] Through the above transformation, the grasping geometry analysis results are uniformly mapped to the end-effector pose that can be directly used by the robot control layer. Furthermore, a fixed distance is recoiled in the opposite direction from the approach direction of the optimal grasping posture. Constructing pre-grasp pose After obtaining the pre-grasping pose and the final grasping pose, the robotic arm end effector is first controlled to move to the pre-grasping pose; then it is advanced from the pre-grasping pose to the grasping pose along the approach direction; after reaching the grasping pose, the gripper is controlled to complete the closing action; after gripping the target, the lifting action is performed along the predetermined lifting direction, thus forming a sequence of four stages: pre-grasping, approach, closing and lifting.

[0118] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A TSDF and semantic constraint based grasping method, characterized in that, Includes the following steps: S1: Scene Observation and Point Cloud Construction Multi-view RGB images, depth images and camera pose information of a target scene are acquired by using an RGB-D camera, and effective depth observation data under each view are restored to scene point clouds in a unified reference coordinate system ; S2: Scene Distance Field Construction Scene-based point cloud construct a truncated signed distance field TSDF of the scene voxel, and generate an Euclidean signed distance field ESDF through the TSDF; S3: Target Region Extraction According to the semantic region corresponding to the target category, a target point cloud corresponding to the current task target is extracted from the scene point cloud , the target point cloud is purified to obtain a purified target point cloud ;​​ S4: Candidate Grasping Pose Generation Based on target point cloud Generate candidate grasping poses, which include at least the grasping center, closing direction, approach direction, grasping pose, pre-grab pose, and gripper opening width; S5: Perform volume building For each candidate grasping posture, construct the proximity channel volume, the closed corridor volume, and the gripper body volume respectively; S6: Minimum Clearance Calculation ESDF is used to calculate the minimum clearance corresponding to the volume of the approach channel, the volume of the closed corridor, and the volume of the gripper body, respectively. S7: Geometric Feasibility Screening and Determination of Optimal Grasping Posture Based on the minimum clearance, the candidate grasping postures are geometrically screened to obtain a set of feasible candidate grasping postures. The system performs redundancy suppression and comprehensive scoring on the feasible candidate grasping set to determine the optimal grasping posture. ; S8: Execute mapping mapping the optimal grasp pose to a sequence of pre-grasp, approach, close, and lift stages executable by the end-of-arm tool.

2. The grasping method based on TSDF and semantic constraints according to claim 1, characterized in that, In step S1, the specific processing procedure is as follows: S11: according to the depth image and the camera intrinsic parameters projecting the valid depth pixels under each view angle to three-dimensional points in the camera coordinate system; S12: according to the camera coordinate system to the robot base coordinate system of the external parameter transformation matrix mapping the three-dimensional points under each perspective to the robot base coordinate system; S13: Unify and fuse 3D points from multiple perspectives to obtain scene point clouds. ,in, Indicates the first The set of spatial points corresponding to the frame, and has .

3. The method of claim 1, wherein, In step S2, the specific processing procedure is as follows: S21: a scene point cloud under a unified reference coordinate system A voxel fusion is performed to update the truncated signed distance value and the cumulative weight of each voxel in a recursive manner to construct a truncated signed distance field TSDF of the scene voxel. S22: Restore the scene surface based on TSDF; S23: Based on the restored scene surface and the occupied voxels and free space boundaries, generate an ESDF through Euclidean distance propagation or neighborhood update to obtain any query location in the scene. Euclidean distance to the nearest obstacle .

4. The grasping method based on TSDF and semantic constraints according to claim 1, characterized in that, In step S3, the specific processing procedure is as follows: S31: Based on target category Semantic regions corresponding to each perspective Constrained recovery is performed on multi-view effective depth observation data to obtain a preliminary target point cloud. ; S32: performing at least one of connected component consistency processing, outlier rejection, and voxel down-sampling on the preliminary target point cloud performing at least one of connected component consistency processing, outlier rejection, and voxel down-sampling on the preliminary target point cloud .

5. The method of claim 1, wherein, In step S4, the specific processing procedure is as follows: S41: Based on target point cloud Left and right contact points in and Determine the contact center , closing direction and gripper opening width ; S42: Based on left and right contact normals and Constructing approach direction And establish a local coordinate system for the candidate grasping postures; S43: determining the grasping pose of the candidate grasping pose based on the local coordinate system and the pre-grasping pose ; S44: Based on the gripper opening width range, left and right contact normal matching relationship, and local contact quality threshold, preliminary screening of candidate gripping postures is performed, eliminating candidate gripping postures that do not meet the gripper opening range constraint or local contact quality requirements, forming an original candidate gripping posture set. .

6. The crawling method based on TSDF and semantic constraints according to claim 1, characterized in that, In step S5, the specific processing procedure is as follows: S51: Construct approach channel volume for describing the swept volume as the jaws are advanced from the pre- grasp pose to the grasp pose; S52: Construct closed corridor volume , to describe the unobstructed space required for the jaw fingers to complete the closing action; S53: Constructing the gripper body volume , for describing the overall occupied space of the gripper palm and finger base in the grasping pose.

7. The grasping method based on TSDF and semantic constraints according to claim 6, characterized in that, In step S6, the specific processing procedure is as follows: S61: Calculate approach channel volume Corresponding minimum clearance ; S62: Calculate the volume of a closed corridor Corresponding minimum clearance ; S63: Calculate gripper body volume Corresponding minimum clearance .

8. The grasping method based on TSDF and semantic constraints according to claim 1, characterized in that, In step S7, the geometric feasibility screening process includes the following steps: S71: Minimum clearance when approaching the channel volume Not less than the safety margin threshold of the approach stage Minimum clearance for the volume of a closed corridor Not less than the safety margin threshold during the closing phase And the minimum clearance of the gripper body volume Not less than the security margin threshold of the main body occupation phase When the time is right, the corresponding candidate grasping posture is determined to be a feasible candidate grasping posture; S72: Minimum clearance margin based on feasible candidate grasping postures during the approach, closing, and body occupancy phases. Form a set of feasible candidate grasping postures .

9. The grasping method based on TSDF and semantic constraints according to claim 8, characterized in that, In step S7, the redundancy suppression and comprehensive scoring process includes the following steps: S73: Set of feasible candidate grasping postures For any two candidate grasping poses, the difference in translation between their grasping centers is considered. and posture differences A similarity judgment is performed. When both the translation difference and the pose difference are less than the corresponding threshold, redundant candidate grasping poses are deleted to obtain a set of feasible candidate grasping poses after redundancy removal. S74: Based on at least two of the following indices—normal consistency, contact symmetry, contact balance, minimum clearance margin, gripping consistency, and attitude stability—comprehensively score the candidate gripping postures in the set of feasible candidate gripping postures after redundancy removal, and determine the candidate gripping posture with the highest score as the optimal gripping posture. .

10. The crawling method based on TSDF and semantic constraints according to claim 1, characterized in that, In step S8, the specific processing procedure is as follows: S81: the optimal grasp pose the grasp pose in the camera coordinate system mapping to the end-effector target pose in the robot base coordinate system , i.e., the grasp pose ; S82: reverse retreat by a predetermined distance along the approaching direction of the optimal grasping pose , configure the pre-grasping pose ; S83: Based on pre-captured pose and grasping pose The robot arm generates a sequence of actions that sequentially complete the four stages of pre-grabbing, approaching, closing, and lifting.

Images