Interactive grasp generation method fusing visual language model and key point learning

Abstract

The application provides an interactive grasping generation method combining a visual language model and key point learning. The application uses a visual language model to determine a target object according to a natural language instruction and generate target region constraint information containing a name and a bounding box; limits a two-dimensional projection region of a candidate grasping center point according to the information, predicts a grasping center, key points and a gripper width based on an RGB image and a depth map, and corrects a confidence according to the distance, spacing and symmetry of the key points relative to the center point to obtain multiple candidate grasps; obtains an initial pose according to a two-dimensional-three-dimensional key point correspondence, and performs a top view grasping screening according to the consistency of the gripper orientation and the gravity direction and the horizontal alignment degree of the key points; determines a weight according to the key point correction confidence and the two-dimensional projection distance of the key points to the center point, performs a weighted PnP optimization by minimizing a weighted projection error, and obtains a target gripper six-degree-of-freedom pose. The stability of the grasping pose generation can be improved.

Description

Technical Field

[0001] This invention relates to the field of interactive grasping technology for robots, and more specifically to an interactive grasping generation method that integrates visual language models and key point learning. Background Technology

[0002] In applications where human-machine collaboration is becoming increasingly prevalent, robots' ability to autonomously understand their environment, identify task objectives, and perform stable grasping has become an important research direction in fields such as intelligent manufacturing, service robots, and home assistants.

[0003] In related technologies, keypoint-based grasping methods typically detect gripper keypoints in images and use the PnP algorithm to recover the grasping posture, thereby reducing the computational cost of directly processing dense point clouds. Some methods improve grasping stability under occlusion, illumination changes, and migration from simulated to real-world environments by redesigning the keypoint output space, incorporating depth information, or employing domain randomization training.

[0004] However, existing solutions may still have the following shortcomings: there is a lack of stable constraints between the target region determined by the language instructions and the subsequent candidate grabbing generation; when key point prediction is affected by occlusion or observation noise, the reliability of the candidates is difficult to accurately characterize by the network response value; the quality differences of different corresponding points in the pose solution are not fully utilized, and abnormal key points can easily reduce the stability of the six-degree-of-freedom pose estimation. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides an interactive grasping generation method that integrates visual language models and key point learning, in order to solve the technical problems of insufficient connection between language target region constraints and candidate grasping generation, inadequate key point reliability evaluation, and the impact of abnormal corresponding points on the stability of pose solution.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides an interactive capture generation method that integrates visual language models and keypoint learning, comprising: Acquire RGB images, depth maps, and natural language instructions; use a visual language model to determine the target object that matches the natural language instructions; and generate target region constraint information that includes the target object name and bounding box. Based on the target region constraint information, the region where the two-dimensional projection of the center point of the candidate grab is located is defined, and multiple candidate grabs are generated within the defined region, including: A multi-task convolutional neural network is used to predict the grabbing center heatmap, center sub-pixel offset, 2D offset from the center to predefined 3D key points of the gripper, and gripper width based on the RGB image and depth map. Multiple 2D projections of the center points are extracted and sub-pixel corrected from the grabbing center heatmap. Corresponding 2D projections of key points are obtained based on the 2D offset. The heatmap response value is corrected based on the distance, spacing, and symmetry of the 2D projections of key points relative to their corresponding 2D projections of center points to obtain key point correction confidence and candidate confidence. Each center point 2D projection is combined with its corresponding key point 2D projection, gripper width, key point correction confidence, and candidate confidence to form multiple candidate grabs. PnP is solved based on the correspondence between the 2D projection of the key points and the 3D key points of the gripper to obtain the initial six-degree-of-freedom pose of each candidate gripper; based on the consistency of the gripper orientation with the direction of gravity and the horizontal alignment of the key points, candidate grippers that meet the top-view gripping constraints are selected. The key point weights are determined based on the key point correction confidence and the two-dimensional projection distance from the key point to the center point after screening. The initial six-degree-of-freedom attitude is then optimized by weighted PnP by minimizing the weighted projection error to obtain the target gripper's six-degree-of-freedom attitude.

[0007] Secondly, the present invention provides an interactive grasping and generation device that integrates visual language models and key point learning, comprising: The target region determination module is used to acquire RGB images, depth maps and natural language instructions, use a visual language model to determine the target object that matches the natural language instructions, and generate target region constraint information containing the target object name and bounding box. A candidate capture generation module is used to define the region where the two-dimensional projection of the center point of the candidate capture is located based on the target region constraint information, and to generate multiple candidate captures within the defined region, including: A multi-task convolutional neural network is used to predict the center heatmap, center sub-pixel offset, two-dimensional offset from the center to the predefined three-dimensional key points of the gripper, and gripper width based on the RGB image and depth map. Multiple center point 2D projections are extracted and subpixel corrected from the grabbing center heatmap. The corresponding key point 2D projections are obtained based on the 2D offset. The heatmap response values ​​are corrected based on the distance, spacing, and symmetry of the key point 2D projections relative to the corresponding center point 2D projections to obtain key point correction confidence and candidate confidence. Each center point 2D projection is combined with its corresponding key point 2D projection, gripper width, key point correction confidence, and candidate confidence to form multiple candidate grabs. The candidate selection module is used to perform PnP solution based on the correspondence between the 2D projection of key points and the 3D key points of grippers to obtain the initial six-degree-of-freedom pose of each candidate gripper; and to select candidate grippers that meet the top-view gripping constraints based on the consistency of gripper orientation with the direction of gravity and the horizontal alignment of key points. The attitude optimization module is used to determine the key point weights based on the key point correction confidence and the two-dimensional projection distance from the key point to the center point after screening the candidate grasping key points. By minimizing the weighted projection error, the initial six-degree-of-freedom attitude is optimized by weighted PnP to obtain the target gripper's six-degree-of-freedom attitude.

[0008] As a further improvement to the above-mentioned interactive grasping generation method or device, the initial six-degree-of-freedom posture is obtained by using the standard PnP algorithm; if the weighted PnP optimization fails to converge or the weighted projection error is greater than a preset threshold, the initial six-degree-of-freedom posture is taken as the target gripper's six-degree-of-freedom posture.

[0009] As a further improvement to the above-mentioned interactive crawling generation method or device, the loss function construction process of the multi-task convolutional neural network includes: An improved Focal Loss is used to construct a center heatmap loss, wherein the improved Focal Loss adjusts the gradient weights of positive and negative samples separately through two parameters. L1 loss is used to construct center offset loss, keypoint offset loss and gripper width loss; The center heatmap loss, the center offset loss, the key point offset loss, and the gripper width loss are weighted and summed using preset weight coefficients to obtain the comprehensive training loss.

[0010] As a further improvement to the above-mentioned interactive grasping generation method or device, the predefined gripper 3D key points include four key points: left and right top and left and right bottom; the process of determining the key point correction confidence and candidate confidence includes: The response value of the captured center heatmap at the corresponding center position is used as the initial confidence level; The distance from each key point to the center point is normalized based on the gripper width to determine the key point distance; the spacing anomaly is determined based on the deviation of the spacing between adjacent key points relative to the preset spacing; and the symmetry deviation is determined based on the difference in the horizontal distances from the corresponding left and right key points to the center point. The initial confidence level is corrected using the key point distance, the spacing anomaly, and the symmetry deviation, and the correction result is limited to between 0 and 1 to obtain the adjusted confidence level of each key point; The distance weight and symmetry weight are determined based on the key point distance and the symmetry deviation, respectively. The adjusted confidence is then weighted and normalized to obtain the key point correction confidence. The candidate confidence score is obtained by aggregating the corrected confidence scores of multiple key points captured from the same candidate.

[0011] As a further improvement to the above-mentioned interactive grasping generation method or device, the initial six-degree-of-freedom posture includes an initial rotation matrix and an initial translation vector; screening candidate grasps that satisfy the top-view grasping constraints includes: Transform the end normal vector in the gripper coordinate system to the world coordinate system based on the initial rotation matrix; Calculate the cosine of the angle between the transformed end normal vector and the gravity direction in the world coordinate system, and restrict the cosine of the angle to a non-negative value to obtain the direction consistency score; Calculate the horizontal deviations of the top key points on the left and right sides and the bottom key points on the left and right sides respectively, take the larger value of the two and obtain the horizontal alignment score through the exponential decay function; The comprehensive geometric score is determined based on the geometric mean of the directional consistency score and the horizontal alignment score; Candidate crawls with a comprehensive geometric score lower than a preset top-down crawl threshold are removed, and the remaining candidate crawls are sorted according to the candidate confidence level.

[0012] As a further improvement to the above-mentioned interactive crawling generation method or device, the weighted PnP optimization is performed on each of the filtered candidate crawls, including: Construct a set of corresponding 2D and 3D points based on the 2D projection of each key point and its corresponding predefined 3D key points of the gripper, and obtain the camera intrinsic parameters; The confidence scores of the key point corrections are normalized to obtain the confidence scores weights; The distance weight is obtained by Gaussian attenuation and normalization based on the distance from the 2D projection of the key point to the 2D projection of the center point; The geometric mean of the confidence weight and the distance weight is normalized to obtain the mixed weight, and the confidence weight, the distance weight, or the mixed weight is used as the key point weight; The initial six-degree-of-freedom attitude is used as the initial value for the weighted PnP solver iteration. Based on the camera intrinsic parameters, the current rotation matrix, and the current translation vector, project the three-dimensional key points of each predefined gripper onto the image plane; Calculate the reprojection error between each projection point and the corresponding key point's two-dimensional projection, and use the key point weights to weight the reprojection error; The current rotation matrix and the current translation vector are iteratively adjusted to reduce the sum of the weighted projection errors, thereby obtaining the six-degree-of-freedom pose of the target gripper.

[0013] As a further improvement to the above-mentioned interactive grasping generation method or device, the origin of the gripper coordinate system is taken as the three-dimensional center point of the gripper, and the three-dimensional center point of the gripper and its corresponding two-dimensional projection are added to the set of two-dimensional corresponding points as additional two-dimensional corresponding points.

[0014] Thirdly, the present invention provides a non-transient computer-readable storage medium storing a computer program, which, when executed by a processor, implements the interactive crawling and generation method described in the first aspect.

[0015] Fourthly, the present invention provides an electronic device, including a processor and a memory; The memory stores computer programs; The processor is used to execute the computer program to implement the interactive crawling and generation method described in the first aspect.

[0016] (III) Beneficial Effects This invention provides an interactive crawling generation method that integrates visual language models and keypoint learning. Compared with existing technologies, it has the following advantages: This invention utilizes a visual language model to convert natural language instructions into target region constraint information, thereby limiting the two-dimensional projection area of ​​candidate grasping center points and reducing interference from non-target regions in candidate generation. It corrects keypoint confidence based on the distance, spacing, and symmetry of keypoints relative to the center point and forms candidate confidence, ensuring that candidate ranking reflects the geometric quality of keypoints. Based on the initial pose, it filters candidate grasping based on gripper orientation, gravity direction, and horizontal alignment, and performs weighted PnP optimization based on keypoint correction confidence and the two-dimensional projection distance from keypoints to the center point. This reduces the impact of abnormal keypoints on pose solving and improves the stability of grasping pose generation under occlusion or observation noise conditions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an interactive capture generation method that integrates visual language models and keypoint learning, provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a gripper coordinate system provided in an embodiment of the present invention; Figure 3A comparison chart of the capture success rate and average inference time between the method provided in this embodiment of the invention and existing methods; Figure 4 This is a graph showing the relationship between the number of Top-K candidate hypotheses and the geometric screening strategy provided in this embodiment of the invention on the crawling success rate. Figure 5 A comparison chart of the capture success rates of different key point weighting strategies provided in the embodiments of the present invention under different noise levels; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention; Component labeling explanation: 100 - electronic device, 101 - memory, 102 - processor, 103 - display. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This embodiment proposes an interactive grasping generation method that integrates visual language models and keypoint learning. The method first uses a visual language model to determine the target object based on natural language instructions and forms target region constraint information. Then, based on this information, it defines the two-dimensional projection region of candidate grasping center points, predicts the grasping center, keypoints, and gripper width within the defined region, and corrects the confidence level based on the distance, spacing, and symmetry of keypoints relative to the center point, resulting in multiple candidate grasps. Next, it obtains an initial six-DOF pose based on the correspondence between two-dimensional and three-dimensional keypoints, and performs top-down grasping filtering based on gripper orientation, gravity direction, and horizontal alignment. Finally, it determines weights based on the keypoint correction confidence level and the two-dimensional projection distance from the keypoint to the center point, and optimizes the initial pose using weighted PnP.

[0021] The beneficial effects of this embodiment are mainly reflected in the following aspects: 1) Utilize a visual language model to determine the target object based on natural language instructions, and use the bounding box of the target object or the target region formed by the bounding box as the region constraint for downstream candidate crawling to reduce background interference.

[0022] 2) The network confidence is corrected by using the distance, spacing and left-right symmetry of key points, and then aggregated to form candidate confidence, so as to improve the consistency between candidate reliability evaluation and key point geometric quality.

[0023] 3) Perform top-down grasping and filtering based on the initial pose, and perform weighted PnP optimization based on the key point correction confidence and the two-dimensional projection distance from the key point to the center point to reduce the impact of abnormal key points on the six-degree-of-freedom pose estimation.

[0024] Example 1: like Figure 1 As shown, this embodiment of the invention provides an interactive capture generation method that integrates visual language models and keypoint learning, including: like Figure 1 As shown, this embodiment of the invention provides an interactive capture generation method that integrates visual language models and keypoint learning, including: S1. Acquire RGB image, depth map and natural language command, use visual language model to determine target object matching natural language command, and generate target region constraint information containing target object name and bounding box; S2. Based on the target region constraint information, define the region where the two-dimensional projection of the center point of the candidate grabbing point is located, and generate multiple candidate grabbing points within the defined region, including: S21. Using a multi-task convolutional neural network, predict the center heatmap, center sub-pixel offset, center to predefined gripper 3D keypoints and gripper width based on RGB image and depth map; S22. Extract and subpixel correct the two-dimensional projection of multiple center points from the captured center heat map, and obtain the corresponding two-dimensional projection of key points based on the two-dimensional offset. S23. Based on the distance, spacing, and symmetry correction heatmap response value of the two-dimensional projection of the key point relative to the two-dimensional projection of the corresponding center point, obtain the key point correction confidence and candidate confidence, and combine each center point two-dimensional projection with its corresponding key point two-dimensional projection, gripper width, key point correction confidence and candidate confidence to form multiple candidate grabs. S3. Perform PnP solution based on the correspondence between the 2D projection of the key points and the 3D key points of the gripper to obtain the initial six-degree-of-freedom posture of each candidate gripper; based on the consistency between the gripper orientation and the gravity direction and the horizontal alignment of the key points, select candidate grippers that meet the top-view gripping constraints. S4. Determine the key point weights based on the key point correction confidence and the two-dimensional projection distance from the key point to the center point after screening. Perform weighted PnP optimization on the initial six-degree-of-freedom posture by minimizing the weighted projection error to obtain the target gripper's six-degree-of-freedom posture.

[0025] Through the above steps, target area constraint, candidate grab generation, key point geometric quality evaluation, initial pose determination, candidate screening and weighted pose optimization form a continuous processing chain, which can reduce the impact of non-target areas and abnormal key points on grab pose generation.

[0026] Overall, this embodiment proposes an interactive image capture and generation method that integrates visual language models and keypoint learning, with RGB images as input. and its depth map The natural language instruction T outputs a set of 6-DoF candidate grasp poses: in, It is the set of real numbers; and These are the height and width of the image, respectively; The first The six-DOF pose, gripper width, and candidate confidence of each candidate grasp; This represents the special three-dimensional Euclidean group used to describe the rotation and translation of a rigid body. It should be noted that the candidate indices are omitted in the following formulas for keypoint calculation and attitude determination for a single candidate grasp.

[0027] The overall process of this embodiment is as follows: semantic target localization and region constraint, candidate crawling and key point generation, key point and candidate confidence correction, initial PnP pose estimation, top-down geometric filtering, and weighted PnP pose optimization.

[0028] Based on this, the following will provide a detailed explanation of each step of the above technical solution: In step S1, RGB images, depth maps, and natural language instructions are acquired. A visual language model is used to determine the target object that matches the natural language instructions, and target region constraint information containing the target object name and bounding box is generated.

[0029] This step combines image content with natural language instructions to output structured information of target objects in the current image that conform to human intent, including the target object name and bounding box coordinates (bbox), for use in downstream candidate crawling generation.

[0030] To locate the target object corresponding to the natural language command and form region constraints, this step can use a Vision-Language Model (VLM) to process the input RGB image, depth map, and natural language command to generate structured target data: ,in This represents a visual language model. For implementations that use only RGB images and natural language commands as model inputs, the depth map is used for subsequent candidate capture generation and pose determination; the structured target data includes at least the target object name and bounding box coordinates.

[0031] For example, in this embodiment, the visual language model can be Qwen3.5-Plus, the corresponding structured data is in JSON format, and the bounding box can be represented by the image coordinates of the upper left and lower right corners. Cropping the RGB or RGB-D region of interest based on the bounding box, masking the region outside the box, or using the bounding box as a cue for the segmentation model to obtain a binary target mask can all form target region constraints for use by the downstream network; the actual form used is determined by the specific implementation.

[0032] In step S2, the two-dimensional projection area of ​​the candidate grab center point is defined according to the target area constraint information, and multiple candidate grabs are generated based on key points within the defined area.

[0033] To achieve sub-pixel-level keypoint localization and more accurate candidate capture, this step is as follows: S21. Based on predefined gripper 3D key points, under the constraint of the target area, a multi-task convolutional neural network is used to jointly predict the gripper center heatmap, center sub-pixel offset map, center-to-key point offset map, and gripper width based on RGB image and depth map.

[0034] like Figure 2 As shown, four 3D key points can be predefined in the gripper's own coordinate system. The following implementation uses four key points—the top left and right sides and the bottom left and right sides—as an example. These four key points correspond to the 2D key points predicted by the network in a fixed order. Figure 2 The plane square shown in (a) is Figure 2 The non-planar tetrahedral shape shown in (b) or Figure 2 The planar tail shape shown in (c) can be used as the key point configuration; when using different configurations, the corresponding three-dimensional coordinates are preset and the PnP solution method suitable for the configuration is selected.

[0035] Specifically, the predefined 3D key points of the gripper are four key points: the top of the left and right sides, and the bottom of the left and right sides, which can be represented as follows: in, A predefined set of 3D key points for the gripper; For the first The three-dimensional coordinates of the key points in the gripper coordinate system.

[0036] Under the constraint of the target region, using RGB images and depth maps as network inputs, a multi-task convolutional neural network outputs the following four types of prediction results: 1. Capture the central heat map Predict the coordinates of each pixel. The probability distribution represents the center of the crawl. To adapt to different crawling directions, the network discretizes the crawling direction into... Each category outputs the center probability in each direction. This enables multi-directional grasping and detection at the same location; Represents pixels In direction category The predicted probability is used as the center of the capture.

[0037] 2. Center subpixel offset map Further regress the 2D refinement offset at the pixel-level predicted grasping center position. This is to improve the accuracy of the center positioning. Among them, These represent the offsets of the central subpixel in the x and y coordinates, respectively.

[0038] 3. Center to Keypoint Offset Map : For each predicted capture center, regress its two-dimensional offset to the four keypoints in the image plane. There are a total of 2*4=8 channels, which are used to assist in restoring the grasping posture.

[0039] 4. Gripper width Single-channel prediction Each candidate gripper corresponds to a gripper opening and closing range, used to determine the gripper opening and closing amount and adapt to the object size.

[0040] Furthermore, the network architecture employs an encoder-decoder structure, combining multi-scale feature fusion with convolutional layers to achieve high-resolution pixel-level prediction. The total number of output channels is: in One channel is used to capture the center probability, 8 channels are used for the two-dimensional offset from the center to the four key points, and 1 channel is used for the opening and closing amplitude of the gripper.

[0041] To ensure the model can accurately learn the positions of the gripping center and key points, and reasonably predict the opening and closing amplitude of the gripper, in an optional implementation, the loss function construction process of the multi-task convolutional neural network includes: S101. An improved Focal Loss (a loss function that improves classification accuracy by reducing the weight of simple samples and focusing on learning from difficult samples) is used to construct a center-grabbing heatmap loss. The improved Focal Loss adjusts the gradient weights of positive and negative samples using two parameters; expressed as: in, For pixel coordinates, To retrieve the direction category index, The total number of positive sample centers; This indicates the pixel coordinates in the captured center heatmap. In the The predicted probability value for each grasping direction category is used to characterize the confidence that the pixel will become the grasping center in the corresponding direction; for the ground truth label... This indicates that the location is a true positive sample at the center of the crawl. Indicates a negative sample; and These are preset hyperparameters, where Used to adjust the gradient weights of positive samples This is used to adjust the gradient weights of negative samples, and to alleviate the imbalance problem by independently controlling the loss contribution of positive and negative samples through two parameters. For example, , .

[0042] S102. The L1 loss is used to construct the center offset loss, keypoint offset loss, and gripper width loss; these are expressed as follows: in, This represents the true sub-pixel offset of the k-th capture center. This represents the corresponding center subpixel offset predicted by the multi-task convolutional neural network. This represents the offset from the real center to the key point corresponding to the k-th capture center. This represents the offset from the corresponding center to the key point predicted by the network. This represents the actual gripper width corresponding to the k-th grasping posture. The corresponding gripper width predicted by the network; This represents the L1 norm.

[0043] S103. The center heatmap loss, the center offset loss, the key point offset loss, and the gripper width loss are weighted and summed using preset weight coefficients to obtain the comprehensive training loss: in, These are the preset weighting coefficients for the corresponding loss terms. For example, , , , This is used to balance the impact of gradients from different branches on network training.

[0044] S22. Non-maximum suppression is applied to multiple local extreme points of the captured center heatmap, and sub-pixel correction is performed using the center sub-pixel offset map to obtain multiple sets of center point 2D projections of the gripper. Combined with the offset map from the center to the key point, the corresponding key point 2D projection is generated.

[0045] Specifically, local maximum suppression is performed on the center heatmap to retain the center position with the largest response value within a preset neighborhood, and the Top-K positions with higher response values ​​are selected as initial crawling center candidates. The neighborhood size, response threshold, and number of Top-K positions can be preset based on the verification data.

[0046] Based on this, the retained center point coordinates are corrected using the center sub-pixel offset map to eliminate pixel-level positioning errors and establish the pixel-level center point. The correction is to a high-precision 2D projection of the gripper center point, specifically represented as follows: in, The corrected coordinates of the gripper center point (i.e., the 2D projection of the gripper center point).

[0047] Subsequently, based on the 2D projections of the precise center points and the offset maps from the center to the key points in each group, the 2D projection coordinates of the four key points of the gripper were calculated point by point, providing an accurate 2D observation basis for subsequent key point confidence calculation and pose determination. The 2D projection coordinates of the four key points of the gripper are specifically represented as follows: in, This represents the 2D projected coordinates of the j-th key point.

[0048] By removing redundant candidates through nonmaximum suppression and eliminating pixel positioning errors by combining subpixel correction, the positioning accuracy of the 2D projection of the grasping center and key points is significantly improved, providing reliable 2D observation data for subsequent geometric evaluation, attitude selection and pose calculation.

[0049] S23. Construct a geometric evaluation function based on the distance, spacing and symmetry of the two-dimensional projection of the key point relative to the two-dimensional projection of the corresponding center point, correct the heat map response value, obtain the key point correction confidence and candidate confidence, and obtain multiple candidate grabs based on this and the gripper width.

[0050] In an optional implementation, the process of constructing and calculating the geometric evaluation function includes: S101', Use the response value of the grabbing center heatmap at the corresponding grabbing center position as the initial confidence level. This response value represents the credibility of the current location as the crawling center, and comes directly from the network output without relying on manual rule calculation.

[0051] S102', Based on the 2D projection of the key points and the width of the gripper, calculate respectively: 1. Geometric distance from each key point to the center .in, Represents the L2 norm. For the first The predicted gripper width for each candidate grab.

[0052] 2. Spacing between adjacent key points .in, The two-dimensional projection spacing is formed by preset pairs of adjacent key points. The preset spacing for the corresponding key point pairs.

[0053] 3. Key points exhibit left-right symmetry deviation .in, These are the horizontal coordinates of the top or bottom key points corresponding to the left and right sides, including the top key points on the left and right sides and the bottom key points on the left and right sides; The x-coordinate of the center point.

[0054] S103', Combining the aforementioned geometric distances The spacing between adjacent key points The key points have left and right symmetrical deviations. With the initial confidence level Obtain the adjusted keypoint confidence level: in, , , These are the preset non-negative weighting coefficients for the corresponding items, used to balance the correction magnitude of each geometric feature on the confidence score. It should be noted that, as shown in the above formula, the larger the geometric distance, the more anomalous the spacing, and the greater the symmetry deviation, the lower the corrected confidence score will be, ensuring that the confidence score is consistent with the geometric quality of the keypoint. When the above calculation result is less than 0, it is set to 0; when it is greater than 1, it is set to 1, thus limiting the adjusted keypoint confidence score to between 0 and 1.

[0055] S104'. Using the grasping center as a reference, the closer the geometric distance between the key point and the center, the higher the distance weight. The larger the value, the smaller the left-right symmetry deviation of the key point, and thus the greater the symmetry weight. The larger it is. For example, a monotonically decaying function can be used. , ,in and A non-negative attenuation coefficient is preset. The adjusted keypoint confidence scores are weighted using the distance weight and symmetry weight to generate a comprehensive keypoint confidence score. S105', Comprehensive confidence level of the key points Normalization is performed to obtain the final keypoint confidence scores: in, Indicates the first in a single candidate crawl Correction confidence level of each key point.

[0056] As mentioned above, for simplicity, the candidate crawl index is omitted from the keypoint confidence symbol. Based on this, it is possible to further define for the first... For each candidate crawl, the candidate confidence level can be determined by the arithmetic mean of the corrected confidence levels of four key points. Keypoint correction confidence is used for keypoint weighting, and candidate confidence is used for candidate ranking; other preset aggregation rules can also be used depending on the actual implementation.

[0057] The two-dimensional projection of the keypoint corresponding to the same center point, the predicted gripper width, the keypoint correction confidence, and the candidate confidence are combined with the two-dimensional projection of the center point to form a candidate gripper; the above combination is performed on each center point to obtain multiple candidate grippers. The initial six-degree-of-freedom pose of each candidate gripper is obtained in step S3.

[0058] In step S3, the initial six-degree-of-freedom pose is obtained based on the correspondence between the two-dimensional and three-dimensional key points of each candidate grasp; then, the top-view grasping direction is verified and candidate grasps are selected based on the consistency between the gripper orientation and the gravity direction and the horizontal alignment of the key points.

[0059] In an optional implementation, the initial rotation matrix for each candidate gripper is obtained using the standard PnP algorithm based on predefined 3D keypoints of the gripper, the corresponding 2D projections of the keypoints, and camera intrinsic parameters. and the initial translation vector Furthermore, before the initial PnP, candidates that clearly do not meet the top-view grasping characteristics can be coarsely screened based on the two-dimensional projection directions of the front and rear edges; this two-dimensional coarse screening does not replace subsequent orientation verification based on the initial posture. Specifically, this step includes: S31. For each candidate grasping posture, calculate the corresponding comprehensive geometric score, including: The end-effector normal vector in the gripper coordinate system, used to characterize the gripper orientation. Transform to the camera coordinate system using the initial rotation matrix, and then transform to the world coordinate system using the calibration extrinsic parameters from the camera coordinate system to the world coordinate system, to obtain the end-point normal vector. Obtain the gravity direction in the world coordinate system. Calculate the cosine of the angle between the two: Calculate the horizontal deviations of the top key points on the left and right sides of the gripper and the horizontal deviations of the bottom key points on the left and right sides respectively, and take the larger deviation value. And generate a horizontal alignment score using an exponential function: in, To align the reference pixel value horizontally. It is an exponential function.

[0060] And calculate the cosine value of the included angle. The horizontal alignment score The geometric mean of the average values, after normalization, is used to obtain the comprehensive geometric score. Specifically, the value of the cosine of the included angle is taken as 0 when it is less than 0; comprehensive geometric scoring. The value range is 0-1.

[0061] S32. Eliminate candidate grasping postures whose comprehensive geometric score is lower than a preset top-down grasping threshold, and sort the remaining candidate grasping postures according to their candidate confidence levels. Sort by high to low. The top-down grab threshold can be predetermined based on the verification data. The top-down grab threshold is preset based on the verification data; for example, the top-down grab threshold can be any value between 0.1 and 0.6.

[0062] In step S4, weighted PnP optimization is performed on each candidate capture after filtering, specifically including: S41. Based on the two-dimensional projection of the four key points captured by the candidate and their corresponding three-dimensional key points of the gripper, construct a two-dimensional-three-dimensional corresponding point set and obtain the calibrated camera intrinsic parameter matrix K.

[0063] Optionally, the origin of the gripper coordinate system is taken as the three-dimensional center point of the gripper, and it is paired with the two-dimensional projection of the center point obtained in step S22, and added to the corresponding point set as an additional two-dimensional-three-dimensional corresponding point.

[0064] The key point weight design adopts any of the following methods: (i) Normalize the confidence scores of key points and calculate the confidence weights: (ii) Calculate the distance weight by applying Gaussian attenuation to the Euclidean distance between the 2D projections of the keypoints and the 2D projection of the center point: in, This represents the Euclidean distance between the two-dimensional projections of the key points and the two-dimensional projection of the center point. The distance standard deviation is used to control the range of weight decay and can be preset according to the gripper scale or validation data.

[0065] (iii) Calculate the geometric mean normalized result of the confidence weight and the distance weight to obtain the mixed weight: S42. Using the initial pose obtained from standard PnP as the initial value for iteration, and based on the correspondence between each 2D-3D keypoint, estimate the six-DOF pose of the candidate grasping object using a weighted PnP solver, including the rotation matrix. Translation vector .

[0066] S43. Determine the camera projection function based on the camera intrinsic parameter matrix K. , the three-dimensional key points of the gripper The coordinates of the projected points are obtained by projecting the currently estimated rotation matrix and translation vector onto the image plane: S44. Calculate the projection point obtained from the above formula. Corresponding 2D observation points obtained in step S22 The Euclidean distance between them, and according to the assigned weights. Weighting is applied.

[0067] S45, Iteratively adjust the rotation matrix Translation vector The optimal six-DOF pose is obtained by minimizing the sum of the weighted projection errors of all key points.

[0068] The above attitude optimization can be expressed as the following weighted projection error minimization problem: in, Let represent the optimal rotation matrix and translation vector, respectively. This represents the value of the independent variable that minimizes the objective function; For the first 3D coordinates of key points; This represents the camera projection function.

[0069] The base value of the keypoint weight is determined by the keypoint correction confidence, the distance from the keypoint to the center, or a combination of the two, and remains unchanged during the weighted PnP optimization process of the candidate capture.

[0070] When the weighted PnP optimization fails to converge or the weighted projection error exceeds a preset threshold, the initial pose obtained by the standard PnP is output; when the initial PnP fails to solve due to insufficient effective two-dimensional-three-dimensional corresponding points or degradation of key point configuration, the corresponding candidate capture is removed.

[0071] Thus, the target region constraint, candidate capture generation, key point geometric quality evaluation, initial pose determination, candidate screening, and weighted pose optimization form a complete processing flow, which can reduce the impact of non-target regions and low-quality key points on pose generation.

[0072] To verify the beneficial effects of the method provided in this embodiment of the invention (hereinafter referred to as "this method"), under the same RGB-D acquisition device, camera calibration, robotic arm hardware, and execution threshold conditions, this method was compared with existing representative methods (KGN, GraspNet, MegaPose), and the candidate selection strategy and keypoint weight design were verified. The experimental results are as follows: Figures 3 to 5 As shown.

[0073] like Figure 3 As shown, in the scenario of industrial part grasping, this method (corresponding to...) Figure 3 The success rate of our method in grasping cubes reached 72.0%, significantly higher than GraspNet (63.0%), KGN (58.0%), and MegaPose (8.0%). Meanwhile, the average inference time of our method was only 3.65s, far lower than MegaPose's 22.34s, and more efficient than GraspNet (5.74s). In the cube grasping scenario, the success rate of our method further reached 75.6%. These figures reflect… Figure 3 The comparison results of each method under the experimental conditions shown are presented.

[0074] like Figure 4 As shown, after introducing top-down grabbing geometry filtering, in Figure 4 The success rate for each number of Top-K candidates shown is higher than that of the scheme without geometric filtering; the success rate is 78.2% when Top-K is 3, and the success rate decreases as Top-K continues to increase. Under the conditions of this experiment, Top-K is preferably 3.

[0075] like Figure 5 As shown, a comparison of four keypoint weighting strategies—uniform weight, confidence weight, distance weight, and mixed weight—is presented under different normalized noise levels: uniform weight decreases in success rate as noise increases; both confidence weight and distance weight exhibit some robustness; and the mixed weight, obtained by the geometric mean of confidence weight and distance weight, shows... Figure 5 Under the experimental conditions shown, the overall performance was better, with a success rate of 79.6% in the absence of noise.

[0076] Furthermore, to evaluate the robustness of the weighted PnP algorithm when there are errors in key point localization, while keeping the target area constraints and candidate capture unchanged, zero-mean Gaussian noise with standard deviations of 1 pixel, 3 pixels and 5 pixels were injected into the decoded key points. The capture success rates of the standard EPnP and the weighted PnP of this invention were compared, and the results are shown in Table 1.

[0077] Table 1. Comparison of capture success rates (%) between weighted PnP and EPnP under different keypoint noise levels. As shown in Table 1, as the standard deviation of keypoint noise increases, the success rate of EPnP with uniform weights decreases from 74.5% to 63.8%. The success rate of the weighted PnP of this invention under 5-pixel noise is 72.5%, reflecting the influence of keypoint weights on the stability of attitude solution under the noise conditions shown in Table 1.

[0078] Furthermore, to evaluate the stability of this method under simulated sensing degradation conditions, RGB Gaussian noise, depth bias, random depth loss, and partial occlusion were injected into the RGB-D observations before candidate capture generation. The average capture success rate of EPnP and the weighted PnP of this invention were compared, and the results are shown in Table 2.

[0079] Table 2. Average grasping success rate (%) under different sensor degradation conditions As shown in Table 2, under the simulated various sensor degradation conditions, the capture success rate of the weighted PnP method of this invention is higher than that of EPnP; under the partial occlusion condition, the success rates are 69.7% and 58.3% respectively, indicating that the method has good stability under the simulated observation degradation conditions.

[0080] Furthermore, to evaluate the role of each component of this method, ablation experiments were conducted on a cubic datum to remove target region constraints, candidate confidence evaluation, and weighted PnP respectively, and the corresponding average capture success rates were compared. The results are shown in Table 3.

[0081] Table 3. Ablation experiment results on the cube baseline (average capture success rate %) As shown in Table 3, the success rate of crawling was 44.2% after removing the target region constraint, 61.8% after removing the candidate confidence evaluation, and 70.4% after removing the weighted PnP, all of which were lower than the 75.6% of the complete frame, reflecting the impact of the above components on crawling performance under the experimental conditions shown in Table 3.

[0082] Based on the same inventive concept, this embodiment also provides an interactive grasping and generation device that integrates visual language models and key point learning, including: The target region determination module is used to acquire RGB images, depth maps and natural language instructions, use a visual language model to determine the target object that matches the natural language instructions, and generate target region constraint information containing the target object name and bounding box. A candidate capture generation module is used to define the region where the two-dimensional projection of the center point of the candidate capture is located based on the target region constraint information, and to generate multiple candidate captures within the defined region, including: A multi-task convolutional neural network is used to predict the center heatmap, center sub-pixel offset, two-dimensional offset from the center to the predefined three-dimensional key points of the gripper, and gripper width based on the RGB image and depth map. Multiple center point 2D projections are extracted and subpixel corrected from the grabbing center heatmap. The corresponding key point 2D projections are obtained based on the 2D offset. The heatmap response values ​​are corrected based on the distance, spacing, and symmetry of the key point 2D projections relative to the corresponding center point 2D projections to obtain key point correction confidence and candidate confidence. Each center point 2D projection is combined with its corresponding key point 2D projection, gripper width, key point correction confidence, and candidate confidence to form multiple candidate grabs. The candidate selection module is used to perform PnP solution based on the correspondence between the 2D projection of key points and the 3D key points of grippers to obtain the initial six-degree-of-freedom pose of each candidate gripper; and to select candidate grippers that meet the top-view gripping constraints based on the consistency of gripper orientation with the direction of gravity and the horizontal alignment of key points. The attitude optimization module is used to determine the key point weights based on the key point correction confidence and the two-dimensional projection distance from the key point to the center point after screening the candidate grasping key points. By minimizing the weighted projection error, the initial six-degree-of-freedom attitude is optimized by weighted PnP to obtain the target gripper's six-degree-of-freedom attitude.

[0083] The target area determination module, candidate capture generation module, initial posture determination module, candidate capture filtering module, and posture optimization module correspond to the relevant functions of steps S1, S2, S3, and S4, respectively. Each module can be called by the same processor in the above order, or it can be implemented by software components or hardware units that communicate with each other.

[0084] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the interactive capture generation method that integrates visual language models and key point learning as provided in any embodiment of the present invention.

[0085] In embodiments of the present invention, any combination of one or more storage media may be used. The storage medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0086] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 6 The diagram shown is a structural schematic of the electronic device 100 provided in an embodiment of the present invention. In some embodiments, the electronic device may be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), or other terminal device. Furthermore, the interactive crawling generation method integrating visual language models and keypoint learning provided in this embodiment of the present invention can also be applied to databases, servers, and service response systems based on terminal artificial intelligence. This embodiment of the present invention does not limit the specific application scenarios of the interactive crawling generation method integrating visual language models and keypoint learning.

[0087] like Figure 6 As shown, the electronic device 100 provided in this embodiment of the invention includes a memory 101 and a processor 102.

[0088] The memory 101 is used to store computer programs; preferably, the memory 101 includes various media that can store program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk.

[0089] Specifically, memory 101 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Electronic device 100 may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 101 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0090] The processor 102 is connected to the memory 101 and is used to execute the computer program stored in the memory 101 to implement the interactive capture generation method that integrates visual language model and key point learning provided in any embodiment of the present invention.

[0091] In an optional implementation, the processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, and discrete hardware components.

[0092] In an optional embodiment of the present invention, the electronic device 100 may further include a display 103. The display 103 is communicatively connected to the memory 101 and the processor 102, and is used to display the relevant GUI interface of the interactive grasping and generation method that integrates visual language models and key point learning.

[0093] It is understood that the interactive crawling generation device, storage medium, and electronic device that integrate visual language model and key point learning provided in the embodiments of the present invention correspond to the interactive crawling generation method that integrates visual language model and key point learning provided in the embodiments of the present invention. The explanation, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the method, and will not be repeated here.

[0094] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0095] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An interactive grasping and generation method integrating visual language models and keypoint learning, characterized in that, include: Acquire RGB images, depth maps, and natural language instructions; use a visual language model to determine the target object that matches the natural language instructions; and generate target region constraint information that includes the target object name and bounding box. Based on the target region constraint information, the region where the two-dimensional projection of the center point of the candidate grab is located is defined, and multiple candidate grabs are generated within the defined region, including: A multi-task convolutional neural network is used to predict the grabbing center heatmap, center sub-pixel offset, 2D offset from the center to predefined 3D key points of the gripper, and gripper width based on the RGB image and depth map. Multiple 2D projections of the center points are extracted and sub-pixel corrected from the grabbing center heatmap. Corresponding 2D projections of key points are obtained based on the 2D offset. The heatmap response value is corrected based on the distance, spacing, and symmetry of the 2D projections of key points relative to their corresponding 2D projections of center points to obtain key point correction confidence and candidate confidence. Each center point 2D projection is combined with its corresponding key point 2D projection, gripper width, key point correction confidence, and candidate confidence to form multiple candidate grabs. PnP is solved based on the correspondence between the 2D projection of the key points and the 3D key points of the gripper to obtain the initial six-degree-of-freedom pose of each candidate gripper; based on the consistency of the gripper orientation with the direction of gravity and the horizontal alignment of the key points, candidate grippers that meet the top-view gripping constraints are selected. The key point weights are determined based on the key point correction confidence and the two-dimensional projection distance from the key point to the center point after screening. The initial six-degree-of-freedom attitude is then optimized by weighted PnP by minimizing the weighted projection error to obtain the target gripper's six-degree-of-freedom attitude.

2. The interactive crawling generation method as described in claim 1, characterized in that, The initial six-degree-of-freedom pose is obtained using the standard PnP algorithm; if the weighted PnP optimization fails to converge or the weighted projection error exceeds a preset threshold, the initial six-degree-of-freedom pose is taken as the target gripper's six-degree-of-freedom pose.

3. The interactive crawling generation method as described in claim 1, characterized in that, The loss function construction process of the multi-task convolutional neural network includes: An improved Focal Loss is used to construct a center heatmap loss, wherein the improved Focal Loss adjusts the gradient weights of positive and negative samples separately through two parameters. L1 loss is used to construct center offset loss, keypoint offset loss and gripper width loss; The center heatmap loss, the center offset loss, the key point offset loss, and the gripper width loss are weighted and summed using preset weight coefficients to obtain the comprehensive training loss.

4. The interactive crawling generation method as described in claim 1, characterized in that, The predefined gripper's three-dimensional key points include four key points: the top left and right sides and the bottom left and right sides; The process of determining the keypoint correction confidence and candidate confidence includes: The response value of the captured center heatmap at the corresponding center position is used as the initial confidence level; The distance from each key point to the center point is normalized based on the gripper width to determine the key point distance; the spacing anomaly is determined based on the deviation of the spacing between adjacent key points relative to the preset spacing; and the symmetry deviation is determined based on the difference in the horizontal distances from the corresponding left and right key points to the center point. The initial confidence level is corrected using the key point distance, the spacing anomaly, and the symmetry deviation, and the correction result is limited to between 0 and 1 to obtain the adjusted confidence level of each key point; The distance weight and symmetry weight are determined based on the key point distance and the symmetry deviation, respectively. The adjusted confidence is then weighted and normalized to obtain the key point correction confidence. The candidate confidence score is obtained by aggregating the corrected confidence scores of multiple key points captured from the same candidate.

5. The interactive crawling generation method as described in claim 1, characterized in that, The initial six-degree-of-freedom pose includes an initial rotation matrix and an initial translation vector; candidate grasping methods that satisfy the top-view grasping constraints are selected, including: Transform the end normal vector in the gripper coordinate system to the world coordinate system based on the initial rotation matrix; Calculate the cosine of the angle between the transformed end normal vector and the gravity direction in the world coordinate system, and restrict the cosine of the angle to a non-negative value to obtain the direction consistency score; Calculate the horizontal deviations of the top key points on the left and right sides and the bottom key points on the left and right sides respectively, take the larger value of the two and obtain the horizontal alignment score through the exponential decay function; The comprehensive geometric score is determined based on the geometric mean of the directional consistency score and the horizontal alignment score; Candidate crawls with a comprehensive geometric score lower than a preset top-down crawl threshold are removed, and the remaining candidate crawls are sorted according to the candidate confidence level.

6. The interactive crawling generation method as described in claim 1, characterized in that, For each candidate crawled after filtering, the weighted PnP optimization is performed separately, including: Construct a set of corresponding 2D and 3D points based on the 2D projection of each key point and its corresponding predefined 3D key points of the gripper, and obtain the camera intrinsic parameters; The confidence scores of the key point corrections are normalized to obtain the confidence scores weights; The distance weight is obtained by Gaussian attenuation and normalization based on the distance from the 2D projection of the key point to the 2D projection of the center point; The geometric mean of the confidence weight and the distance weight is normalized to obtain the mixed weight, and the confidence weight, the distance weight, or the mixed weight is used as the key point weight; The initial six-degree-of-freedom attitude is used as the initial value for the weighted PnP solver iteration. Based on the camera intrinsic parameters, the current rotation matrix, and the current translation vector, project the three-dimensional key points of each predefined gripper onto the image plane; Calculate the reprojection error between each projection point and the corresponding key point's two-dimensional projection, and use the key point weights to weight the reprojection error; The current rotation matrix and the current translation vector are iteratively adjusted to reduce the sum of the weighted projection errors, thereby obtaining the six-degree-of-freedom pose of the target gripper.

7. The interactive crawling generation method as described in claim 6, characterized in that, The origin of the gripper coordinate system is taken as the three-dimensional center point of the gripper, and the three-dimensional center point of the gripper and its corresponding two-dimensional projection are added to the set of two-dimensional corresponding points as additional two-dimensional corresponding points.

8. An interactive grasping and generation device that integrates visual language models and key point learning, characterized in that, include: The target region determination module is used to acquire RGB images, depth maps and natural language instructions, use a visual language model to determine the target object that matches the natural language instructions, and generate target region constraint information containing the target object name and bounding box. A candidate capture generation module is used to define the region where the two-dimensional projection of the center point of the candidate capture is located based on the target region constraint information, and to generate multiple candidate captures within the defined region, including: A multi-task convolutional neural network is used to predict the center heatmap, center sub-pixel offset, two-dimensional offset from the center to the predefined three-dimensional key points of the gripper, and gripper width based on the RGB image and depth map. Multiple center point 2D projections are extracted and subpixel corrected from the grabbing center heatmap. The corresponding key point 2D projections are obtained based on the 2D offset. The heatmap response values ​​are corrected based on the distance, spacing, and symmetry of the key point 2D projections relative to the corresponding center point 2D projections to obtain key point correction confidence and candidate confidence. Each center point 2D projection is combined with its corresponding key point 2D projection, gripper width, key point correction confidence, and candidate confidence to form multiple candidate grabs. The candidate selection module is used to perform PnP solution based on the correspondence between the 2D projection of key points and the 3D key points of grippers to obtain the initial six-degree-of-freedom pose of each candidate gripper; and to select candidate grippers that meet the top-view gripping constraints based on the consistency of gripper orientation with the direction of gravity and the horizontal alignment of key points. The attitude optimization module is used to determine the key point weights based on the key point correction confidence and the two-dimensional projection distance from the key point to the center point after screening the candidate grasping key points. By minimizing the weighted projection error, the initial six-degree-of-freedom attitude is optimized by weighted PnP to obtain the target gripper's six-degree-of-freedom attitude.

9. A non-transient computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the interactive crawling generation method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, Including processor and memory; The memory stores computer programs; The processor is used to execute the computer program to implement the interactive crawling generation method according to any one of claims 1 to 7.

Images