A target object context-aware graspability prediction method and system

Abstract

The application discloses a target object context perception-based graspability prediction method and system, which comprises the following steps: acquiring and preprocessing local color point clouds of a target box and global color point clouds of the surrounding environment; determining prior information of the target box, and registering the preprocessed local color point clouds into the global color point clouds based on the prior information to obtain fused global color point clouds; determining surface points of the target box based on the fused global color point clouds and the prior information, and determining candidate grasp points that can be used for grasping among the surface points; generating grasp vectors of each candidate grasp point, and constructing a multi-dimensional feature vector of a candidate grasp pose on the target box based on the grasp vectors; inputting the multi-dimensional feature vector of each candidate grasp pose into a graspability prediction model to obtain a graspability score, and determining an optimal grasp pose based on the graspability score. The application significantly improves the grasping success rate, adaptability and operation safety of a robot in a complex scene through multi-level fusion of perception and intelligent decision-making.

Description

Technical Field

[0001] This invention relates to the field of logistics automation technology, and in particular to a method and system for predicting the graspability of a target object based on its context awareness. Background Technology

[0002] In intelligent manufacturing, smart logistics, and flexible automated production scenarios, the stable and reliable grasping of regular-shaped boxes by robots is a fundamental and crucial task.

[0003] However, traditional grasping planning methods rely heavily on the geometric features of the target object, lacking sufficient awareness of the environmental context. They only consider the target object itself, ignoring its spatial relationship with the surrounding environment. In complex, dynamic, and unstructured real-world work scenarios, their generalization ability and robustness are significantly limited. Moreover, traditional point cloud processing methods mostly rely on point clouds from a single viewpoint, ignoring rich color information. They also struggle to consider both target surface details and global environmental constraints, and often neglect physical stability and semantic priors during the grasping process, resulting in a high grasping failure rate or difficulty adapting to changing industrial scenarios. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method for predicting graspability based on target object context awareness, comprising:

[0005] Acquire the local color point cloud of the target box and the global color point cloud of the surrounding environment, and preprocess the local color point cloud and the global color point cloud;

[0006] The prior information of the target box is determined, and the preprocessed local color point cloud is registered to the global color point cloud based on the prior information to obtain the fused global color point cloud;

[0007] The surface points of the target box are determined by fusing global color point cloud and prior information, and candidate grasping points that can be used for grasping are identified from the surface points.

[0008] Generate the grasp vectors for each candidate grasp point, and construct a multi-dimensional feature vector of the candidate grasp pose on the target box based on each candidate grasp point and grasp vector;

[0009] The multidimensional feature vectors of each candidate grasping pose are input into the graspability prediction model to obtain a graspability score, and the optimal grasping pose is determined based on the graspability score.

[0010] Furthermore, the acquisition of the local color point cloud of the target container and the global color point cloud of the surrounding environment, and the preprocessing of the local color point cloud and the global color point cloud, include:

[0011] The local color point cloud of the target box and the global color point cloud of the surrounding environment are acquired, and the local color point cloud and the global color point cloud are preprocessed, including noise reduction and downsampling.

[0012] Furthermore, the step of determining the prior information of the target box and registering the preprocessed local color point cloud to the global color point cloud based on the prior information to obtain the fused global color point cloud includes:

[0013] Determine the prior information of the target box, which includes the 6D pose of the target box, the box specification list, the center point coordinates, and the three-dimensional dimensions.

[0014] Based on prior information, a transformation matrix is ​​constructed from the target box coordinate system to the global coordinate system, and the local color point cloud is transformed to the global coordinate system through the transformation matrix to obtain the local color transformed point cloud;

[0015] Remove points belonging to the target box region from the global color point cloud to obtain the global color filtered point cloud, and merge the local color transformed point cloud with the global color filtered point cloud to obtain the fused global color point cloud.

[0016] Furthermore, the determination of the surface points of the target box based on the fusion of global color point cloud and prior information includes:

[0017] Based on prior information, each point on the target box is determined from the fused global color point cloud, and the normal vector of each point is calculated.

[0018] Determine the six directional axes in the coordinate system corresponding to the fused global color point cloud, and calculate the angle between the normal vector of the point and each directional axis;

[0019] Determine the smallest included angle among the calculated included angles, and determine the preset included angle;

[0020] Select the points whose minimum included angle is less than the preset included angle, and determine the selected points on the target box as the surface points of the target box.

[0021] Furthermore, determining candidate gripping points among the surface points that can be used for gripping includes:

[0022] Determine the normal vector of each surface point on the target box, and divide each surface point based on the normal vector to obtain the surface points on different surfaces;

[0023] Select surface points with opposite and parallel normal vectors on different surfaces, and determine the positional distances between these surface points;

[0024] These surface points are matched one by one according to the minimum positional distance to obtain multiple surface point groups, and the centroid of the target box is determined.

[0025] Project the centroid onto each surface of the target box to obtain the centroid projection point on each surface, and determine the distance between each surface point in the surface point group and the corresponding centroid projection point on the surface.

[0026] Determine a preset distance value and filter out surface points with a distance less than the preset distance value as candidate gripping points that can be used for gripping.

[0027] Furthermore, the step of generating the grasping vectors for each candidate grasping point, and constructing a multi-dimensional feature vector of the candidate grasping pose on the target box based on each candidate grasping point and grasping vector, includes:

[0028] Determine the crawling direction for each candidate crawling point, and generate a crawling vector for each candidate crawling point based on the crawling direction;

[0029] Determine the coordinates of each candidate grab point, and construct the candidate grab pose on the target box based on the coordinates of each candidate grab point and the grab vector;

[0030] Based on context awareness, the geometric features, environmental collision features, color semantic features, and stability features of the target box in the fused global color point cloud are determined, and the geometric features, environmental collision features, color semantic features, and stability features are converted into vectors to form a multi-dimensional feature vector of the candidate grasping pose.

[0031] Furthermore, the step of inputting the multidimensional feature vectors of each candidate grasping pose into the graspability prediction model to obtain a graspability score includes:

[0032] The multidimensional feature vectors of each candidate grasping pose are input into the graspability prediction model, and the graspability prediction model makes predictions to obtain the graspability score of each candidate grasping pose.

[0033] Furthermore, the process of constructing the crawlability prediction model includes:

[0034] The target box is simulated to grasp in a preset grasping simulation environment, and the result data of the grasping simulation is determined.

[0035] A dataset is constructed based on the results data, the data features of the dataset are extracted, and the data features of the dataset are input into a preset neural network model to construct an initial graspability prediction model for the target box.

[0036] The dataset is divided into training and test sets according to a preset ratio, and the training and test sets are input into the initial graspability prediction model of the target box.

[0037] The initial graspability prediction model of the target box is trained and tested until it meets the preset convergence condition, thus obtaining the graspability prediction model of the target box.

[0038] Furthermore, the determination of the optimal grasping pose based on the graspability score includes:

[0039] The graspability scores of all candidate grasping poses are sorted in ascending order to obtain a score sequence, and the top three candidate grasping poses in the score sequence are determined as the preferred grasping poses for the target box.

[0040] The present invention also provides a target object context-aware graspability prediction system, comprising:

[0041] The acquisition module is used to acquire the local color point cloud of the target box and the global color point cloud of the surrounding environment, and to preprocess the local color point cloud and the global color point cloud.

[0042] The fusion module is used to determine the prior information of the target box and register the preprocessed local color point cloud to the global color point cloud based on the prior information to obtain the fused global color point cloud;

[0043] The determination module is used to determine the surface points of the target box based on the fusion of global color point cloud and prior information, and to determine the candidate grasping points among the surface points that can be used for grasping;

[0044] The module is used to generate the grasp vector of each candidate grasp point, and to construct a multi-dimensional feature vector of the candidate grasp pose on the target box based on each candidate grasp point and the grasp vector.

[0045] The prediction module is used to input the multidimensional feature vectors of each candidate grasping pose into the graspability prediction model to obtain a graspability score, and to determine the preferred grasping pose based on the graspability score.

[0046] Compared with existing technologies, the graspability prediction method and system based on target object context awareness of this invention have the following advantages:

[0047] This invention constructs a unified scene representation that combines the integrity of target details with the semantic consistency of environmental context by fusing and registering local high-precision point clouds with global environmental point clouds, effectively overcoming the occlusion defects and resolution limitations of single-view point clouds.

[0048] This invention is based on the description of grasping pose using multi-dimensional feature vectors, which can simulate the multi-factor comprehensive decision-making process of human grasping and quantitatively evaluate the feasibility of each potential grasping pose within the framework of mechanical constraints and physical laws.

[0049] This invention performs end-to-end mapping and sorting of multi-dimensional features through a graspability prediction model, and can autonomously identify the optimal grasping scheme with high success rate, low collision risk and strong stability from many candidate poses.

[0050] Overall, this invention significantly improves the robustness, adaptability, and intelligence of robots in grasping regular boxes in real-world scenarios such as logistics sorting and warehousing by integrating multi-level perception and intelligent decision-making. It reduces reliance on preset templates and manual parameter tuning, and achieves high success rate and high adaptability of robot grasping of target boxes in complex and unstructured environments. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the flow structure of the graspability prediction method based on target object context awareness in an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the composition of the graspability prediction system based on target object context awareness in an embodiment of the present invention. Detailed Implementation

[0053] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0054] like Figure 1 As shown in the embodiments of this application, a graspability prediction method based on target object context awareness is provided, including: S100: acquiring the local color point cloud of the target box and the global color point cloud of the surrounding environment, and preprocessing the local color point cloud and the global color point cloud; S200: determining the prior information of the target box, and registering the preprocessed local color point cloud to the global color point cloud based on the prior information to obtain a fused global color point cloud; S300: determining the surface points of the target box based on the fused global color point cloud and the prior information, and determining the candidate grasping points that can be grasped among the surface points; S400: generating the grasping vector of each candidate grasping point, and constructing a multidimensional feature vector of the candidate grasping pose on the target box based on each candidate grasping point and the grasping vector; S500: inputting the multidimensional feature vector of each candidate grasping pose into the graspability prediction model to obtain a graspability score, and determining the preferred grasping pose based on the graspability score.

[0055] Furthermore, this invention constructs a unified scene representation that combines target detail integrity with environmental context semantic consistency through the fusion and registration of local high-precision point clouds and global environment point clouds, effectively overcoming the occlusion defects and resolution limitations of single-viewpoint point clouds. Based on multi-dimensional feature vector-based grasping pose description, this invention can simulate the multi-factor comprehensive decision-making process of human grasping, quantitatively evaluating the feasibility of each potential grasping pose within the framework of mechanical constraints and physical laws. Through an graspability prediction model, this invention performs end-to-end mapping and sorting of multi-dimensional features, autonomously identifying the optimal grasping scheme from numerous candidate poses that combines high success rate, low collision risk, and strong stability. Overall, this invention, through multi-level fusion perception and intelligent decision-making, significantly improves the robustness, adaptability, and intelligence of robots grasping regular boxes in practical scenarios such as logistics sorting and warehousing, reducing reliance on preset templates and manual parameter tuning, and achieving high success rate and high adaptability robot grasping of target boxes in complex unstructured environments.

[0056] In the embodiments of this application, a target object context-aware graspability prediction method is provided. The method of acquiring the local color point cloud of the target box and the global color point cloud of the surrounding environment, and preprocessing the local color point cloud and the global color point cloud, includes: acquiring the local color point cloud of the target box and the global color point cloud of the surrounding environment, and preprocessing the local color point cloud and the global color point cloud, wherein the preprocessing includes noise reduction processing and downsampling processing.

[0057] Specifically, the process involves acquiring local color point clouds of the target container and global color point clouds of its surrounding environment. Both local and global point clouds undergo preprocessing, including denoising and downsampling. Denoising effectively filters out sensor noise and measurement outliers, significantly improving the purity of the point cloud and the reliability of its geometric features, thus eliminating interference for subsequent accurate registration and feature extraction. Downsampling, while preserving key geometric structures and semantic information, drastically reduces the amount of point cloud data, optimizing computational and storage efficiency and ensuring the real-time performance of subsequent registration, fusion, and analysis processes. This step provides a unified, clean, and efficient multi-source data entry point, a crucial prerequisite for achieving subsequent high-precision fusion and intelligent decision-making.

[0058] In embodiments of this application, a target object context-aware graspability prediction method is provided. The method involves determining prior information about the target container and registering a preprocessed local color point cloud to a global color point cloud based on this prior information to obtain a fused global color point cloud. This includes: determining prior information about the target container, including its 6D pose, box size list, center point coordinates, and three-dimensional dimensions; constructing a transformation matrix from the target container coordinate system to the global coordinate system based on the prior information, and transforming the local color point cloud to the global coordinate system using the transformation matrix to obtain a local color transformed point cloud; removing points belonging to the target container region from the global color point cloud to obtain a global color filtered point cloud, and merging the local color transformed point cloud with the global color filtered point cloud to obtain a fused global color point cloud.

[0059] Specifically, the prior information of the target box is determined, including its 6D pose, box size list, center point coordinates, and 3D dimensions. Using the pose information, a precise transformation matrix from the target box coordinate system to the global coordinate system can be directly constructed. This matrix is ​​then used to transform the high-precision local color point cloud to the global coordinate system in one step, resulting in a locally color-transformed point cloud with accurate pose. In the global color point cloud, based on the prior dimensions and position information of the target box, point cloud data that originally belonged to the box area is identified and removed, thus obtaining a global color-filtered point cloud that eliminates low-quality or duplicate data in the target area. The local color-transformed point cloud and the global color-filtered point cloud are then merged to generate a fused global color point cloud. This step directly drives registration with prior information, avoiding the cumulative errors and computational delays that may occur with iterative registration algorithms, and achieving efficient and accurate coordinate alignment. By adopting the strategy of "removing first and then fusing", it effectively solves the data conflict and redundancy problem between local and global point clouds in the target area, ensuring the high resolution and integrity of the fused data on the target subject. The generated fused point cloud retains the fine geometric and texture details of the local point cloud in the target box area, and completely retains the obstacle and spatial layout information of the global point cloud in the environment area, thus constructing a unified, coherent and high-quality scene digital base, providing a reliable and information-complete environmental perception foundation for subsequent grasping point analysis, collision detection and stability assessment.

[0060] In the embodiments of this application, a target object context-aware graspability prediction method is provided. The method for determining the surface points of the target box based on the fusion of global color point cloud and prior information includes: determining each point on the target box from the fusion of global color point cloud based on prior information, and calculating the normal vector of each point; determining the six direction axes in the coordinate axis corresponding to the fusion of global color point cloud, and calculating the angle between the normal vector of the point and each direction axis; determining the minimum angle among the calculated angles, and determining a preset angle; selecting the points whose minimum angle is less than the preset angle, and determining the selected points on the target box as the surface points of the target box.

[0061] Specifically, based on the known location and size of the box, all points belonging to the target box are segmented from the fused point cloud; the normal vector of each point is calculated to represent its local surface orientation; six coordinate axes (+X, -X, +Y, -Y, +Z, -Z) are defined in the global coordinate system, and the angle between the normal vector of each point and these six axes is calculated one by one; the minimum value of the six angles corresponding to each point is found and compared with a pre-set angle threshold; points with a minimum angle less than the threshold are determined to be surface points of the box, which means that the local surface orientation of the point is close enough to a certain coordinate axis direction, thus confirming that it is located on a flat main surface of the box. This step uses the angle between the normal vector and the coordinate axis to efficiently and accurately identify flat surface areas on the box, excluding atypical surface points located at edges, corners, or noise points. By using preset thresholds for filtering, it can flexibly adapt to the requirements of different box surface flatness, balancing recognition accuracy and robustness. The resulting set of surface points directly corresponds to the six principal planes of the box, providing clear and standardized input for subsequent generation of grasping points based on regular geometry, thereby supporting the stability and repeatability of the grasping pose.

[0062] In embodiments of this application, a graspability prediction method based on target object context awareness is provided. The method for determining candidate grasping points among surface points includes: determining the normal vectors of each surface point on the target box, and dividing the surface points based on the normal vectors to obtain surface points on different surfaces; filtering surface points with opposite and parallel normal vectors on different surfaces, and determining the positional distances between these surface points; matching these surface points one-to-one according to the minimum positional distance to obtain multiple surface point groups, and determining the centroid of the target box; projecting the centroid onto each surface of the target box to obtain centroid projection points on each surface, and determining the distance between each surface point in the surface point group and the corresponding centroid projection point on the surface; determining a pre-set preset distance value, and filtering surface points with distances less than the preset distance value as candidate grasping points among the surface points.

[0063] Specifically, points are assigned to different surfaces of the box based on their normal vector directions, thus organizing the surface point set. Points with opposite and parallel normal vectors are actively selected on different surfaces. The spatial distance between all potential point pairs is calculated, and the points with the smallest distance are precisely matched to form multiple potential "gripping point pairs." The three-dimensional centroid of the target box is calculated and projected perpendicularly onto each surface to obtain the centroid projection point of each surface. The distance between each point in each candidate "gripping point pair" and the centroid projection point of its respective surface is calculated, and points with distances less than a preset threshold are selected. These points, being closer to the surface center, provide better torque balance and gripping stability. This step, by automatically matching point pairs with parallel and opposite normal vectors, directly ensures the feasibility of gripper force closure from a geometric perspective, laying the physical foundation for the gripping action. Introducing centroid projection constraints, prioritizing gripping points close to the center of each surface, greatly enhances the stability against rotation and overturning after gripping. The entire selection process is entirely based on geometric calculations, without relying on complex models or large amounts of labeled data, combining algorithmic efficiency with interpretable results.

[0064] In the embodiments of this application, a target object context-aware graspability prediction method is provided. The step of generating grasp vectors for each candidate grasp point and constructing a multi-dimensional feature vector of candidate grasp poses on the target box based on each candidate grasp point and grasp vector includes: determining the grasping direction of each candidate grasp point and generating grasp vectors for each candidate grasp point based on the grasping direction; determining the coordinate position of each candidate grasp point and constructing candidate grasp poses on the target box based on the coordinate position of each candidate grasp point and grasp vectors; determining the geometric features, environmental collision features, color semantic features, and stability features of the target box in the fused global color point cloud based on context awareness, and converting the geometric features, environmental collision features, color semantic features, and stability features into vectors to construct a multi-dimensional feature vector of candidate grasp poses.

[0065] Specifically, the grasping direction is determined based on the surface normal vector of each candidate grasping point, and the corresponding grasping vector is generated. Combined with the coordinate position of the point, a complete candidate grasping pose is formed. Relying on the rich contextual information contained in the fused global color point cloud, four types of key features are extracted: geometric features (including surface curvature, size matching degree, etc.), environmental collision features (the degree of interference between the grasping path and surrounding obstacles), color semantic features (such as the recognition of labels, tape or functional patterns), and stability features (physical stability assessment based on center of gravity and torque). These features are quantified into numerical values ​​and organized into multi-dimensional feature vectors, which serve as a comprehensive descriptor for each candidate grasping pose. This step elevates discrete grasping points into complete poses containing directional information, enabling grasping planning to directly interface with the robot's motion control interface. Through the fusion of multi-dimensional features, it achieves comprehensive modeling of grasping success rate, environmental safety, semantic rationality, and physical stability, making the evaluation criteria go beyond simple geometric constraints and more closely reflect the complex needs of real-world scenarios. The structured feature vectors provide standardized input for subsequent graspability prediction models, enabling machine learning models to efficiently learn the mapping relationship between features and grasping results, thereby supporting data-driven intelligent grasping decisions.

[0066] In the embodiments of this application, a graspability prediction method based on target object context awareness is provided. The step of inputting the multidimensional feature vectors of each candidate grasping pose into the graspability prediction model to obtain a graspability score includes: inputting the multidimensional feature vectors of each candidate grasping pose into the graspability prediction model, and having the graspability prediction model perform prediction output to obtain the graspability score of each candidate grasping pose.

[0067] In the embodiments of this application, a graspability prediction method based on target object context awareness is provided. The process of constructing the graspability prediction model includes: performing a grasping simulation on the target box in a preset grasping simulation environment and determining the result data of the grasping simulation; constructing a dataset based on the result data, extracting the data features of the dataset, and inputting the data features of the dataset into a preset neural network model to construct an initial graspability prediction model for the target box; dividing the dataset into a training set and a test set according to a preset ratio, and inputting the training set and the test set into the initial graspability prediction model for the target box; training and testing the initial graspability prediction model for the target box until the initial graspability prediction model for the target box meets a preset convergence condition, thereby obtaining the graspability prediction model for the target box.

[0068] Specifically, in a pre-defined grasping simulation environment, large-scale and diverse grasping simulations are performed on the target box to generate grasping attempt data containing both successful and failed results. A dataset is constructed based on these simulation results, and multi-dimensional data features corresponding to candidate grasping poses are extracted from it. These features are input into a pre-defined neural network model to construct an initial graspability prediction model. The dataset is divided into a training set and a test set according to a pre-defined ratio, and the initial model is iteratively trained and validated. By continuously optimizing the model parameters, the prediction accuracy, loss function, and other indicators meet the pre-defined convergence conditions, and finally a fully trained and stable graspability prediction model is obtained. This step utilizes a simulation environment to replace a large number of real physical experiments, enabling efficient and low-cost generation of training data covering various poses, environments, and box states. This solves the problems of difficult real data acquisition and high annotation costs. The neural network-based model can automatically learn complex nonlinear relationships from multidimensional features, capturing the subtle interactions between geometric, physical, and environmental factors, achieving more accurate success rate predictions than traditional analytical models. The rigorous training and testing process ensures the model's generalization ability and robustness, enabling it to maintain reliable predictive performance even when faced with unseen box poses or environmental configurations. This provides a real-time, data-driven intelligent decision-making core for robot grasping planning.

[0069] In the embodiments of this application, a graspability prediction method based on target object context awareness is provided. The method for determining the preferred grasping pose based on the graspability score includes: sorting the graspability scores of all candidate grasping poses in ascending order to obtain a score sequence, and determining the top three candidate grasping poses in the score sequence as the preferred grasping poses of the target box.

[0070] Specifically, all candidate grasping poses are sorted in ascending or descending order based on their graspability scores, forming an ordered scoring sequence. The top three poses with the highest scores in this sequence are selected as the preferred grasping poses for the target box. This step places all candidate solutions under a unified quantitative evaluation framework through the sorting process, transforming the selection process from subjective experience-based judgment to objective data-driven decision-making, enhancing the systematic nature and interpretability of the decision. Selecting the top three, rather than just the highest score, provides crucial operational redundancy and flexibility. When the optimal pose cannot be executed due to dynamic constraints such as real-time collisions or motion inaccessibility, it can be immediately switched to the second or third best pose, thereby significantly improving the robustness and success rate of the overall grasping task. This multi-option strategy provides optimization space for the subsequent motion planning and execution modules, allowing for final fine-tuning of the actual feasibility among multiple high-quality grasping options, further enhancing the system's adaptability in complex scenarios.

[0071] like Figure 2As shown in the embodiments of this application, a graspability prediction system based on target object context awareness is provided, comprising: an acquisition module, used to acquire local color point clouds of a target box and global color point clouds of the surrounding environment, and preprocess the local color point clouds and global color point clouds; a fusion module, used to determine prior information of the target box, and register the preprocessed local color point clouds to the global color point clouds based on the prior information to obtain a fused global color point cloud; a determination module, used to determine surface points of the target box based on the fused global color point cloud and prior information, and determine candidate grasping points among the surface points that can be grasped; a construction module, used to generate grasping vectors for each candidate grasping point, and construct multidimensional feature vectors of candidate grasping poses on the target box based on each candidate grasping point and grasping vector; and a prediction module, used to input the multidimensional feature vectors of each candidate grasping pose into a graspability prediction model to obtain a graspability score, and determine the preferred grasping pose based on the graspability score.

[0072] In summary, this invention provides a graspability prediction method based on target object context awareness, comprising: acquiring and preprocessing a local color point cloud of a target container and a global color point cloud of the surrounding environment; determining prior information about the target container and registering the preprocessed local color point cloud to the global color point cloud to obtain a fused global color point cloud; determining surface points of the target container based on the fused global color point cloud and prior information, and identifying candidate grasping points that can be grasped; generating grasping vectors for each candidate grasping point and constructing multidimensional feature vectors of candidate grasping poses on the target container based on these vectors; inputting the multidimensional feature vectors of each candidate grasping pose into a graspability prediction model to obtain a graspability score, and determining the preferred grasping pose based on the score. This invention significantly improves the grasping success rate, adaptability, and operational safety of robots in complex scenarios through multi-level fusion perception and intelligent decision-making.

[0073] Finally, it should be noted that those skilled in the art can obviously make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for predicting graspability based on target object context awareness, characterized in that, include: Acquire the local color point cloud of the target box and the global color point cloud of the surrounding environment, and preprocess the local color point cloud and the global color point cloud; The prior information of the target box is determined, and the preprocessed local color point cloud is registered to the global color point cloud based on the prior information to obtain the fused global color point cloud; The surface points of the target box are determined by fusing global color point cloud and prior information, and candidate grasping points that can be used for grasping are identified from the surface points. Generate the grasp vectors for each candidate grasp point, and construct a multi-dimensional feature vector of the candidate grasp pose on the target box based on each candidate grasp point and grasp vector; The multidimensional feature vectors of each candidate grasping pose are input into the graspability prediction model to obtain a graspability score, and the optimal grasping pose is determined based on the graspability score. The process of determining the prior information of the target box and registering the preprocessed local color point cloud to the global color point cloud based on the prior information to obtain the fused global color point cloud includes: Determine the prior information of the target box, which includes the 6D pose of the target box, the box specification list, the center point coordinates, and the three-dimensional dimensions. Based on prior information, a transformation matrix is ​​constructed from the target box coordinate system to the global coordinate system, and the local color point cloud is transformed to the global coordinate system through the transformation matrix to obtain the local color transformed point cloud; Remove points belonging to the target box region from the global color point cloud to obtain the global color filtered point cloud, and merge the local color transformed point cloud with the global color filtered point cloud to obtain the fused global color point cloud; The process of generating grasping vectors for each candidate grasping point and constructing multi-dimensional feature vectors of candidate grasping poses on the target box based on each candidate grasping point and grasping vector includes: Determine the crawling direction for each candidate crawling point, and generate a crawling vector for each candidate crawling point based on the crawling direction; Determine the coordinates of each candidate grab point, and construct the candidate grab pose on the target box based on the coordinates of each candidate grab point and the grab vector; Based on context awareness, the geometric features, environmental collision features, color semantic features, and stability features of the target box in the fused global color point cloud are determined, and the geometric features, environmental collision features, color semantic features, and stability features are converted into vectors to form a multi-dimensional feature vector of the candidate grasping pose. The step of inputting the multidimensional feature vectors of each candidate grasping pose into the graspability prediction model to obtain a graspability score includes: The multidimensional feature vectors of each candidate grasping pose are input into the graspability prediction model, and the graspability prediction model makes predictions to obtain the graspability score of each candidate grasping pose.

2. The graspability prediction method based on target object context awareness according to claim 1, characterized in that, The process of acquiring the local color point cloud of the target box and the global color point cloud of the surrounding environment, and preprocessing the local and global color point clouds, includes: Noise reduction and downsampling.

3. The graspability prediction method based on target object context awareness according to claim 1, characterized in that, The surface points of the target box are determined based on the fusion of global color point cloud and prior information, including: Based on prior information, each point on the target box is determined from the fused global color point cloud, and the normal vector of each point is calculated. Determine the six directional axes in the coordinate system corresponding to the fused global color point cloud, and calculate the angle between the normal vector of the point and each directional axis; Determine the smallest included angle among the calculated included angles, and determine the preset included angle; Select the points whose minimum included angle is less than the preset included angle, and determine the selected points on the target box as the surface points of the target box.

4. The graspability prediction method based on target object context awareness according to claim 1, characterized in that, The determination of candidate gripping points among surface points that can be used for gripping includes: Determine the normal vector of each surface point on the target box, and divide each surface point based on the normal vector to obtain the surface points on different surfaces; Select surface points with opposite and parallel normal vectors on different surfaces, and determine the positional distances between these surface points; These surface points are matched one by one according to the minimum positional distance to obtain multiple surface point groups, and the centroid of the target box is determined. Project the centroid onto each surface of the target box to obtain the centroid projection point on each surface, and determine the distance between each surface point in the surface point group and the corresponding centroid projection point on the surface. Determine a preset distance value and filter out surface points with a distance less than the preset distance value as candidate gripping points that can be used for gripping.

5. The graspability prediction method based on target object context awareness according to claim 1, characterized in that, The process of constructing the crawlability prediction model includes: The target box is simulated to grasp in a preset grasping simulation environment, and the result data of the grasping simulation is determined. A dataset is constructed based on the results data, the data features of the dataset are extracted, and the data features of the dataset are input into a preset neural network model to construct an initial graspability prediction model for the target box. The dataset is divided into training and test sets according to a preset ratio, and the training and test sets are input into the initial graspability prediction model of the target box. The initial graspability prediction model of the target box is trained and tested until it meets the preset convergence condition, thus obtaining the graspability prediction model of the target box.

6. The graspability prediction method based on target object context awareness according to claim 1, characterized in that, The process of determining the optimal grasping pose based on graspability scoring includes: The graspability scores of all candidate grasping poses are sorted in ascending order to obtain a score sequence, and the top three candidate grasping poses in the score sequence are determined as the preferred grasping poses for the target box.

7. A target object context-aware graspability prediction system, characterized in that, include: The acquisition module is used to acquire the local color point cloud of the target box and the global color point cloud of the surrounding environment, and to preprocess the local color point cloud and the global color point cloud. The fusion module is used to determine the prior information of the target box and register the preprocessed local color point cloud to the global color point cloud based on the prior information to obtain the fused global color point cloud; The determination module is used to determine the surface points of the target box based on the fusion of global color point cloud and prior information, and to determine the candidate grasping points among the surface points that can be used for grasping; The module is used to generate the grasp vector of each candidate grasp point, and to construct a multi-dimensional feature vector of the candidate grasp pose on the target box based on each candidate grasp point and the grasp vector. The prediction module is used to input the multidimensional feature vectors of each candidate grasping pose into the graspability prediction model to obtain a graspability score, and to determine the preferred grasping pose based on the graspability score.

Images