Mechanical arm scattering and grabbing planning method based on multi-mode sensing

Abstract

The invention relates to a mechanical arm scattering and grabbing planning method based on multi-mode sensing, and belongs to the technical field of mechanical arm control. The problems that in the waste steel recovery process, sensing is not accurate due to the fact that workpieces are complex in shape and tightly stacked, structure collapse is caused by direct grabbing, and the generalization ability of a traditional algorithm is poor are solved. According to the technical scheme, the method comprises the steps that an image and point cloud data are synchronously obtained through a structured light sensor, and after a unified pose is calibrated through hands and eyes, a workpiece is segmented through super voxel clustering; constructing a grabability index based on the spatial topological relation, the support stability, the pose and the surface area ratio, generating a layered scattering cost map, and planning a minimum disturbance path; and a surface normal vector is adopted to generate candidate grabbing point pairs, and the optimal grabbing pose is evaluated through the grabbing quality convolutional network. And high-precision workpiece segmentation, scientific scattering decision making and self-adaptive grabbing planning are achieved, and the operation safety and efficiency are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robotic arm control and relates to a robotic arm scattering grasping planning method based on multimodal perception. Background Art

[0002] In the field of scrap steel recycling technology, scrap steel materials have different forms but highly similar physical properties, resulting in two major technical bottlenecks:

[0003] Existing robotic arm systems rely on single-modality sensors, such as 2D images or 3D point cloud data. Multimodal sensing technology, a key means of fusing two or more sensory data types, offers significant advantages in complex scenarios. However, current systems fail to effectively integrate image and point cloud information. 2D images lack depth, and 3D point clouds are susceptible to noise. Neither can accurately segment the spatial topology of stacked workpieces. Spatial topology refers to the three-dimensional positional logic and support structure between objects. Its absence makes it impossible to quantify the operability of objects.

[0004] For scenes with densely stacked objects, traditional methods have three drawbacks:

[0005] The necessity of fragmentation evaluation has not been established for cost maps. As a grid map that integrates multi-sensor information, the cost map is a core tool for quantifying scene complexity.

[0006] Point cloud capture algorithms based on predefined object models cannot adapt to the randomness of scrap steel shapes;

[0007] The separation of the fragmentation operation and grasp planning design lacks a mechanism to link graspability indicators. Graspability requires a comprehensive understanding of parameters such as spatial topology, support stability, posture, and surface area, but existing technologies do not provide a quantitative model.

[0008] In the current solution, the perception module and the grasping planning module operate separately, making it difficult for the system to dynamically respond to changing scenarios. In particular, the lack of a multimodal perception-driven, collaborative decision-making framework for breaking up and grasping severely restricts the automation level of scrap steel recycling.

[0009] Explanation of terms:

[0010] 1. Multimodal Perception

[0011] Modality refers to sensory forms and is categorized by human sensory experience into vision, hearing, smell, taste, temperature, touch and pressure, vibration, and movement. Multimodal perception involves using data from two or more of these modalities to acquire characteristic information about the environment.

[0012] 2. Spatial topological relationship

[0013] Spatial topology is a spatial relationship in the Geographic Information System (GIS) that determines the positional relationship between individuals in a space and another individual, and clearly shows the logical relationship between each individual.

[0014] 3. Costmap

[0015] The cost map is improved from the grid map, and multiple sensor information is superimposed and stored in the same grid map, making the map information richer. Summary of the Invention

[0016] In view of this, the object of the present invention is to provide a robotic arm scattered grasping planning method based on multimodal perception. The scattered grasping planning method of the present invention comprises three parts: a multimodal perception module, a robotic arm scattered module and a robotic arm grasping module. The multimodal perception module obtains point cloud and image information in the scene through a structured light sensor and processes this information. Based on the multi-module perception system, the robotic arm scattered module adopts a suitable scattered degree evaluation index to quantify the degree of scattered object and perform scattered path planning. The robotic arm grasping module adopts a two-stage grasping planning method based on deep learning for grasping.

[0017] In order to achieve the above object, the present invention provides the following technical solutions:

[0018] A method for planning a robot arm's scattered grasping based on multimodal perception includes the following steps:

[0019] S1: Synchronously acquire a two-dimensional image and three-dimensional point cloud data of a scene through a structured light sensor; preprocess the three-dimensional point cloud data to remove noise and compress it; establish a unified pose mapping between the three-dimensional point cloud and the two-dimensional image based on the hand-eye transformation relationship; and cluster and segment the point cloud based on the concavity and convexity of the point cloud using a supervoxel clustering algorithm to obtain a segmented object point cloud;

[0020] S2: Based on the segmented object point cloud, calculate the graspability index of each object, the graspability index is based on the spatial topological relationship, the support relationship between objects, the object posture stability and the surface area ratio; construct a layered scattering cost map according to the graspability index; generate a scattering path based on the scattering cost map, and control the robot arm to perform the scattering operation;

[0021] S3: Generate candidate grasping point pairs on the surface of the object after being broken up; evaluate the candidate grasping point pairs through the grasping quality convolutional network and select the optimal grasping posture; control the robotic arm to perform the grasping operation of the optimal grasping posture.

[0022] Furthermore, in S1, the preprocessing includes:

[0023] Remove invalid areas of the point cloud through pass filtering;

[0024] Reduce point cloud density by downsampling;

[0025] Eliminate outlier noise points through statistical filtering;

[0026] Among them, the establishment of the hand-eye transformation relationship satisfies:

[0027]

[0028] Where, P world is the world coordinate point cloud, is the hand-eye transformation matrix, P camera is the camera coordinate system point cloud.

[0029] Furthermore, the calculation formula of the crawlability index is:

[0030] G=α·R t +β·R s +γ·S p +δ·A r

[0031] Where R t Scoring spatial topological relationships, R s Score the support relationship between objects, S p Score the object’s posture stability, A r is the surface area ratio of the object, α is the spatial topological relationship score R t The weight coefficient reflects the influence of the degree of spatial isolation of the object on the graspability; β is the support relationship score R between objects. s The weight coefficient reflects the influence of the degree of mechanical constraint release on graspability; γ is the object posture stability score S p The weight coefficient reflects the influence of posture stability on graspability; δ is the surface area ratio of the object A r The weight coefficient reflects the influence of the exposed surface area ratio on the graspability, and satisfies α+β+γ+δ=1.

[0032] Furthermore, the construction of the layered scattered costmap includes:

[0033] The scene is layered by height, and the average graspability of objects in each layer is calculated;

[0034] Generate a grid map using the inverse of the crawlability as the cost, satisfying:

[0035]

[0036] Where C ij is the cost value of grid (i, j), G ij is the graspability of the corresponding position.

[0037] Furthermore, the scattering path generation adopts any of the following methods:

[0038] (i) Traversal algorithm based on spatial curves: a smooth path is generated with the curvature of the end-of-arm motion trajectory minimized as the optimization goal;

[0039] (ii) Cost map-based traversal algorithm: A* algorithm is used to search for a path with the minimum sum of cost values on the scattered cost map.

[0040] Furthermore, in S3, generating candidate grasping point pairs includes:

[0041] Extract the surface normal vector of the object point cloud;

[0042] With the normal vector direction as the grab axis, a grab point pair set {(p a ,p b )},satisfy:

[0043] ||p a -p b ||=D g

[0044] Where p a and p b It represents the three-dimensional coordinate position of the two contact points of the end effector of the robot arm on the surface of the object, D g The preset gripping distance of the robot end effector.

[0045] Furthermore, the grab quality convolutional network is an improved grab quality convolutional neural network (GQ-CNN), whose input includes:

[0046] Local point cloud depth map of candidate grasping point pairs;

[0047] Grasp the gripper posture parameters;

[0048] The output is the crawl success rate score, which satisfies:

[0049]

[0050] Where, is the optimal grasping pose vector obtained by evaluation, which includes the position coordinates and posture angles of the gripper in three-dimensional space; g is a single candidate grasping pose in the candidate grasping set G; Q is the grasping quality evaluation function, which is implemented by GQ-CNN and outputs a grasping success rate prediction value with a value range of [0,1]; D is the input data tuple, defined as D=(I d ,θ), where Id is the local point cloud depth map, θ is the gripper posture parameter; G is the candidate grasping set, which contains the generated grasping point pairs {(p a ,p b )} corresponding pose set.

[0051] Furthermore, after executing S3, an iterative step is also included:

[0052] Recalculate the graspability index for the remaining objects;

[0053] If the graspability index is lower than the preset threshold, the process returns to S2 to perform an iterative scattering operation.

[0054] The beneficial effects of the present invention are:

[0055] (1) A multimodal perception framework that integrates 2D images and 3D point clouds overcomes the limitations of single-modal data. By unifying supervoxel clustering and hand-eye calibration, high-precision segmentation of stacked workpieces is achieved, significantly improving the ability to model spatial topological relationships and providing reliable environmental representation for subsequent operations.

[0056] (2) A pioneering graspable quantitative indicator system was developed, integrating four parameters: spatial topology, support stability, object pose, and surface area ratio, to construct a hierarchical fragmentation cost map. This indicator objectively assesses the necessity of fragmentation and guides the robot arm to reconstruct the scene along the path with minimal disturbance, completely avoiding the risk of structural collapse caused by traditional blind fragmentation.

[0057] (3) A deep learning-based grasp planning module generates candidate grasp point pairs using surface normal vectors. Combined with an improved grasp quality convolutional network evaluation mechanism, it eliminates the reliance on predefined object models. This design adapts to the randomness of scrap steel morphology and enables stable grasping in complex stacking scenarios.

[0058] (4) The fragmentation module and the grasping module form a closed-loop linkage: the fragmentation operation dynamically optimizes the scene state based on the graspability, and the grasping plan responds to scene changes in real time. The iterative fragmentation grasping strategy significantly reduces invalid operations and improves overall operation efficiency.

[0059] (5) The lightweight design of the entire algorithm meets real-time requirements, and the cost map and layering strategy reduce computational complexity. The system can be deployed in standard scrap steel recycling production lines, promoting the unmanned process in this field.

[0060] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0062] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0063] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0064] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0065] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0066] See also Figure 1 In the multimodal perception module, the entire system first uses a depth or structured light camera to acquire the corresponding measurement point cloud and depth image of the scene. Secondly, preprocessing methods such as pass-through filtering, downsampling, and statistical filtering are used to denoise and compress the raw point cloud. The hand-eye transformation relationship between the camera and the robotic arm is then used to construct a correspondence between the 3D point cloud and the 2D image of the material pile, achieving the unification of position and posture. Finally, a supervoxel point cloud clustering method is used to cluster and segment the point cloud based on its convexity and concavity, obtaining point clouds under different classifications.

[0067] The robot arm's fragmentation module first uses the spatial topology of objects as a foundation, combining surface area ratio, support relationships, workpiece posture stability, and surface area stability to characterize the graspability index and quantify the degree of object fragmentation. Secondly, a hierarchical fragmentation strategy is employed based on the graspability of each object to construct a fragmentation cost map. Based on this fragmentation map, a fragmentation traversal planning algorithm is constructed based on spatial curves and graspability. Finally, the scrap pile is fragmented according to the planned algorithm's path.

[0068] The robotic arm grasping module begins by analyzing the nature of the grasping problem. By analyzing the conditions for successful grasping, it employs a method for generating candidate grasp points based on surface normals in point clouds to generate several candidate grasp point pairs on the surface of the object to be grasped. It then evaluates these candidate grasps using an improved GQ-CNN network model to estimate the optimal grasp. Finally, the optimal grasp is executed and re-evaluated to achieve full grasping of the entire workpiece.

[0069] To address the problem that single images or point cloud information are difficult to accurately represent the characteristic information of scrap steel workpieces, the multimodal perception module combines images and point clouds to calibrate and acquire the position and posture of scrap steel workpieces, effectively improving the accuracy of point cloud clustering and segmentation, thereby improving the utilization rate of scrap steel workpiece features.

[0070] To address the problem that stacked scrap steel has a complex configuration and is difficult to grasp directly using conventional grasping methods, the robotic arm's breakup module uses a graspability index represented by the spatial topological relationship of objects, the support relationship between objects, the object's posture, and the object's surface area ratio to quantify the degree of object breakup. Based on this, a breakup cost map is constructed, which provides a basis for breakup path planning and converts the complex three-dimensional object grasping problem into a simpler plane grasping problem.

[0071] To address the limited generalization capabilities of traditional point cloud-based grasping methods, the robotic arm grasping module employs a two-stage grasp planning approach based on 3D point clouds, powered by deep learning. A candidate grasp point generation method based on point cloud surface normal vectors generates a sufficient number of candidate grasps on the surface of the workpiece to be grasped, increasing the sample size. An improved grasp evaluation network, based on a grasp quality convolutional network architecture, is trained using large datasets to identify common workpiece characteristics and corresponding grasping methods.

[0072] The multimodal perception module combines the unique feature information of two-dimensional images and three-dimensional point clouds, integrating the advantages of both and achieving scrap steel workpiece segmentation with high accuracy.

[0073] The robot arm's fragmentation module proposes a unique graspability metric based on the spatial topology of objects, inter-object support relationships, object pose, and object surface area ratio. This allows for an assessment of the degree of scrap fragmentation and creates a cost map based on this metric. Finally, fragmentation path planning is performed under the guidance of this map.

[0074] The robotic arm grasping module uses an original candidate grasping point generation method based on the point cloud surface normal vector to generate grasping points for subsequent optimal grasping candidates.

[0075] Example 1: Multimodal Perception and Layered Breakup of Heavy Scrap Steel Plates

[0076] Scenario characteristics: 5 tons of hydraulically baled scrap steel plates are stacked at a height of 1.2m, with workpieces tightly fitted together.

[0077] 1.1 3D scene reconstruction

[0078] The line laser structured light sensor carried by the robotic arm scans the scrap steel pile along a Z-shaped trajectory with a scanning interval of 10 cm;

[0079] Synchronous acquisition: 1280×960 resolution RGB image, exposure time 5ms; 500,000-point high-density point cloud, accuracy ±0.5mm;

[0080] Dynamic compensation: The vibration error of the robotic arm is corrected through the IMU data;

[0081] 1.2 Intelligent Point Cloud Processing

[0082] Noise reduction stage:

[0083] The straight-through filter removes the ground point cloud with Z < 0.2m;

[0084] Voxel downsampling compresses the point cloud to 120,000 points with a voxel size of 8mm;

[0085] Statistical filtering removes outliers outside the mean by 3σ, with 50 K nearest neighbors;

[0086] Coordinate unification:

[0087] Loading the Eye-to-Hand calibration matrix

[0088] Perform a coordinate transformation:

[0089] Multimodal Segmentation:

[0090] Supervoxel clustering: voxel size 15 mm, color weight 0.4, spatial weight 0.6;

[0091] Convexity analysis: calculate the local curvature and use a curvature threshold of 0.25 as the segmentation boundary;

[0092] Image assistance: Canny edge detection results are projected onto the point cloud to correct segmentation;

[0093] Output: 8 independent workpiece point cloud clusters, maximum steel plate size 600×400mm;

[0094] 1.3 Graspability-driven breakup

[0095] The indicator calculation is shown in Table 1.

[0096] Where R t Scoring spatial topological relationships, R s Score the support relationship between objects, S p Score the object’s posture stability, A r is the surface area ratio of the object, α is the spatial topological relationship score R t The weight coefficient reflects the influence of the degree of spatial isolation of the object on the graspability; β is the support relationship score R between objects. s The weight coefficient reflects the influence of the degree of mechanical constraint release on graspability; γ is the object posture stability score S p The weight coefficient reflects the influence of posture stability on graspability; δ is the surface area ratio of the object A r The weight coefficient reflects the influence of the exposed surface area ratio on the graspability, and satisfies α+β+γ+δ=1.

[0097] Table 1

[0098]

[0099] Cost map construction:

[0100] Layering strategy: Every 200mm on the Z axis is a layer, for a total of 6 layers;

[0101] Rasterization: 10cm×10cm grid, cost value

[0102] Heat map generation: low cost (blue) at the top, high cost (red) at the bottom;

[0103] Optimal path planning:

[0104] Algorithm selection: A* algorithm, heuristic function Manhattan distance;

[0105] Path characteristics: Total length 2.3m; Avoids high-cost area #5; Curvature continuous <0.15rad / m; Execution parameters: End effector: Flat pusher; Speed curve: S-shaped acceleration and deceleration, maximum 0.6m / s; Contact force control: Constant force mode, 50N±5N;

[0106] 1.4 Verification of the Scattering Effect

[0107] Post-processing scans show: top workpiece displacement: 400 mm; number of newly exposed workpieces: 3; average graspability improvement: 0.41 → 0.69.

[0108] Example 2: Iterative operation of the entire process of automobile crushing materials

[0109] Scene characteristics: Mixed metal parts (engine / transmission / gear) stacked volume 3m 3

[0110] 2.1 Initial Perception and Decision-Making

[0111] Multimodal scanning: TOF camera + structured light fusion point cloud, accuracy ±1mm;

[0112] Segmentation results: 14 workpieces were identified, including 3 key parts;

[0113] Crawlability Assessment:

[0114] Engine block: G = 0.76 > threshold 0.7 → skip breaking up;

[0115] Transmission assembly: G = 0.42 < threshold → mark needs to be broken up;

[0116] 2.2 Two-stage crawl execution

[0117] Candidate point generation:

[0118] Normal vector calculation: PCA algorithm is used, with a neighborhood radius of 30 mm;

[0119] Grab point pairs: generate 56 sets of candidate points; spacing D g =150mm (suitable for heavy-duty clamping jaws); axial tolerance: ±5°;

[0120] GQ-CNN Evaluation:

[0121] Input data: local depth map: 256×256 pixels; gripper parameters: opening and closing angle [15°, 75°];

[0122] Network reasoning: Convolutional layer: 5×5 kernel × 32 channels; Fully connected layer: 1024 neurons;

[0123] Optimal grasp: pose Quality score Q = 0.91;

[0124] Crawl execution:

[0125] Motion planning: RRT* algorithm obstacle avoidance;

[0126] Force control parameters: approach speed: 0.2m / s; clamping force: 300N; lifting acceleration: 0.5g;

[0127] 2.3 Dynamic Scene Iteration

[0128] Condition Monitoring:

[0129] Vibration sensor detects changes in the support structure (>0.5g vibration);

[0130] Real-time recalculation of the gearbox graspability: G = 0.58 → still < threshold;

[0131] Scatter optimization:

[0132] Cost map update: Added high cost area for gear group, C max =3.2;

[0133] Path planning: Method: spatial curve fitting (B-spline curve order 3); Key points: [1.0, 0.7, 0.5] → [1.3, 0.9, 0.6] → [1.5, 1.1, 0.7];

[0134] Breakup execution: Tool replacement: sharp tooth pusher; Action mode: high frequency micro vibration (100Hz, amplitude 5mm)

[0135] Second crawl:

[0136] Release single gear 12;

[0137] Optimal crawling success rate: 87.5% (7 / 8 successful);

[0138] 2.4 Performance Statistics

[0139] The performance statistics are shown in Table 2.

[0140] Table 2

[0141] stage Operation time Energy consumption Success rate Initial crawl 28s 1.2kW 100％ Scattering operation 41s 2.5kW - Second crawl 19s×8 times 0.8kW 87.5％

[0142] Example 3: Optimization of Precision Operations for Electronic Waste

[0143] Scene characteristics: 0.5m 3 Circuit board scrap containing micro components, minimum size 2mm;

[0144] 3.1 Parameter Adaptive Configuration

[0145] Enhanced Perception:

[0146] Point cloud density: 0.1mm resolution, 5 million points;

[0147] Filter optimization: bilateral filtering preserves edge features;

[0148] Crawlability weight:

[0149] Small workpiece strategy: α = 0.1, β = 0.1, γ = 0.2, δ = 0.6;

[0150] Surface area calculation: visible surface: depth map projection method; hidden surface: Poisson reconstruction estimation;

[0151] Grabber configuration:

[0152] Micro gripper: stroke 0~15mm;

[0153] Vacuum suction cup: diameter 0.5mm;

[0154] 3.2 Micro-scale fragmented decision making

[0155] The crawlability analysis is shown in Table 3.

[0156] Table 3

[0157]

[0158]

[0159] Break up path planning:

[0160] Vibration trace: Lissajous graph. X-axis 100Hz, Y-axis 105Hz;

[0161] Energy control: piezoelectric ceramic drive, maximum 20V;

[0162] 3.3 Precision Grasping Implementation

[0163] Candidate point generation:

[0164] Normal vector optimization: neighborhood radius: 0.5mm; anti-noise processing: RANSAC plane fitting

[0165] Point pair density: Generate 20 to 30 candidate pairs per component;

[0166] GQ-CNN fine-tuning:

[0167] Input improvements: Depth map super-resolution: 4x SRCNN enhancement; Gripper parameters: increase surface friction coefficient;

[0168] Lightweight network: MobileNetV3 backbone; the number of parameters is reduced to 35% of the original version;

[0169] Execution monitoring:

[0170] Visual servoing: 1000fps high-speed camera closed-loop control;

[0171] Contact detection: nanoscale strain gauge, sensitivity 0.01N;

[0172] 3.4 Quality Verification

[0173] X-ray test results:

[0174] Component breakage rate: <0.1%;

[0175] Grasping position deviation: ±0.05mm;

[0176] The classification and recovery purity are shown in Table 4.

[0177] Table 4

[0178] Material Type Sorting accuracy Gold contacts 99.2％ tin solder 97.8％ Ceramic substrate 98.5％

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A robotic arm fragmentation grasping planning method based on multimodal perception, characterized by: The following steps are involved: S1: Synchronously acquires 2D images and 3D point cloud data of the scene through a structured light sensor; Preprocessing the three-dimensional point cloud data to remove noise and compress it; establishing a unified posture mapping between the three-dimensional point cloud and the two-dimensional image based on the hand-eye transformation relationship; clustering and segmenting the point cloud according to the concavity and convexity of the point cloud using a supervoxel clustering algorithm to obtain a segmented object point cloud; S2: Based on the segmented object point cloud, calculate the graspability index of each object, the graspability index is based on the spatial topological relationship, the support relationship between objects, the object posture stability and the surface area ratio; construct a layered scattering cost map according to the graspability index; generate a scattering path based on the scattering cost map, and control the robot arm to perform the scattering operation; S3: Generate candidate grasping point pairs on the surface of the object after being broken up; evaluate the candidate grasping point pairs through the grasping quality convolutional network and select the optimal grasping posture; control the robotic arm to perform the grasping operation of the optimal grasping posture.

2. The method for planning a robot arm's scattered grasping based on multimodal perception according to claim 1 is characterized in that: In S1, the pre-processing includes: Remove invalid areas of the point cloud through pass filtering; Reduce point cloud density by downsampling; Eliminate outlier noise points through statistical filtering; Among them, the establishment of the hand-eye transformation relationship satisfies: Where, P world is the world coordinate point cloud, is the hand-eye transformation matrix, P camera is the camera coordinate system point cloud.

3. The method for planning a robot arm's scattered grasping based on multimodal perception according to claim 1, characterized in that: The calculation formula of the crawlability index is: G=α·R t +β·R s +γ·S p +δ·A r Where R t Scoring spatial topological relationships, R s Score the support relationship between objects, S p Score the object’s posture stability, A r is the surface area ratio of the object, α is the spatial topological relationship score R t The weight coefficient reflects the influence of the spatial isolation of the object on its graspability; β is the support relationship score R between objects s The weight coefficient reflects the influence of the degree of mechanical constraint release on graspability; γ is the object posture stability score S p The weight coefficient reflects the influence of posture stability on graspability; δ is the surface area ratio A of the object r The weight coefficient reflects the influence of the exposed surface area ratio on the graspability, and satisfies α+β+γ+δ=1.

4. The method for planning a robot arm's scattered grasping based on multimodal perception according to claim 1, characterized in that: The construction of the layered scattered costmap includes: The scene is layered by height, and the average graspability of objects in each layer is calculated; Generate a grid map using the inverse of the crawlability as the cost, satisfying: Where C ij is the cost value of grid (i, j), G ij is the graspability of the corresponding position.

5. The method for planning a robot arm's scattered grasping based on multimodal perception according to claim 1, characterized in that: The scattering path is generated by any of the following methods: (i) Traversal algorithm based on spatial curves: a smooth path is generated with the curvature of the end-of-arm motion trajectory minimized as the optimization goal; (ii) Cost map-based traversal algorithm: A* algorithm is used to search for a path with the minimum sum of cost values on the scattered cost map.

6. The method for planning a robot arm's scattered grasping based on multimodal perception according to claim 1, characterized in that: In S3, generating candidate grasping point pairs includes: Extract the surface normal vector of the object point cloud; With the normal vector direction as the grab axis, a grab point pair set {(p a ,p b )},satisfy: ||p a -p b ||=D g Where p a and p b It represents the three-dimensional coordinate position of the two contact points of the end effector of the robot arm on the surface of the object, D g The preset gripping distance of the robot end effector.

7. The method for planning a robot arm's scattered grasping based on multimodal perception according to claim 1, characterized in that: The Grasp Quality Convolutional Neural Network (GQ-CNN) is an improved Grasp Quality Convolutional Neural Network (GQ-CNN), and its input includes: Local point cloud depth map of candidate grasping point pairs; Grasp the gripper posture parameters; The output is the crawl success rate score, which satisfies: Where, is the optimal grasping pose vector obtained by evaluation, which includes the position coordinates and posture angles of the gripper in three-dimensional space; g is a single candidate grasping pose in the candidate grasping set G; Q is the grasping quality evaluation function, which is implemented by GQ-CNN and outputs a grasping success rate prediction value with a value range of [0,1]; D is the input data tuple, defined as D=(I d ,θ), where I d is the local point cloud depth map, θ is the gripper posture parameter; G is the candidate grasping set, which contains the generated grasping point pairs {(p a ,p b )} corresponding pose set.

8. The method for planning a robot arm's scattered grasping based on multimodal perception according to claim 1, characterized in that: After executing S3, the following iterative steps are also included: Recalculate the graspability index for the remaining objects; If the graspability index is lower than the preset threshold, the process returns to S2 to perform an iterative scattering operation.

Images