Mixing disorder grabbing control system and method based on three-dimensional point cloud

Abstract

The application relates to the field of computer vision and automation, and particularly discloses a mixed-disorder grabbing control system and method based on a three-dimensional point cloud, which comprises a three-dimensional camera and a data processing unit electrically connected with the three-dimensional camera; the data processing unit comprises: a point cloud preprocessing module, which acquires scene point clouds and carries out denoising and downsampling to obtain workpiece point clouds and original scene point clouds; a cascaded segmentation module, which carries out hierarchical cascaded segmentation on the workpiece point clouds to obtain independent workpiece point cloud clusters; and a candidate rating module, which calculates a comprehensive score according to the height score and point number ratio of each independent workpiece point cloud cluster, and sorts the independent workpiece point cloud clusters in descending order of the comprehensive score to obtain a priority queue; the technical scheme can adapt to complex industrial site environments, is suitable for non-standard workpieces (especially steel plate workpieces), and can distinguish closely stacked workpieces.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and automation, and in particular to a mixed material disorder grasping control system and method based on three-dimensional point clouds. Background Technology

[0002] In the field of industrial automation, robotic unordered grasping technology is a key link in realizing automated material loading, unloading, and sorting, especially in the context of smart manufacturing and Industry 4.0, where its importance is increasingly prominent. Traditional automated material loading methods, such as vibratory feeders, suffer from problems such as high degree of equipment customization, cumbersome adjustments, inability to adapt to various flexible production processes, and easy scratching of workpieces. Therefore, robotic systems with visual inspection and judgment capabilities have become an important technological path to solve the problem of automated material loading and unloading.

[0003] Currently, disordered grasping technology mainly relies on machine vision systems, and its development has evolved from two-dimensional vision to three-dimensional vision. Early two-dimensional vision methods mainly used template matching, edge detection, and other methods for object recognition, but due to the lack of depth information, they were unable to effectively deal with problems such as object stacking, occlusion, and changes in lighting, and their application in complex industrial environments was limited.

[0004] To overcome the limitations of two-dimensional vision, three-dimensional vision technology has gradually become mainstream. Among them, recognition and localization methods based on three-dimensional point clouds are widely used because they can acquire complete depth information. For example, Chinese Patent Publication No. CN110342153A discloses a method for garbage bin recognition and grabbing based on three-dimensional point clouds. This method adds a three-dimensional point cloud LiDAR module to a traditional hook-lift garbage truck, replaces the hydraulic device with a robotic arm execution unit, uses LiDAR scanning to acquire point cloud data, and performs garbage bin recognition and localization through point feature histogram (PFH) feature extraction and iterative nearest point algorithm (ICP), ultimately achieving automatic garbage bin grabbing.

[0005] However, the above-mentioned solution has significant limitations in scenarios involving mixed and disordered material handling. Firstly, this solution primarily targets standard-shaped, single-type trash cans, employing a standard model matching-based identification method. This involves ICP registration between the point cloud to be identified and the source point cloud set of a standard trash can. When faced with non-standard, irregularly shaped industrial workpieces with uniform or smooth surface textures (such as steel plates), this method suffers a significant drop in accuracy and robustness due to a lack of sufficient feature differentiation. Especially in complex industrial scenarios such as tightly stacked workpieces, reflective surfaces, or the presence of holes, the standard model matching method often struggles to accurately distinguish the boundaries between adjacent workpieces, easily leading to undersegmentation (misclassifying multiple workpieces as a single entity) or oversegmentation (fragmenting the same workpiece into multiple small clusters).

[0006] Secondly, the gripping point determination method used in the above scheme is relatively simple, mainly relying on the centroid position of the target object, without considering the actual geometric features of the workpiece surface and the physical constraints of the end effector. For workpieces with irregular shapes, holes, or complex edges, the centroid position may be located in the suspended area or the center of the hole. Directly gripping by the centroid will lead to problems such as insufficient adsorption area, unstable magnetic attraction, or workpiece falling, seriously affecting the gripping success rate and system safety.

[0007] In addition to the target workpiece itself, industrial environments present potential obstacles such as material frames, other workpieces, and the surrounding environment. The system needs to avoid collisions with these obstacles during the grasping process. However, the aforementioned solution only achieves simple contact detection through a contact sensor at the end of the robotic arm, lacking a systematic detection mechanism for workpiece occlusion and environmental collisions. This makes it difficult to guarantee operational safety in complex stacking scenarios. Furthermore, it does not consider handling common abnormal states in industrial environments, such as empty material frames, numerous noise points, or point cloud acquisition failures. In actual production, if the system cannot effectively identify these abnormal states, the robot may perform ineffective grasping operations on empty material frames, wasting time and resources and potentially damaging equipment.

[0008] Furthermore, while existing deep learning-based end-to-end crawling and detection methods perform well on public datasets, they face challenges in industrial applications, such as strong data dependence, limited generalization ability, black-box characteristics that make it difficult to trace problems, and high computational resource consumption. These challenges make it difficult to meet the stringent requirements of industrial sites for system robustness, interpretability, and real-time performance.

[0009] Therefore, there is an urgent need for a system that can adapt to complex industrial environments, distinguish between closely stacked workpieces (especially steel plate workpieces), determine a safe and reliable gripping posture based on the geometric features of the workpiece surface and the physical constraints of the end effector, establish a multi-level safety detection mechanism to ensure the safety of the gripping process, and also have the ability to identify and handle abnormal states to improve the industrial applicability of the system. Summary of the Invention

[0010] This invention provides a mixed material disorder gripping control system based on three-dimensional point cloud, which can adapt to complex industrial environments, target non-standard workpieces (especially steel plate workpieces), and distinguish closely stacked workpieces.

[0011] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0012] A mixing and disordered grasping control system based on 3D point cloud includes a 3D camera and a data processing unit electrically connected to the 3D camera.

[0013] The data processing unit includes:

[0014] The point cloud preprocessing module acquires scene point clouds and performs noise reduction and downsampling to obtain workpiece point clouds and original scene point clouds.

[0015] The cascaded segmentation module performs hierarchical cascaded segmentation of the workpiece point cloud to obtain independent workpiece point cloud clusters;

[0016] The candidate rating module calculates a comprehensive score based on the height score and the proportion of points in each independent workpiece point cloud cluster, and sorts the independent workpiece point cloud clusters in descending order of comprehensive score to obtain a priority queue.

[0017] The occlusion and collision detection module performs inter-board occlusion detection and environmental collision detection on the point cloud clusters of independent workpieces in the priority queue based on the original scene point cloud, and filters out the workpieces that can be safely grasped.

[0018] The pose calculation module performs local coordinate system projection based on the local geometric features of the workpiece being safely gripped, and performs morphological mesh search in the local coordinate system to determine the final adsorption center point, thereby calculating the six-degree-of-freedom gripping pose of the workpiece.

[0019] The basic principles and beneficial effects of the scheme are as follows:

[0020] The present invention provides a mixed material disorder gripping control system based on three-dimensional point cloud. Through the coordinated cooperation and data flow coupling between the three-dimensional camera and the modules of the specific data processing unit, it realizes workpiece gripping pose recognition and path decision with high robustness and high safety.

[0021] The point cloud preprocessing module denoises and downsamples the raw point cloud acquired by the 3D camera, and then separates and outputs two data streams: the workpiece point cloud and the original scene point cloud. This design establishes a dual-track parallel analysis logic for the system. The refined and denoised workpiece point cloud is directly input into the subsequent segmentation and calculation modules to ensure the efficiency and real-time performance of the core positioning algorithm; at the same time, the original scene point cloud, containing complete environmental interference and material frame boundaries, is retained and input into the collision detection module, establishing an absolute physical defense for system safety.

[0022] The cascaded segmentation module takes the denoised workpiece point cloud as input. This invention breaks through the barriers of traditional single segmentation algorithms that are sensitive to material, reflection, and adhesion. Through a hierarchical cascaded segmentation mechanism, it sequentially superimposes and absolves physical geometric and surface property constraints of different dimensions at multiple stages. Based on the coarse segmentation in the previous stage, the subsequent stage performs microscopic peeling of the adhesion interface based on local feature changes and surface consistency, thereby stably dividing the massive chaotic point cloud into independent workpiece point cloud clusters that correspond one-to-one with the physical entities.

[0023] The candidate rating module does not blindly perform 3D collision and pose calculations on all segmented point cloud clusters. Instead, it comprehensively scores and ranks the individual workpiece point cloud clusters in descending order based on their height score (spatial position) and point count ratio (visibility and integrity), constructing a priority queue. This design establishes a serial truncation mechanism for subsequent modules, allowing the occlusion and collision detection modules to perform safety checks on the top candidates in the queue in ascending priority order. Once the check passes, the result is output, thus avoiding computationally intensive collision interference calculations on all cluttered targets in the entire image, allowing for highly focused computing power.

[0024] The occlusion and collision detection module incorporates dual verification through inter-plate occlusion detection and environmental collision detection. Inter-plate occlusion detection determines whether there is physical stacking of other mixed materials above the target workpiece to prevent dragging and pulling during the grasping process. Environmental collision detection, based on the initial scene point cloud, detects whether the end effector and its connectors interfere with the material frame wall during approach and lifting. These two cross-detections completely eliminate the safety hazards of impact damage and workpiece falls from a spatial perspective.

[0025] The pose calculation module does not rely on the centroid method, which is easily affected by workpiece reflection defects, holes, and non-standard irregularities, when determining the final adsorption point. Instead, it projects the workpiece's 3D information matrix onto a local coordinate system consistent with its own orientation, based on the workpiece's local geometric features. This reduces the high-dimensional six-degree-of-freedom space search to a local 2D grid map. Based on this, the actual physical force dimensions of the adsorption actuator are transformed into morphological structural operators for soft mesh erosion. This process introduces a suspension loss constraint in physical space, eliminating mesh cells that do not meet the contact ratio requirements of the suction cup or magnetic surface. Finally, it backtracks to calculate the optimal 6D gripping pose that balances the physical shape of the suction cup with irregular holes and edges on the workpiece.

[0026] Traditional point cloud segmentation suffers from severe undersegmentation (mistaking multiple workpieces for a single unit) in scenarios where workpieces are tightly packed and have extremely narrow gaps. This is because spatial distance is obscured by noise. This invention addresses this issue by employing hierarchical cascaded segmentation, utilizing the multi-stage evolution of local geometric features to perform multi-dimensional judgment and filtering of adhesion boundaries. This allows for the precise segmentation of contacting plates that cannot be distinguished by spatial distance alone into independent units. This improves the system's instance segmentation success rate in scenarios with tightly packed and chaotic workpieces, enhances overall grasping accuracy, and prevents grasping failures or slippage caused by the robotic arm grasping multiple adhered plates.

[0027] This invention does not rely on any offline CAD model matching algorithms (such as template matching or deep learning prior template training). Through local coordinate system projection and morphological mesh search in the pose calculation module, it can automatically calculate and reduce the dimensionality of any irregular, flat, non-standard workpiece. By scanning the projected raster image using physical size convolution kernels, it can naturally detect various irregular holes, internal cavities, edge slits, and irregular cut corners on the surface of the steel plate workpiece. Based on this, the system can automatically bypass suspended meshes without physical support, thereby determining the geometrically safe position that allows the end suction cup to provide maximum magnetic flux or maximum vacuum sealing, completely eliminating the problems of missed suction, empty gripping, and flipping / falling caused by local torque imbalance when gripping non-standard steel plates with weight-reducing holes, crescent shapes, or strips at the center of gravity.

[0028] In industrial settings where bins are stacked haphazardly, obstacles intersect densely. The occlusion and collision detection module of this invention independently detects occlusion and interference, simultaneously performing environmental envelope collision avoidance detection based on the original scene point cloud and the overlapping relationships between workpieces and the actuator's movement path. This enables the robotic arm to intelligently avoid potential mechanical impacts when reaching into deep bins, narrow slits, or gripping against bin edges, effectively protecting both the optical camera and the robotic arm, and reducing the incidence of abnormal stops or mechanical collisions during system operation.

[0029] The core bottleneck of unordered data capture lies in the system latency caused by the calculation of normals and spatial matrix operations of a large number of 3D point clouds. This invention performs voxel redundancy removal in the preprocessing stage, reducing the amount of data participating in cascaded segmentation;

[0030] During the solution phase, the 3D point cloud is compressed into a local 2D raster image by local coordinate system projection, which transforms the originally extremely time-consuming 3D spatial envelope interference calculation and optimization point finding into an efficient 2D image matrix slider erosion search.

[0031] Because this invention is built upon model-free general morphological search, when factories replace non-standard parts of different shapes and sizes, or when the end effector of a robot is replaced with an electromagnetic chuck of different lengths and widths, there is no need to re-register 3D model data and create sampling templates, nor is there any need to re-annotate and retrain the black-box deep learning neural network. Relying on the physical mapping of the structural kernel size in the morphological mesh search, the path can be adaptively adjusted and optimized, greatly reducing the threshold for system commissioning and deployment in multi-batch, small-volume flexible manufacturing scenarios.

[0032] In summary, this invention achieves the effects of adapting to complex industrial environments, targeting non-standard workpieces (especially steel plate workpieces), and distinguishing closely stacked workpieces.

[0033] Furthermore, the cascaded segmentation module includes:

[0034] The first-level segmentation unit clusters the workpiece point cloud within a limited spatial range and separates coarsely segmented point cloud clusters.

[0035] The second-level segmentation unit segments the coarse segmentation point cloud clusters that are in contact with each other. It calculates the angle between the normals of the surfaces of adjacent small blocks in the coarse segmentation point cloud clusters to determine the geometric concave-convex connection characteristics and cuts off the adhesion boundary according to the concave connection relationship.

[0036] The third-level segmentation unit performs surface merging on the point cloud after cutting off the adhesion boundary based on normal consistency, smoothing surface crack lines; and...

[0037] The fourth-level segmentation unit performs color segmentation and re-merging on surface areas with obvious color differences based on the color difference distance in the color space to obtain independent workpiece point cloud clusters.

[0038] Furthermore, the cascaded segmentation module also includes a material shortage detection unit. When the first-level segmentation unit fails to separate coarse segmentation point cloud clusters with a number of points exceeding the preset volume point number threshold and the number of remaining noise points in the workpiece point cloud exceeds the noise limit, the material shortage detection unit determines that the system state is a material shortage noise saturation state, outputs a material change alarm signal, and blocks the robotic arm.

[0039] Furthermore, the candidate rating module extracts the maximum height of all points in each independent workpiece point cloud cluster, performs a normalization transformation on the maximum height within the entire scene height range to generate a vertical allocation index, counts the total number of points contained in each independent workpiece point cloud cluster to generate a scale allocation index, and then multiplies the vertical allocation index and the scale allocation index by the corresponding priority bias coefficient and sums them by weight to obtain the final comprehensive score.

[0040] Furthermore, the occlusion and collision detection module includes an adaptive occlusion detection unit. The adaptive occlusion detection unit calculates the surface normal axis pointing to space based on the outer contour of the safe gripping workpiece, and generates a prism-shaped detection area on the surface of the safe gripping workpiece in the positive direction of the surface normal axis. The adaptive occlusion detection unit searches in the original scene point cloud for occlusion interference points belonging to the point cloud of non-safe gripping workpieces. When an occlusion interference point is detected falling into the prism-shaped detection area, the safe gripping workpiece is marked as an occlusion danger workpiece.

[0041] Furthermore, the occlusion and collision detection module also includes an environmental collision detection unit. The environmental collision detection unit simulates the maximum envelope radius and vertical lifting height of the adsorption end effector to construct a cylindrical obstacle avoidance zone. The cylindrical obstacle avoidance zone is placed on the calculated proposed adsorption center point. The environmental collision detection unit searches within the original scene point cloud for environmental obstacle points that fall into the cylindrical obstacle avoidance zone. If an environmental obstacle point is found, the proposed adsorption center point is discarded.

[0042] Furthermore, the pose calculation module includes a principal component analysis unit, which is used to obtain the geometric center of the independent workpiece point cloud cluster, calculate the covariance matrix of the independent workpiece point cloud cluster in each direction, decompose the covariance matrix to obtain three mutually orthogonal principal eigencomponent directions, and set them as the principal direction axis, secondary direction axis and plane vertical normal axis respectively according to the density of spatial dispersion from large to small, and verify the plane vertical normal axis to make the plane vertical normal axis point upward, and construct a right-handed orthogonal local coordinate system.

[0043] Furthermore, the pose calculation module also includes a projection rasterization unit. The projection rasterization unit extracts the transformation parameters from the independent workpiece point cloud cluster to the right-hand orthogonal local coordinate system, projects and rotates all three-dimensional coordinates in the independent workpiece point cloud cluster to the local two-dimensional plane, ignores the thickness term to restore the three-dimensional workpiece to the two-dimensional plane, and calculates the grid position number associated with each test projection point according to the discrete raster unit size, and generates a two-dimensional grid image with position occupancy marks mapped in the local two-dimensional plane.

[0044] The pose calculation module also includes a morphological erosion unit. The morphological erosion unit obtains a simulated magnetic pattern according to the actual working projection range of the robotic arm's adsorption disk. The simulated magnetic pattern is then slid across the two-dimensional mesh image at specified intervals to perform soft erosion judgment. The soft erosion judgment includes accumulating the total area of ​​the hole mesh and edge cavity mesh within the simulated magnetic pattern. If the proportion of the total area of ​​the hole mesh and edge cavity mesh to the total area of ​​the simulated magnetic pattern is less than the upper limit of the suspension loss, the center of symmetry of the simulated magnetic pattern is extracted and determined as a feasible adsorption center point.

[0045] Furthermore, the pose calculation module also includes a pose output unit. The pose output unit traverses all feasible adsorption center points found in the two-dimensional mesh image, calculates the two-dimensional Euclidean distance between the feasible adsorption center point and the geometric center of the independent workpiece point cloud cluster, takes the feasible adsorption center point with the shortest two-dimensional Euclidean distance as the final adsorption center point, extracts the original spatial position corresponding to the final adsorption center point, and refines the grasping height in the direction of the final adsorption center point based on the height value of the point closest to the final adsorption center point. The direction of the right-handed orthogonal local coordinate system is fused to generate a six-degree-of-freedom grasping pose signal and outputs it to the industrial robotic arm controller. Attached Figure Description

[0046] Figure 1 This is a logic block diagram of an embodiment of a material mixing and disordered grasping control system based on 3D point clouds;

[0047] Figure 2 Original point cloud scene image used for the experiment;

[0048] Figure 3 A point cloud representation using only Euclidean clustering segmentation;

[0049] Figure 4 Provides real-time monitoring images for automated operation in industrial settings;

[0050] Figure 5 Capture footage from industrial sites;

[0051] Figure 6 Point cloud diagram of the material frame and sheet metal;

[0052] Figure 7 This is a rendering of a point cloud segmented using LCCP.

[0053] Figure 8 The image shows the search results for morphological mesh corrosion in the context of irregularly shaped workpieces with holes.

[0054] Figure 9 This is a sensitivity analysis diagram of the LCCP concavity / convexity threshold parameter on segmentation performance. Detailed Implementation

[0055] The following detailed description illustrates the specific implementation method:

[0056] A mixing and disordered grasping control system based on 3D point clouds (e.g.) Figure 1 (As shown), it includes a 3D camera and a data processing unit electrically connected to the 3D camera;

[0057] The data processing unit includes:

[0058] The point cloud preprocessing module acquires scene point clouds and performs noise reduction and downsampling to obtain workpiece point clouds and original scene point clouds.

[0059] The cascaded segmentation module performs hierarchical cascaded segmentation of the workpiece point cloud to obtain independent workpiece point cloud clusters;

[0060] The candidate rating module calculates a comprehensive score based on the height score and the proportion of points in each independent workpiece point cloud cluster, and sorts the independent workpiece point cloud clusters in descending order of comprehensive score to obtain a priority queue.

[0061] The occlusion and collision detection module performs inter-board occlusion detection and environmental collision detection on the point cloud clusters of independent workpieces in the priority queue based on the original scene point cloud, and filters out the workpieces that can be safely grasped.

[0062] The pose calculation module performs local coordinate system projection based on the local geometric features of the workpiece being safely gripped, and performs morphological mesh search in the local coordinate system to determine the final adsorption center point, thereby calculating the six-degree-of-freedom gripping pose of the workpiece.

[0063] In practical use, it is deployed in automated industrial robot workstations to identify, locate, and safely pick up non-standard steel plate workpieces that are stacked disorderly in the material frame.

[0064] The system mainly includes a 3D camera (such as a high-precision 3D structured light camera, which is usually installed in the eye-to-hand manner above the material frame where the mixed workpiece is placed), an industrial robotic arm, an adsorption end effector installed at the end of the robotic arm (such as an electromagnetic chuck or vacuum chuck assembly; for the sake of convenience in the manual, magnetic chucks or magnets will be used as examples below), and a data processing unit (such as an industrial control computer loaded with visual analysis algorithms) that is electrically connected to both the 3D camera and the robotic arm controller.

[0065] When the hardware device starts up, the 3D camera first performs active light projection and depth image capture on the material frame area to obtain the original depth information including the material frame body and the disordered stacked workpieces inside. The data processing unit performs preliminary transformation on the depth information to establish an initial noisy point cloud in 3D space, which is then used as the input to the core algorithm to initiate the subsequent perception, decision-making, and control output process.

[0066] During operation, the various functional modules contained in the data processing unit will work according to the following logical flow and calculation logic.

[0067] The point cloud preprocessing module receives coordinate data output from the 3D camera. Its primary task is to reduce noise and compress the massive amount of raw data to decrease the computational burden of subsequent complex geometric calculations. Specifically, this module includes the following sub-steps executed sequentially:

[0068] Region of Interest (ROI) clipping: Based on the preset physical three-dimensional size boundary of the material frame, a three-dimensional pass-through filter is used to initially segment the spatial coordinates of the point cloud, directly discarding useless background and workshop environment point clouds outside the material frame.

[0069] Outlier Removal: A statistical outlier removal filter is used for denoising. The principle is as follows: for each point in the point cloud, the average distance to a specified number of neighboring points is calculated. If the number of neighboring points is set to 30, the calculated average distance follows a Gaussian distribution. If the average distance of a point exceeds the global average distance plus a certain multiple of the standard deviation, the point is determined to be an isolated noise point caused by reflections, dust, etc., and is removed.

[0070] Voxel downsampling: To avoid system timeouts caused by directly entering massive point clouds into cascaded segmentation, a voxel center point replacement method is used for downsampling. Assuming the voxel mesh size is set to 5.0 mm × 5.0 mm × 5.0 mm, the algorithm calculates the average coordinates of all contained point clouds within this tiny cubic space, and uses this average point as the unique representative of the voxel. This effectively compresses the point cloud density to about one-tenth of its original size while preserving the workpiece topology very well.

[0071] Coordinate system transformation: By multiplying the extrinsic parameter matrix, the point cloud information under the reference coordinate system of the 3D camera is uniformly transformed to the base coordinate system (i.e., the world coordinate system) of the industrial robot arm, so as to obtain the workpiece point cloud (for segmentation and extraction) and the original scene point cloud (preserving the material frame to prevent collision) which are convenient for the robot arm to perform.

[0072] The cascaded segmentation module works as follows: To completely solve the problem of local under-segmentation caused by the adhesion of stacked steel plates at the edges, this embodiment designs a cascaded segmentation module that proceeds from coarse to fine, comprising four sequentially cascaded segmentation stages:

[0073] (1) First stage: Spatial preliminary screening based on Euclidean clustering

[0074] First, the first-level segmentation unit clusters the preprocessed workpiece point cloud within a defined spatial range. This method is an accelerated Euclidean clustering approach based on KD-Tree (K-Dimensional Tree), specifically: for the point cloud set... The algorithm generates a KD-Tree spatial index to reduce time overhead. The clustering process is as follows: for each core point, a cluster is formed with that point as the center and a radius of... Find the set of points surrounding it within the range:

[0075] : (1)

[0076] If the data is within the range, it is grouped into a cluster, and breadth-first search (BFS) is used to continue searching for points that meet the criteria until no more data is found. During this process, a no-data detection mechanism is added: if the clustering is empty and there are still a large number of points in the scene (such as 300,000 points), it is considered to be noise or no data, and the operation ends.

[0077] In equation (1), P represents the input point cloud set; and A point in the set; Indicated by The set of points in the neighborhood of the center; This is the threshold for the search radius in Euclidean clustering. This represents the Euclidean distance between two points. If a clustering tolerance threshold is set... With a resolution of 7.0 mm, Euclidean clustering can quickly generate several coarse segmentation point clusters for workpieces with clearly isolated physical spaces within the frame.

[0078] (2) Second stage: Concavity-convexity segmentation of stacked objects based on LCCP

[0079] For objects that are tightly packed together in a scattered stacking state (with gaps smaller than the clustering tolerance) For steel plates, ordinary Euclidean clustering cannot separate them. At this point, the second-level segmentation unit initiates Local Convex Connectivity Patch (LCCP) determination:

[0080] Generating supervoxels: To reduce computational cost without sacrificing local geometric properties, the point cloud is over-segmented into a set of supervoxels using the VCCS (Voxel Cloud Connectivity Segmentation) method. Each supervoxel has attributes such as position, color, and normal. The distance definition in VCCS is:

[0081] (2)

[0082] The hypervoxel size is adjusted using lccpVoxelSize (20mm) and lccpSeedSize (50mm) in the code.

[0083] In equation (2), D is the supervoxel composite distance; , and These represent differences in color, space, and normal, respectively. , , These are the corresponding weighting coefficients.

[0084] Concavity / convexity check: for two adjacent supervoxels and Their centers of mass are respectively The normal is Define the center-connect vector. Whether to perform a connection operation is determined by the relationship between the connection vector and the normal.

[0085] The included angle is defined as the angle between the normal and the connection vector. If the following conditions are met, it is considered a concave connection.

[0086] (3)

[0087] Furthermore, the concavity angle is greater than the set threshold (i.e., lccpConcavityThresh=10°).

[0088] (4)

[0089] Once a concave connection is detected, the algorithm deletes the adjacent edges, separating the overlapping steel plates.

[0090] In equations (3) and (4), , It is the normal vector; For the reason point to The center connection vector; This represents the angle between the normal vector and the connection vector. For the concave angle, The threshold for determining concave connections is set as follows: Since the overlapping edges of stacked steel plates often exhibit abrupt changes in normal, the algorithm cuts off the common edge at the concave connection, thus separating the multiple stacked steel plates. In practice, the concave / convex analysis neighborhood size can be set to 50 mm, and the concave determination angle threshold can be set to... .

[0091] (3) Third stage: Smoothness subdivision based on normal region growth

[0092] To address the potential issues of oversegmentation or poor fitting performance on smooth surfaces caused by LCCP, a supplementary process is added:

[0093] Normal region growth: For regions with small curvature changes, a merging operation is performed based on the consistency of the normal angle, at the seed point. and neighboring points If the conditions are met If so, a merging operation is performed, which helps to compensate for the surface disconnection problem caused by LCCP.

[0094] in, and These represent the seed point and its neighboring points in the region growth process; and The corresponding normal vector; This is the normal smoothing threshold.

[0095] (4) Fourth stage: Region growth and subdivision based on color attributes

[0096] If the steel plate surface is coated with paint to distinguish markings or the surface reflection has different oxidation color levels, the fourth-level segmentation unit is segmented according to the Euclidean distance in the RGB space.

[0097] (5)

[0098] This step ensures that workpieces with obvious visual characteristics can be correctly distinguished.

[0099] In equation (5), and Point , Color vector in the RGB color space; This is the color difference threshold.

[0100] In particular, this cascade module also incorporates a No Material Check mechanism. If no coarse segmentation point cloud cluster with more than 500 points is extracted after the first-level Euclidean clustering, but the number of remaining noise points in the entire ROI region exceeds 300,000, the system will skip all subsequent useless calculations, directly set the HasMaterial status flag to False, output a no-material-check alarm, and suspend robot actions.

[0101] The working logic of the candidate rating module is as follows: After obtaining several independent workpiece point cloud clusters through cascaded segmentation, the optimal grasping target needs to be selected from a large number of point cloud clusters. Here, a one-dimensional evaluation criterion based on height and visibility is proposed:

[0102] (4.6)

[0103] Top-level object, visibility score (points) It tends to favor complete objects. Additionally, there's adaptive prism occlusion detection, which creates a virtual prism along the OBB normal direction of each candidate object and uses an Octree to check if there are obstacle points within that prism, thus eliminating targets with a higher risk of physical occlusion.

[0104] In equation (6), The overall score for candidate targets to be captured; This represents the maximum value of the cloud cluster at that point along the Z direction (i.e., the highest point of the object). and Represents the lowest point of the scene and the Z-axis height range, which are used together to normalize the height to the entire scene; Represents high weight, Represents visibility weight; The number of point clouds representing the largest cluster in the scene. This represents the number of points contained in a point cloud cluster.

[0105] The height and visibility weighted scoring model, after cascaded segmentation, yields a series of isolated workpiece point cloud clusters. Based on this, a weighted evaluation method based on physical height and visual integrity is proposed.

[0106] (1) Altitude ( In stacked scenes, objects at the top are generally less occluded and easier to pick up. Therefore, for each point cloud cluster, calculate its maximum value along the Z-direction and then normalize it to the Z-axis of the entire scene. Objects at higher positions have larger scores.

[0107] (2) Visibility score ( The number of point clouds represents the size of the visible portion of an object. The system calculates the number of point clouds in each cluster and normalizes it by dividing by the maximum number of point clouds in the scene. The larger the number of point clouds, the more complete the object is, and the more planes are available for grasping.

[0108] Final score Calculated using the weighted formula:

[0109] (7)

[0110] The weighting coefficients can be set in the parameters (the default values ​​are 0.7 and 0.3 respectively), which means "prioritizing high priority while considering completeness".

[0111] In equations (6) and (7), and Each represents the overall score of the candidate target to be captured; , Weights are assigned based on height and visibility. and These are the normalized height and visibility scores.

[0112] The working logic of the occlusion and collision detection module is as follows: In order to prevent the physical danger of lifting the upper occlusion plate during the grabbing process, the high-scoring workpieces in the queue are checked for occlusion.

[0113] First, construct the minimum oriented bounding box (OBB) of the workpiece and extract the principal normal axes perpendicular to the surface of the part.

[0114] Establish a cross section along the positive direction of the normal axis that is equivalent to the surface area of ​​the workpiece and extends outwards in elevation. The virtual three-dimensional polyprism occlusion area.

[0115] Using a spatial octree search, we search the original scene point cloud for points that are not part of the workpiece body but fall within the detection area of ​​the prism.

[0116] If other workpieces are found to obscure this workpiece, the workpiece is identified as a "risky target" and its priority is lowered to prevent the robot from operating blindly. Only workpieces without any obstructions are classified into the "clear target queue".

[0117] The working logic of the pose calculation module is as follows: Generally speaking, the traditional grasping method selects the center of mass of the object for grasping. However, for steel plates with irregular shapes or holes on the surface, the center of mass may be in a suspended position, causing the suction cup to leak air or the magnet to be too attractive. Therefore, this embodiment proposes and implements a morphological corrosion method based on 2D projection mesh to find a suitable adsorption position on the object, including the following points.

[0118] Construction of local coordinate system based on PCA

[0119] When obtaining the local coordinate system of the workpiece, obtaining the main orientation of the point cloud through Principal Component Analysis (PCA) is an effective and stable method. Essentially, it maps a high-dimensional data to a low-dimensional subspace with the largest variance, preserving as much information as possible from the original dataset.

[0120] For the set of independent workpiece points obtained through cascading segmentation Each point in the point cloud satisfies Then find the geometric center (Centroid) of the point cloud containing that point.

[0121] (8)

[0122] Then, a decentering operation is performed on all points to remove the interference caused by the translation and obtain the covariance matrix. :

[0123] (9)

[0124] matrix This describes the variance of the point cloud distribution in various directions within three-dimensional space and the covariance between them. Because Since it is a real symmetric matrix, we can find an orthogonal matrix and a diagonal matrix that satisfy:

[0125] (10)

[0126] Singular Value Decomposition (SVD) or eigenvalue decomposition of a matrix yields a matrix consisting of an eigenvalue diagonal matrix and the corresponding eigenvectors.

[0127] In equations (8) and (10), P is the point cloud set of a single workpiece, and N is the number of points. For the first A three-dimensional point, Let C be the geometric center, V be the covariance matrix, Λ be the eigenvector matrix, and Λ be the eigenvalue diagonal matrix.

[0128] In 3D point cloud processing, there are three eigenvectors (XYZ) with clear geometric and physical meaning:

[0129] (1) Medium eigenvalues Corresponding feature vector and Orthogonal is defined as the secondary axis (X-axis) of the workpiece.

[0130] (2) The eigenvector corresponding to the largest eigenvalue represents the direction in which the point cloud distribution is most dispersed and the length is the longest, i.e. the workpiece main axis (Y-axis).

[0131] (3) The eigenvector corresponding to the minimum eigenvalue represents the direction in which the point cloud distribution is flattest and has the smallest variance. For a planar steel plate, it is the normal direction of the plane, which is also the workpiece thickness direction (Z-axis).

[0132] Since the direction of the eigenvector is uncertain (i.e. and (All of these are reasonable solutions). In order to meet the requirements of subsequent robotic arm posture control and the right-hand rule, the direction of the coordinate axes is limited here. If the sign is a negative sign, then let Force the normal vector to point upwards (away from the bottom of the material frame). Recalculate the X-axis using the vector cross product to make the resulting local coordinate system a standard right-handed orthogonal coordinate system.

[0133] in, , , The eigenvalues ​​are arranged in descending order. , , For the corresponding feature vector; It is the unit vector in the vertical direction of the world coordinate system.

[0134] 2D Mesh Projection and Discretization

[0135] To find the safest adsorption sites on irregular steel plate surfaces that may have pores, a projection-discretization-erosion method is proposed.

[0136] Step 1. Establish and project the local coordinate system, and obtain the rotation matrix using PCA. and center of mass Transform the point cloud from the world coordinate system to the local planar coordinate system. For each point in the point cloud... Its coordinates in the local coordinate system The calculation method is as follows

[0137] (11)

[0138] In the local coordinate system, the object's thickness information is compressed, thus solving the problem of converting 3D point clouds into 2D planes.

[0139] In equation (11), Represents the rotation matrix from the world coordinate system to the local coordinate system; The center of mass of the workpiece; A point in the world coordinate system; These are the points after transformation to the local coordinate system.

[0140] Step 2. Grid Discretization: In this embodiment, a two-dimensional raster map with a resolution of 5.0 mm is used. To avoid array out-of-bounds errors, the boundaries of the projected points need to be obtained beforehand. .

[0141] Mapping function Defined as:

[0142] (12)

[0143] The code uses std::vector<int8_t> The grid stores the binary image, where 1 indicates that there is a point cloud at that location, and 0 indicates that there is no point cloud.

[0144] Step 3. Morphological Etching with Physical Constraints This is the most important step, in which the magnet (size) is shaped and etched. To adhere to a surface, the program defines a rectangular structure element with the following size in the grid coordinate system:

[0145] (13)

[0146] The erosion operation is not ordinary binary erosion, but rather incorporates a soft constraint with overhang tolerance. For each candidate center point... The program checks the state of the eight points surrounding it or all the grids contained within the entire structural element. The program flow is as follows: iterate through all the grid points covered by the structural element and calculate the number of points in state 0 (dangling). If the proportion of dangling points is less than (here we take 0.2, which means 20% dangling is allowed), then the center point is considered a "feasible adsorption site".

[0147] (4.14)

[0148] Finally, the point closest to the geometric center among all these points is selected as the optimal solution, and an inverse transformation is performed to return it to the world coordinate system, which is the final 6D grab pose.

[0149] In equations (13) and (14), Φ is the mapping from continuous planar coordinates to discrete grid coordinates; Index for grid columns; The projection point in the local coordinate system Direction coordinates; Minimum boundary of direction; For grid resolution; Leave blank space at the boundaries; and The size of the adsorption area of ​​the end effector; and The structural element is half-width and half-length; As a candidate center; The number of spaces within the area covered by the structuring element; Area of ​​the structural element; This is the threshold for the proportion of suspended elements.

[0150] Safe zone search based on morphological corrosion

[0151] To ensure the magnets are reliably attracted, the algorithm performs physically constrained morphological erosion operations on the mesh. The structural elements... It is a rectangular core of the corresponding size.

[0152] Unlike standard etching, this embodiment uses overhang tolerance. (Default is 0.2), for a candidate center Calculate the effective fill rate within the area occupied by the structural element:

[0153] : (15)

[0154] Only when At this point, the point is considered a feasible grasping point. Then, among all feasible points, the one closest to the geometric center is selected as the optimal grasping point, and an inverse transformation is performed to return to the world coordinate system to obtain the final 6D pose. This method theoretically ensures that the grasping point avoids holes and meets the torque requirements for physical adsorption.

[0155] In equation (15), For The effective fill rate centered on; structural element The binary occupancy state of the corresponding grid within; The relative mesh offset within the structuring element; when Not less than At that time, the candidate point satisfies the adsorption area constraint.

[0156] After obtaining the set of all feasible grab points, the system executes the final optimization strategy for optimal pose calculation and Z-axis refinement.

[0157] (1) Center offset principle. Calculate the distance from all possible positions to the geometric center (Visual Center) of the object, and select the position with the smallest distance as the optimal gripping center. This allows the gripping point to be as close as possible to the object's center of gravity, avoiding excessive rotation during the lifting process.

[0158] in, and These represent the grid. The optimal grab center coordinates in the direction.

[0159] (2) Coordinate backtracking and Z-axis refinement. The selected local 2D coordinates are inversely transformed to the world coordinate system. The point closest to the grab center is found in the origin cloud, and its Z value is used as the final grab height.

[0160] (3) Finally, a complete 6D pose is obtained, including position coordinates (x, y, z) and rotation quaternions (qx, qy, qz, qw). Since the steel plate is a planar object, the rotation range around the Z-axis is limited in the algorithm and the normal is consistent to ensure the uniqueness of the pose of the robot end effector.

[0161] Environmental collision detection and intelligent decision-making: In order to ensure that the robotic arm does not collide with the side or bottom of the bin during the grasping operation, it is necessary to perform environmental interference detection on the obtained grasping pose. Based on the above, a complete logical chain consisting of "no-material self-check" and "hierarchical decision-making" is proposed.

[0162] End effector collision detection based on cylindrical envelope

[0163] In the second stage of detection, the system uses real point cloud data (Original SceneCloud) with material frame environment for calculation to simulate the real physical picking process. An adjustable-size cylinder is defined in the system as the end effector of the robot (such as a magnetic chuck and its connecting flange).

[0164] The collision detection model is defined by three geometric parameters:

[0165] (1) Detection radius (binCheckRadius). This parameter represents the radius of the bottom surface of the cylinder. The value of this parameter should be slightly larger than the maximum outer diameter of the actual end effector, leaving a certain margin.

[0166] (2) Check Height. This indicates the distance the cylinder travels along the axial direction from contacting the bottom of the workpiece until it is lifted.

[0167] (3) Starting offset (binCheckStartOffset). To prevent the workpiece surface from being identified as an obstacle in the environment due to point cloud noise, the bottom of the cylinder is raised by a certain height (e.g., 20mm) towards the normal direction.

[0168] The algorithm places this cylinder along the workpiece grasping normal direction and obtains all points in the cylinder's envelope under the Octree spatial index. If it finds any one or more points in the envelope that belong to the original scene (excluding the workpiece's own point cloud), then it considers this grasping path infeasible and discards this candidate target.

[0169] "Free Judgment" and Multi-level Intelligent Decision Logic

[0170] In this embodiment, a deterministic state machine serves as its decision center, integrating cascaded segmentation results, scoring and ranking, inter-board occlusion, and environmental collision information. Its working principle is as follows:

[0171] (1) No Material Check. At the start of cascade segmentation (Euclidean clustering stage), the system will detect the point cloud situation of the scene. If the clustering algorithm cannot find a valid cluster and there are many isolated points in the scene (remainingPointsThreshold, default value is 300,000 points), then it is considered to be "no material" or "only noise". At this time, HasMaterial is set to False and subsequent operations are immediately terminated to avoid unnecessary processing of empty boxes or a large amount of noise by the robot. It can be judged as no material and a material replacement signal is sent.

[0172] (2) Priority path. Clear target selection. If the no-material detection is successful, the "clear candidate list" (no occlusion between boards) is searched first. The candidate objects are checked for environmental collision according to the score from high to low. If the first object that meets the environmental conditions is detected, it is taken as the best safe target and the position information of the object is given. The current loop is then exited.

[0173] (3) Alternative Paths. The risk target assessment will only activate alternative paths if all "clear" targets cannot be environmentally detected or if there are no "clear" targets. The system selects the highest-scoring risk from the "risk candidate list" (with inter-board occlusion) for environmental detection. If successful, it will be used as the risk capture target and output.

[0174] (4) Failure response. If none of the above methods can obtain a feasible solution, the program will display "Failed to capture", prompting the operator to intervene or start the jitter mechanism.

[0175] Key parameter configuration

[0176] In complex industrial flexible manufacturing scenarios, the robustness and final picking success rate of a vision grasping system largely depend on the high degree of adaptability between the core algorithm parameters and the characteristics of the actual physical environment. Variable lighting conditions in industrial settings, the imaging noise level of 3D cameras, and physical environmental variables such as the surface material (e.g., reflectivity, oil stains) and random stacking density of non-standard steel plate workpieces all significantly affect the geometric features and topological structure of point cloud data.

[0177] Meanwhile, the system must have strong adaptability and compatibility to handle the gripping tasks of various workpieces (covering different geometric dimensions, thicknesses and weights) in different production lines, and to match the physical force constraints of different types of robotic arm end effectors (such as magnetic chucks or vacuum chucks of different lengths and widths).

[0178] Based on these engineering requirements, this embodiment designs all key core parameters as dynamically adjustable open configuration items in its software architecture design, as shown in Table 1. This highly flexible configurable parameterization mechanism not only effectively bridges the gap between theoretical algorithms and physical reality, but also enables field engineers to perform targeted optimizations based on specific tooling fixtures and workpiece characteristics, thereby maximizing the long-term operational stability and environmental generalization capability of the disordered grasping system in multi-task, cross-scenario applications.

[0179] Table 1 Key Algorithm Parameter Configuration and Physical Meaning

[0180]

[0181] To verify the effectiveness of this embodiment, experiments were conducted on the following hardware and software platforms.

[0182] Hardware configuration: Intel Core i7-12700H processor, 16GB DDR4 memory, NVIDIA GeForce RTX3060 graphics card.

[0183] Software environment: Windows 10 operating system, Visual Studio 2026 development environment (MSVC 19.xx), C++17 standard.

[0184] Dependencies: PCL 1.12.1 (core library for point cloud processing), Qt 5.15.2 (interface and plugin framework), OpenCASCADE 7.7 (geometric kernel).

[0185] System integration and online data acquisition: This system has been successfully applied in a real industrial robot workstation. It uses a high-precision 3D structured light camera (e.g., Mech-Eye Pro) as a vision sensor, mounted above the material frame in an "eye-to-hand" configuration. During operation, the system acquires real-time 3D point cloud information of stacked steel plates in the work area via industrial communication methods such as TCP / IP and controls a six-axis industrial robot to perform grasping and loading / unloading operations. All experimental results in this embodiment are derived from real grasping data obtained during the online operation of this actual production line.

[0186] To enable the system to achieve "perception" and "obstacle avoidance," each set of experimental data consists of two parts. The first is a processed point cloud, which has undergone ROI cropping, background removal, and voxel downsampling (workpiece point cloud only, used for cascade segmentation and gripping point calculation); the second is the original scene point cloud, including the bin, surrounding objects, and all other point clouds. This is used for environmental collision detection in the second stage. A typical original scene point cloud is shown below. Figure 2 As shown.

[0187] To fully test the system's performance, experiments were conducted in three representative industrial environments: a standard scattered scenario, a tightly stacked scenario, and a scenario with irregularly shaped workpieces with holes.

[0188] (1) Scene 1

[0189] Standard scattered component scene (can be separated using Euclidean clustering). For example... Figure 3 As shown, in a standard scenario, Euclidean clustering can be used to directly separate them.

[0190] The system has been applied in industrial settings, and its operation under continuous production cycles is as follows: Figure 4 and 5 The image shown is a screenshot of the software monitoring the automatic capture process and the surrounding environment.

[0191] In practical applications, the system utilizes a 3D camera to acquire point clouds of the material frame and the sheet metal, such as... Figure 6As shown. Within one second, a 6D grasping posture with safety guarantees is calculated, and a log file is used to record whether the grasping was successful, whether there was any obstruction between boards, and whether no material was detected.

[0192] (2) Scene Two

[0193] Closely stacked scenario (LCCP algorithm verification). In areas where multiple steel plates are tightly stacked (gap < 3mm), traditional Euclidean clustering methods fail to identify object edges, causing two steel plates to be mistakenly identified as a large connected region (undersegmentation). After the system triggers the second-level LCCP segmentation, the method can correctly identify a 15° concave angle change in the normal at the object connection point (exceeding a set threshold), thus accurately segmenting the adhered group into different workpieces. The resulting image is shown in Figure 7.

[0194] (3) Scene 3

[0195] Irregularly shaped workpiece with holes (mesh corrosion algorithm verification). For irregularly shaped steel plates with weight-reducing holes on the surface, the traditional centroid method typically finds the gripping point at the center of the hole (i.e., air), which fails to achieve gripping. The morphological mesh search method proposed in this paper rasterizes the point cloud projection and then uses a 100mm×100mm virtual magnet core to move and find the optimal position, such as... Figure 8 As shown, this method avoids the location of the geometric center hole and finds a suitable maximum inscribed rectangular area in the solid part of the edge, ensuring the safety and reliability of physical adsorption.

[0196] To fairly compare the performance of the various systems, comparative experiments were conducted on a real dataset containing 500 crawl attempts (results are shown in Table 2). The metrics used were:

[0197] (1) Success rate of segmentation: the proportion of independent workpieces that are correctly separated.

[0198] (2) Grasping success rate: The generated grasping pose is a reasonable proportion (no collision and sufficient contact area).

[0199] (3) Success rate of capturing irregular scene: the proportion of correctly capturing irregular parts.

[0200] Table 2 Algorithm Performance Comparison Data

[0201]

[0202] As shown in Table 2, in handling tightly stacked cases, the four-level cascaded algorithm proposed in this embodiment achieves a segmentation success rate of 89.4%, a significant improvement compared to the previous 45.2%, indicating that LCCP and region growing play an important role in solving the "undersegmentation" problem. The morphological erosion algorithm can also correctly find reliable grasping points, greatly improving the stability of the system.

[0203] System robustness largely depends on some important parameter settings. In order to study the impact of different parameters on the algorithm performance, we will conduct a controlled variable experiment, namely the effect test of the "LCCP concavity threshold".

[0204] When the setting is too small (<5°), minute rust spots or deformations on the steel plate surface will be misjudged as boundaries, resulting in severe "over-segmentation." Conversely, when the setting is too large (>20°), gently transitioning stacked boundaries cannot be detected, resulting in "under-segmentation." Figure 9 As shown.

[0205] The above results show that the optimal balance point under the current depth camera accuracy can effectively remove surface noise and detect stacking boundaries.

[0206] The above are merely embodiments of the present invention. The invention is not limited to the fields covered by these embodiments. Commonly known structures and characteristics in the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are able to access all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A mixed-material disorder grasping control system based on 3D point clouds, comprising a 3D camera and a data processing unit electrically connected to the 3D camera, characterized in that, The data processing unit includes: The point cloud preprocessing module acquires scene point clouds and performs noise reduction and downsampling to obtain workpiece point clouds and original scene point clouds. The cascaded segmentation module performs hierarchical cascaded segmentation of the workpiece point cloud to obtain independent workpiece point cloud clusters; The candidate rating module calculates a comprehensive score based on the height score and the proportion of points in each independent workpiece point cloud cluster, and sorts the independent workpiece point cloud clusters in descending order of comprehensive score to obtain a priority queue. The occlusion and collision detection module performs inter-board occlusion detection and environmental collision detection on the point cloud clusters of independent workpieces in the priority queue based on the original scene point cloud, and filters out the workpieces that can be safely grasped. The pose calculation module performs local coordinate system projection based on the local geometric features of the workpiece being safely gripped, and performs morphological mesh search in the local coordinate system to determine the final adsorption center point, thereby calculating the six-degree-of-freedom gripping pose of the workpiece.

2. The mixing and disordered grasping control system based on three-dimensional point cloud according to claim 1, characterized in that: The cascaded splitting module includes: The first-level segmentation unit clusters the workpiece point cloud within a limited spatial range and separates coarsely segmented point cloud clusters. The second-level segmentation unit segments the coarse segmentation point cloud clusters that are in contact with each other. It calculates the angle between the normals of the surfaces of adjacent small blocks in the coarse segmentation point cloud clusters to determine the geometric concave-convex connection characteristics and cuts off the adhesion boundary according to the concave connection relationship. The third-level segmentation unit performs surface merging on the point cloud after cutting off the adhesion boundary based on normal consistency, smoothing surface crack lines; and... The fourth-level segmentation unit performs color segmentation and re-merging on surface areas with obvious color differences based on the color difference distance in the color space to obtain independent workpiece point cloud clusters.

3. The mixing and disordered grasping control system based on three-dimensional point cloud according to claim 2, characterized in that: The cascaded segmentation module also includes a material shortage detection unit. When the first-level segmentation unit fails to separate coarse segmentation point cloud clusters with a number of points exceeding the preset volume point number threshold and the number of remaining noise points in the workpiece point cloud exceeds the noise limit, the material shortage detection unit determines that the system state is a material shortage noise saturation state, outputs a material change alarm signal, and blocks the robotic arm.

4. The mixing and disordered grasping control system based on three-dimensional point cloud according to claim 1, characterized in that: The candidate rating module extracts the maximum height of all points in each independent workpiece point cloud cluster, performs a normalization transformation on the maximum height within the entire scene height range to generate a vertical allocation index, counts the total number of points contained in each independent workpiece point cloud cluster to generate a scale allocation index, and then multiplies the vertical allocation index and the scale allocation index by the corresponding priority bias coefficients and sums them by weight to obtain the final comprehensive score.

5. The mixing and disordered grasping control system based on three-dimensional point cloud according to claim 1, characterized in that: The occlusion and collision detection module includes an adaptive occlusion detection unit. The adaptive occlusion detection unit calculates the surface normal axis pointing to space based on the outer contour of the safe gripping workpiece. It then expands outward in the positive direction of the surface normal axis on the surface of the safe gripping workpiece to generate a prism-shaped detection area. The adaptive occlusion detection unit searches the original scene point cloud for occlusion interference points that belong to the point cloud of the non-safe gripping workpiece. When an occlusion interference point is detected falling into the prism-shaped detection area, the safe gripping workpiece is marked as an occlusion danger workpiece.

6. The mixing and disordered grasping control system based on three-dimensional point cloud according to claim 5, characterized in that: The occlusion and collision detection module also includes an environmental collision detection unit. The environmental collision detection unit simulates the maximum envelope radius and vertical lifting height of the adsorption end effector to construct a cylindrical obstacle avoidance zone. The cylindrical obstacle avoidance zone is placed on the calculated proposed adsorption center point. The environmental collision detection unit searches within the original scene point cloud for environmental obstacle points that fall into the cylindrical obstacle avoidance zone. If an environmental obstacle point is found, the proposed adsorption center point is discarded.

7. The mixing and disordered grasping control system based on three-dimensional point cloud according to claim 1, characterized in that: The pose calculation module includes a principal component analysis unit, which is used to find the geometric center of the point cloud cluster of independent workpieces, calculate the covariance matrix of the distribution of the point cloud cluster of independent workpieces in various directions, decompose the covariance matrix to obtain three mutually orthogonal principal eigencomponent directions, and set them as the principal direction axis, secondary direction axis and plane vertical normal axis respectively according to the density of spatial dispersion from large to small, and verify the plane vertical normal axis to make the plane vertical normal axis point upward, and construct a right-handed orthogonal local coordinate system.

8. The mixing and disordered grasping control system based on three-dimensional point cloud according to claim 7, characterized in that: The pose calculation module also includes a projection rasterization unit. The projection rasterization unit extracts the transformation parameters from the independent workpiece point cloud cluster to the right-hand orthogonal local coordinate system, projects and rotates all three-dimensional coordinates in the independent workpiece point cloud cluster to the local two-dimensional plane, ignores the thickness term to restore the three-dimensional workpiece to the two-dimensional plane, and calculates the grid position number associated with each test projection point according to the discrete raster unit size, and generates a two-dimensional grid image with position occupancy marks mapped in the local two-dimensional plane. The pose calculation module also includes a morphological erosion unit. The morphological erosion unit obtains a simulated magnetic pattern according to the actual working projection range of the robotic arm's adsorption disk. The simulated magnetic pattern is then slid across the two-dimensional mesh image at specified intervals to perform soft erosion judgment. The soft erosion judgment includes accumulating the total area of ​​the hole mesh and edge cavity mesh within the simulated magnetic pattern. If the proportion of the total area of ​​the hole mesh and edge cavity mesh to the total area of ​​the simulated magnetic pattern is less than the upper limit of the suspension loss, the center of symmetry of the simulated magnetic pattern is extracted and determined as a feasible adsorption center point.

9. The mixing and disordered grasping control system based on three-dimensional point cloud according to claim 8, characterized in that: The pose calculation module also includes a pose output unit. The pose output unit traverses all feasible adsorption center points found in the two-dimensional mesh image, calculates the two-dimensional Euclidean distance between the feasible adsorption center point and the geometric center of the independent workpiece point cloud cluster, takes the feasible adsorption center point with the shortest two-dimensional Euclidean distance as the final adsorption center point, extracts the original spatial position corresponding to the final adsorption center point, and refines the gripping height in the direction of the final adsorption center point based on the height value of the point closest to the final adsorption center point. It then fuses the direction of the right-handed orthogonal local coordinate system to generate a six-degree-of-freedom gripping pose signal and outputs it to the industrial robotic arm controller.

10. A method for controlling disordered grasping of mixed materials based on 3D point clouds, characterized in that: The system described in any one of claims 1-9 is employed.

Images