Intelligent disordered workpiece grabbing system based on three-dimensional vision

Abstract

The application provides a three-dimensional vision-based intelligent grabbing system for unordered workpieces, and relates to the technical fields of three-dimensional vision perception and automatic grabbing, and comprises sequentially connected data acquisition module, three-dimensional perception module, pose estimation module and robot execution module, wherein the data acquisition module synchronously acquires two-dimensional images and three-dimensional point clouds of unordered workpieces; the three-dimensional perception module guides three-dimensional point clouds to perform instance segmentation by using semantic features of two-dimensional images, obtains point cloud subsets of each independent workpiece, and generates stacking occlusion information between workpieces; the pose estimation module performs six-dimensional pose estimation on each workpiece according to the stacking occlusion information; and the robot execution module performs grabbing after converting the six-dimensional pose to a robot coordinate system according to hand-eye calibration; through directional data coupling between modules, the application can realize automatic layer-by-layer grabbing of stacked workpieces, and is especially suitable for pipe joint workpieces with rotational symmetry.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional vision perception and automated grasping technology, and in particular to an intelligent grasping system for disordered workpieces based on three-dimensional vision. Background Technology

[0002] In industrial automated sorting and loading / unloading scenarios, using 3D vision to guide robots to grasp disordered workpieces has become an important technical means. However, existing grasping systems for stacked workpieces generally adopt a modular, serial architecture of segmentation, pose estimation, and grasping. The modules only pass the final calculation results to each other. Intermediate layer information, such as stacked occlusion areas and hierarchical order, is discarded after segmentation and cannot be used as prior knowledge in subsequent pose estimation. This architecture, which lacks information interaction between modules, requires indiscriminate searching of all regions in the pose estimation stage. Occlusion areas and unocclusion areas are assigned the same matching weight. For workpieces with rotational symmetry, it is easy to output incorrect pose assumptions due to misleading constraints of occlusion point clouds. Moreover, this architecture has a high data dependency, requiring a large amount of manually labeled stacked scene data for model training, making it difficult to deploy quickly in actual industrial applications. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a three-dimensional vision-based intelligent grasping system for disordered workpieces that can overcome or at least partially solve the above problems.

[0004] To address the aforementioned technical problems, this invention provides an intelligent grasping system for disordered workpieces based on 3D vision, comprising: a data acquisition module for simultaneously acquiring 2D images of disordered workpieces and corresponding 3D point clouds; a 3D perception module connected to the data acquisition module for guiding instance segmentation of the 3D point clouds based on the semantic features of the 2D images to obtain point cloud subsets for each independent workpiece, and generating stacking occlusion information between workpieces based on the spatial distribution of the point cloud subsets; a pose estimation module connected to both the 3D perception module and the data acquisition module for performing six-dimensional pose estimation for each workpiece based on the point cloud subsets and the stacking occlusion information, and outputting the position and orientation of each workpiece in the camera coordinate system; and a robot execution module connected to the pose estimation module for converting the six-dimensional pose to the robot coordinate system based on a hand-eye calibration matrix, and controlling the robot end effector to grasp the workpiece.

[0005] The above technical solution, through targeted data coupling between multiple modules, enables the output of the upstream module not only to serve as the input of the downstream module, but also to participate in the decision-making process of the downstream module as prior knowledge with specific physical constraints. This is specifically manifested in the following three levels of collaborative relationships:

[0006] Level 1: Cross-modal collaboration within the perception layer, i.e., collaboration within the 3D perception module.

[0007] Within the 3D perception module, the feature extraction unit extracts semantic feature maps from the 2D image, and the feature fusion unit maps these semantic feature maps as spatial guiding weights to the spatial coordinate features of the 3D point cloud. The essence of this collaborative relationship is to use the rich semantic information such as color, texture, and edge in the 2D image to make up for the fuzzy geometric features of the 3D point cloud at the stacking boundary, while using the spatial coordinate information of the 3D point cloud to make up for the lack of depth dimension in the 2D image. The two are fused at the feature level rather than the data level to obtain enhanced point cloud features that contain both semantic annotation information and accurate spatial coordinates, thereby achieving accurate instance segmentation at the stacking boundary.

[0008] Second level: Constraint coordination between the perception module and the pose estimation module

[0009] The 3D perception module outputs stacked occlusion information, which includes the location of the occluded areas and the stacking order. This information is not simply output as a segmentation result and then stops there. Instead, it is passed to the pose estimation module in an explicit and structured form. The occlusion recognition unit in the pose estimation module uses this information to determine the occluded and unoccluded parts in the point cloud of each workpiece. The technical effect of this collaborative relationship is that it injects the prior knowledge of which areas are unreliable into the pose estimation process in advance, so that subsequent matching calculations can allocate computing resources in a targeted manner and not give equal matching weight to occluded areas.

[0010] Third level: Mathematical coordination between weight modulation unit and attitude calculation unit

[0011] The weight modulation unit applies a first weight to the matching score of the occluded part and a second weight to the unoccluded part based on the judgment result of the occlusion recognition unit. The value of the first weight is less than that of the second weight. The essence of this cooperative relationship is to adopt a mathematical processing strategy of weight reduction rather than point deletion. On the one hand, it retains the geometric constraint information that still has reference value in the occluded point cloud. On the other hand, it suppresses the interference of the occluded part on the pose calculation result by reducing the weight. It achieves the optimal balance between information preservation and noise suppression. The pose calculation unit performs the calculation work based on the weighted matching score. Since the weight distribution is obviously biased towards the high confidence region of the unoccluded part, the pose search space is effectively pruned. For rotationally symmetric workpieces, incorrect pose assumptions will be automatically eliminated under the weight suppression effect, and finally an unambiguous six-dimensional pose is output.

[0012] Specifically, the 3D perception module includes a feature extraction unit for semantic segmentation of the 2D image and outputting a 2D semantic feature map; a feature fusion unit for mapping the 2D semantic feature map to the spatial coordinates of the 3D point cloud to obtain enhanced point cloud features. Specifically, the feature fusion unit upsamples the 2D semantic feature map to match the projection size of the 3D point cloud and then fuses it element-wise with the coordinate features of the 3D point cloud to obtain the enhanced point cloud features. This element-wise fusion can be any of the following forms: element-wise addition, element-wise multiplication, or channel concatenation followed by mapping through a convolutional layer; and an instance segmentation unit for clustering and segmenting the point cloud subsets of each independent workpiece based on the enhanced point cloud features. This clustering can use a base... The algorithm can be any one of Euclidean distance-based clustering, density-based spatial clustering, or deep learning-based point cloud instance segmentation networks. A stacking analysis unit is used to generate the stacking occlusion information based on the overlapping areas of each point cloud subset in the spatial height direction. The stacking occlusion information includes the location of the occlusion area and the stacking hierarchy order. Specifically, the stacking analysis unit projects each point cloud subset onto a horizontal plane, determines whether there is an upper-lower overlap relationship based on the intersection of the projected areas, and sorts the point cloud subsets according to their center height values ​​in the height direction to determine the stacking hierarchy order. The subset with the larger height value is determined to be the upper layer, and the subset with the smaller height value is determined to be the lower layer. The area in the lower layer point cloud covered by the horizontal projection of the upper layer point cloud is marked as the location of the occlusion area.

[0013] Specifically, the pose estimation module includes an occlusion recognition unit, used to determine the occluded and unoccluded parts in the point cloud subsets of each workpiece based on the stacked occlusion information. Specifically, the occlusion recognition unit maps the occlusion region position back to the three-dimensional coordinate space of the corresponding workpiece point cloud subset, marks the three-dimensional points located within the occlusion region position as occluded parts, and marks the three-dimensional points located outside the occlusion region position as unoccluded parts. A matching calculation unit is used to perform pose matching between the point cloud subsets of each workpiece and a preset workpiece model, and calculate the matching score. In one specific embodiment, the matching calculation unit uses an iterative nearest-point algorithm to perform rigid body transformation registration between the point cloud subsets and the preset workpiece model, using the reciprocal of the registration error as the matching score. During the registration process, for each point pair, only the error between the point cloud of the unoccluded part and the corresponding point of the model is calculated; the point cloud of the occluded part is not considered. Error calculation; a weight modulation unit, used to apply a first weight to the contribution of the occluded part to the matching score and a second weight to the unoccluded part, wherein the first weight is less than the second weight. In one specific embodiment, the first weight is 0 and the second weight is 1, at which time the matching score of the occluded part is 0, which is equivalent to not participating in the attitude calculation. In another specific embodiment, the first weight is a positive number greater than 0 and less than 1, at which time the occluded part participates in the attitude calculation with a reduced weight; an attitude calculation unit, used to perform attitude calculation based on the weighted matching score and output the six-dimensional pose. Specifically, the attitude calculation unit traverses multiple attitude assumptions of the preset workpiece model in six-dimensional space, calculates the weighted matching score under each attitude assumption, and selects the attitude assumption corresponding to the highest matching score as the final output six-dimensional pose.

[0014] Specifically, the robot execution module includes a coordinate transformation unit for transforming the six-dimensional pose from the camera coordinate system to the robot base coordinate system according to the hand-eye calibration matrix; a trajectory planning unit for generating a collision-free grasping path based on the transformed pose and the geometric model of the workpiece; and a motion control unit for driving the robot end effector to move to the target grasping point and perform the grasping action according to the collision-free grasping path.

[0015] In a preferred embodiment, the pose estimation module is implemented using a neural network pre-trained on a synthetic dataset. The synthetic dataset is generated by randomly setting the pose parameters and stacking state parameters of the workpiece in a virtual simulation environment. The pose parameters include position coordinates and Euler angles, and the stacking state parameters include the number of stacking layers and the relative displacement.

[0016] The specific process of generating this synthetic dataset is as follows: First, load the 3D computer-aided design model of the workpiece in the virtual simulation environment; then, randomly generate the pose parameters of each workpiece so that each workpiece is distributed in space with random positions and orientations; then, randomly generate the stacking state parameters so that some workpieces overlap in the height direction; finally, according to the set stacking state, place each workpiece layer by layer, and record the 3D point cloud of the current scene and the true six-dimensional pose label of each workpiece for each layer.

[0017] In a further preferred embodiment, the models of each workpiece in the synthetic dataset have rotational symmetry, which means that the geometry remains unchanged after rotating 180 degrees around the central axis of the workpiece. For this type of workpiece with 180-degree rotational symmetry, when generating pose parameters, the synthetic dataset restricts the rotation angle around the central axis to a half-open interval of 0 to 180 degrees to avoid generating duplicate samples of equivalent poses.

[0018] In a preferred embodiment, the data acquisition module includes a structured light 3D camera and a color camera. The structured light 3D camera is used to project a structured light pattern and acquire a depth image of the workpiece, and reconstruct the 3D point cloud based on the depth image. The color camera is used to acquire the 2D image. The structured light 3D camera and the color camera are rigidly fixedly connected, and the coordinate transformation relationship between the two is pre-calibrated.

[0019] In a preferred embodiment, the point cloud subsets of each workpiece output by the 3D perception module have individual identification tags. The pose estimation module independently estimates the corresponding six-dimensional pose for each workpiece based on the individual identification tags. The robot execution module performs grasping sequentially from top to bottom according to the stacking order of each workpiece. The stacking order output by the 3D perception module is passed to the robot execution module. The robot execution module determines the target workpiece to be grasped based on this order. After each grasping action is completed, a trigger signal is sent to the data acquisition module to trigger the next acquisition-perception-pose-grabbing cycle.

[0020] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0021] First, through cross-modal feature fusion within the 3D perception module, precise guidance of 2D semantic features for 3D point cloud instance segmentation is achieved. At the stacking boundary, the semantic information provided by the 2D image effectively fills in the fuzzy areas of the 3D point cloud geometric features, significantly improving the instance segmentation accuracy of the stacked workpiece. This effect is not the superposition effect of the 2D vision module and the 3D point cloud module operating independently, but rather a systematic gain generated after the two work together at the feature level.

[0022] Second, by explicitly transferring stacked occlusion information from the 3D perception module to the pose estimation module, the pose estimation module can know the reliable and unreliable regions in the point cloud of each workpiece in advance. This enables the directional allocation of computing resources and the pre-clipping of the pose search space, reducing the computational complexity of pose estimation and improving robustness under occlusion conditions.

[0023] Third, by using the dual-weighting mechanism of the weight modulation unit, the mathematical strategy of "reducing weights rather than deleting points" is adopted to process the occluded point cloud. This avoids the information loss caused by directly deleting the point cloud and effectively suppresses the interference of the occluded area on the attitude calculation. This mechanism, in conjunction with the occlusion recognition unit, filters out misleading information while retaining effective geometric constraints, and realizes unambiguous attitude estimation of rotationally symmetric workpieces in stacked scenarios.

[0024] Fourth, by using a targeted generation strategy for synthetic datasets, training data with precise pose labels is automatically generated in a virtual simulation environment for specific combinations of stacked scenarios and rotationally symmetric workpieces. This solves the technical problem of accurately labeling poses in stacked states during real-world data acquisition. At the same time, by limiting the rotation angle to a half-open interval, redundancy of equivalent pose samples is avoided, thus improving the utilization efficiency of training data.

[0025] Fifth, through directional data coupling and data flow closure between modules, a linkage mechanism of "perception-driven estimation, estimation constraint execution, and execution feedback perception" is formed. The individual identity labels and stacking hierarchy output by the 3D perception module simultaneously serve pose estimation and grasping decision, enabling the system to clear stacked workpieces layer by layer without human intervention. Attached Figure Description

[0026] In the attached diagram:

[0027] Figure 1 This is a schematic diagram of the overall module connection of an intelligent grasping system for disordered workpieces based on three-dimensional vision, provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0029] Example 1: As Figure 1 As shown, this embodiment provides a disordered workpiece grasping system based on three-dimensional vision, including a three-dimensional perception module, a pose estimation module, and a robot execution module. This system is used to automatically identify and grasp stacked pipe joint workpieces.

[0030] The data acquisition module uses an integrated RGB-D camera. The color camera is used to acquire two-dimensional color images with a resolution of 1920×1080. The structured light projector works with the depth camera to acquire depth images and reconstruct three-dimensional point clouds. The RGB-D camera is installed directly above the robot's workspace, covering the stacking area of ​​the workpieces.

[0031] The robot execution module adopts a six-degree-of-freedom industrial robot, and the end effector is a parallel pneumatic gripper. The hand-eye calibration matrix between the robot base coordinate system and the RGB-D camera coordinate system is obtained in advance through the standard chessboard calibration method.

[0032] The workflow of the above system is as follows:

[0033] The first step is for the data acquisition module to simultaneously acquire two-dimensional images and three-dimensional point clouds of the stacking area, and output the two-dimensional images and three-dimensional point clouds to the three-dimensional perception module at the same time.

[0034] The second step involves the feature extraction unit of the 3D perception module using a pre-trained semantic segmentation neural network to perform pixel-by-pixel classification on the 2D image, outputting a 2D semantic feature map. The value of each pixel in this semantic feature map represents the probability confidence that the pixel belongs to the workpiece region. The feature fusion unit then upsamples this 2D semantic feature map to match the projection size of the 3D point cloud, and then fuses it with the coordinate features of the 3D point cloud. After concatenation of the channel dimensions, the points are mapped by a convolutional layer to output enhanced point cloud features. The instance segmentation unit, based on the enhanced point cloud features, uses a clustering algorithm based on Euclidean distance to segment point clouds belonging to different workpieces into independent point cloud subsets. Each point cloud subset corresponds to an independent workpiece. The stacking analysis unit projects each point cloud subset onto a horizontal plane and detects the overlap of the projected areas. If the horizontal projections of two point cloud subsets intersect, it is determined that they are stacked. The center height values ​​of the two are compared to determine the stacking order. The spatial area in the lower point cloud subset that is covered by the horizontal projection of the upper point cloud subset is marked as the occlusion area location. The 3D perception module outputs the point cloud subsets of each independent workpiece. Each point cloud subset is accompanied by stacking occlusion information, including the occlusion area location, stacking order, and individual identity label.

[0035] The third step involves the occlusion recognition unit of the pose estimation module receiving the point cloud subsets and corresponding stacked occlusion information output by the 3D perception module. Based on the spatial coordinate range of the occlusion area, the 3D points marked as occluded areas in each point cloud subset are identified as occluded parts, and the remaining 3D points are identified as unoccluded parts. The matching calculation unit iteratively registers the point cloud subset of each workpiece with the preset 3D computer-aided design model of the pipe joint, and calculates the registration error. During the registration process, only the point cloud of the unoccluded parts participates in the point pair search and error calculation. The weight modulation unit applies a weight of 1 to the registration error score of the unoccluded parts and a weight of 0.2 to the registration error score of the occluded parts, and calculates the weighted comprehensive matching score. The attitude calculation unit traverses multiple initial attitude assumptions in the six-dimensional attitude space, performs the above matching calculation and weight modulation respectively, and selects the six-dimensional attitude corresponding to the highest weighted comprehensive matching score as the final output.

[0036] In the fourth step, the coordinate transformation unit of the robot execution module receives the six-dimensional pose of each workpiece output by the pose estimation module, and transforms it from the camera coordinate system to the robot base coordinate system according to the hand-eye calibration matrix. The trajectory planning unit plans a collision-free path from the current robot end effector position to the target gripping point in the robot workspace based on the transformed pose and the geometric dimensions of the pipe joint. The motion control unit drives the robot end effector to move to the target gripping point according to the planned path, and closes the gripper to perform the gripping action.

[0037] Fifth, after the grasping is completed, the robot execution module sends a trigger signal to the data acquisition module, and the system returns to the first step to re-execute the acquisition and perception pose grasping loop for the remaining workpieces until the stacking area is cleared of workpieces.

[0038] Example 2: The main difference between this example and Example 1 lies in the specific implementation of the pose estimation module.

[0039] In this embodiment, the pose estimation module uses a PointNet++ network pre-trained on a synthetic dataset to achieve end-to-end six-dimensional pose estimation. The input of the network is a subset of the point cloud of a single workpiece and its corresponding stacking occlusion information, and the output is the six-dimensional pose of the workpiece.

[0040] First, a 3D computer-aided design model of the pipe fitting workpiece is obtained and imported into the Blender physics simulation environment. The workpiece is designed to have the characteristic that its geometry remains unchanged after rotating 180 degrees around its central axis. In the simulation environment, a cuboid stacking area is randomly generated, and the number of workpieces is randomly generated, ranging from 5 to 20 per batch. For each workpiece, its position coordinates are randomly generated. Distribute evenly within the stacking area Determined by the number of stacked layers, and Euler angles, the rotation angles around the central axis are sampled uniformly between 0 and 180 degrees, while the other two rotation angles are sampled uniformly between 0 and 360 degrees. Workpieces are placed layer by layer from smallest to largest value. After each workpiece is placed, its true six-dimensional pose is recorded as a label. At the same time, the simulation outputs the three-dimensional point cloud and two-dimensional image of the scene. In this way, a synthetic dataset containing 50,000 samples is automatically generated. Each sample contains a three-dimensional point cloud, a two-dimensional image, stacking occlusion information, a subset of point clouds of each independent workpiece and its corresponding true six-dimensional pose label. All annotations can be completed without manual intervention.

[0041] The synthetic dataset was divided into training, validation, and test sets in an 8:1:1 ratio. The PointNet++ network was trained under supervision using the training set, hyperparameters were tuned using the validation set, and network performance was evaluated using the test set. After training, the trained network parameters were deployed to the pose estimation module of the actual system.

[0042] The beneficial effects of this embodiment are that it achieves fully automated generation and annotation of training samples through a synthetic dataset, avoiding the difficulties of manually collecting stacked scene data and the subjective errors of manual pose annotation. At the same time, the size of the synthetic dataset can be infinitely expanded as needed, significantly improving the training efficiency and generalization ability of the pose estimation model.

[0043] Example 3: The difference between this example and Example 1 lies in the specific weight value selection strategy of the weight modulation unit.

[0044] In this embodiment, the weight modulation unit applies dynamic weights to the occluded and unoccluded parts during the registration process. Specifically, the weight modulation unit determines the weight values ​​according to the stacking hierarchy: for the workpiece located at the top layer, its point cloud subset has no occluded parts, and all point clouds are unoccluded parts, so a weight of 1 is uniformly applied; for the workpiece located in the middle layer, the proportion of occluded parts to the total point cloud is usually between 10% and 30%, so a weight of 0.3 is applied to the occluded parts, and a weight of 1 is applied to the unoccluded parts; for the workpiece located at the bottom layer, the proportion of occluded parts to the total point cloud can reach more than 50%, so a weight of 0.1 is applied to the occluded parts, and a weight of 1 is applied to the unoccluded parts.

[0045] Compared to the fixed weight strategy in Embodiment 1, the dynamic weight strategy in this embodiment has a fixed weight of 0.2 for the occluded part. It can adaptively adjust the weight reduction range according to the actual degree of occlusion of different workpieces. When the occlusion is slight, more constraint information of the occluded part is retained. When the occlusion is severe, the interference of the occluded part is suppressed to a greater extent, achieving a more refined balance between information preservation and noise suppression.

[0046] Example 4: The difference between this example and Example 1 lies in the specific fusion method of the feature fusion unit.

[0047] In this embodiment, the feature fusion unit fuses the two-dimensional semantic feature map and the three-dimensional point cloud coordinate features by element-wise multiplication rather than channel concatenation followed by mapping. Specifically, the feature fusion unit uses the upsampled two-dimensional semantic feature map as attention weights. This feature map is single-channel with a value range of 0 to 1, and is correlated with the coordinate features of each point in the three-dimensional point cloud. The mathematical expression for element-wise multiplication of the three channels is:

[0048]

[0049] in, To enhance the point cloud features, Two-dimensional semantic features in pixel coordinates The semantic confidence weight at the location, For 3D point clouds in spatial coordinates The original coordinate features at the location.

[0050] This element-wise multiplication fusion method has a clear physical meaning: regions with high semantic confidence in the 2D semantic feature map, that is, regions judged as workpiece boundaries in the 2D image, are assigned higher feature responses to their corresponding spatial points in the 3D point cloud, giving these points higher clustering weights in subsequent clustering segmentation; conversely, regions with low semantic confidence, that is, regions judged as background or non-boundary areas in the 2D image, are assigned lower clustering weights to their corresponding 3D points in clustering segmentation. Through this element-wise multiplication attention modulation mechanism, the stacking boundaries in the 3D point cloud are "explicitly labeled" by the 2D semantic feature map, significantly improving the accuracy of instance segmentation at stacking boundaries.

[0051] In various embodiments of the present invention, the grasping order decision of the robot execution module depends on the stacking hierarchy order output by the three-dimensional perception module.

[0052] Specifically, during the process of generating stacked occlusion information by the stacking analysis unit of the 3D perception module, the point cloud subsets of each independent workpiece are sorted from high to low according to the center height value to generate a stacking hierarchy order list. This list, together with the point cloud subsets of each workpiece and the individual identity label, is output to the pose estimation module and the robot execution module.

[0053] After receiving the point cloud subsets with stacking hierarchy order labels, the pose estimation module performs pose estimation from top to bottom according to the stacking hierarchy order. It prioritizes processing the upper-layer workpieces that are ranked earlier in the stacking hierarchy, and then processes the lower-layer workpieces that are ranked later. For the areas of the lower-layer workpieces that are occluded by the upper-layer workpieces, the occlusion recognition unit of the pose estimation module pre-marks the occluded areas based on the location of the occluded areas in the stacked occlusion information, and the weight modulation unit sets a lower matching weight for this part.

[0054] After receiving the six-dimensional poses of each workpiece and the corresponding stacking order from the pose estimation module, the robot execution module performs sequential grasping from top to bottom. After each grasping action, the robot execution module sends a trigger signal to the data acquisition module. The system re-acquires two-dimensional images and three-dimensional point clouds of the current stacking scene. The three-dimensional perception module recalculates the stacking order of the remaining workpieces, and the pose estimation module re-solves the six-dimensional poses of the remaining workpieces. The robot execution module then grasps the topmost workpiece. This cycle continues until all workpieces in the stacking scene have been removed one by one.

[0055] The grasping order decision and dynamic update mechanism can achieve the following technical effects: In each round of the cycle, the system only performs perception and pose estimation for the workpieces remaining in the current scene. As the top workpiece is removed one by one, the occlusion state of the lower workpieces changes continuously. The originally occluded areas are gradually revealed, and the number of unoccluded parts that the pose estimation module can use continues to increase. The pose calculation accuracy is gradually improved in each round of the cycle.

[0056] The disordered workpiece gripping system provided by this invention can be widely applied to stacked workpiece identification and gripping scenarios in the fields of automotive parts manufacturing, electronic component sorting, and logistics warehousing automation. Through deep data collaboration and directional constraint transmission between modules, the system can automatically clear stacked workpieces layer by layer without manual intervention. It is especially suitable for the automated sorting of pipe joint-type workpieces with rotational symmetry characteristics that are difficult to distinguish using traditional vision methods.

[0057] Based on the description of the above embodiments, the scope of protection of the present invention is not limited to the contents described in the specific embodiments above, but is defined by the claims and their equivalents. The core inventive concept of the present invention lies in the synergistic effect of the following five aspects:

[0058] First, the 2D semantic features within the 3D perception module guide the cross-modal fusion of 3D point cloud features, including but not limited to element-wise multiplication, channel stitching and mapping, element-wise addition, and all other fusion methods that can map semantic features to spatial coordinate features.

[0059] Second, the 3D perception module outputs stacked occlusion information, which includes the location of the occlusion area and the stacking order. This information is explicitly passed to the pose estimation module to construct the corresponding data flow architecture.

[0060] Third, the pose estimation module employs a dual-weight soft suppression mechanism where the weight modulation unit applies a first weight to the occluded part and a second weight to the unoccluded part, with the first weight being less than the second weight. This mechanism includes two strategies: fixed weight and dynamic weight.

[0061] Fourth, the method for generating pose parameters for rotationally symmetric workpieces in the synthetic dataset includes strategies for limiting rotation angles to non-equivalent intervals to avoid redundant samples.

[0062] Fifth, the complete system architecture and data flow closed loop formed by the combination of the above solutions.

[0063] Any non-substantial modifications made based on the above-mentioned core inventive concept, including but not limited to the specific algorithm implementation of each module, the numerical adjustment of weight values, the selection of the simulation environment for the synthetic dataset, and the specific brand and model of the hardware device, shall fall within the protection scope of this invention.

Claims

1. A three-dimensional vision-based intelligent workpiece grasping system, characterized in that, include: The data acquisition module is used to simultaneously acquire two-dimensional images and corresponding three-dimensional point clouds of disordered workpieces; The three-dimensional perception module, connected to the data acquisition module, is used to guide the three-dimensional point cloud to perform instance segmentation based on the semantic features of the two-dimensional image, obtain a subset of the point cloud of each independent workpiece, and generate stacking occlusion information between the workpieces based on the spatial distribution of the point cloud subset. The pose estimation module is connected to the 3D perception module and the data acquisition module respectively. It is used to perform six-dimensional pose estimation for each workpiece based on the point cloud subset and the stacked occlusion information, and output the position and orientation of each workpiece in the camera coordinate system. The robot execution module, connected to the pose estimation module, is used to convert the six-dimensional pose to the robot coordinate system according to the hand-eye calibration matrix, and control the robot end effector to grasp the workpiece.

2. The intelligent grasping system for disordered workpieces based on three-dimensional vision according to claim 1, characterized in that, The three-dimensional perception module includes: The feature extraction unit is used to perform semantic segmentation on the two-dimensional image and output a two-dimensional semantic feature map. The feature fusion unit is used to map the two-dimensional semantic feature map to the spatial coordinates of the three-dimensional point cloud to obtain enhanced point cloud features. An instance segmentation unit is used to cluster and segment the point cloud subsets of each independent workpiece based on the enhanced point cloud features. The stacking analysis unit is used to generate the stacking occlusion information based on the overlapping areas of each point cloud subset in the spatial height direction. The stacking occlusion information includes the location of the occlusion area and the stacking hierarchy order.

3. The intelligent grasping system for disordered workpieces based on three-dimensional vision according to claim 2, characterized in that, The feature fusion unit upsamples the two-dimensional semantic feature map to match the projection size of the three-dimensional point cloud, and then fuses it element-by-element with the coordinate features of the three-dimensional point cloud to obtain the enhanced point cloud features.

4. The intelligent grasping system for disordered workpieces based on three-dimensional vision according to claim 1, characterized in that, The pose estimation module includes: The occlusion recognition unit is used to determine the occluded and unoccluded parts in the point cloud subset of each workpiece based on the stacked occlusion information. The matching calculation unit is used to perform pose matching between the point cloud subset of each workpiece and the preset workpiece model, and calculate the matching score. A weight modulation unit is used to apply a first weight to the contribution of the occluded part to the matching score and a second weight to the unoccluded part, wherein the first weight is less than the second weight; The attitude calculation unit is used to calculate the attitude based on the weighted matching score and output the six-dimensional pose.

5. The intelligent grasping system for disordered workpieces based on three-dimensional vision according to claim 4, characterized in that, The matching calculation unit uses an iterative nearest point algorithm to perform rigid body transformation registration between the point cloud subset and the preset workpiece model, and uses the reciprocal of the registration error as the matching score.

6. The intelligent grasping system for disordered workpieces based on three-dimensional vision according to claim 1, characterized in that, The robot execution module includes: A coordinate transformation unit is used to transform the six-dimensional pose from the camera coordinate system to the robot base coordinate system according to the hand-eye calibration matrix; The trajectory planning unit is used to generate a collision-free grasping path based on the transformed pose and the geometric model of the workpiece. The motion control unit is used to drive the robot end effector to move to the target grasping point and perform the grasping action according to the collision-free grasping path.

7. The intelligent grasping system for disordered workpieces based on three-dimensional vision according to claim 1, characterized in that, The pose estimation module is implemented using a neural network pre-trained on a synthetic dataset. The synthetic dataset is generated by randomly setting the pose parameters and stacking state parameters of the workpiece in a virtual simulation environment. The pose parameters include position coordinates and Euler angles, and the stacking state parameters include the number of stacking layers and the relative displacement.

8. The intelligent grasping system for disordered workpieces based on three-dimensional vision according to claim 7, characterized in that, The models of each workpiece in the synthetic dataset have rotational symmetry, which means that the geometry remains unchanged after rotating 180 degrees around the central axis of the workpiece.

9. The intelligent grasping system for disordered workpieces based on three-dimensional vision according to claim 1, characterized in that, The data acquisition module includes: A structured light 3D camera is used to project a structured light pattern and acquire a depth image of the workpiece, and to reconstruct the 3D point cloud based on the depth image; A color camera is used to acquire the two-dimensional image; The structured light 3D camera and the color camera are rigidly fixedly connected, and the coordinate transformation relationship between the two is pre-calibrated.

10. The intelligent grasping system for disordered workpieces based on three-dimensional vision according to claim 1, characterized in that, The point cloud subsets of each workpiece output by the three-dimensional perception module have individual identity tags. The pose estimation module independently estimates the corresponding six-dimensional pose for each workpiece based on the individual identity tags. The robot execution module performs grasping sequentially from the top to the bottom according to the stacking order of each workpiece.

Images