Rapid pose detection method based on workpiece stacking scene
By processing point cloud data using the PointNet++ network, the problems of low target recognition rate and long segmentation time in workpiece stacking scenarios are solved, realizing fast and accurate workpiece pose detection and supporting efficient sorting by robotic arms.
Patent Information
- Application Number
- CN202511142650.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-15
- Publication Date
- 2025-12-12
AI Technical Summary
Existing technologies suffer from low target recognition rates and long instance segmentation times in workpiece stacking scenarios, making it difficult to meet the real-time requirements of automated sorting.
A point cloud processing method based on PointNet++ network is adopted, including point cloud filtering, denoising, downsampling, feature extraction, semantic segmentation and center point prediction, to generate high-quality point cloud features to calculate homogeneous transformation matrix to describe workpiece pose.
It enables rapid identification of workpiece pose without sacrificing accuracy, ensuring stable gripping by the robotic arm and improving sorting efficiency and accuracy.
Smart Images
Figure CN121120771A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of visual target detection, and particularly relates to a fast pose detection method based on a workpiece stacking scene. BACKGROUND
[0002] At present, many instance segmentation and pose detection methods have been proposed for workpiece stacking scenes, mainly including traditional two-dimensional methods based on RGB images, three-dimensional grid-based methods, voxel-based methods, and three-dimensional point cloud-based methods. However, these methods have obvious limitations in industrial stacking scenes: on the one hand, in the face of serious part occlusion and blurred instance boundary, their recognition rate is often low; on the other hand, they perform poorly in the capture ability of local geometric details and the robustness of feature expression. In addition, these methods consume a lot of time in the instance segmentation process, and often cannot meet the real-time requirements of automatic sorting. SUMMARY
[0003] The technical problem to be solved by the application is how to realize fast recognition of the pose of a target in a workpiece stacking scene.
[0004] To achieve the above purpose, the technical scheme of the application is as follows:
[0005] In a first aspect, the application provides a fast pose detection method based on a workpiece stacking scene, comprising the following steps:
[0006] Obtaining a three-dimensional image of a target workpiece, converting the three-dimensional image into an original point cloud image, and pre-processing the original point cloud image to generate a point cloud image of the target workpiece;
[0007] Extracting features from the point cloud image to output geometric features and depth features of the target workpiece;
[0008] Performing instance segmentation on the geometric features and depth features of the target workpiece to output high-quality point cloud features;
[0009] Calculating a homogeneous transformation matrix of the target workpiece according to the output high-quality point cloud features, and describing the pose of the target workpiece according to the homogeneous transformation matrix.
[0010] The preprocessing includes:
[0011] Filtering the original point cloud image based on a pass-through filtering technique to remove irrelevant noise parts in the original point cloud image;
[0012] Denoising the original point cloud image after filtering based on a statistical outlier removal algorithm;
[0013] Sampling the original point cloud image after denoising based on a farthest point downsampling to output a point cloud image of a preset size.
[0014] The method for feature extraction of the point cloud image comprises:
[0015] inputting the preprocessed point cloud image into a pre-constructed PointNet++ network for feature extraction;
[0016] wherein the PointNet++ network comprises cascaded feature extraction layers and reverse residual layers, and a channel attention module is embedded in the feature extraction layers.
[0017] The instance segmentation comprises:
[0018] inputting the geometric features and the depth features of the target workpiece into a semantic segmentation branch for semantic segmentation, and outputting a semantic perception feature matrix;
[0019] inputting the semantic perception feature matrix into a point pair similarity branch and a center point prediction branch respectively, converting the semantic perception feature matrix into a point pair similarity distance vector through feature embedding in the point pair similarity branch, and inputting the point pair similarity distance vector into the center point prediction branch;
[0020] The center point prediction branch performs instance segmentation based on the input semantic perception feature matrix and the point pair similarity distance vector, and outputs corresponding high-quality point cloud data.
[0021] The semantic segmentation comprises: mapping and converting the geometric features and the depth features into a semantic probability distribution through a multi-layer perception machine, and generating a corresponding semantic perception feature matrix according to the semantic probability distribution;
[0022] The geometric features and the depth features are abstract feature vectors.
[0023] The method for instance segmentation of the center point prediction branch based on the input semantic perception feature matrix and the point pair similarity distance vector comprises:
[0024] calculating a center point score of each point in the point cloud image according to the semantic perception feature matrix, screening each point according to the center point score through a non-maximum suppression operation, taking the screened points as initial centers, describing distances according to the point pair similarity distance vector, and performing instance segmentation based on a K-Means clustering method.
[0025] The high-quality point cloud features are converted into a displacement vector and a rotation matrix through two independent multi-layer perception machines, and a homogeneous transformation matrix representing the pose of the workpiece is generated according to the displacement vector and the rotation matrix.
[0026] In a second aspect, the present application provides a fast pose detection device based on a workpiece stacking scene, comprising:
[0027] The point cloud acquisition module is configured to acquire a three-dimensional image of the target workpiece, convert the three-dimensional image into an original point cloud image, and generate a point cloud image of the target workpiece by preprocessing the original point cloud image.
[0028] The feature extraction module is configured to perform feature extraction on the point cloud image and output geometric features and depth features of the target workpiece.
[0029] The instance segmentation module is configured to perform instance segmentation on the geometric features and the depth features of the target workpiece and output high-quality point cloud features.
[0030] The pose detection module is configured to calculate a homogeneous transformation matrix of the target workpiece according to the output high-quality point cloud features and describe the pose of the target workpiece according to the homogeneous transformation matrix.
[0031] In a third aspect, a computer-readable storage medium storing one or more programs is provided, the one or more programs comprising instructions which, when executed by a computing device, cause the computing device to perform any of the above-described methods.
[0032] In a fourth aspect, a computer device is provided, comprising,
[0033] one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise instructions for performing any of the above-described methods.
[0034] The present application has the advantages that the present application outputs high-quality workpiece pose estimation results without sacrificing accuracy through single-view point cloud image data. The present application can quickly identify different types of workpieces in the presence of serious part occlusion and blurred instance boundaries in the industrial stacking scene, ensuring stable grasping and operation of the target workpiece by the mechanical arm, and achieving efficient and accurate sorting. BRIEF DESCRIPTION OF DRAWINGS
[0035] Figure 1 A flowchart of the fast pose detection method for the workpiece stacking scene of the present application is shown in the figure.
[0036] Figure 2 A preprocessing process schematic diagram is shown in the figure.
[0037] Figure 3 An instance segmentation process schematic diagram is shown in the figure.
[0038] Figure 4 A grasping point schematic diagram is shown in the figure.
[0039] Figure 5 A grasping pose schematic diagram is shown in the figure. DETAILED DESCRIPTION
[0040] The technical solutions of the present application will be described in detail below with reference to the drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solutions of the present application, and are not limitations of the technical solutions of the present application. In the case of no conflict, the technical features in the embodiments of the present application and the embodiments can be combined with each other.
[0041] Embodiment 1
[0042] Reference Figures 1-5 As shown in the figure, a fast pose detection method based on workpiece stacking scene includes the following steps:
[0043] Obtain a three-dimensional image of a target workpiece, convert the three-dimensional image into an original point cloud image, and pre-process the original point cloud image to generate a point cloud image of the target workpiece;
[0044] Feature extraction is performed on the point cloud image to output geometric features and depth features of the target workpiece;
[0045] Instance segmentation is performed on the geometric features and depth features of the target workpiece to output high-quality point cloud features;
[0046] A homogeneous transformation matrix of the target workpiece is calculated according to the output high-quality point cloud features, and the pose of the target workpiece is described according to the homogeneous transformation matrix.
[0047] The pre-processing process includes point cloud filtering, point cloud denoising, and point cloud downsampling.
[0048] Point cloud filtering: straight-through filtering technology is used for filtering processing. By setting appropriate threshold values on the point cloud coordinate axes, only point cloud images within the threshold value range are retained, thereby quickly eliminating irrelevant noise parts in the original point cloud image under the workpiece stacking scene.
[0049] In this embodiment, the point cloud coordinate system can be a world coordinate system, a Cartesian coordinate system, a camera coordinate system, or a sensor coordinate system.
[0050] Point cloud denoising: for the filtered point cloud data, a statistical outlier removal (SOR) algorithm is used. Based on the statistical distribution characteristics of the local neighborhood distance, a dynamic threshold is constructed by calculating the mean and standard deviation of the neighborhood distance of each point, and the outlier points deviating from the statistical characteristics are removed accordingly, thereby realizing accurate denoising of the original point cloud image.
[0051] Point cloud downsampling: farthest point sampling (FPS) is adopted, a point is randomly selected from the data points of the input denoised original point cloud image as the first sampling point, and it is recorded in the sampling point set; for each data point in the point cloud image, the Euclidean distance between it and all data points in the current sampling point set is calculated; the point with the maximum median value in the current distance array is selected from the remaining points, and it is added to the sampling point set; the distance array is updated so that the distance value of each unselected point remains the same as the distance to the nearest point in the sampling point set; repeat the above steps until the sampling point set reaches the preset size.
[0052] The preprocessed point cloud image is input into the pre-constructed PointNet++ network for feature extraction, and the feature vector of the target workpiece is output, including geometric features and depth features.
[0053] The PointNet++ network includes a feature extraction layer (Squeeze-and-Excitation SetAbstraction; SESA) and an inverse residual layer (InvResMLP), which can effectively improve the feature extraction capability in complex stacking scenarios.
[0054] The SESA layer replaces the original SA layer for feature extraction by embedding a channel attention module (Squeeze-and-Excitation Module; SE-Module) into each Set Abstraction (SA) layer; the InvResMLP layer is connected in series after the SESA layer, forming a cascaded structure of "feature dimension extraction-deep feature refinement".
[0055] The InvResMLP layer first expands the channel number, then reduces the calculation amount through depth separable convolution, and finally compresses back to the original channel number, while maintaining the performance of the model and reducing the computational complexity.
[0056] The instance segmentation process includes: inputting the feature vector into the semantic segmentation branch for semantic segmentation, and outputting a semantic perception feature matrix; inputting the semantic perception feature matrix into the point pair similarity branch and the center point prediction branch respectively, the point pair similarity branch converts the semantic perception feature matrix into a point pair similarity distance vector, so that the similarity between point pairs can be described by the distance between vectors, and the point pair similarity distance vector is input into the center point prediction branch; the center point prediction branch realizes instance segmentation of the target workpiece based on the input semantic perception feature matrix and point pair similarity distance vector, and outputs corresponding high-quality point cloud data.
[0057] The semantic segmentation branch: taking the feature vector as input, mapping the extracted features into semantic probability distribution through MLP, i.e. semantic-aware feature matrix, to provide class prior information for subsequent instance segmentation task.
[0058] The point pair similarity branch: converting the abstract features into vector form through feature embedding of the semantic-aware feature matrix, so that the similarity between point pairs can be described by the distance between vectors.
[0059] The center point prediction branch: obtaining the center point score of each point in the point cloud image according to the semantic-aware feature matrix, and screening the center point score of each point through the Non-Maximum Suppression (NMS) operation, taking the screened candidate center points as the initial center, combining the distance description of the point pair similarity distance vector, and adopting the K-Means clustering method to realize accurate instance segmentation and output high-quality point cloud features.
[0060] Converting the high-quality point cloud features into displacement vectors and rotation matrices through two independent MLPs, and generating the homogeneous transformation matrix representing the pose of the workpiece according to the displacement vectors and the rotation matrices.
[0061] The evaluation indexes of the pose detection result include Overall Accuracy (OA) and mean Intersection over Union (mIoU). It is verified through experiments that the method of the present application has achieved very good results under different test data sets, among which, for the Fraunhofer IPA Bin-Picking data set, the OA is 86.4% and the mIoU is 75.2%; for the Siléane data set, the OA is 89.1% and the mIoU is 77.4%.
[0062] Embodiment 2
[0063] A fast pose detection method based on workpiece stacking scene, comprising the following steps:
[0064] Obtaining the three-dimensional image of the workpiece through the IntelRealSense L515 camera, converting the three-dimensional image into a point cloud image, and sequentially performing filtering, denoising and downsampling processing on the point cloud image, inputting the processed point cloud image into the PointNet++ network for feature extraction, capturing the multi-level geometric semantic features of the target workpiece, performing instance segmentation on the multi-level geometric semantic features, obtaining high-quality point cloud features, calculating the homogeneous transformation matrix of the target workpiece according to the output high-quality point cloud features, and describing the pose of the target workpiece according to the homogeneous transformation matrix.
[0065] The quaternion attitude parameter and the three-dimensional displacement amount are extracted from the homogeneous transformation matrix to jointly describe the target pose of the end effector of the mechanical arm, and the highest score is taken as the optimal grasping target to be sent to the mechanical arm to complete the first grasping.
[0066] Embodiment 3
[0067] A fast pose detection device based on a workpiece stacking scene, comprising:
[0068] A point cloud acquisition module is configured to acquire a three-dimensional image of a target workpiece, convert the three-dimensional image into an original point cloud image, and generate a point cloud image of the target workpiece by preprocessing the original point cloud image.
[0069] A feature extraction module is configured to perform feature extraction on the point cloud image and output geometric features and depth features of the target workpiece.
[0070] An instance segmentation module is configured to perform instance segmentation on the geometric features and the depth features of the target workpiece and output high-quality point cloud features.
[0071] A pose detection module is configured to calculate a homogeneous transformation matrix of the target workpiece according to the output high-quality point cloud features and describe the pose of the target workpiece according to the homogeneous transformation matrix.
[0072] Embodiment 4
[0073] A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by a computing device, cause the computing device to perform any of the above-described methods.
[0074] Embodiment 5
[0075] A computer device, comprising,
[0076] One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise instructions for performing any of the above-described methods.
[0077] In summary, the present application outputs high-quality workpiece pose estimation results under the premise of not sacrificing accuracy through single-view point cloud image data. The present application can quickly identify different types of workpieces in the presence of serious part occlusion and blurred instance boundaries in the industrial stacking scene, ensure stable grasping and operation of the mechanical arm on the target workpiece, and achieve efficient and accurate sorting.
[0078] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, the methods can be tangibly embodied in a machine-readable storage medium having stored thereon instructions that can be used to program a computer to perform any of the methods. The software implementation can be initialized by loading and executing a set of instructions arranged to perform one of the methods into the computer's memory. Alternatively, hard-wired circuitry can be used in place of, or in combination with, software instructions. Thus, the
[0079] The present application is described in relation to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It is understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in one or more of the flow diagrams and / or block diagrams. Figure 1 one or more functions specified in one or more of the flow diagrams and / or block diagrams.
[0080] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in one or more of the flow diagrams and / or block diagrams. Figure 1 one or more functions specified in one or more of the flow diagrams and / or block diagrams.
[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in one or more of the flow diagrams and / or block diagrams. Figure 1 one or more functions specified in one or more of the flow diagrams and / or block diagrams.
[0082] The embodiments of the present application described above are merely illustrative and not limiting. Numerous modifications and adaptations will be apparent to those skilled in the art without departing from the spirit and scope of the present application.
Claims
1. A fast pose detection method based on workpiece stacking scenarios, characterized in that, Includes the following steps: A three-dimensional image of the target workpiece is acquired, the three-dimensional image is converted into a raw point cloud image, and the raw point cloud image is preprocessed to generate a point cloud image of the target workpiece. Feature extraction is performed on the point cloud image to output the geometric and depth features of the target workpiece; Perform instance segmentation on the geometric and depth features of the target workpiece to output high-quality point cloud features; The homogeneous transformation matrix of the target workpiece is calculated based on the high-quality point cloud features output, and the pose of the target workpiece is described based on the homogeneous transformation matrix.
2. The fast pose detection method based on workpiece stacking scenarios according to claim 1, characterized in that, The preprocessing includes: The original point cloud image is filtered using a pass-through filtering technique to remove irrelevant noise from the original point cloud image. The original point cloud image after filtering is denoised based on a statistical outlier removal algorithm. The original point cloud image after denoising is sampled based on the farthest point downsampling, and a point cloud image of a preset size is output.
3. The fast pose detection method based on workpiece stacking scenarios according to claim 1, characterized in that, The method for feature extraction from point cloud images includes: The preprocessed point cloud image is input into a pre-built PointNet++ network for feature extraction; The PointNet++ network consists of cascaded feature extraction layers and inverse residual layers, with channel attention modules embedded in the feature extraction layers.
4. The fast pose detection method based on workpiece stacking scenarios according to claim 1, characterized in that, The instance segmentation includes: The geometric and depth features of the target workpiece are input into the semantic segmentation branch for semantic segmentation, and the semantic-aware feature matrix is output. The semantically aware feature matrix is input into the point-to-point similarity branch and the center point prediction branch, respectively. The point-to-point similarity branch transforms the semantically aware feature matrix into a point-to-point similarity distance vector through feature embedding, and then inputs the point-to-point similarity distance vector into the center point prediction branch. The center point prediction branch performs instance segmentation based on the input semantically aware feature matrix and point pair similarity distance vector, and outputs the corresponding high-quality point cloud data.
5. The fast pose detection method based on workpiece stacking scenarios according to claim 4, characterized in that, The semantic segmentation includes: mapping the geometric features and depth features into a semantic probability distribution through a multilayer perceptron, and generating a corresponding semantic perceptron feature matrix based on the semantic probability distribution; The geometric and depth features are abstract feature vectors.
6. The fast pose detection method based on workpiece stacking scenarios according to claim 4, characterized in that, The method for instance segmentation based on the input semantically aware feature matrix and point pair similarity distance vector in the center point prediction branch includes: The center point score of each point in the point cloud image is calculated based on the semantically aware feature matrix. Non-maximum suppression is used to filter points according to their center point scores. The selected points are used as initial centers. The distance is described based on the point pair similarity distance vector. Instance segmentation is performed based on the K-Means clustering method.
7. The fast pose detection method based on workpiece stacking scenarios according to claim 1, characterized in that, High-quality point cloud features are transformed into displacement vectors and rotation matrices through two independent multilayer perceptrons, and a homogeneous transformation matrix representing the workpiece pose is generated based on the displacement vectors and rotation matrices.
8. A rapid pose detection device based on a workpiece stacking scenario, characterized in that, include: The point cloud acquisition module is used to acquire a three-dimensional image of the target workpiece, convert the three-dimensional image into a raw point cloud image, and preprocess the raw point cloud image to generate a point cloud image of the target workpiece. The feature extraction module is used to extract features from the point cloud image and output the geometric and depth features of the target workpiece. The instance segmentation module is used to segment the geometric and depth features of the target workpiece into high-quality point cloud features. The pose detection module is used to calculate the homogeneous transformation matrix of the target workpiece based on the output high-quality point cloud features, and to describe the pose of the target workpiece based on the homogeneous transformation matrix.
9. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods of claims 1 to 7.
10. A computer device, characterized in that, include, One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any of claims 1 to 7.