Stacked workpiece grabbing method and system based on machine vision and storage medium

Abstract

According to the stacked workpiece grabbing method and system based on machine vision and the storage medium, a 3D camera, a mechanical arm, an end effector and an improved target detection and point cloud registration algorithm are combined, and by collecting and processing images and point cloud data, high-precision target recognition and positioning are achieved, the grabbing path is optimized, the efficiency is improved, and the working efficiency is improved. The problems that a traditional grabbing system is low in detection precision, insufficient in point cloud registration precision, low in grabbing efficiency and the like in a complex workpiece stacking scene are effectively solved. The system aims at improving the industrial production efficiency, reducing the labor cost and achieving efficient and intelligent automatic grabbing.

Description

Technical Field

[0001] This invention relates to the field of industrial robot automation technology, specifically to an intelligent grasping system and method for stacked workpieces based on multimodal perception, which is particularly suitable for rapid identification, accurate positioning and stable grasping of multi-target stacked workpieces in complex industrial scenarios. Background Technology

[0002] With the rapid development of industrial automation, the application of robotics technology in manufacturing is becoming increasingly widespread, especially in the fields of high-precision grasping and intelligent production. For example, the published document with authorization announcement number CN116330245B, entitled "An Automated Hand-Eye Calibration Device and Method for Industrial Vision Robots," discloses an automated hand-eye calibration device for industrial vision robots, including a base plate. A slide is fixedly mounted on the top of the base plate, a pulley is rotatably connected to the top of the inner wall of the slide, an electromagnet is fixedly mounted on the bottom of the inner wall of the slide, a slider is slidably sleeved on the inner wall of the slide, a panel is fixedly mounted on the top of the slider, a bracket is fixedly mounted on one side of the top of the panel, and a fixing frame is fixedly mounted on the other side of the top of the panel. By sliding the slider on the inner wall of the carriage, the electromagnet resists the magnetic force of the slider, allowing the panel to move by controlling the magnitude and direction of the electromagnet's magnetic force through the control components. By connecting the panel to the carriage via the slider, the equipment can move via the electromagnet when gripping and scanning workpieces, thereby enabling the equipment to move over a wide range and transport workpieces.

[0003] The core of the intelligent control described in the aforementioned public documents utilizes a vision system for identification and judgment. Therefore, machine vision, as a key technology, provides industrial robots with workpiece positioning and identification information through image acquisition and processing, thereby achieving automated grasping. In actual industrial environments, workpieces are often present in a complex and disordered stacked state, which places higher demands on the accuracy and efficiency of the robot grasping system. The core technologies of the robot grasping system include target detection, point cloud registration, and robotic arm control, which are crucial in the workpiece identification, positioning, and grasping process. In recent years, the application of deep learning algorithms in target detection and pose estimation has significantly improved system performance; however, in complex stacked scenarios, challenges remain, such as insufficient detection accuracy, low registration efficiency, and insufficient flexibility in robotic arm control.

[0004] Disadvantages of existing technology:

[0005] Disadvantage 1: Insufficient accuracy of target detection in complex stacked scenarios. In complex and disordered stacked workpiece scenarios, existing target detection algorithms often have difficulty accurately identifying target objects.

[0006] Disadvantage 2: The accuracy and speed of pose estimation algorithms cannot meet industrial needs. Existing pose estimation algorithms often face the dual challenges of accuracy and efficiency when processing complex point cloud data.

[0007] Disadvantage 3: The robotic arm control lacks flexibility and real-time performance. In actual grasping processes, the trajectory planning of the robotic arm and the design of the end effector often cannot adapt well to complex scenarios. Summary of the Invention

[0008] The improved target detection and point cloud registration algorithm of this invention achieves high-precision target recognition and positioning by acquiring and processing image and point cloud data, optimizes the grasping path, improves efficiency, and effectively solves the problems of low detection accuracy, insufficient point cloud registration accuracy, and low grasping efficiency of traditional grasping systems in complex stacked workpiece scenarios.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a method for grasping stacked workpieces based on machine vision, comprising the following steps:

[0010] Step 1: Initialize the hardware device;

[0011] Step 2: Data Acquisition. Collect and capture color images, depth images, and point cloud data from the scene.

[0012] Step 3: Target detection. The acquired color image is processed using the improved YOLOv5 target detection algorithm to identify the 2D pixel ROI bounding box and stacking status of the target object to be captured.

[0013] Step 4: Point cloud preprocessing and cropping. The collected point cloud data is voxelized and downsampled, and outlier denoising is performed. The topmost complete target to be captured is selected, and then the point cloud of the target to be captured within the 3D ROI bounding box of the target to be captured is obtained.

[0014] Step 5: Point cloud registration. The target point cloud to be captured is matched with the preset template point cloud using a coarse registration algorithm to improve the overlap between the two sets of point clouds.

[0015] Step 6: Perform the capture. Based on the pose estimation results of the target object, combined with the camera calibration parameters and rigid body changes, calculate the true physical position of the target object in the world coordinate system.

[0016] In step 3, the specific method for processing using the improved YOLOv5 object detection algorithm is as follows:

[0017] Based on the ShuffleNetv2 backbone network, the number of model parameters is reduced;

[0018] The Slim Neck structure of GSConv is used for feature fusion to optimize feature extraction;

[0019] Embedded SimAM attention mechanism;

[0020] Using the SIoU loss function:

[0021] in:

[0022] Intersection, union, and comparison;

[0023] : The distance between the center of the predicted bounding box and the center of the ground truth bounding box;

[0024] : The diagonal length between the predicted bounding box and the ground truth bounding box;

[0025] and Balance coefficient.

[0026] YOLOv5 is an existing technology based on the ShuffleNetv2 backbone network, reducing the number of model parameters; it uses the Slim Neck structure of GSConv for feature fusion and optimizes feature extraction; and it embeds the SimAM attention mechanism. These are all new additions, forming the new YOLOv5 model.

[0027] The ShuffleNetv2 backbone network divides the feature map into two parts: one part is directly retained, and the other part is convolved with the original feature map after convolution. This method achieves feature reuse, reduces the number of required parameters, and maintains the network's expressive power.

[0028] The convolution operation is broken down into group convolution and point convolution, thereby reducing computational complexity and the number of parameters. Group convolution reduces the number of parameters while maintaining network performance.

[0029] The method of feature fusion to optimize feature extraction involves organically combining the GSConv module and the Slim Neck structure to design a lightweight and efficient network architecture for model training, achieving superior performance while maintaining a lightweight design. Model performance is evaluated on a validation set, and the structure and parameters are adjusted based on feedback to optimize feature extraction capabilities.

[0030] SimAM aims to improve computational efficiency and model performance while reducing computational complexity and memory usage by simplifying traditional attention mechanisms. Inserting a SimAM module into the model is typically done in the feature extraction section, such as after convolutional layers, to enhance the expressive power of features.

[0031] The quality of features provided by the preceding feature extraction modules (such as ShuffleNetv2, GSConv, and SimAM) directly affects the accuracy of bounding box prediction. High-quality features help the model better locate the target, thereby reducing SIoU loss. The SimAM attention mechanism highlights key feature regions, enabling the model to focus more on important regions of the target when predicting bounding boxes, thus reducing SIoU loss.

[0032] Step 5 employs the PCA coarse registration algorithm, with the following matching formula:

[0033]

[0034] in:

[0035] and Point cloud to be registered

[0036] and : The mean of the point cloud;

[0037] and Corresponding point;

[0038] Translation vector;

[0039] and Corresponding point.

[0040] Fine registration involves inputting the coarsely registered point cloud pairs into the improved K-TEASER++ fine registration algorithm. By pruning feature correspondences, the rotation and translation matrix of the target point cloud is calculated to obtain the final pose estimate of the target object.

[0041]

[0042] in:

[0043] Rotation matrix

[0044] Translation vector

[0045] n: Represents the number of samples in the dataset.

[0046] i: is an index variable used to iterate through all samples from 1 to n.

[0047] T: Indicates transpose;

[0048] R is a matrix, and m is the number of point clouds in the sample;

[0049] In step 6, the rotation and translation matrix is ​​converted into a quaternion to adjust the attitude of the end effector;

[0050]

[0051] in:

[0052] : Quaternion scalar part;

[0053] : The unit vector of the rotation axis;

[0054] : Rotation angle.

[0055] In step 6, the robotic arm is controlled to perform a gripping action according to the planned motion trajectory to complete the workpiece gripping task. After gripping, the robotic arm transports the workpiece to the designated placement area, completing the overall process.

[0056] A machine vision-based stacked workpiece gripping system;

[0057] The target detection module is used to identify workpieces using an improved YOLOv5 algorithm.

[0058] The point cloud registration module combines PCA coarse registration and TEASER++ fine registration techniques to achieve high-precision pose estimation of the workpiece.

[0059] The gripping execution module, including a six-axis robotic arm and an end effector suction cup, is used to perform flexible gripping of workpieces;

[0060] The system control module is used to integrate the aforementioned modules to achieve intelligent gripping of stacked workpieces;

[0061] The system executes the machine vision-based stacked workpiece gripping method as described.

[0062] A storage medium, the storage medium being a computer-readable storage medium for storing software program code for executing the machine vision-based stacked workpiece gripping method.

[0063] This invention relates to an intelligent gripping system for industrial stacked workpieces based on machine vision. It combines a 3D camera, a robotic arm, an end effector, and improved target detection and point cloud registration algorithms. By acquiring and processing image and point cloud data, it achieves high-precision target recognition and positioning, optimizes the gripping path, and improves efficiency. It effectively solves the problems of low detection accuracy, insufficient point cloud registration accuracy, and low gripping efficiency of traditional gripping systems in complex stacked workpiece scenarios. This system aims to improve industrial production efficiency, reduce labor costs, and achieve efficient and intelligent automated gripping. Attached Figure Description

[0064] The following is a brief explanation of the content represented by each figure in this specification:

[0065] Figure 1 This is a hardware framework diagram of the present invention;

[0066] Figure 2 This is a flowchart of the workflow of the present invention;

[0067] Figure 3 This is a layout diagram for the present invention;

[0068] Figure 4 This is a schematic diagram of the lightweight improved algorithm of this invention;

[0069] Figure 5 This is a schematic diagram of the registration optimization algorithm of the present invention;

[0070] Figure 6 This is a flowchart of the algorithm of the present invention. Detailed Implementation

[0071] The following description, with reference to the accompanying drawings, details the specific implementation of the present invention, including the shape and structure of each component, the relative positions and connections between the parts, the function and working principle of each part, the manufacturing process, and the operation and use methods, to help those skilled in the art to have a more complete, accurate, and in-depth understanding of the inventive concept and technical solution of the present invention.

[0072] The reasons for the shortcomings of current technologies:

[0073] Reason 1: The object detection algorithm model has high complexity, making it difficult to balance between lightweight design and accuracy. Existing object detection algorithms often reduce computational complexity by reducing model parameters in lightweight design, but this may lead to a decrease in detection accuracy.

[0074] Reason 2: Pose estimation algorithms are highly dependent on the quality of the initial point cloud and the environment. The performance of pose estimation algorithms depends to a large extent on the quality of the point cloud data and environmental conditions.

[0075] Reason 3: The robotic arm control algorithm is out of touch with the actual needs of the scenario. The robotic arm control algorithm usually prioritizes safety and ignores the optimization of efficiency.

[0076] Based on the above reasons, the present invention provides a machine vision-based method, system, and storage medium for grasping stacked workpieces.

[0077] The machine vision-based intelligent gripping system for stacked industrial workpieces includes:

[0078] The target detection module is used to achieve high-precision identification of workpieces through an improved YOLOv5 algorithm. The target detection module uses ShuffleNetv2 as the backbone network and reduces model complexity through lightweight design. It adopts the SlimNeck structure of GSConv for feature fusion and embeds the SimAM attention mechanism to improve detection accuracy. The SIoU loss function is used to optimize the prediction results and improve the prediction accuracy.

[0079] Point cloud registration module, combined with PCA

[0080] The coarse registration and TEASER++ fine registration techniques achieve high-precision pose estimation of the workpiece. The point cloud registration module preprocesses the point cloud data through voxelization downsampling and outlier denoising, uses PCA coarse registration to improve the initial overlap of the point cloud, and achieves precise registration through the TEASER++ algorithm. The K-TEASER++ algorithm is introduced to optimize the feature correspondence in the registration process, improving the accuracy and robustness of the registration. The point cloud registration module uses the FPFH algorithm to extract features from two sets of point cloud data. The KCP feature correspondence pruning method is used to select and prune the correct feature correspondence. The relative transformation is calculated through the optimization problem in TEASER++ to achieve accurate registration results.

[0081] The gripping execution module, including a six-axis robotic arm and an end effector suction cup actuator, is used to flexibly grip the workpiece. The gripping execution module includes a six-axis robotic arm with a working radius of 1100mm and a repeatability of ±0.05mm. The end effector suction cup actuator can adapt to the gripping requirements of both forward and reverse positions of the workpiece. The software platform enables the design and optimization of the gripping process.

[0082] The system control module integrates the aforementioned modules to achieve intelligent grasping of stacked workpieces. It embeds improved target detection and point cloud registration algorithms into the software to achieve overall system control. Through hardware and software communication, it enables efficient system collaboration to complete the intelligent grasping task of stacked workpieces. The system control module adjusts camera parameters to acquire high-quality images and point cloud data. By adding corresponding script modules to the software, it achieves efficient system collaboration and process optimization.

[0083] A machine vision-based intelligent gripping system for stacked industrial workpieces and its implementation method include the following steps:

[0084] S1 Initialize hardware devices

[0085] S2 Data Acquisition: The 3D camera captures color images, depth images, and point cloud data of the scene. This data provides the foundation for subsequent object detection and point cloud registration.

[0086] S3 Object Detection: The acquired color image is processed using the improved YOLOv5 object detection algorithm to identify the 2D pixel ROI bounding box and stacking status of the target object to be captured.

[0087] Based on the ShuffleNetv2 backbone network, the number of model parameters is reduced and the detection speed is improved.

[0088] We employ the Slim Neck structure of GSConv for feature fusion and optimize feature extraction.

[0089] Embedding the SimAM attention mechanism improves detection accuracy.

[0090] Using the SIoU loss function optimizes prediction results and improves accuracy.

[0091]

[0092] in:

[0093] Intersection and Union

[0094] Distance between the center of the predicted bounding box and the center of the ground truth bounding box

[0095] : Diagonal length between the predicted bounding box and the ground truth bounding box

[0096] and Balance coefficient

[0097] S4 point cloud preprocessing and cropping:

[0098] The collected point cloud data is voxelized downsampled and outlier denoising is performed to remove cluttered point cloud data and improve data quality.

[0099] By combining the 2D bounding box and depth information in the target detection results, the topmost and most complete target is selected, and the point cloud clipping method proposed in this paper is used to obtain the point cloud data within the 3D ROI bounding box of the target to be captured.

[0100] S5 Point Cloud Registration: Coarse registration, which uses the PCA coarse registration algorithm to improve the overlap between the pre-processed target point cloud to be captured and the template point cloud extracted in advance through CAD, laying the foundation for subsequent fine registration.

[0101]

[0102] in:

[0103] and Point cloud to be registered

[0104] and Mean value of point cloud

[0105] Fine registration involves inputting the coarsely registered point cloud pairs into the improved K-TEASER++ fine registration algorithm. By pruning the feature correspondences, the rotation and translation matrix of the target point cloud is calculated to obtain the final pose estimate of the target object.

[0106]

[0107] in:

[0108] Rotation matrix

[0109] Translation vector

[0110] and Corresponding point

[0111] S6 Grab Execution: Based on the pose estimation results of the target object, combined with camera calibration parameters and rigid body changes, calculate the true physical position of the target object in the world coordinate system.

[0112] Convert the rotation and translation matrix into quaternions to adjust the attitude of the end effector.

[0113]

[0114] in:

[0115] Quaternion scalar part

[0116] The unit vector of the rotation axis

[0117] Rotation angle

[0118] The robotic arm is controlled to perform gripping actions according to the planned motion trajectory to complete the task of gripping the workpiece.

[0119] After the gripping is completed, the robotic arm moves the workpiece to the designated placement area, completing the entire process.

[0120] This invention relates to an improved YOLOv5 algorithm-based target detection module: using ShuffleNetv2 as the backbone network and reducing model complexity through lightweight design. It employs the Slim Neck structure of GSConv for feature fusion and embeds the SimAM attention mechanism to improve detection accuracy. The SIoU loss function is used to optimize prediction results and improve prediction accuracy. Especially in scenarios with complex stacked workpieces, it significantly reduces false negatives and false positives, achieving high-precision target recognition.

[0121] A point cloud registration module combining PCA coarse registration and K-TEASER++ fine registration techniques is implemented. Preprocessing of the point cloud data involves voxelization downsampling and outlier denoising. PCA coarse registration improves the initial overlap of the point cloud, while an improved K-TEASER++ fine registration algorithm achieves accurate pose estimation. The FPFH algorithm is used for feature extraction, and the KCP feature correspondence pruning method optimizes the feature correspondence during registration, improving both accuracy and robustness.

[0122] Intelligent system control module: Improved target detection and point cloud registration algorithms are embedded in the software to achieve overall system control. Efficient system collaboration is achieved through hardware and software communication to complete the intelligent grasping task of stacked workpieces. Camera parameters are adjusted to acquire high-quality images and point cloud data; efficient collaboration and process optimization are achieved by adding corresponding script modules to the software.

[0123] High-precision pose estimation and grasping execution: Based on the pose estimation results of the target object, combined with camera calibration parameters and rigid body changes, the true physical position of the target object in the world coordinate system is calculated. The rotation and translation matrices are converted into quaternions, and the attitude of the end effector is adjusted. The robotic arm is controlled to execute the grasping action according to the planned motion trajectory to complete the workpiece grasping task.

[0124] The aforementioned machine vision-based method and system for picking up stacked workpieces has the following advantages:

[0125] Target detection accuracy has been significantly improved:

[0126] By improving the YOLOv5 algorithm, combining the ShuffleNetv2 backbone network, the Slim Neck structure of GSConv, the SimAM attention mechanism, and the SIoU loss function, higher detection accuracy and lower false negative and false positive rates are achieved.

[0127] In complex stacked workpiece scenarios, it can accurately identify the 2D pixel ROI bounding box of the target object and its stacking status, ensuring the accuracy and reliability of the detection results.

[0128] Enhanced point cloud registration accuracy and robustness:

[0129] Using PCA coarse registration and K-TEASER++ fine registration techniques, the source point cloud and the target point cloud are accurately registered by voxel downsampling and outlier denoising of the point cloud data.

[0130] The KCP feature correspondence trimming method is introduced to optimize the feature correspondence in the registration process, improve the registration accuracy and robustness, and adapt to the registration needs of workpieces with different shapes and complex environments.

[0131] Significantly improved crawling efficiency:

[0132] The robotic arm has a working radius of 1100mm and a repeatability of ±0.05mm. Combined with an end effector, it can flexibly and efficiently complete the gripping tasks of different workpieces.

[0133] The system control module enables efficient collaboration among various modules, optimizes the crawling process, and significantly improves crawling efficiency.

[0134] Enhanced system flexibility and adaptability:

[0135] The system can adapt to the needs of grasping workpieces in both forward and reverse states, and is suitable for workpieces of different shapes and sizes, with high versatility and adaptability.

[0136] The software platform enabled the design and optimization of the data crawling process, improving the system's flexibility and configurability.

[0137] Efficient integration of system control modules:

[0138] The system control module achieves efficient collaboration among modules through hardware and software communication, ensuring overall system control and process optimization, and improving system response speed and coordination.

[0139] By adding corresponding script modules to the software, the system's efficient collaboration and processes were further optimized, ensuring the overall system's stability and efficiency.

[0140] High-precision pose estimation and grasping execution:

[0141] By using PCA coarse registration and the improved K-TEASER++ fine registration algorithm, high-precision pose estimation of the target object is achieved, providing accurate reference information for grasping execution.

[0142] Based on the pose estimation results, the grasping execution module precisely adjusts the movement trajectory of the robotic arm, ensuring the efficient and accurate completion of the grasping task.

[0143] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A method for grasping stacked workpieces based on machine vision, characterized in that, Includes the following steps: Step 1: Initialize the hardware device; Step 2: Data Acquisition. Collect and capture color images, depth images, and point cloud data from the scene. Step 3: Target detection. The acquired color image is processed using the improved YOLOv5 target detection algorithm to identify the 2D pixel ROI bounding box and stacking status of the target object to be captured. Step 4: Point cloud preprocessing and cropping. The collected point cloud data is voxelized and downsampled, and outlier denoising is performed. The topmost complete target to be captured is selected, and then the point cloud of the target to be captured within the 3D ROI bounding box of the target to be captured is obtained. Step 5: Point cloud registration. The target point cloud to be captured is registered with the preset template point cloud using a coarse registration algorithm to improve the overlap between the two sets of point clouds. Step 6: Perform the capture. Based on the pose estimation results of the target object, combined with the camera calibration parameters and rigid body changes, calculate the true physical position of the target object in the world coordinate system.

2. The machine vision-based method for grasping stacked workpieces according to claim 1, characterized in that: In step 3, the specific method for processing using the improved YOLOv5 object detection algorithm is as follows: Based on the ShuffleNetv2 backbone network, the number of model parameters is reduced; The Slim Neck structure of GSConv is used for feature fusion to optimize feature extraction; Embedded SimAM attention mechanism; Using the SIoU loss function: ; in: Intersection, union, and comparison; : The distance between the center of the predicted bounding box and the center of the ground truth bounding box; : The diagonal length between the predicted bounding box and the ground truth bounding box; and Balance coefficient.

3. The machine vision-based method for grasping stacked workpieces according to claim 2, characterized in that: The ShuffleNetv2 backbone network divides the feature map into two parts: one part is directly retained, and the other part is convolved with the original feature map after convolution operation.

4. The machine vision-based method for grasping stacked workpieces according to claim 3, characterized in that: The embedded SimAM attention mechanism simplifies the traditional attention mechanism, improves computational efficiency and model performance, reduces computational complexity and memory usage, and inserts a SimAM module into the model for feature extraction.

5. The machine vision-based method for grasping stacked workpieces according to claim 4, characterized in that: Feature fusion Optimization involves organically combining the GSConv module and the Slim Neck structure to design a lightweight and efficient network architecture for training the model. This allows the model to maintain its lightweight nature while achieving superior performance. The model's performance is then evaluated on a validation set, and the structure and parameters are adjusted based on the feedback to optimize feature extraction capabilities.

6. The machine vision-based method for grasping stacked workpieces according to any one of claims 1-5, characterized in that: Step 5 employs the PCA coarse registration algorithm, with the following matching formula: ； in: and Point cloud to be registered and : The mean of the point cloud; Fine registration involves inputting the coarsely registered point cloud pairs into the improved K-TEASER++ fine registration algorithm. By pruning feature correspondences, the rotation and translation matrix of the target point cloud is calculated to obtain the final pose estimate of the target object. ； in: Rotation matrix; Translation vector; and Corresponding point.

7. The machine vision-based method for grasping stacked workpieces according to claim 6, characterized in that: In step 6, the rotation and translation matrix is ​​converted into a quaternion to adjust the attitude of the end effector; ； in: : Quaternion scalar part; : The unit vector of the rotation axis; : Rotation angle.

8. The machine vision-based method for grasping stacked workpieces according to claim 7, characterized in that: In step 6, the robotic arm is controlled to perform a gripping action according to the planned motion trajectory to complete the workpiece gripping task. After the gripping is completed, the robotic arm transports the workpiece to the designated placement area to complete the overall process.

9. A machine vision-based stacked workpiece gripping system, characterized in that: The target detection module is used to identify workpieces using an improved YOLOv5 algorithm. The point cloud registration module combines PCA coarse registration and TEASER++ fine registration techniques to achieve high-precision pose estimation of the workpiece. The gripping execution module, including a six-axis robotic arm and an end effector suction cup, is used to perform flexible gripping of workpieces; The system control module is used to integrate the aforementioned modules to achieve intelligent gripping of stacked workpieces; The system executes the machine vision-based stacked workpiece gripping method as described in any one of claims 1-8.

10. A storage medium, said storage medium being a computer-readable storage medium for storing software program code, characterized in that: The software program code is used to execute the machine vision-based stacked workpiece grasping method as described in any one of claims 1-8.

Images