Garbage sorting method based on vision-driven robot

Abstract

The invention discloses a garbage sorting method based on a vision-driven robot, and relates to the technical field of mechanical arm target grabbing, and the garbage sorting method comprises the following steps: collecting and preprocessing RGB image data and depth image data of a to-be-detected target; a target detection frame is obtained through the GrondingDINO large model after LoRa fine adjustment is carried out on the GrondingDINO large model; obtaining a segmentation mask of a to-be-detected target through the SAM large model; inputting the RGB image, the depth image and the segmentation mask of the to-be-detected target into a GraspNet network, and speculating a final grabbing posture of the to-be-detected target; calculating an actual grabbing pose matrix of the mechanical arm; the actual grabbing pose matrix of the mechanical arm drives a mechanical arm motor, and the mechanical arm is controlled to move downwards to the to-be-detected target to complete the grabbing task. Through multi-modal fusion, LoRa fine adjustment and SAM / GraspNet cooperation, robust and accurate detection segmentation and pixel-level grabbing planning are achieved, a new category is rapidly adapted, and the grabbing success rate and the sorting efficiency are improved based on the mechanical arm pose geometric accuracy.

Description

Technical Field

[0001] This invention relates to the field of robotic arm target grasping technology, specifically to a garbage sorting method based on vision-driven robots. Background Technology

[0002] In recent years, the rapid development of artificial intelligence and robotics has driven the widespread application of robotic arms in manufacturing, warehousing, and service industries. Particularly in target grasping, solutions combining visual perception and deep learning have significantly improved the recognition, positioning, and grasping accuracy of robotic arms, enabling them to play a crucial role in achieving automated sorting and intelligent operation.

[0003] The existing technology has the following shortcomings:

[0004] (1) Existing detection and segmentation technologies are not robust enough in complex lighting, occlusion, or backgrounds with similar colors, leading to capture failure;

[0005] (2) Target grasping and detection models usually require a large amount of labeled data and high-cost fine-tuning, making it difficult to quickly adapt to new object categories;

[0006] (3) The lack of tight coupling between pixel-level segmentation and grasping decision results in insufficient utilization of the target's true boundary and 3D information by the grasping strategy;

[0007] (4) When the coordinate transformation link from the visual result to the motion control of the robotic arm is imperfect, geometric errors will be introduced, affecting the grasping accuracy.

[0008] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0009] The purpose of this invention is to provide a waste sorting method based on vision-driven robots to solve the problems mentioned in the background art.

[0010] To achieve the above objectives, the present invention provides the following technical solution: a waste sorting method based on a vision-driven robot, specifically comprising:

[0011] Acquire and preprocess RGB image data and depth image data of the target under test;

[0012] Extract RGB image features and depth image features and input them into the GroundingDINO large model after LoRa fine-tuning to obtain the target detection box;

[0013] The target detection bounding box is used as a prompt input and passed to the SAM large model to obtain the segmentation mask of the target to be tested;

[0014] The RGB image, depth image, and segmentation mask of the target are input into the GraspNet network to infer the final grasping pose of the target.

[0015] The actual grasping pose matrix of the robotic arm is calculated based on the pose matrix from the robotic arm base to the robotic arm end effector, the transformation matrix between the camera coordinate system and the image coordinate system, and the transformation matrix between the robotic arm end effector coordinate system and the camera coordinate system.

[0016] The actual grasping pose matrix of the robotic arm drives the robotic arm motor, controlling the robotic arm to move downward to the target to complete the grasping task.

[0017] As a preferred embodiment of the vision-driven robot-based waste sorting method described in this invention, wherein:

[0018] Select an RGB-D camera, mount it to the robot, and connect it to the robot control center via USB.

[0019] Initialize all devices and check if all devices are properly connected;

[0020] The specific parameters for setting the RGB-D camera include a resolution of 1280x720, a frame rate of 30FPS, and a synchronous acquisition mode for RGB image data and depth image data.

[0021] Activate the RGB-D camera via API call to acquire RGB image data and depth image data in real time in the form of continuous frames;

[0022] Align the depth image data with the RGB image data; if they are not aligned, adjust the depth-RGB mapping using the camera's calibration parameters.

[0023] The RGB image data and depth image data are filtered by median filtering, the RGB image is color corrected, and the RGB image data and depth image data are saved to the image buffer.

[0024] As a preferred embodiment of the vision-driven robot-based waste sorting method described in this invention, wherein:

[0025] Extract RGB image features and depth image features, extract text prompt features based on image features, and perform feature enhancement through a self-attention mechanism;

[0026] Image features and text cue features are fused through cross-modal attention mechanisms between images and text and between text and images;

[0027] By guiding query selection through language, the similarity between image features and text prompt features is calculated, and the top n features with the highest similarity are selected as the query vector.

[0028] The query vector is used as input, and cross-modal attention is used to calculate image features and text prompt features in sequence, and the feature fusion representation is updated.

[0029] The LoRa fine-tuning specifically includes...

[0030] Load the pre-trained weight file for the GroundingDINO large model;

[0031] Insert the LoRa adapter into the weight matrix of the multi-head attention layer;

[0032] The LoRa adapter is used to synthesize two low-rank matrices to replace the original weight matrix, freezing the original weights of the GroundingDINO large model, and only fine-tuning the weights of the LoRa adapter.

[0033] The acquisition of the target detection box specifically includes,

[0034] The feature fusion representation is input into the GroundingDINO model after LoRa fine-tuning for prediction and inference, and the output is the coordinates of the detection box of the target, the target confidence score of each detection box and the corresponding class label.

[0035] The target confidence score threshold is defined as 0.5. The target confidence scores of each candidate detection box are sorted by non-maximum suppression, and the overlap between each detection box and each other candidate detection box is calculated. If the overlap between two candidate boxes exceeds the confidence threshold, the detection box with the relatively low confidence score is removed.

[0036] As a preferred embodiment of the vision-driven robot-based waste sorting method described in this invention, wherein:

[0037] The image data of the target under test is cropped based on the RGB image data and depth image data of the target detection box, and only the image area within the target detection box is retained;

[0038] The image region is passed as a cue to the SAM model and a segmentation mask is generated, outputting the pixel-level segmentation of the target to be tested;

[0039] As a preferred embodiment of the vision-driven robot-based waste sorting method described in this invention, wherein:

[0040] Based on the segmentation mask of the target, the grasping posture of the target is inferred from the different points on the surface of the target.

[0041] Feature maps of the RGB and depth image data of the target under test are obtained through a CNN model;

[0042] Each pixel corresponds to a position on the surface of the target object;

[0043] The feature map is used to determine the suitable area to be grasped on the surface of the target object.

[0044] The judgment specifically includes using a confidence map of the feature map output by the GraspNet network, where each point represents the probability value of grasping on the surface of the target to be tested;

[0045] The point with the highest probability value is selected as the final crawling point;

[0046] The angle space is discretized by using the GraspNet network, which discretizes the continuous angle range, outputs the confidence score corresponding to each discrete angle, and selects the angle with the highest confidence score as the final grasping angle.

[0047] The final grasping posture is determined by the final grasping point and final grasping angle that do not overlap.

[0048] As a preferred embodiment of the vision-driven robot-based waste sorting method described in this invention, wherein:

[0049] An image coordinate system is constructed based on the RGB image data and depth image data of the target to be tested;

[0050] Construct a coordinate system for the robot arm model based on the DH parameters of the robot arm, and calculate the pose matrix from the robot arm base to the end effector of the robot arm through forward kinematics.

[0051] The coordinate distribution of the camera coordinate system is represented as: X:Right, Y:Down, Z:Forward;

[0052] The coordinate distribution of the robotic arm model coordinate system is represented as: X: Down, Y: Left, Z: Forward;

[0053] Based on the coordinate distribution described above, the transformation matrix between the robotic arm's end effector coordinate system and the camera coordinate system, around the X, Y, and Z coordinates, is: ,pass Calculate the transformation matrix between the robot arm's end effector coordinate system and the camera coordinate system;

[0054] Based on the coordinate distribution described above, the transformation matrix between the image coordinate system and the camera coordinate system, around the Euler angles of X, Y, and Z, is... ,pass Calculate the transformation matrix between the image coordinate system and the camera coordinate system;

[0055] The actual grasping pose matrix of the robotic arm is obtained by multiplying the pose matrix from the base of the robotic arm to the end effector, the transformation matrix between the end effector coordinate system and the camera coordinate system, and the transformation matrix between the camera coordinate system and the image coordinate system.

[0056] As a preferred embodiment of the vision-driven robot-based waste sorting method described in this invention, wherein:

[0057] Before grasping, the position of the robotic arm is set directly above the final grasping point. Based on the actual grasping pose matrix of the robotic arm, the robotic arm is driven by the robotic arm motor to move downward to the final grasping point and adjust the final grasping angle to complete the grasping. After the grasping is completed, the target is placed in the target cylinder and returned to the initial position to wait for subsequent grasping tasks.

[0058] On the other hand, the present invention provides an electronic device and a robot chassis. The electronic device includes an RGB-D camera, an industrial control computer, and a robot chassis. The RGB-D camera is used to acquire real-time image and video data. The industrial control computer is used to build a program running environment and perform computational operations, including an internal graphics card for algorithm inference and processing. The robot chassis is used to patrol, detect targets to be grasped, move to a certain distance in front of the target, and wait to complete the target grasping task. The computer program, when executed by a processor, implements the steps of the vision-driven robot-based garbage sorting method described above.

[0059] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements the steps of the vision-driven robot-based waste sorting method as described above in the present invention.

[0060] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0061] (1) GroundingDINO, which is jointly extracted from RGB and depth features and finely tuned by LoRa, is used for detection. The detection box is used as a prompt to call SAM to obtain a pixel-level segmentation mask. Multimodal fusion can maintain high target localization and segmentation accuracy even under conditions of light change, similar color or occlusion, thereby improving the reliability and robustness of detection and segmentation in complex scenes.

[0062] (2) LoRa fine-tuning GroundingDINO is used to train a small number of adapter parameters to quickly adapt to new scenes / new target categories, ensuring the original capabilities of the large model while significantly reducing training costs and deployment difficulty, which is more economical and efficient than traditional full fine-tuning or training from scratch.

[0063] (3) The pixel-level segmentation mask generated by SAM is combined with RGB-D information and input into GraspNet so that the grab candidate point and angle judgment are based on the real boundary and depth information of the target, reducing false grabs, collisions and repeated attempts, and significantly improving the single grab success rate and overall sorting throughput.

[0064] (4) Based on the segmentation mask, pixel-level candidate points are established on the target surface. The confidence map output by GraspNet and the discretized angle confidence are used to select the final grasping point and angle. This can simultaneously consider the surface graspability and posture stability, reduce the grasping failure rate and improve the repeatability of the grasping action.

[0065] (5) The end-to-end coordinate transformation link ensures the geometric accuracy of the robotic arm's grasping posture and reduces grasping deviation caused by coordinate error. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0067] Figure 1 This is a flowchart of the waste sorting method based on a vision-driven robot according to the present invention.

[0068] Tables 1, 2, 3, and 4 are simulation experiment results data tables for the vision-driven robot-based garbage sorting method of the present invention under different scenarios. Detailed Implementation

[0069] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0070] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a waste sorting method based on a vision-driven robot.

[0071] Acquire and preprocess RGB image data and depth image data of the target under test;

[0072] Select an RGB-D camera, mount it to the robot, and connect it to the robot control center via USB.

[0073] Initialize all devices and check if all devices are properly connected;

[0074] The specific parameters for setting the RGB-D camera include a resolution of 1280x720, a frame rate of 30FPS, and a synchronous acquisition mode for RGB image data and depth image data.

[0075] Activate the RGB-D camera via API call to acquire RGB image data and depth image data in real time in the form of continuous frames;

[0076] Align the depth image data with the RGB image data; if they are not aligned, adjust the depth-RGB mapping using the camera's calibration parameters.

[0077] The RGB image data and depth image data are filtered by median filtering, the RGB image is color corrected, and the RGB image data and depth image data are saved to the image buffer.

[0078] Extract RGB image features and depth image features and input them into the GroundingDINO large model after LoRa fine-tuning to obtain the target detection box;

[0079] Extract RGB image features and depth image features, extract text prompt features based on image features, and perform feature enhancement through a self-attention mechanism;

[0080] Image features and text cue features are fused through cross-modal attention mechanisms between images and text and between text and images;

[0081] By guiding query selection through language, the similarity between image features and text prompt features is calculated, and the top n features with the highest similarity are selected as the query vector.

[0082] The query vector is used as input, and cross-modal attention is used to calculate image features and text prompt features in sequence, and the feature fusion representation is updated.

[0083] The LoRa fine-tuning specifically includes...

[0084] Load the pre-trained weight file for the GroundingDINO large model;

[0085] Insert the LoRa adapter into the weight matrix of the multi-head attention layer;

[0086] The LoRa adapter is used to synthesize two low-rank matrices to replace the original weight matrix, freezing the original weights of the GroundingDINO large model, and only fine-tuning the weights of the LoRa adapter.

[0087] The acquisition of the target detection box specifically includes,

[0088] The feature fusion representation is input into the GroundingDINO model after LoRa fine-tuning for prediction and inference, and the output is the coordinates of the detection box of the target, the target confidence score of each detection box and the corresponding class label.

[0089] The target confidence score threshold is defined as 0.5. The target confidence scores of each candidate detection box are sorted by non-maximum suppression, and the overlap between each detection box and each other candidate detection box is calculated. If the overlap between two candidate boxes exceeds the confidence threshold, the detection box with the relatively low confidence score is removed.

[0090] The target detection bounding box is used as a prompt input and passed to the SAM large model to obtain the segmentation mask of the target to be tested;

[0091] The image data of the target under test is cropped based on the RGB image data and depth image data of the target detection box, and only the image area within the target detection box is retained;

[0092] The image region is passed as a cue to the SAM model and a segmentation mask is generated, outputting the pixel-level segmentation of the target to be tested;

[0093] It should also be noted that the mask is a binary image of the same size as the image data, where the image region with a value of 1 represents the segmented region of the target to be tested, and the image region with a value of 0 represents the background.

[0094] The RGB image, depth image, and segmentation mask of the target are input into the GraspNet network to infer the final grasping pose of the target.

[0095] Based on the segmentation mask of the target, the grasping posture of the target is inferred from the different points on the surface of the target.

[0096] Feature maps of the RGB and depth image data of the target under test are obtained through a CNN model;

[0097] Each pixel corresponds to a position on the surface of the target object;

[0098] The feature map is used to determine the suitable area to be grasped on the surface of the target object.

[0099] The judgment specifically includes using a confidence map of the feature map output by the GraspNet network, where each point represents the probability value of grasping on the surface of the target to be tested;

[0100] The point with the highest probability value is selected as the final crawling point;

[0101] The angle space is discretized by using the GraspNet network, which discretizes the continuous angle range, outputs the confidence score corresponding to each discrete angle, and selects the angle with the highest confidence score as the final grasping angle.

[0102] The final grasping posture is determined by the final grasping point and final grasping angle that do not overlap.

[0103] The actual grasping pose matrix of the robotic arm is calculated based on the pose matrix from the robotic arm base to the robotic arm end effector, the transformation matrix between the camera coordinate system and the image coordinate system, and the transformation matrix between the robotic arm end effector coordinate system and the camera coordinate system.

[0104] An image coordinate system is constructed based on the RGB image data and depth image data of the target to be tested;

[0105] Construct a coordinate system for the robot arm model based on the DH parameters of the robot arm, and calculate the pose matrix from the robot arm base to the end effector of the robot arm through forward kinematics.

[0106] The coordinate distribution of the camera coordinate system is represented as: X:Right, Y:Down, Z:Forward;

[0107] The coordinate distribution of the robotic arm model coordinate system is represented as: X: Down, Y: Left, Z: Forward;

[0108] Based on the coordinate distribution described above, the transformation matrix between the robotic arm's end effector coordinate system and the camera coordinate system, around the X, Y, and Z coordinates, is: ,pass Calculate the transformation matrix between the robot arm's end effector coordinate system and the camera coordinate system;

[0109] Based on the coordinate distribution described above, the transformation matrix between the image coordinate system and the camera coordinate system, around the Euler angles of X, Y, and Z, is... ,pass Calculate the transformation matrix between the image coordinate system and the camera coordinate system;

[0110] The actual grasping pose matrix of the robotic arm is obtained by multiplying the pose matrix from the base of the robotic arm to the end effector, the transformation matrix between the coordinate system of the end effector and the camera coordinate system, and the transformation matrix between the camera coordinate system and the image coordinate system.

[0111] It should be further noted that the transformation matrix between the image coordinate system and the camera coordinate system specifically includes,

[0112] ;

[0113] ;

[0114] Translation matrix ;

[0115] Transformation matrix between image coordinate system and camera coordinate system for:

[0116] = ;

[0117] It should be further noted that the transformation matrix between the image coordinate system and the camera coordinate system specifically includes,

[0118] = ;

[0119] = ;

[0120] Translation matrix ;

[0121] Transformation matrix between image coordinate system and camera coordinate system for:

[0122] = = ;

[0123] The actual grasping pose matrix of the robotic arm drives the robotic arm motor, controlling the robotic arm to move downward to the target to complete the grasping task;

[0124] Before grasping, the position of the robotic arm is set directly above the final grasping point. Based on the actual grasping pose matrix of the robotic arm, the robotic arm is driven by the robotic arm motor to move downward to the final grasping point and adjust the final grasping angle to complete the grasping. After the grasping is completed, the target is placed in the target cylinder and returned to the initial position to wait for subsequent grasping tasks.

[0125] Example 2

[0126] The following is another embodiment of the present invention, which provides a waste sorting method based on a vision-driven robot. In order to verify the beneficial effects of the present invention, a simulation experiment is conducted for scientific demonstration.

[0127] The robot includes a six-degree-of-freedom robotic arm (including an end effector and gripper) and supports joint control, collision detection, force and torque output;

[0128] Physical scenario: A waste sorting workbench with various types of waste objects (such as plastic bottles, cardboard, metal cans, glass shards, organic matter, etc.), including obstructions, objects of similar color, and changes in lighting;

[0129] Grouped physical scenarios, specifically including:

[0130] Scene S1: Simple lighting, no occlusion;

[0131] Scene S2: Medium lighting, partial occlusion;

[0132] Scene S3: Strong light / shadow, background with similar color;

[0133] Scene S4: Random occlusion, mixed background;

[0134] The target objects to be tested are 5–8 types of waste, and there are 6–12 targets to be sorted in each scenario, which are randomly placed, have different orientations, and different levels of obstruction.

[0135] Set up the core comparison group, specifically including:

[0136] C1: RGB-D joint features + LoRa fine-tuning GroundingDINO + SAM + GraspNet (complete workflow, default settings).

[0137] C2: GroundingDINO+SAM+GraspNet with only RGB features (without using depth information);

[0138] C3: Full fine-tuning of GroundingDINO+SAM+GraspNet (comparing training cost and performance with LoRa).

[0139] C4: GroundingDINO (without LoRa, without cross-modal fusion) + SAM + GraspNet (traditional baseline).

[0140] C5: GroundingDINO+SAM segmentation stage uses manually labeled masks instead of SAM (compare the impact of pixel-level segmentation quality on crawling).

[0141] C6:GraspNet uses only RGB-D geometric information and does not introduce segmentation mask constraints;

[0142] C7: Cyclic baseline (without visual perception, relying solely on preset points of the robotic arm) is used to evaluate the gain of perception on system performance.

[0143] C8: SAM Loss Variable Evaluation: Comparing the impact of different segmentation qualities on crawling, using pure geometric constraints of coarse masks.

[0144] C9: Coordinate Transformation Error Comparison: Sensitivity of end-to-end grasping pose and success rate under calibration error simulation with moderate error assessment.

[0145] The steps of the vision-driven robot-based waste sorting method described in this invention are executed, and statistical indicators are collected, specifically including:

[0146] Single crawl success rate (success / failure sample ratio)

[0147] Average crawling time (total time from recognition to completion)

[0148] Robotic arm end-effector pose error (distance from the closest point to the real target surface, mm);

[0149] The significance of the indicators was compared between the pairwise comparison groups using a paired t-test.

[0150] The simulation results specifically include:

[0151] In scene S1 (simple lighting, no occlusion)

[0152] Core Comparison Group Single crawl success rate Average fetch time End-effector pose error C1 94% 6.2s 4.1mm / 1.2° C2 80% 6.8s 6.5mm / 2.1° C3 92% 6.4s 4.8mm / 1.3° C4 78% 7.5s 7.2mm / 2.4° C5 95% 6.6s 4.2mm / 1.1° C6 79% 6.9s 7.0mm / 2.3° C7 45% 9.0s 12.0mm / 4.5° C8 72% 7.0s 6.0mm / 2.0° C9 93% 6.3s 4.6mm / 1.3°

[0153] Table 1

[0154] In scene S2 (medium lighting, partial occlusion)

[0155] Core Comparison Group Single crawl success rate Average fetch time End-effector pose error C1 95% 6.1s 5.2mm / 1.6° C2 72% 7.4s 7.0mm / 2.2° C3 87% 6.4s 4.9mm / 1.5° C4 70% 8.0s 7.6mm / 2.5° C5 89% 6.7s 5.0mm / 1.4° C6 78% 6.9s 7.2mm / 2.3° C7 46% 9.1s 9.8mm / 3.2° C8 77% 7.1s 6.3mm / 2.0° C9 93% 6.3s 4.4mm / 1.3°

[0156] Table 2

[0157] In scene S3 (strong light / shadow, background of similar color)

[0158] Core Comparison Group Single crawl success rate Average fetch time End-effector pose error C1 92% 6.2s 3.6mm / 1.7° C2 66% 7.8s 7.2mm / 2.3° C3 85% 6.8s 4.9mm / 1.5° C4 68% 8.4s 7.1mm / 2.4° C5 90% 6.5s 4.6mm / 1.4° C6 75% 7.0s 7.4mm / 2.4° C7 50% 9.0s 9.2mm / 3.0° C8 72% 7.4s 6.6mm / 2.1° C9 93% 6.3s 4.4mm / 1.3°

[0159] Table 3

[0160] In scenario S4 (random occlusion, mixed background)

[0161] Core Comparison Group Single crawl success rate Average fetch time End-effector pose error C1 89% 6.0s 3.8mm / 1.7° C2 63% 7.5s 7.3mm / 2.2° C3 89% 6.6s 4.6mm / 1.5° C4 68% 8.2s 7.4mm / 2.5° C5 92% 6.3s 4.4mm / 1.4° C6 77% 6.9s 7.1mm / 2.2° C7 52% 9.0s 9.4mm / 3.1° C8 70% 7.6s 6.4mm / 2.1° C9 84% 6.2s 4.2mm / 1.3°

[0162] Table 4

[0163] The simulation results show that, in any scenario, the complete workflow (C1) is significantly better than C2-C9, and it has a significant improvement in detection and grasping robustness compared to pure RGB (C2).

[0164] GroundingDINO, which is jointly extracted from RGB and depth features and finely tuned by LoRa, is used for detection. The detection bounding box is used as a cue to call SAM to obtain a pixel-level segmentation mask. Multimodal fusion can maintain high target localization and segmentation accuracy even under conditions of lighting changes, similar colors, or occlusion, thereby improving the reliability and robustness of detection and segmentation in complex scenes.

[0165] By using LoRa to fine-tune GroundingDINO, a small number of adapter parameters can be trained to quickly adapt to new scenes / new target categories, ensuring the original capabilities of the large model while significantly reducing training costs and deployment difficulty. This is more economical and efficient than traditional full fine-tuning or training from scratch.

[0166] By combining the pixel-level segmentation mask generated by SAM with RGB-D information and inputting it into GraspNet, the grab candidate point and angle judgment are based on the target's true boundary and depth information, reducing false grabs, collisions and repeated attempts, and significantly improving the single grab success rate and overall sorting throughput.

[0167] By establishing pixel-level candidate points on the target surface based on segmentation masks and selecting the final grasping point and angle using the confidence map output by GraspNet and discretized angle confidence, it can simultaneously consider surface graspability and posture stability, reduce the grasping failure rate and improve the repeatability of grasping actions.

[0168] The end-to-end coordinate transformation link ensures the geometric accuracy of the robotic arm's grasping pose and reduces grasping deviations caused by coordinate errors.

[0169] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A waste sorting method based on vision-driven robots, characterized in that, Specifically, it includes: Acquire and preprocess RGB image data and depth image data of the target under test; Extract RGB image features and depth image features and input them into the GroundingDINO large model after LoRa fine-tuning to obtain the target detection box; The target detection bounding box is used as a prompt input and passed to the SAM large model to obtain the segmentation mask of the target to be tested; The RGB image, depth image, and segmentation mask of the target are input into the GraspNet network to infer the final grasping pose of the target. The actual grasping pose matrix of the robotic arm is calculated based on the pose matrix from the robotic arm base to the robotic arm end effector, the transformation matrix between the camera coordinate system and the image coordinate system, and the transformation matrix between the robotic arm end effector coordinate system and the camera coordinate system. The actual grasping pose matrix of the robotic arm drives the robotic arm motor, controlling the robotic arm to move downward to the target to complete the grasping task.

2. The waste sorting method based on a vision-driven robot according to claim 1, characterized in that: The extraction of RGB image features and depth image features specifically includes, Extract RGB image features and depth image features, extract text prompt features based on image features, and perform feature enhancement through a self-attention mechanism; Image features and text cue features are fused through cross-modal attention mechanisms between images and text and between text and images; By guiding query selection through language, the similarity between image features and text prompt features is calculated, and the top n features with the highest similarity are selected as the query vector. The query vector is used as input, and cross-modal attention is used to calculate image features and text prompt features in sequence, and the feature fusion representation is updated.

3. The waste sorting method based on a vision-driven robot according to claim 1, characterized in that: The acquisition of the target detection box specifically includes, The feature fusion representation is input into the GroundingDINO model after LoRa fine-tuning for prediction and inference, and the output is the coordinates of the detection box of the target, the target confidence score of each detection box and the corresponding class label. The target confidence score threshold is defined as 0.

5. The target confidence scores of each candidate detection box are sorted by non-maximum suppression, and the overlap between each detection box and each other candidate detection box is calculated. If the overlap between two candidate boxes exceeds the confidence threshold, the detection box with the relatively low confidence score is removed.

4. The waste sorting method based on a vision-driven robot according to claim 3, characterized in that: The acquisition of the segmentation mask of the target to be tested specifically includes, The image data of the target under test is cropped based on the RGB image data and depth image data of the target detection box, and only the image area within the target detection box is retained; The image region is passed as cue information to the SAM model to generate a segmentation mask, and the pixel-level segmentation of the target to be tested is output.

5. The waste sorting method based on a vision-driven robot according to claim 4, characterized in that: The inferred final grasping posture of the target to be measured specifically includes, Based on the segmentation mask of the target, the grasping posture of the target is inferred from the different points on the surface of the target. Feature maps of the RGB and depth image data of the target under test are obtained through a CNN model; Each pixel corresponds to a position on the surface of the target object; The feature map is used to determine the suitable area on the surface of the target object to be grasped.

6. The waste sorting method based on a vision-driven robot according to claim 1, characterized in that: The judgment specifically includes using a confidence map of the feature map output by the GraspNet network, where each point represents the probability value of grasping on the surface of the target to be tested; The point with the highest probability value is selected as the final crawling point; The angle space is discretized by using the GraspNet network, which discretizes the continuous angle range, outputs the confidence score corresponding to each discrete angle, and selects the angle with the highest confidence score as the final grasping angle. The final grasping posture is determined by the final grasping point and final grasping angle that do not overlap.

7. The waste sorting method based on a vision-driven robot according to claim 1, characterized in that: The computational robotic arm's actual grasping pose matrix specifically includes, An image coordinate system is constructed based on the RGB image data and depth image data of the target to be tested; Construct a coordinate system for the robot arm model based on the DH parameters of the robot arm, and calculate the pose matrix from the robot arm base to the end effector of the robot arm through forward kinematics. The coordinate distribution of the camera coordinate system is represented as: X:Right, Y:Down, Z:Forward; The coordinate distribution of the robotic arm model coordinate system is represented as: X: Down, Y: Left, Z: Forward; The coordinate distribution of the image coordinate system is represented as: X:Forward, Y:Right, Z:Down; Based on the coordinate distribution described above, the transformation matrix between the robotic arm's end effector coordinate system and the camera coordinate system, around the X, Y, and Z coordinates, is: ,pass Calculate the transformation matrix between the robot arm's end effector coordinate system and the camera coordinate system; Based on the coordinate distribution described above, the transformation matrix between the image coordinate system and the camera coordinate system, around the Euler angles of X, Y, and Z, is... ,pass Calculate the transformation matrix between the image coordinate system and the camera coordinate system; The actual grasping pose matrix of the robotic arm is obtained by multiplying the pose matrix from the base of the robotic arm to the end effector, the transformation matrix between the end effector coordinate system and the camera coordinate system, and the transformation matrix between the camera coordinate system and the image coordinate system.

8. An electronic device and a robot chassis, the electronic device comprising an RGB-D camera, an industrial control computer, and a robot chassis, wherein the RGB-D camera is used to acquire real-time image and video data; the industrial control computer is used to build a program running environment and perform computational operations, and includes an internal graphics card for algorithm inference and processing; the robot chassis is used to patrol and detect a target to be grasped, and after moving a certain distance in front of the target, wait to complete the target grasping task, characterized in that: When the processor executes the computer program, it implements the steps of the waste sorting method based on a vision-driven robot as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the waste sorting method based on a vision-driven robot as described in any one of claims 1 to 7.

Images