Robot 3D visual guidance grabbing method

Abstract

The invention discloses a robot 3D visual guide grabbing method, which belongs to the field of robot application, and comprises the following steps of: comprehensively scanning a working environment by using a high-resolution 3D visual sensor to obtain point cloud data of a scene; carrying out object identification on the processed point cloud data by using a deep learning algorithm; a model capable of recognizing a target object is trained, the model can distinguish objects of different types, shapes, colors and textures, point cloud data are obtained through omnibearing scanning of a high-resolution 3D vision sensor, professional preprocessing is carried out, and the working environment can be accurately perceived. The deep learning algorithm is combined with multiple segmentation methods, objects can be accurately recognized and segmented under the complex background and shielding, meanwhile, geometric models of the objects and the environment are created through a specific algorithm, a comprehensive environment map is formed, an accurate basis is provided for follow-up operation, and object characteristics, gripper capacity and multiple grabbing strategies are comprehensively considered.

Description

Technical Field

[0001] The present invention relates to the field of robot applications, and in particular to a robot 3D vision-guided grasping method. Background Art

[0002] In the field of robot applications, the use of 3D vision guidance for grasping operations is a key technology. With the development of industries such as industrial automation, logistics warehousing, and service robots, the demand for robots to accurately grasp target objects is growing. However, the actual working environment is often complex and changeable, with problems such as a variety of object types, complex backgrounds, occlusions, and dynamic obstacles. When faced with these complex situations, traditional robot grasping methods may not be able to accurately identify objects, plan reasonable paths, and grasp targets stably, resulting in low grasping efficiency and low success rate, which is difficult to meet actual application needs. Therefore, there is a need for a robot 3D vision-guided grasping method that can adapt to complex environments and improve grasping accuracy and success rate. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a robot 3D vision-guided grasping method, which solves the problems raised in the above-mentioned background technology.

[0004] Technical solution: To solve the above technical problems, according to one aspect of the present invention, more specifically, a robot 3D vision-guided grasping method comprises the following steps: S1, environment acquisition and modeling

[0005] A. Data Collection:

[0006] Use high-resolution 3D vision sensors to fully scan the working environment and obtain point cloud data of the scene;

[0007] Preprocess the collected point cloud data, including noise removal, hole filling and data smoothing, to improve data quality; use statistical filtering algorithm to remove isolated noise points, and perform smoothing through moving least squares method;

[0008] B. Object recognition and segmentation:

[0009] Apply deep learning algorithms to perform object recognition on the processed point cloud data; train a model that can recognize target objects, which can distinguish objects of different types, shapes, colors, and textures, including in complex backgrounds and occlusion situations;

[0010] According to the recognition results, different objects are segmented and the point cloud data of each object is separated from the scene; accurate object segmentation is achieved using region growing, clustering algorithm, and deep learning segmentation algorithm;

[0011] C. Environmental Modeling:

[0012] Create a 3D geometric model for each segmented object. Use convex hull algorithms or fitting methods based on geometric primitives to convert the point cloud of the object into a more concise geometric representation.

[0013] At the same time, considering the obstacles and supporting surfaces in the scene, the corresponding geometric models of the workbenches, shelves, and other equipment in the environment are established to form a complete environmental map, which includes the position, posture, and size information of objects, as well as the position and range information of obstacles;

[0014] S2. Path planning and optimization

[0015] A. Grab point selection:

[0016] For each identified target object, multiple possible grasping points are calculated based on its geometric model and physical characteristics, combined with the robot's gripper shape and grasping ability; these grasping points are evaluated using grasping quality evaluation indicators;

[0017] Adopt multimodal grasping strategies, considering different grasping postures and gripper configurations, including parallel grasping, encircling grasping, and fingertip grasping, to cope with objects of different shapes and surface features;

[0018] A machine learning-based method is introduced to train a grasping point prediction model through a large amount of simulation data and some actual grasping data, so that the robot can automatically select the optimal grasping point according to the shape, texture and environmental information of the object;

[0019] B. Path planning:

[0020] Based on the robot's kinematic and dynamic models, a path planning algorithm is used to plan a collision-free path for the robot from its current position to the target grasping point.

[0021] Considering dynamic obstacles in the environment, the predicted obstacle trajectory is combined when planning the path to generate a time-elastic path to ensure that the robot can avoid collisions with dynamic obstacles during movement;

[0022] C. Posture adjustment and approach strategy:

[0023] When approaching the target object, the robot's posture is adjusted in real time based on 3D visual feedback to ensure that the gripper is accurately aligned with the grasping point. A visual servo-based method is used to control the robot's joint angles through visual feedback to achieve precise grasping posture adjustment.

[0024] Using a progressive approach strategy, the robot first approaches the target object at a slower speed, adjusts the movement speed and posture according to the distance and visual error, and switches to a more precise control mode when it approaches a certain distance to ensure the accuracy and stability of the grasping action;

[0025] S3. Capture, Execution and Adjustment

[0026] A. Grab operation:

[0027] Control the robot gripper to approach and grasp the target object with a predetermined grasping force and speed; use a force sensor or tactile sensor to measure the grasping force, and adjust the gripping degree of the gripper in real time according to the grasping force feedback to avoid damaging the object or grasping failure;

[0028] Using compliant control algorithms, the robot can adapt to the slight displacement and deformation of the object during the grasping process to ensure a secure grasp;

[0029] B. Crawl verification and adjustment:

[0030] Use 3D vision to check the grasped object again to verify whether the target object is grasped correctly; by comparing the point cloud data before and after grasping, analyze the posture and position changes of the object to determine whether sliding or incomplete grasping occurs;

[0031] If the grasping fails, adjust the grasping point or grasping strategy according to the feedback information and re-perform the grasping operation; at the same time, analyze the cause of the failure, update the grasping point prediction model and grasping strategy selection algorithm to improve the success rate of subsequent grasping;

[0032] S4. Update and learn

[0033] A. Environmental Update:

[0034] During the grasping process, the 3D vision sensor is continuously used to monitor changes in the environment. When changes in the environment are detected, the environmental model is updated in real time;

[0035] For moving objects, use tracking algorithms to predict and update their trajectories in order to update the robot’s path planning and grasping strategies;

[0036] B. Adaptive learning:

[0037] Record the success and failure information of each grasping operation; use reinforcement learning algorithms to train and update the robot's grasping strategy, so that the robot can continuously optimize its grasping behavior based on past experience;

[0038] Deep learning models are regularly updated and fine-tuned to adapt to new object types and environmental changes.

[0039] Furthermore, the 3D vision sensor is any one or more combinations of a structured light sensor, a binocular camera, and a ToF camera. The 3D vision sensor can cover the working range of the robot and collect data at a higher frame rate to ensure adaptability to dynamic scenes.

[0040] Furthermore, the physical properties of the object in the path planning and optimization in step S2 include but are not limited to the center of gravity, friction, and surface texture; the quality evaluation indicators include grasping stability indicators and anti-interference ability indicators.

[0041] Furthermore, in the path planning step, the path is optimized so that the movement time is shortened as much as possible and the range of joint movement is reduced while satisfying the collision-free condition. The trajectory optimization algorithm can be used to smooth and optimize the path.

[0042] Furthermore, in the environment updating step, when the 3D vision sensor monitors changes in the environment, the change data includes the movement of objects, the appearance of new objects, or the addition of obstacles.

[0043] Furthermore, the success and failure information of the grasping operation in the adaptive learning step includes but is not limited to object features, environmental information, grasping points, paths and grasping results.

[0044] The beneficial effects of the robot 3D vision-guided grasping method of the present invention are:

[0045] (1) The present invention uses high-resolution 3D vision sensors to scan all directions to obtain point cloud data, and performs professional preprocessing to accurately perceive the working environment. The deep learning algorithm combines multiple segmentation methods to accurately identify and segment objects under complex backgrounds and occlusions. At the same time, it uses specific algorithms to create geometric models of objects and environments, forming a comprehensive environmental map, providing an accurate basis for subsequent operations.

[0046] (2) The present invention comprehensively considers the characteristics of the object, the gripper's ability and a variety of grasping strategies, and combines the prediction model trained by machine learning to intelligently select the optimal grasping point. Path planning not only considers collision-free, but also combines the motion trajectory of dynamic obstacles to generate a time-elastic path, and shortens the motion time and reduces the joint amplitude through optimization algorithms, thereby improving the grasping efficiency and the stability of the robot's operation. Posture adjustment and progressive approach strategies ensure that the grasping action is accurate and stable.

[0047] (3) The present invention uses force sensors or tactile sensors to adjust the grasping force in real time, combined with a compliant control algorithm, so that the robot can adapt to slight changes in the object and firmly grasp the target. After grasping, the robot will check again through 3D vision, make timely adjustments based on feedback, analyze the reasons for failure, and update the model and strategy, which significantly improves the grasping success rate.

[0048] (4) The present invention continuously monitors environmental changes during the grasping process, updates the environmental model in real time, predicts the trajectory of moving objects, and adjusts path planning and grasping strategies in a timely manner. The grasping information is recorded and the grasping strategy is updated using the reinforcement learning algorithm. The deep learning model is regularly fine-tuned so that the robot can continuously adapt to new environments and objects and improve long-term working performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0050] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0052] In order to make the technical solution of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Reference Figure 1 A robot 3D vision guided grasping method comprises the following steps: S1, environment acquisition and modeling

[0054] A. Data Collection:

[0055] Use high-resolution 3D vision sensors to fully scan the working environment and obtain point cloud data of the scene;

[0056] Preprocess the collected point cloud data, including noise removal, hole filling and data smoothing, to improve data quality; use statistical filtering algorithm to remove isolated noise points, and perform smoothing through moving least squares method;

[0057] B. Object recognition and segmentation:

[0058] Apply deep learning algorithms to the processed point cloud data for object recognition; train a model that can recognize target objects and distinguish between objects of different types, shapes, colors, and textures, including in complex backgrounds and under occlusion;

[0059] According to the recognition results, different objects are segmented and the point cloud data of each object is separated from the scene; accurate object segmentation is achieved using region growing, clustering algorithm, and deep learning segmentation algorithm;

[0060] C. Environmental Modeling:

[0061] Create a 3D geometric model for each segmented object. You can use the convex hull algorithm or the fitting method based on geometric primitives (such as cylinders, cuboids, etc.) to convert the point cloud of the object into a more concise geometric representation.

[0062] At the same time, considering the obstacles and supporting surfaces in the scene, the corresponding geometric models of the workbenches, shelves, and other equipment in the environment are established to form a complete environmental map, which includes the position, posture, and size information of objects, as well as the position and range information of obstacles;

[0063] S2. Path planning and optimization

[0064] A. Grab point selection:

[0065] For each identified target object, multiple possible grasping points are calculated based on its geometric model and physical characteristics, combined with the robot's gripper shape and grasping ability; these grasping points are evaluated using grasping quality evaluation indicators;

[0066] Adopt multimodal grasping strategies, considering different grasping postures and gripper configurations, including parallel grasping, encircling grasping, and fingertip grasping, to cope with objects of different shapes and surface features;

[0067] A machine learning-based method is introduced to train a grasping point prediction model through a large amount of simulation data and some actual grasping data, so that the robot can automatically select the optimal grasping point according to the shape, texture and environmental information of the object;

[0068] B. Path planning:

[0069] Based on the robot's kinematic and dynamic models, a path planning algorithm is used to plan a collision-free path for the robot from its current position to the target grasping point.

[0070] Considering dynamic obstacles in the environment, the predicted obstacle trajectory is combined when planning the path to generate a time-elastic path to ensure that the robot can avoid collisions with dynamic obstacles during movement;

[0071] C. Posture adjustment and approach strategy:

[0072] When approaching the target object, the robot's posture is adjusted in real time based on 3D visual feedback to ensure that the gripper is accurately aligned with the grasping point. A visual servo-based method is used to control the robot's joint angles through visual feedback to achieve precise grasping posture adjustment.

[0073] Using a progressive approach strategy, the robot first approaches the target object at a slower speed, adjusts the movement speed and posture according to the distance and visual error, and switches to a more precise control mode when it approaches a certain distance to ensure the accuracy and stability of the grasping action;

[0074] S3. Capture, Execution and Adjustment

[0075] A. Grab operation:

[0076] Control the robot gripper to approach and grasp the target object with a predetermined grasping force and speed; use a force sensor or tactile sensor to measure the grasping force, and adjust the gripping degree of the gripper in real time according to the grasping force feedback to avoid damaging the object or grasping failure;

[0077] Using compliant control algorithms, the robot can adapt to the slight displacement and deformation of the object during the grasping process to ensure a secure grasp;

[0078] B. Crawl verification and adjustment:

[0079] Use 3D vision to check the grasped object again to verify whether the target object is grasped correctly; by comparing the point cloud data before and after grasping, analyze the posture and position changes of the object to determine whether sliding or incomplete grasping occurs;

[0080] If the grasping fails, adjust the grasping point or grasping strategy according to the feedback information and re-perform the grasping operation; at the same time, analyze the cause of the failure, update the grasping point prediction model and grasping strategy selection algorithm to improve the success rate of subsequent grasping;

[0081] S4. Update and learn

[0082] A. Environmental Update:

[0083] During the grasping process, the 3D vision sensor is continuously used to monitor changes in the environment. When changes in the environment are detected, the environmental model is updated in real time;

[0084] For moving objects, use tracking algorithms to predict and update their trajectories in order to update the robot’s path planning and grasping strategies;

[0085] B. Adaptive learning:

[0086] Record the success and failure information of each grasping operation; use reinforcement learning algorithms to train and update the robot's grasping strategy, so that the robot can continuously optimize its grasping behavior based on past experience;

[0087] Deep learning models (for object recognition, grasp point prediction, etc.) are regularly updated and fine-tuned to adapt to new object types and environmental changes.

[0088] Reference Figure 1 The 3D vision sensor is any one or more combinations of structured light sensors, binocular cameras, and ToF cameras. The 3D vision sensor can cover the working range of the robot and collect data at a higher frame rate to ensure adaptability to dynamic scenes.

[0089] Reference Figure 1In step S2, the physical properties of the object in path planning and optimization include but are not limited to center of gravity, friction, and surface texture; the quality evaluation indicators include grasping stability indicators and anti-interference ability indicators.

[0090] Reference Figure 1 ,In the path planning step, the path is optimized so that it can ,shorten the movement time and reduce the joint movement range as much as possible while meeting ,the collision-free conditions. The trajectory optimization algorithm (quadratic ,programming or gradient descent method) can be used to smooth and ,optimize the path.

[0091] Reference Figure 1 ,In the environment update step, when the 3D vision sensor monitors the ,environmental changes, the change data includes the movement of objects, the ,appearance of new objects, or the addition of obstacles.

[0092] Reference Figure 1 ,The success and failure information of the grasping operation in the ,adaptive learning step includes but is not limited to object features, ,environmental information, grasping points, paths and grasping results.

[0093] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A robot 3D vision-guided grasping method, characterized in that: It includes the following steps: S1, environment collection and modeling A. Data Collection: Use high-resolution 3D vision sensors to fully scan the working environment and obtain point cloud data of the scene; Preprocess the collected point cloud data, including noise removal, hole filling and data smoothing, to improve data quality; Statistical filtering algorithm is used to remove isolated noise points, and moving least squares method is used for smoothing; B. Object recognition and segmentation: Apply deep learning algorithms to perform object recognition on the processed point cloud data; train a model that can recognize target objects, which can distinguish objects of different types, shapes, colors, and textures, including in complex backgrounds and occlusion situations; According to the recognition results, different objects are segmented and the point cloud data of each object is separated from the scene; accurate object segmentation is achieved using region growing, clustering algorithm, and deep learning segmentation algorithm; C. Environmental Modeling: Create a 3D geometric model for each segmented object. Use convex hull algorithms or fitting methods based on geometric primitives to convert the point cloud of the object into a more concise geometric representation. At the same time, considering the obstacles and supporting surfaces in the scene, the corresponding geometric models of the workbenches, shelves, and other equipment in the environment are established to form a complete environmental map, which includes the position, posture, and size information of objects, as well as the position and range information of obstacles; S2. Path planning and optimization A. Grab point selection: For each identified target object, multiple possible grasping points are calculated based on its geometric model and physical characteristics, combined with the robot's gripper shape and grasping ability; these grasping points are evaluated using grasping quality evaluation indicators; Adopt multimodal grasping strategies, considering different grasping postures and gripper configurations, including parallel grasping, encircling grasping, and fingertip grasping, to cope with objects of different shapes and surface features; A machine learning-based method is introduced to train a grasping point prediction model through a large amount of simulation data and some actual grasping data, so that the robot can automatically select the optimal grasping point according to the shape, texture and environmental information of the object; B. Path planning: Based on the robot's kinematic and dynamic models, a path planning algorithm is used to plan a collision-free path for the robot from its current position to the target grasping point. Considering dynamic obstacles in the environment, the predicted obstacle trajectory is combined when planning the path to generate a time-elastic path to ensure that the robot can avoid collisions with dynamic obstacles during movement; C. Posture adjustment and approach strategy: When approaching the target object, the robot's posture is adjusted in real time based on 3D visual feedback to ensure that the gripper is accurately aligned with the grasping point. A visual servo-based method is used to control the robot's joint angles through visual feedback to achieve precise grasping posture adjustment. Using a progressive approach strategy, the robot first approaches the target object at a slower speed, adjusts the movement speed and posture according to the distance and visual error, and switches to a more precise control mode when it approaches a certain distance to ensure the accuracy and stability of the grasping action; S3. Capture, Execution and Adjustment A. Grab operation: Control the robot gripper to approach and grasp the target object with a predetermined grasping force and speed; use a force sensor or tactile sensor to measure the grasping force, and adjust the gripping degree of the gripper in real time according to the grasping force feedback to avoid damaging the object or grasping failure; Using compliant control algorithms, the robot can adapt to the slight displacement and deformation of the object during the grasping process to ensure a secure grasp; B. Crawl verification and adjustment: Use 3D vision to check the grasped object again to verify whether the target object is grasped correctly; by comparing the point cloud data before and after grasping, analyze the posture and position changes of the object to determine whether sliding or incomplete grasping occurs; If the grasping fails, adjust the grasping point or grasping strategy according to the feedback information and re-perform the grasping operation; at the same time, analyze the cause of the failure, update the grasping point prediction model and grasping strategy selection algorithm to improve the success rate of subsequent grasping; S4. Update and learn A. Environmental Update: During the grasping process, the 3D vision sensor is continuously used to monitor changes in the environment. When changes in the environment are detected, the environmental model is updated in real time; For moving objects, use tracking algorithms to predict and update their trajectories in order to update the robot’s path planning and grasping strategies; B. Adaptive learning: Record the success and failure information of each grasping operation; use reinforcement learning algorithms to train and update the robot's grasping strategy, so that the robot can continuously optimize its grasping behavior based on past experience; Deep learning models are regularly updated and fine-tuned to adapt to new object types and environmental changes.

2. A robot 3D vision-guided grasping method according to claim 1, characterized in that: The 3D vision sensor is any one or more combinations of a structured light sensor, a binocular camera, and a ToF camera. The 3D vision sensor can cover the working range of the robot and collect data at a higher frame rate to ensure adaptability to dynamic scenes.

3. A robot 3D vision-guided grasping method according to claim 1, characterized in that: The physical properties of the object in the path planning and optimization in step S2 include but are not limited to the center of gravity, friction, and surface texture; the quality evaluation indicators include the grasping stability index and the anti-interference ability index.

4. A robot 3D vision-guided grasping method according to claim 1, characterized in that: In the path planning step, the path is optimized so that the movement time is shortened as much as possible and the range of joint movement is reduced while satisfying the collision-free condition. The trajectory optimization algorithm can be used to smooth and optimize the path.

5. The robot 3D vision-guided grasping method according to claim 1, characterized in that: In the environment updating step, when the 3D vision sensor monitors changes in the environment, the change data includes the movement of objects, the appearance of new objects, or the addition of obstacles.

6. A robot 3D vision-guided grasping method according to claim 1, characterized in that: The success and failure information of the grasping operation in the adaptive learning step includes but is not limited to object features, environmental information, grasping points, paths and grasping results.

Images