Humanoid robot grabbing method based on three-dimensional vision

Abstract

The invention discloses a humanoid robot grabbing method based on three-dimensional vision, and the method comprises the following specific steps: S1, environment modeling and perception: carrying out the three-dimensional scanning of a working environment, and constructing a three-dimensional map of the environment; s2, object identification and 6D attitude estimation: identifying a target object, and estimating the position and attitude of the object in a camera coordinate system; s3, coordinate conversion and target positioning, wherein the posture of the object is converted into a robot base coordinate system from a camera coordinate system through TF conversion; and S4, path planning and motion control are conducted, specifically, path planning is conducted through a cumotion algorithm, a grabbing path is generated, and the two arms are driven to move according to the planned track to achieve grabbing. By means of environment modeling and sensing, the robot can update environment information in real time, conduct three-dimensional scanning on the working environment and generate the three-dimensional grid model of the environment in real time, the model is used for obstacle detection and avoidance planning, and the dynamic environment adaptability of the robot to object grabbing is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot grasping, and in particular to a grasping method of a humanoid robot based on three-dimensional vision. Background Art

[0002] In recent years, with the continuous advancement of industrial automation and intelligence, robotics has been widely applied in various fields. In particular, in the field of humanoid robots, achieving efficient and autonomous grasping has become a hot research topic. Traditional robotic grasping methods typically rely on pre-defined trajectories and fixed environmental structures, making them difficult to adapt to complex and changing working environments. This is especially true in dual-arm collaborative scenarios, where humanoid robots face a range of technical challenges, including multi-degree-of-freedom motion control, complex environmental perception, and multi-sensor fusion.

[0003] Most existing robotic grasping systems are based on two-dimensional vision or lidar sensors, which have limited environmental perception capabilities and find it difficult to accurately identify the three-dimensional position and posture of target objects.

[0004] The defects in the existing robot grasping field are: 1. Patent document CN108274476B proposes a method for a humanoid robot to grasp a sphere. However, the robot in this patent document has poor environmental adaptability in grasping. In a dynamic and complex environment, the robot has difficulty updating environmental information in real time and planning an effective grasping path, resulting in a high grasping failure rate. 2. Existing robots may lack multi-sensor fusion in the grasping field. Existing systems generally rely on a single type of sensor and lack the effective fusion of multi-sensor information such as vision, force, and touch. This results in insufficient robot perception and interaction capabilities. 3. Application document CN118636158A discloses a method and related device for carrying camping items based on a humanoid robot. However, the collaborative control of the two arms of the humanoid robot described in this patent document is complex. The collaborative control of the two arms of the humanoid robot requires high-precision motion synchronization and path planning. Existing control strategies often lack an effective coordination mechanism when facing the collaborative grasping task of the two arms, making it difficult to achieve efficient and stable collaboration. 4. Traditional path planning methods mostly use classic sampling or search algorithms. These methods have problems such as high computational complexity and long planning time when dealing with high-degree-of-freedom robots, and are difficult to meet the needs of real-time grasping tasks. Summary of the Invention

[0005] The purpose of the present invention is to provide a humanoid robot grasping method based on three-dimensional vision to solve the problems raised in the above background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solutions: The present invention provides a humanoid robot grasping method based on three-dimensional vision, the specific steps of which are as follows: Step S1, environment modeling and perception: Use an RGB-D camera to perform a three-dimensional scan of the working environment, and use the nvblox octree map generation algorithm to build a three-dimensional map of the environment; Step S2: Object recognition and 6D pose estimation: Use a deep learning algorithm to identify the target object and estimate the object's position and pose in the camera coordinate system; Step S3, coordinate conversion and target positioning: convert the object's posture from the camera coordinate system to the robot base coordinate system through TF transformation; Step S4, path planning and motion control: Use the cumotion algorithm for path planning, generate a grasping path, and drive the arms to move along the planned trajectory to achieve grasping.

[0007] Preferably, in step S1, the nvblox algorithm generates a three-dimensional mesh model of the environment by fusing the depth map and RGB image information, and uses it for obstacle detection and avoidance planning.

[0008] Preferably, the specific steps of generating the three-dimensional network model of the environment in step S1 include: the nvblox algorithm first voxelizes the depth map, divides the environment into multiple three-dimensional voxel grids, each three-dimensional voxel grid contains obstacle information in the environment, and at the same time, the nvblox algorithm combines RGB image information to perform texture mapping on the obstacles.

[0009] Preferably, the perception process of step S1 is further combined with a segmentation and classification algorithm of point cloud data to distinguish between static and dynamic obstacles in the environment.

[0010] Preferably, in step S2, object recognition performs feature extraction and posture estimation through a deep learning network to obtain six-degree-of-freedom information of the object in the camera coordinate system.

[0011] Preferably, in step S2, the collected image is preprocessed before target object recognition, and the image preprocessing method includes filtering of the depth map, noise reduction of the point cloud data, and edge detection of the RGB image.

[0012] Preferably, in step S3, the relative position relationship between the camera and the robot is pre-calibrated, and the TF tree is used to achieve accurate conversion from the camera coordinate system to the robot base coordinate system.

[0013] Preferably, in step S4, the cumotion algorithm combines the deep learning model's prediction of the environment and target object to quickly generate multiple paths and select the optimal path for execution.

[0014] Preferably, in step S4, the MoveIt2 framework is combined to plan and control the movement of the joint space and task space of the humanoid robot's arms to achieve flexible grasping of the target object.

[0015] Preferably, in step S4, the motion control of the humanoid robot also includes setting path constraints and speed limits, and performing error correction in combination with real-time joint state feedback. Compared with the prior art, the present invention has the following beneficial effects: 1. Through environmental modeling and perception, the present invention enables the robot to update environmental information in real time. An RGB-D camera is used to perform three-dimensional scanning of the working environment. The nvblox algorithm is then used to generate a three-dimensional mesh model of the environment in real time by fusing depth maps and RGB image information. This model is then used for obstacle detection and avoidance planning. During the real-time generation of the three-dimensional mesh model of the environment, texture mapping combined with RGB images further enhances obstacle recognition accuracy. Furthermore, during obstacle perception, point cloud data segmentation and classification algorithms are combined to distinguish between static and dynamic obstacles in the environment, enabling different processing during path planning and improving the robot's ability to adapt to dynamic environments when grasping objects.

[0016] 2. The present invention identifies the target object based on the 6D posture estimation algorithm of deep learning, identifies the target object through feature extraction and posture estimation through the deep learning network, and obtains the six-degree-of-freedom information of the target object in the camera coordinate system. The six-degree-of-freedom information of the target object in the camera coordinate system can more accurately locate the three-dimensional spatial position and posture of the target object, facilitate the humanoid robot to accurately grasp the target object, and improve the success rate of the humanoid robot in grasping the target object.

[0017] 3. The present invention combines the MoveIt2 framework to plan and control the movement of the joint space and task space of the two arms, thereby enabling the humanoid robot to flexibly grasp the target object. At the same time, the system also sets path constraints and speed limits, and combines real-time joint state feedback to perform error correction to prevent the two arms from colliding with the environment during the grasping process, thereby ensuring the safety of the humanoid robot during operation.

[0018] 4. The present invention adopts the cumotion algorithm in path planning. The cumotion algorithm first extracts features of the environment through a deep convolutional neural network to generate a set of candidate paths, and then optimizes the path based on the reinforcement learning strategy to select the path with the lowest collision risk and the highest grasping efficiency. The planning time is within 0.1s, which can meet the needs of real-time response. At the same time, the collision detection module is also combined in the path planning process to ensure that the generated path is collision-free. The path planning time of the present invention is short and can avoid collisions. It can not only meet the needs of real-time grasping tasks, but also ensure the safety of the grasping path. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings: See also Figure 1 The humanoid robot grasping method based on three-dimensional vision consists of a visual perception part and a path planning part. The visual perception part includes an environment perception module for generating an octree map and a three-dimensional pose estimation module for identifying the three-dimensional pose of the object; the path planning part includes a scene planner, a collision detector and a path planner.

[0022] The present invention proposes a specific embodiment of a humanoid robot grasping method based on three-dimensional vision. The system hardware deployment of the present invention includes: Hardware platform: This invention uses NVIDIA Orin NX as the main processing platform. Orin NX has high-performance GPU acceleration capabilities, can handle complex deep learning computing tasks, and ensure real-time path planning and environmental perception.

[0023] Vision sensor: The RealSense D435i RGB-D camera is used as a vision sensor to collect depth information and RGB images of the environment. The D435i camera's depth accuracy and wide field of view enable the system to efficiently perceive the working environment and accurately locate target objects.

[0024] Robot controller: The robot controller uses a control framework based on ROS2 and combines it with MoveIt2 for motion planning and control. The controller sends the planned joint angle path to the robot actuator to drive the dual arms to perform grasping tasks.

[0025] The humanoid robot grasping method of the present invention comprises the following steps: Step S1, environment modeling and perception: Specifically, use an RGB-D camera to perform a three-dimensional scan of the working environment, and use the nvblox octree map generation algorithm to build a three-dimensional map of the environment; Specifically, after the system starts, it first scans the environment using the RealSense D435i camera to obtain a depth map and RGB image of the working area. This depth information is then passed to the environmental perception module, which generates a 3D octree map of the environment in real time. This real-time 3D octree map can be used for path planning and obstacle avoidance for the robot. Step S2: Object recognition and 6D pose estimation: Use a deep learning algorithm to identify the target object and estimate the object's position and pose in the camera coordinate system; Specifically, the depth map and RGB image are simultaneously passed to the 3D pose estimation module, which pre-models the object to be grasped, trains the corresponding deep learning model, and pre-processes the captured image after acquisition, including operations such as filtering the depth map, denoising the point cloud data, and edge detection of the RGB image. The deep learning model is used to infer the pre-processed image, extract the object's features, and estimate its 6D pose, which includes 3D spatial coordinates and Euler angles. Step S3: coordinate conversion and target positioning, converting the object's posture from the camera coordinate system to the robot base coordinate system through TF transformation; Specifically, the object's posture and octree map are converted from the camera coordinate system to the robot base coordinate system through the TF coordinate transformation module provided by ROS2; Step S4: Path planning and motion control: Use the cumotion algorithm to perform path planning, generate a grasping path, and drive the arms to move along the planned trajectory to achieve grasping; Specifically, the converted octree map, object position and posture, and object collision volume will be passed to the scene planner, and the object position will be passed to the path planning module. The path planning module calls the cumotion algorithm and performs path planning based on deep learning and reinforcement learning methods. The planning time is within 0.1s. The short path planning time can meet the needs of real-time response. The path planning module will combine with the collision detection module while planning the path. The system establishes a virtual model through the collision detection module to simulate the movement route of the humanoid robot's two arms in all two-arm grasping paths. Through real-time planning of the two-arm movement path, the collision situation of the robot's two-arm posture in the movement path is simulated according to the specific situation of whether the humanoid robot currently grasps the target object. When the humanoid robot grasps the target object and moves on the specified path, the surrounding environment of the two-arm movement path, the volume of the target object, the volume of the humanoid robot, and the two-arm posture of the humanoid robot will affect the collision situation. The path planning module is combined with the collision detection module to ensure that the humanoid robot will not collide during the two-arm movement process on the generated path.

[0026] After planning is completed, the generated joint angle path is sent to the robot controller, driving the arms to move along the planned trajectory, gradually approaching the target object and finally achieving grasping.

[0027] In terms of environmental modeling and perception, the present invention uses an RGB-D camera to perform a three-dimensional scan of the working environment and utilizes the nvblox octree map generation algorithm to construct a three-dimensional map of the environment. The nvblox algorithm can generate a three-dimensional mesh model of the environment in real time by fusing depth maps and RGB image information, and use this model for obstacle detection and avoidance planning. Specifically, the nvblox algorithm first voxelizes the depth map, dividing the environment into multiple three-dimensional voxel grids, each of which contains information about obstacles in the environment. The present invention further enhances the accuracy of obstacle recognition by combining RGB images for texture mapping. In the perception process, it also combines the segmentation and classification algorithms of point cloud data to distinguish between static and dynamic obstacles in the environment, so that different treatments can be made during path planning.

[0028] In terms of object recognition and 6D pose estimation, the present invention adopts a 6D pose estimation algorithm based on deep learning to identify the target object and estimate its position and pose in the camera coordinate system. Specifically, the images captured by the RGB-D camera are subjected to feature extraction and pose estimation through a deep learning network to obtain the six-degree-of-freedom information of the object in the camera coordinate system. To ensure the accuracy of target object recognition, the system also preprocesses the captured images, including filtering of the depth map, denoising of the point cloud data, and edge detection of the RGB image.

[0029] In terms of motion planning, the present invention adopts the path planning algorithm cumotion provided by NVIDIA, which is based on deep learning and accelerated by CUDA, to ensure that the robot can efficiently plan a suitable grasping path in a complex environment; the cumotion algorithm combines the deep learning model's prediction of the environment and target objects to quickly generate multiple paths and select the optimal path for execution.

[0030] In terms of robot motion control, the present invention combines the MoveIt2 framework to plan and control the motion of the joint space and task space of the two arms to achieve flexible grasping of the target object; specifically, the cumotion algorithm first extracts features from the environment through a deep convolutional neural network to generate a set of candidate paths, and then optimizes the path based on a reinforcement learning strategy to select the path with the lowest collision risk and the highest grasping efficiency; in order to ensure safety during movement, the system also sets path constraints and speed limits, and combines real-time joint state feedback for error correction to prevent the two arms from colliding with the environment during the grasping process; the system's path constraint on the humanoid robot is for the posture of the robot's arm end effector, and the system's speed limit on the humanoid robot is to plan the arm movement speed of the humanoid robot through a deep learning model.

[0031] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A humanoid robot grasping method based on three-dimensional vision, characterized in that: The specific steps are as follows: Step S1, environment modeling and perception: use an RGB-D camera to perform a three-dimensional scan of the working environment, and use the nvblox octree map generation algorithm to build a three-dimensional map of the environment; Step S2, object recognition and 6D posture estimation: Use a deep learning algorithm to recognize the target object and estimate the position and posture of the object in the camera coordinate system; Step S3, coordinate conversion and target positioning: convert the posture of the object from the camera coordinate system to the robot base coordinate system through TF transformation; Step S4, path planning and motion control: Use the cumotion algorithm for path planning, generate a grasping path, and drive the arms to move along the planned trajectory to achieve grasping.

2. A three-dimensional vision-based humanoid robot grasping method according to claim 1, characterized in that: In step S1, the nvblox algorithm generates a 3D mesh model of the environment by fusing the depth map and RGB image information, and uses it for obstacle detection and avoidance planning.

3. A three-dimensional vision-based humanoid robot grasping method according to claim 2, characterized in that: The specific steps of generating the three-dimensional network model of the environment in step S1 include: the nvblox algorithm first voxelizes the depth map to divide the environment into multiple three-dimensional voxel grids, each of which contains obstacle information in the environment. At the same time, the nvblox algorithm combines the RGB image information to perform texture mapping on the obstacles.

4. The three-dimensional vision-based humanoid robot grasping method according to claim 1, characterized in that: The perception process of step S1 also combines the segmentation and classification algorithm of point cloud data to distinguish between static and dynamic obstacles in the environment.

5. The three-dimensional vision-based humanoid robot grasping method according to claim 1, characterized in that: In step S2, object recognition performs feature extraction and posture estimation through a deep learning network to obtain the six-degree-of-freedom information of the object in the camera coordinate system.

6. The three-dimensional vision-based grasping method of a humanoid robot according to claim 5, characterized in that: In step S2, the collected image is preprocessed before the target object is identified. The image preprocessing method includes filtering of the depth map, noise reduction of the point cloud data, and edge detection of the RGB image.

7. The three-dimensional vision-based humanoid robot grasping method according to claim 1, characterized in that: In step S3, the relative position relationship between the camera and the robot is pre-calibrated, and the TF tree is used to achieve accurate conversion from the camera coordinate system to the robot base coordinate system.

8. The three-dimensional vision-based humanoid robot grasping method according to claim 1, characterized in that: In step S4, the cumotion algorithm combines the deep learning model's predictions of the environment and target objects to quickly generate multiple paths and select the optimal path for execution.

9. The three-dimensional vision-based humanoid robot grasping method according to claim 1, characterized in that: In step S4, in combination with the MoveIt2 framework, the movement of the joint space and task space of the humanoid robot's arms is planned and controlled to achieve flexible grasping of the target object.

10. The three-dimensional vision-based grasping method of a humanoid robot according to claim 8, characterized in that: In step S4, the motion control of the humanoid robot also includes setting path constraints and speed limits, and performing error correction in combination with real-time joint state feedback.

Images