A compound robot dynamic grabbing method and system based on visual servoing

Abstract

This invention primarily relates to the field of robot control technology. To address the problem of inconsistent response speeds between the mobile platform and the robotic arm during target grasping in current composite robots, which leads to uncoordinated movements and affects the grasping accuracy of the robotic arm, this invention provides a dynamic grasping method and system for composite robots based on visual servoing. The method involves real-time acquisition of depth images of the composite robot along its travel direction; obtaining the coordinate information of the object to be grasped and the current pose of the composite robot based on the depth information of the depth image and the position of the object in the image; calculating the pose information of the composite robot at the next moment based on the current pose information and the coordinate information of the object to be grasped; determining the position of the composite robot in the world coordinate system; estimating the coordinates of the object to be grasped based on the coordinate information of the object and the world coordinates of the composite robot; and finally, grasping the object.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a dynamic grasping method and system for a composite robot based on visual servoing. Background Technology

[0002] Unlike traditional industrial robotic arms that operate in fixed areas and repeatedly grasp known objects, dynamic grasping by composite robots in complex environments involves several challenging research areas, including environmental perception and processing, mapping perceived information to the robotic arm's operating space, and hand-eye coordination control. Therefore, researching dynamic grasping operations by composite robots is of great significance. Dynamic grasping operations by composite robots include multiple stages such as object recognition and localization, pose calculation, composite robot grasping planning, and robot control. With the continuous improvement of vision sensor manufacturing capabilities and the deepening research into vision algorithms, vision systems have become an important means for composite robots to perceive their environment, effectively expanding the robot's perception capabilities and improving its autonomy and intelligence. The motion planning of composite robots differs from that of traditional fixed robotic arms. The mobile platform expands the robot's workspace, but also brings difficulties to system modeling and motion control.

[0003] Most current research on motion planning for mobile robotic arms plans the motion of the mobile platform and the robotic arm separately. A few studies plan the motion of the mobile robotic arm as a whole, but none of these studies consider the differences between the mobile platform and the robotic arm. Compared with the mobile platform, the robotic arm responds faster and is more accurate. Therefore, how to coordinate the motion between the mobile platform and the robotic arm is a difficult problem faced by composite robots in grasping operations, and it has important research value and practical significance. Summary of the Invention

[0004] The technical problem to be solved by the present invention

[0005] This paper provides a dynamic grasping method and system for a composite robot based on visual servoing, which solves the problem that the different response speeds of the mobile platform and the robotic arm cause uncoordinated movements of the platform and the robotic arm when the composite robot grasps the target, thus affecting the grasping accuracy of the robotic arm.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems

[0007] A dynamic grasping method for a composite robot based on vision servoing is applied to a composite robot including a mobile chassis and a robotic arm.

[0008] Real-time acquisition of depth images of the composite robot in the direction of travel; based on the depth information of the depth images and the position of the object to be grasped in the image, the coordinate information of the object to be grasped and the current pose of the composite robot are obtained.

[0009] The robot's pose information at the next moment is calculated based on the current pose information of the composite robot and the coordinate information of the object to be grasped.

[0010] The movement of the composite robot is controlled based on its pose information, and its position in the world coordinate system is determined.

[0011] The coordinates of the object to be grasped are estimated based on the coordinates of the object and the world coordinates of the composite robot, and then the object is grasped.

[0012] Furthermore, multiple shooting angles are set to acquire depth images of the composite robot in the direction of travel from different angles. The depth images from different shooting angles are fused, and the depth information of the image is extracted from the fused depth image.

[0013] Furthermore, obtaining the coordinate information of the object to be grasped based on the depth image along the direction of the composite robot's movement and the position of the object in the image specifically includes:

[0014] Histograms are used to automatically correct the exposure of the acquired depth images, so that the average brightness and maximum and minimum values ​​of the image meet the preset brightness range.

[0015] Extract the current frame image from the exposure-corrected image and downsample the image using mean pooling;

[0016] The depth values ​​in the depth image are converted into actual distances in the world coordinate system to obtain the true distance from the object to the optical center plane of the image acquisition device.

[0017] Furthermore, the step of calculating the pose information of the composite robot at the next moment based on the current pose information of the composite robot and the coordinate information of the object to be grasped specifically includes:

[0018] Calculate the deflection angle between the desired grasp point and the current velocity direction of the composite robot;

[0019] The time required for the composite robot to reach the graspable area is calculated based on the expected speed of the composite robot's chassis.

[0020] Furthermore, the estimation of the object's coordinates based on the object's coordinate information and the composite robot's world coordinates specifically includes:

[0021] Within the set grasping time, the composite robot is regarded as moving in a straight line. The position of the object to be grasped in the robotic arm coordinate system is estimated by combining the deflection angle between the point to be grasped and the current velocity direction of the composite robot and the time it takes for the composite robot to reach the graspable area.

[0022] Position-based visual servoing is used to calculate the coordinate information of the object to be grasped, and the estimated grasping position of the composite robot is continuously updated.

[0023] The robotic arm of the composite robot is controlled to reach the estimated grasping position and grasp the object to be grasped.

[0024] Furthermore, environmental images of the area where the composite robot walks are acquired in the current environment. The acquired environmental images are input into a trained neural network model. The output of the neural network model is the type of object to be grasped. Different grasping postures are set according to the type of object to be grasped.

[0025] Furthermore, before inputting the RGB image into the neural network model to determine the type of object to be captured, the acquired environmental image is automatically exposed to ensure that the average brightness, maximum brightness, and minimum brightness are within a preset range, and the image is downsampled using the mean pooling method.

[0026] Furthermore, the desired grasping point is 10cm away from the object to be grasped from the end gripper of the robotic arm.

[0027] Based on the aforementioned dynamic grasping method for a visual servoing composite robot, this invention also provides a dynamic grasping system for a visual servoing composite robot, the system comprising a data acquisition device, a computing platform, and a composite robot platform.

[0028] The data acquisition device is used to acquire RGB images and depth maps in the direction of travel of the composite robot in real time and send them to the computing platform.

[0029] The computing platform is used to dynamically identify objects to be grasped in the environment.

[0030] The composite robot platform includes a mobile chassis and a robotic arm, used to grasp objects.

[0031] Furthermore, the data acquisition device is mounted on the gripper of the robotic arm, and the data acquisition device is a depth camera.

[0032] Beneficial effects of the present invention

[0033] This invention discloses a visual servoing-based dynamic grasping method for composite robots, capable of grasping flexible and autonomous moving objects. It designs a hierarchical coordination planning scheme for the mobile platform and the robotic arm. For the mobile platform, a planning strategy of first considering the overall situation and then the local situation is proposed. A depth camera is installed on the robotic arm. Based on the depth images captured by the depth camera, the coordinate information of the object to be grasped and the current pose of the composite robot are obtained. The robot's pose at the next moment is calculated. Based on the calculated pose information, the position of the composite robot in the world coordinate system is calculated. Finally, based on the coordinate information of the object to be grasped and the world coordinates of the composite robot, the coordinates of the object to be grasped are estimated, and the object is grasped.

[0034] To improve the detection efficiency of small objects, a neural network model is established to classify images captured by a depth camera. Different grasping postures are set according to different object types to improve the grasping efficiency of the objects to be grasped. Attached Figure Description

[0035] Figure 1 This is an execution flowchart of the visual servo-based composite robot dynamic grasping system described in this invention.

[0036] Figure 2 This is the camera imaging relationship model described in this invention;

[0037] Figure 3 This is a flowchart of the dynamic grasping process of the composite robot. Detailed Implementation

[0038] The visual servoing composite robot system in this embodiment includes: a data acquisition device, a computer platform, and a composite robot platform.

[0039] The data acquisition device is used to acquire RGB and depth images of the composite robot in the direction of travel in real time, and send the acquired information to the computing platform.

[0040] The computer platform is used to analyze the RGB image and depth map of the current environment to calculate the coordinate information of the image to be captured and the current pose of the composite robot in the environment, so as to determine the next pose of the composite robot and send the determined pose of the composite robot at the next moment to the composite robot.

[0041] The composite robot platform is used to identify objects to be grasped in the environment and to dynamically grasp the objects to be grasped by planning the motion of the composite robot.

[0042] like Figure 1 As shown, the visual servoing-based composite robot system includes the following steps when grasping dynamic objects:

[0043] Step 1: Obtain environmental images of the area where the composite robot walks in the current environment, and input the RGB images provided by the data acquisition device into the neural network model;

[0044] Step 2: Using the 3D coordinate transformation method, obtain the coordinate information of the object to be grabbed based on the depth information collected by the depth camera and the position of the object in the image;

[0045] Step 3: Calculate the pose parameters of the composite robot at the next moment based on the current pose information of the composite robot and the coordinate information of the object to be grasped;

[0046] Step 4: Control the movement of the composite robot based on the calculated pose of the composite robot, and perform odometry to determine the position of the composite robot in the world coordinate system;

[0047] Step 5: Based on the coordinate information of the object to be grasped obtained in Step 2, make a grasping judgment. When the object to be grasped is 10cm away from the end gripper, make a manual grasping.

[0048] Step 6: Based on the coordinate information of the object to be grasped obtained in Step 2 and the world coordinates of the composite robot obtained in Step 4, estimate the coordinates of the object to be grasped and complete the dynamic grasping task during the movement of the composite robot.

[0049] In step 1, when acquiring environmental images of the composite robot in the walking area, a depth camera is used to acquire environmental RGB images from different shooting angles.

[0050] Step 2 includes the following detailed steps:

[0051] Histograms are used to automatically correct the exposure of the RGB images provided by the data acquisition device, so that the average brightness, maximum and minimum values ​​of the image are uniformly within the preset range, thereby reducing the impact of ambient light factors on the extraction of planar attribute information.

[0052] Extract the current frame image from the image after automatic exposure correction, and perform image sampling on it using mean pooling;

[0053] When determining the three-dimensional coordinates of an object in camera coordinates based on its position coordinates in the camera, the true distance from the object to be grasped to the camera's plane of interest is obtained through a linear transformation from the depth value in the depth image to the actual value.

[0054] Step 3, which calculates the pose parameters of the composite robot at the next moment based on the current pose information of the composite robot and the coordinate information of the object to be grasped, specifically includes:

[0055] Calculate the deflection angle between the desired grasp point and the current velocity direction of the composite robot;

[0056] The chassis controller calculates the desired chassis speed and the time required for the composite robot to reach the graspable area.

[0057] Step 6 includes the following detailed steps:

[0058] During the grasping time, the composite robot is treated as linear motion, and the position of the object to be grasped in the robotic arm coordinate system is estimated by combining the pre-grasping time.

[0059] Position-based visual servoing is adopted, and the position of the object to be grasped in the coordinate system of the robotic arm is continuously updated according to the coordinate information of the object to be grasped in step 2.

[0060] Control the robotic arm to reach the predicted grasping position, so as to realize the grasping of objects during the movement of the composite robot. Example

[0061] like Figure 3 As shown, step 1 involves using a 3D coordinate transformation method to obtain the coordinate information of the object to be captured based on the depth information acquired by the depth camera and the position of the object in the image. First, a histogram is used to automatically correct the exposure of the RGB image provided by the data acquisition device, ensuring that the average brightness and maximum / minimum values ​​of the image meet preset ranges to reduce the impact of ambient light factors on the extraction of planar attribute information. The current frame image is then extracted from the exposure-corrected image, and mean pooling is used to downsample it. For example... Figure 2 As shown, when determining the 3D coordinates of an object in the camera coordinate system based on its position coordinates within the camera, the depth value obtained directly from the depth map is the raw depth data and cannot be used to represent physical quantities in the real world. Therefore, a linear transformation from the depth value in the depth map to the actual distance in the real world is needed to obtain the true distance from the object to the camera's optical center plane.

[0062] Preferably, the data acquisition device is a depth camera.

[0063] Step 2: Calculate the pose parameters of the composite robot at the next moment based on the current pose of the composite robot and the coordinate information of the object to be grasped. The pose parameters include the deflection angle between the desired grasping point and the current velocity direction of the composite robot; calculate the desired speed of the chassis and the time it takes for the composite robot to reach the graspable area.

[0064] Step 3: Control the movement of the composite robot based on the calculated pose and perform odometry to determine its position in the world coordinate system. Real-time reading and feedback of information from each degree of freedom during the robot's movement are performed. Odometry estimates the chassis configuration based on wheel motion and involves integrating the effect of wheel speeds. Through continuous integration, the position of the composite robot in the world coordinate system is obtained.

[0065] Step 4: Based on the coordinate information of the object to be grasped and the obtained world coordinates of the composite robot, estimate the coordinates of the object to be grasped. Perform the dynamic grasping task during the composite robot's movement. During the brief grasping time, treat the composite robot as moving in a straight line. Combined with the pre-grabbing time, estimate the position of the object to be grasped in the robotic arm coordinate system. Use position-based visual servoing to continuously update the estimated position from step 6.1 based on the coordinate information calculated in step 2. Control the robotic arm to reach the estimated grasping position, thus achieving object grasping during the composite robot's movement. The estimated grasping position is when the object to be grasped is 10cm away from the end effector.

[0066] To improve the grasping efficiency of different objects, environmental images of the robot's walking area are acquired. RGB images provided by the data acquisition device are input into a trained neural network, which outputs the types of objects to be grasped. Different grasping postures are matched to different object types. The specific method is as follows: Environmental images of the robot within its walking area are acquired and processed. Automatic exposure correction is performed on the environmental images to ensure that the average brightness and maximum / minimum values ​​of the image meet preset ranges, thereby reducing the impact of environmental pipelines on the planar attribute extraction process. Mean pooling is then used to downsample the exposure-corrected images.

[0067] For example, the types of waste in environmental images include masks, cardboard boxes, aluminum cans, crumpled paper, and plastic bags. Consider different planar views, different placement methods, and different storage forms of various objects, including but not limited to (a) masks placed at different angles or with different folds; (b) cardboard boxes placed in an open, closed, folded, or damaged manner; (c) aluminum cans placed directly or after being twisted; (d) crumpled paper in single or multiple sheets, with different degrees of crumpling; and (e) plastic bags placed directly or after being crumpled.

[0068] To accommodate waste detection from different perspectives, the camera has two shooting angles. Angle 1: Top-down or low-angle shooting, with the camera height at 30cm. The horizontal distance between the camera and the object should range from 0.5-5m and be distributed as evenly as possible. Angle 2: Top-down shooting, with the camera height at 65-90cm. The horizontal distance between the camera and the object should also range from 0.5-5m and be distributed as evenly as possible.

[0069] The YOLOv3-Tiny model, pre-trained on ImageNet, is selected as the base model for transfer learning. This means that the network parameters obtained from pre-training on a large dataset are used as the initial values ​​for the detection task, and then further trained on a self-built dataset until the network converges. In this embodiment, the pixel-level model takes as input RGB images acquired in real-time by a data acquisition device, and outputs the category of the object to be captured.

Claims

1. A dynamic grasping method for a composite robot based on visual servoing, applied to a composite robot including a mobile chassis and a robotic arm, characterized in that, include, Real-time acquisition of depth images of the composite robot in the direction of travel; based on the depth information of the depth images and the position of the object to be grasped in the images, the coordinate information of the object to be grasped and the current pose of the composite robot are obtained. The robot's pose information for the next moment is calculated based on its current pose information and the coordinate information of the object to be grasped. Specifically, this includes: Calculate the deflection angle between the desired grasp point and the current velocity direction of the composite robot; The time it takes for the composite robot to reach the graspable area is calculated based on the expected speed of the composite robot's chassis. The movement of the composite robot is controlled based on its pose information, and its position in the world coordinate system is determined. The coordinates of the object to be grasped are estimated based on the coordinates of the object and the world coordinates of the composite robot, and then the object is grasped.

2. The method for dynamic grasping of a composite robot based on visual servoing according to claim 1, characterized in that, The coordinate information of the object to be grasped is obtained based on the depth image of the composite robot's travel direction and the position of the object in the image. Specifically, this includes... Histograms are used to automatically correct the exposure of the acquired depth images, so that the average brightness and maximum and minimum values ​​of the image meet the preset brightness range. Extract the current frame image from the exposure-corrected image and downsample the image using mean pooling; The depth values ​​in the depth image are converted into actual distances in the world coordinate system to obtain the true distance from the object to the optical center plane of the image acquisition device.

3. The method for dynamic grasping of a composite robot based on visual servoing according to claim 1, characterized in that, Estimating the coordinates of the object to be grasped based on its coordinates and the world coordinates of the composite robot specifically includes: Within the set grasping time, the motion of the composite robot is regarded as linear motion. The position of the object to be grasped in the robotic arm coordinate system is estimated by combining the deflection angle between the point to be grasped and the current velocity direction of the composite robot and the time it takes for the composite robot to reach the graspable area. Position-based visual servoing is used to calculate the coordinate information of the object to be grasped, and the estimated grasping position of the composite robot is continuously updated. The robotic arm of the composite robot is controlled to reach the estimated grasping position and grasp the object to be grasped.

4. The method for dynamic grasping of a composite robot based on visual servoing according to claim 1, characterized in that, The system acquires RGB images of the area where the composite robot walks in the current environment. The acquired environmental images are then input into a trained neural network model. The neural network model identifies the type of object to be grasped and sets different grasping postures based on the type of object.

5. The method for dynamic grasping of a composite robot based on visual servoing according to claim 4, characterized in that, The RGB image is input into the neural network model to determine the type of object to be captured. The environmental image is automatically exposed and corrected so that the average brightness, maximum and minimum brightness values ​​of the image are within a preset range. The mean pooling method is then used to downsample the image after exposure correction.

6. The method for dynamic grasping of a composite robot based on visual servoing according to claim 1, characterized in that, The desired gripping point is 10cm away from the gripper at the end of the robotic arm.

7. A visual servoing-based composite robot dynamic grasping system, used to implement the visual servoing-based composite robot dynamic grasping method according to any one of claims 1-6, characterized in that, The system includes a data acquisition device, a computing platform, and a composite robot platform. The data acquisition device is used to acquire RGB images and depth maps in the direction of travel of the composite robot in real time and send them to the computing platform. The computing platform is used to dynamically identify objects to be grasped in the environment, obtain the coordinate information of the objects to be grasped, and calculate the pose information of the composite robot at the next moment based on the current pose information of the composite robot and the coordinate information of the objects to be grasped; specifically including: Calculate the deflection angle between the desired grasp point and the current velocity direction of the composite robot; The time it takes for the composite robot to reach the graspable area is calculated based on the expected speed of the composite robot's chassis. The composite robot platform includes a mobile chassis and a robotic arm; the mobile chassis is used to drive the robotic arm to move in the direction of travel; the robotic arm is used to grasp the object to be grasped according to the coordinate information of the object to be grasped and the world coordinates of the composite robot.

8. A visual servoing-based composite robot dynamic grasping system according to claim 7, characterized in that, The data acquisition device is located at the end of the robotic arm, and the data acquisition device is a depth camera.

Images