A multi-model flexible six-axis robot vision grasping system

Abstract

This invention belongs to the field of industrial automation technology, specifically relating to a multi-model flexible six-axis robot vision grasping system. It includes a data acquisition module for acquiring the target object's mass data, two-dimensional density map, RGB image, and depth image; a target detection module for acquiring depth point cloud data; a weight compensation module for fusing the target object's depth point cloud data and mass data to generate mass distribution data; a pose calculation module for generating several sets of grasping posture samples based on the mass distribution data, and selecting the optimal grasping posture from these samples using the NMS algorithm; and a six-axis robot control module for converting the optimal grasping posture into joint motion commands for a specific model of six-axis robot and controlling the six-axis robot to perform grasping operations based on these joint motion commands. The optimal grasping posture generated by fusing point cloud data and mass data into mass distribution data improves the accuracy of the robot's grasping actions.

Description

Technical Field

[0001] This invention belongs to the field of industrial automation technology, specifically relating to a multi-model flexible six-axis robot vision grasping system. Background Technology

[0002] In automated production, intelligent robots are automated operating devices that use pre-programmed control procedures to grasp, move, or manipulate objects or tools. Robots typically use vision-based grasping systems to identify the surface features of target objects, thus determining the gripping position of the robot's robotic arm and enabling precise grasping of the target object.

[0003] Existing solutions rely on visual information (such as RGB-D images) to plan the grasping pose, taking into account the actual mass distribution characteristics of the object to be grasped, such as the center of gravity and moment of inertia. For non-uniform mass objects (such as containers filled with liquid or irregularly shaped workpieces), slippage or overturning can easily occur during grasping due to torque imbalance. When the recognition system cannot know the weight of the object, the control system port cannot adaptively adjust the clamping force or suction force through the control end effector, resulting in overpressure damage or insufficient grasping force.

[0004] Moreover, when the target object has missing textures or uneven lighting, the edge accuracy of the segmentation mask decreases significantly, leading to contour recognition errors. This causes the recognition system to make a deviation in judging the position of the target object, and the position deviation will further affect the trajectory planning of the robotic arm and reduce the success rate of grasping. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a multi-model flexible six-axis robot vision grasping system. This system solves the problem that when the target object has missing textures or uneven lighting, the edge accuracy of the segmentation mask decreases significantly, leading to contour recognition errors. Consequently, the recognition system's judgment of the target object's position deviates, resulting in unstable grasping.

[0006] The objective of this invention can be achieved through the following technical solution: a multi-model flexible six-axis robot vision grasping system, comprising:

[0007] The data acquisition module is used to acquire the mass data, two-dimensional density map, RGB image and depth image of the target object;

[0008] The target detection module is used to synthesize a point cloud map of the target object based on the RGB image and the depth image, extract the contour feature data of the RGB image using the Transformer architecture of the encoder-decoder structure, and apply the contour feature data to the depth image to extract the depth point cloud data of the target object.

[0009] The weight compensation module is used to fuse the depth point cloud data and mass data of the target object to generate mass distribution data.

[0010] The pose calculation module is used to generate several sets of grasping pose samples based on the mass distribution data, and to select the optimal grasping pose from several grasping pose samples based on the NMS algorithm.

[0011] The six-axis robot control module converts the optimal grasping posture into joint motion commands for a specific model of six-axis robot and controls the six-axis robot to perform grasping operations based on the joint motion commands.

[0012] Preferably, in the weight compensation module, the depth point cloud data and the quality data are fused using the following steps:

[0013] S1: The radix basis function (RBF) interpolation algorithm maps the discrete sampling points of the two-dimensional density map to the surface of the three-dimensional point cloud through the kernel function of the distance weight, generating a continuous mass density field.

[0014] S2: The mass density field is optimized using the variable density SIMP algorithm to minimize the non-uniformity of the mass distribution;

[0015] S3: Obtain the ambient temperature factor and perform temperature compensation correction on each point of the optimized mass density field coordinates;

[0016] S4: The corrected mass density field is fused with the point cloud coordinates corresponding to the depth point cloud data to generate mass distribution data.

[0017] The Open3D algorithm was used to visualize the mass distribution of the target object, and the accuracy was verified by comparing it with mass data.

[0018] Preferably, step S1 includes the following steps:

[0019] The pixel coordinates of the 2D density map are projected onto the 3D point cloud surface using perspective transformation (PT) to generate corresponding sampling points. ;

[0020] Construct an interpolation system to calculate the mass density value for each 3D point cloud coordinate. , The calculation formula is:

[0021] ;

[0022] Where x is the coordinate in the three-dimensional data corresponding to the target object. These are the three-dimensional projected coordinates corresponding to the two-dimensional density sampling points. It is a radial basis function. It is a low-order polynomial; the constant term is taken based on experimental data. The weighting coefficients are solved using the least squares method.

[0023] Preferably, the optimal grasping posture is selected using the NMS algorithm, including:

[0024] Input: All candidate grab poses and their corresponding scores;

[0025] Sort all candidate poses in descending order of score and select the grasping pose with the highest score, then add it to the optimal solution set.

[0026] Traverse the remaining candidate poses:

[0027] Calculate the similarity between the current candidate pose and all existing poses in the optimal solution set;

[0028] If the similarity between the current candidate pose and any pose in the optimal solution set exceeds a preset threshold, then the current candidate pose is suppressed or eliminated.

[0029] If the similarity between the current candidate pose and all poses in the optimal solution set is below a threshold, then add it to the optimal solution set.

[0030] Repeat the traversal process until all candidate poses have been processed.

[0031] Output the optimal grasping posture.

[0032] Preferably, the data acquisition module further includes determining whether the occlusion of a portion of the target object is present. This involves inputting a pre-set generative adversarial network with the contour image of the occluded portion of the target object and outputting the completed contour of the target object. Determining whether the target object is occluded includes:

[0033] The network model is back-projected onto the images of the target object from various viewpoints, and the projection residuals are calculated. If the projection residuals are greater than a preset threshold, it is determined that the target object is occluded at that viewpoint.

[0034] Preferably, the Transformer architecture includes:

[0035] Encoder: Using Vision Transformer (ViT) as the backbone network, the RGB image is segmented into fixed-size tiles, linearly embedded, with positional encoding added, and then processed through multiple layers of Transformer encoder blocks;

[0036] Decoder: Receives the encoder's output, progressively upsamples it, and fuses features from different levels. The Transformer decoder block uses a cross-attention mechanism to output a contour feature map with the same resolution as the input RGB image.

[0037] Preferably, the Transformer architecture includes enhancing the edge gradients of the contour feature map using the Laplacian operator.

[0038] Preferably, the data acquisition module includes:

[0039] Use a weighing sensor to obtain mass data;

[0040] Two-dimensional density maps are obtained using an industrial CT scanner;

[0041] Acquire RGB images using an RGB-D camera

[0042] Use stereo vision technology to acquire depth images;

[0043] It uses GPIO signal lines to connect the trigger interfaces of the industrial CT scanner and RGB-D camera.

[0044] The data acquisition module further includes aligning pixels in the acquired RGB and depth images, including:

[0045] The rotation matrix R and translation vector t of the RGB-D camera and depth sensor are obtained using the hand-eye calibration method;

[0046] Map the RGB pixel coordinates A to the depth camera's coordinate system B using extrinsic parameters: .

[0047] Preferably, the six-axis robot includes different types of end effectors, and the different types of end effectors are used to grasp different target objects.

[0048] The beneficial effects of this invention are as follows:

[0049] This invention constructs a 3D point cloud by fusing RGB images, depth images, and 2D density maps, and utilizes an encoder-decoder Transformer architecture to achieve cross-modal feature alignment. This enables accurate extraction of object contours and spatial location information even in complex scenes, such as those with varying lighting or missing textures. Furthermore, it uses mass distribution data to understand the mass distribution of the target object, thereby improving the robot's grasping accuracy. Attached Figure Description

[0050] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0051] Figure 1 This is a system structure block diagram of the present invention. Detailed Implementation

[0052] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0053] Please see Figure 1 This embodiment provides a multi-model flexible six-axis robot vision grasping system, including:

[0054] The data acquisition module is used to acquire the target object's mass data, two-dimensional density map, RGB image, and depth image, including:

[0055] Use load cells (such as electronic scales integrated under the conveyor belt or on the workbench) to measure the mass of the target object as it enters the gripping station;

[0056] To obtain a two-dimensional density map, specific imaging techniques (such as industrial CT scanners, X-ray imaging, terahertz imaging, acoustic imaging, or density estimation algorithms based on multi-view reconstruction) are used to acquire the density distribution information of the target object in a two-dimensional plane (typically a plane perpendicular to the grasping direction). This generates a two-dimensional array or image DensityMap(x, y), whose values ​​represent local relative densities.

[0057] Use an RGB-D camera to capture RGB images of the target object;

[0058] Use a depth sensor (such as a structured light camera, a binocular stereo vision camera, or a ToF camera) to simultaneously capture a depth image aligned with the RGB image, where each pixel value represents the distance (Z coordinate) of that point from the camera.

[0059] The trigger interface of the industrial CT scanner, RGB-D camera and sensor used to acquire depth images of the target object is connected by GPIO signal lines, so that the data acquisition is aligned in time and synchronously acquired, which facilitates the subsequent fusion of multiple images of the target object.

[0060] Since the data acquisition module uses different cameras to capture and extract RGB and depth images respectively, the target detection module synthesizes a point cloud image of the target object based on the acquired RGB and depth images. To facilitate the next step of stereo matching in the target detection module, the data acquisition module also includes pixel alignment in the acquired RGB and depth images, including:

[0061] The rotation matrix R and translation vector t of the RGB-D camera and depth sensor are obtained using the hand-eye calibration method;

[0062] Map the RGB pixel coordinates A to the depth camera's coordinate system B using extrinsic parameters, and then calculate coordinate system B using the following formula:

[0063] .

[0064] The target detection module synthesizes a point cloud map of the target object based on RGB images and depth images. It uses the Transformer architecture with encoder-decoder structure to extract the contour feature data of the RGB image and applies the contour feature data to the depth image to extract the depth point cloud data corresponding to the target object.

[0065] Point cloud synthesis: Using the camera's intrinsic parameters, each effective depth pixel is back-projected into 3D space to generate an initial point cloud coordinate system containing all objects in the scene (including the background).

[0066] RGB contour feature extraction: Input the RGB image into a Transformer network (e.g., SETR, SegFormer) based on an encoder-decoder structure;

[0067] Encoder: Typically, a Vision Transformer (ViT) or its variants are used as the backbone network. The RGB image is segmented into fixed-size patches, linearly embedded, and positionally encoded before being processed through multiple layers of Transformer encoder blocks. The encoder output is a feature map or feature sequence containing rich contextual information.

[0068] Decoder: Receives the encoder's output, progressively upsamples, and fuses features from different levels. The Transformer decoder block (or a lighter CNN decoder) utilizes self-attention / cross-attention mechanisms to focus on the object's boundary information. The final output is a contour feature map or segmentation mask at the same resolution as the input image. The contour feature map may contain high-dimensional feature vectors, highlighting the object's precise boundaries and shape information. Contour features provide more robust and boundary-focused object location information than the original RGB image.

[0069] Deep point cloud data extraction: Using the contour feature map or segmentation mask obtained in step 2 as a guide, the point cloud data is processed, including:

[0070] Based on segmentation mask: If the decoder output is a segmentation mask SegMask (binary or multi-class image, with the target object region being 1), then the segmentation mask is directly used as the mask to filter out the point cloud data with the corresponding mask value of the target category, and the depth point cloud data of the target object is obtained.

[0071] Based on contour feature maps: If the output is a high-dimensional ContourFeatMap, its information needs to be "fused" or "guided" for deep point cloud segmentation, including:

[0072] Upsample the ContourFeatMap values ​​to the same resolution as the pixels in the depth image;

[0073] The channel dimension of ContourFeatMap (containing contour information) is concatenated or weighted and fused with the pixels of the depth image (or its derived features) at the feature level;

[0074] The fused features are input into a lightweight segmentation head (such as a 1x1 convolutional layer) to predict the probability of each pixel belonging to the target object.

[0075] The segmentation mask SegMask is obtained by thresholding the predicted probability or using argmax;

[0076] Use SegMask to extract the depth point cloud data of the target object from the point cloud data;

[0077] Output depth point cloud data.

[0078] The weight compensation module fuses the depth point cloud data and mass data of the target object to generate mass distribution data, and extracts the center of gravity coordinates of the target object from the mass distribution data.

[0079] The fusion of deep point cloud data and quality data adopts the following method:

[0080] S1: The radix basis function (RBF) interpolation algorithm maps discrete sampling points of a 2D density map to a 3D point cloud surface using a distance-weighted kernel function, generating a continuous mass density field. Specifically, it includes:

[0081] The pixel coordinates of the 2D density map are projected onto the 3D point cloud surface using perspective transformation (PT) to generate corresponding sampling points. ;

[0082] Construct an interpolation system to calculate the mass density value for each 3D point cloud coordinate. , The calculation formula is as follows:

[0083] ;

[0084] Where x is the coordinate in the three-dimensional data corresponding to the target object. These are the three-dimensional projected coordinates corresponding to the two-dimensional density sampling points. It is a radial basis function. It is a low-order polynomial; the constant term is taken based on experimental data. Therefore, it is solved using the least squares method.

[0085] S2: The mass density field is optimized using the variable density SIMP algorithm to minimize the non-uniformity of the mass distribution;

[0086] S3: Obtain the ambient temperature factor and perform temperature compensation correction on each point of the optimized mass density field coordinates;

[0087] S4: The corrected mass density field is fused with the point cloud coordinates corresponding to the depth point cloud data to generate mass distribution data.

[0088] The Open3D algorithm was used to visualize the mass distribution of the target object, and the accuracy was verified by comparing it with mass data.

[0089] The pose calculation module generates and filters the most stable and feasible optimal grasping posture for the six-axis robot based on the mass distribution data of the target object, including:

[0090] The quality distribution data is used as input to a pre-trained neural network, and the point cloud is used as the sampling location for grasping. Feature extraction is performed on a fixed-size image centered on the grasping location, and the grasping angle is sampled from 0° to 170° at each grasping location with a step size of 10°. That is, the angle estimation is regarded as an 18-class classification problem. The NMS algorithm is used to select the grasping location and grasping angle corresponding to the highest score as the optimal grasping posture.

[0091] The neural network here contains convolutional layers and fully connected layers, and is generated by self-supervised learning training using historical grasping position and pose data from different robot models;

[0092] The NMS algorithm is used to remove highly overlapping (similar) candidate grasps, retaining the best grasp pose with the highest score and sufficient differences from each other. The execution process includes:

[0093] Input: All candidate grab poses and their corresponding scores;

[0094] Sort all candidate poses in descending order of score and select the grasping pose with the highest score, then add it to the optimal solution set.

[0095] Traverse the remaining candidate poses:

[0096] Calculate the similarity between the current candidate pose and all existing poses in the optimal solution set (e.g., whether the Euclidean distance of the grab point, the angle between the direction vectors, etc. are less than the threshold);

[0097] If the similarity between the current candidate pose and any pose in the optimal solution set exceeds a preset threshold (i.e., too similar), then the current candidate pose is suppressed (removed).

[0098] If the similarity between the current candidate pose and all poses in the optimal solution set is below a threshold (i.e., sufficiently different), then add it to the optimal solution set.

[0099] Repeat the traversal process until all candidate poses have been processed, and select the first pose in the optimal solution set (i.e. the one with the highest score and not suppressed) as the optimal grasping pose.

[0100] Output: The calculated optimal grasping posture of the robot's end effector.

[0101] The six-axis robot control module converts the optimal grasping posture into joint motion commands for a specific six-axis robot model, and controls the six-axis robot to safely and accurately complete the grasping action based on the joint motion commands. Its execution process includes:

[0102] The target joint position of the robot is obtained by transforming the optimal grasping posture to the base coordinate system of a specific robot model;

[0103] Planning the robot's joint motion trajectory: Plan the end-effector straight line or circular arc path in Cartesian space, and then convert it into a joint trajectory for the robot's motion trajectory from the current joint position to the target joint position;

[0104] The planned motion trajectory is discretized into a series of target joint angles at time points, and these target joint angles (as well as velocities, accelerations, etc.) are packaged into joint motion instructions (such as MoveJ, ServoJ, etc.) that can be understood by a specific robot controller.

[0105] Instructions are sent to the robot controller of a specific target model via the robot control interface;

[0106] For robots of a specific target model, force sensor feedback and position feedback are set in the robot's end effector to detect the gripper status and confirm successful grasping.

[0107] To improve the accuracy of the target detection module in recognizing the contours of target objects, in one embodiment, the self-attention layer of the Transformer encoder uses a deformable attention mechanism, specifically including:

[0108] A lightweight CNN branch is inserted before the encoder's self-attention layer. It contains three 3×3 convolutional layers (with 64, 32, 16 channels respectively) and a ReLU function. Its input is the current Query feature map (size H×W×C), and its output is the spatial offset Δp (size H×W×2N, where N is the number of sampling points). The relative offset is learned through the residual structure to avoid training oscillations caused by directly regressing absolute coordinates.

[0109] The Transformer architecture performs cross-modal fusion of contour feature data and geometric feature data from depth images. It constructs a three-layer feature pyramid, performing Laplacian enhancement on each layer and upsampling it to a uniform size. A channel attention mechanism (SE-Block) is used to weight and fuse the contour feature data and geometric feature data from the depth image. A weight generation network is used, which includes global average pooling and two fully connected layers. The Laplacian operator is used to enhance the gradient of edge contour features. Specifically, by using a three-layer feature pyramid... The Laplacian convolution kernel performs convolution operations on the output feature map of Transformerd, preserving positive gradient responses to enhance salient edges, thereby improving the accuracy of the target object.

[0110] The data acquisition module also includes determining whether the occlusion of a portion of the target object is present. The input of the occluded contour image to a pre-set generative adversarial network outputs the completed contour of the target object. Determining whether the target object is occluded includes:

[0111] The network model is back-projected onto the images of the target object from various viewpoints, and the projection residuals are calculated. If the projection residuals are greater than a preset threshold, it is determined that the target object is occluded at that viewpoint.

[0112] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A multi-model flexible six-axis robot vision grasping system, characterized in that: include: The data acquisition module is used to acquire the mass data, two-dimensional density map, RGB image and depth image of the target object; The target detection module is used to synthesize a point cloud map of the target object based on the RGB image and the depth image, extract the contour feature data of the RGB image using the Transformer architecture of the encoder-decoder structure, and apply the contour feature data to the depth image to extract the depth point cloud data of the target object. The weight compensation module is used to fuse the depth point cloud data and mass data of the target object to generate mass distribution data. The pose calculation module is used to generate several sets of grasping pose samples based on the mass distribution data, and to select the optimal grasping pose from several grasping pose samples based on the NMS algorithm. The six-axis robot control module converts the optimal grasping posture into joint motion commands for the six-axis robot and controls the six-axis robot to perform grasping operations based on the joint motion commands. In the weight compensation module, the depth point cloud data and the mass data are fused using the following steps: S1: Based on the radial basis function difference algorithm, the discrete sampling points of the two-dimensional density map are mapped to the surface of the three-dimensional point cloud through the kernel function of the distance weight, so as to generate a continuous mass density field. S2: The mass density field is optimized using the variable density SIMP algorithm to minimize the non-uniformity of the mass distribution; S3: Obtain the ambient temperature factor and perform temperature compensation correction on each point of the optimized mass density field coordinates; S4: The corrected mass density field is fused with the point cloud coordinates corresponding to the depth point cloud data to generate mass distribution data. The Open3D algorithm was used to visualize the mass distribution of the target object, and the accuracy was verified by comparing it with mass data.

2. The multi-model flexible six-axis robot vision grasping system according to claim 1, characterized in that: Step S1 includes the following steps: The pixel coordinates of the 2D density map are projected onto the 3D point cloud surface using perspective transformation, generating corresponding sampling points. ; Construct an interpolation system to calculate the mass density value for each 3D point cloud coordinate. , The calculation formula is: ； Where x is the coordinate in the three-dimensional data corresponding to the target object. These are the three-dimensional projected coordinates corresponding to the two-dimensional density sampling points. It is a radial basis function. It is a low-order polynomial; the constant term is taken based on experimental data. The weighting coefficients are solved using the least squares method.

3. The multi-model flexible six-axis robot vision grasping system according to claim 1, characterized in that: The NMS algorithm is used to select the optimal grasping posture, including: Input: All candidate grab poses and their corresponding scores; Sort all candidate poses in descending order of score and select the grasping pose with the highest score, then add it to the optimal solution set. Traverse the remaining candidate poses: Calculate the similarity between the current candidate pose and all existing poses in the optimal solution set; If the similarity between the current candidate pose and any pose in the optimal solution set exceeds a preset threshold, then the current candidate pose is suppressed or eliminated. If the similarity between the current candidate pose and all poses in the optimal solution set is below a threshold, then add it to the optimal solution set. Repeat the traversal process until all candidate poses have been processed. Output the optimal grasping posture.

4. The multi-model flexible six-axis robot vision grasping system according to claim 1, characterized in that: The data acquisition module further includes inputting the contour image of the occluded part of the target object into a pre-set generative adversarial network, and outputting the complete contour of the target object after completion. Determining that the target object is occluded includes: The network model is back-projected onto the images of the target object from various viewpoints, and the projection residuals are calculated. If the projection residuals are greater than a preset threshold, it is determined that the target object is occluded at that viewpoint.

5. The multi-model flexible six-axis robot vision grasping system according to claim 1, characterized in that: The Transformer architecture includes: Encoder: Using Vision Transformer as the backbone network, the RGB image is segmented into fixed-size tiles, linearly embedded, and positional encoding is added. Then, it is processed through multiple layers of Transformer encoder blocks. Decoder: Receives the output of the encoder, progressively upsamples and fuses features from different levels. The Transformer decoder block uses a cross-attention mechanism to output a contour feature map with the same resolution as the input RGB image.

6. The multi-model flexible six-axis robot vision grasping system according to claim 5, characterized in that: The Transformer architecture includes enhancing the edge gradients of the contour feature map using the Laplacian operator.

7. The multi-model flexible six-axis robot vision grasping system according to claim 1, characterized in that: The data acquisition module includes: Use a weighing sensor to obtain mass data; Two-dimensional density maps are obtained using an industrial CT scanner; Acquire RGB images using an RGB-D camera Use stereo vision technology to acquire depth images; It uses GPIO signal lines to connect the trigger interfaces of the industrial CT scanner and RGB-D camera.

8. The multi-model flexible six-axis robot vision grasping system according to claim 1, characterized in that: The data acquisition module further includes aligning pixels in the acquired RGB and depth images, including: The rotation matrix R and translation vector t of the RGB-D camera and depth sensor are obtained using the hand-eye calibration method; Map the RGB pixel coordinates A to the depth camera's coordinate system using extrinsic parameters: .

9. The multi-model flexible six-axis robot vision grasping system according to claim 1, characterized in that: The six-axis robot includes different types of end effectors, each used to grasp different target objects.

Images