A robotic arm grasping method, system, and robotic arm based on multi-view fusion

Abstract

This invention provides a robotic arm grasping method, system, and robotic arm based on multi-view fusion. The method includes obtaining voxel view features from point clouds, further obtaining voxel view features under the point view; obtaining data features in high-dimensional space from point clouds, stitching the point clouds together, and then outputting the point view features; projecting the point clouds to obtain a projected view, further obtaining the projected view features under the point view; combining the voxel view features under the point view, converting the feature channel count of the point view features and the projected view features under the point view to 1D, copying the expanded weights of the point view features and the projected view features under the point view, multiplying the expanded weights of the point view features with the point view features, multiplying the expanded weights of the projected view features under the point view with the projected view features under the point view, and adding the corresponding elements to obtain the multi-view fusion features. The robotic arm can then predict the grasping geometry, approach direction, and posture to grasp the target object.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and robotics, specifically relating to a robotic arm grasping method, system, and robotic arm based on multi-view fusion. Background Technology

[0002] With increasing expectations for efficient manufacturing and services, the demand for automation and intelligence is rising. Accurately grasping randomly arranged objects in model-less and complex environments is a fundamental challenge in industrial automation or service robots, as it relates to the robot's ability to learn general object manipulation methods. Specifically, estimating the grasping posture involves determining the position and orientation of the gripper required to grasp the object, which is crucial for collision-free and efficient manipulation. However, limitations in visual sensing accuracy, point cloud defects, and object occlusion in cluttered scenes make robot grasping tasks challenging under these conditions.

[0003] Traditional grasping methods use template matching to estimate the 6-DoF (6D pose tracking) pose of the target object, such as the PPF algorithm and ICP algorithm, and then use the pre-calculated grasping pose to determine the final grasping pose. However, since the grasping pose for each object needs to be manually designed, these methods are difficult to generalize to new objects. To address this issue, some researchers have adopted 2D detection-based methods to locate the object and estimate the gripper pose, such as the novel robot Dex-net 2.0 developed by researchers at UC Berkeley. However, the grasping pose of these methods is also limited to top-down views and is difficult to handle horizontally placed thin objects, such as plates. To overcome the limitations of 2D detection-based grasping methods, researchers have proposed 7-DoF (6-DoF with gripper width) grasping methods, such as Graspnet-1 Billion and Graspness.

[0004] However, current 7-DoF grasping methods employ a single voxel representation, which often lacks modeling and enhancement of geometric features and target-background relationships. This is because during voxelization, points assigned to the same cube are given identical feature vectors, leading to obscuring of the original geometric features. Furthermore, in the application context, these identical features correspond to different true grasping pose labels, which are annotated based on force closure metrics. This can negatively impact gradient descent during training, interfering with subsequent model performance. Since voxel space features are often extracted block by block along nearby cubes, the modeling neglects to model the relationship between the foreground object to be grasped and background-independent parts, making it difficult to accurately estimate the grasping pose. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a robotic arm grasping method, system, and robotic arm based on multi-view fusion, which has high precision. By introducing an attention mechanism, the point view features and the projected view features under the point view can be fused first, and then effectively adaptively combined with the voxel view features under the point view to obtain multi-view fused features, which facilitates the positioning of randomly placed target objects for grasping.

[0006] This invention is achieved through the following technical solution:

[0007] A robotic arm grasping method based on multi-view fusion includes:

[0008] S1, the point cloud of the scene to be captured is voxelized to obtain a series of voxel cubes, the voxel cubes are feature extracted to obtain voxel view features, and the voxel view features are mapped back to the point view to obtain voxel view features under the point view.

[0009] The point cloud is projected into a high-dimensional space to obtain encoded data features. The point cloud is then stitched together with the encoded data features to obtain stitched features. Finally, the stitched features are fed into a multilayer perceptron to output point view features.

[0010] The point cloud is projected along the z-axis using a pinhole camera model to obtain a projected view. The projected view stores the distance between the point cloud and the projected view in the z-axis direction. Then, features are extracted from the projected view to obtain the projected view features. The projected view features are mapped back to the point view to obtain the projected view features under the point view.

[0011] S2, the voxel view features, point view features and projection view features under the point view have the same number of feature channels. Combining the voxel view features under the point view, the number of feature channels of the point view features and the projection view features under the point view are converted into 1 dimension to obtain the weights of the point view features and the projection view features under the point view. Then, the extended weights of the point view features and the projection view features under the point view are obtained by copying. The dimension of the extended weights is the number of feature channels of the point view features.

[0012] S3, multiply the extended weight of the point view feature with the point view feature to obtain the fused point view feature, multiply the extended weight of the projection view feature under the point view with the projection view feature under the point view to obtain the fused projection view feature under the point view, add the corresponding elements in the voxel view feature under the point view, the fused point view feature and the fused projection view feature under the point view to obtain the multi-view fusion feature.

[0013] S4. Based on the multi-view fusion features, predict the grasping geometry region, approach direction, and grasping posture of the robotic arm, and grasp the target item.

[0014] Preferably, the voxel view features, point view features, and projection view features described in S1 have the same number of feature channels.

[0015] Preferably, in S1, the point cloud is projected into a high-dimensional space using a high-frequency function according to the following process:

[0016] Let the point cloud be x i =(k i l i and m i ), k i l i and m i Let represent the coordinates of any point in 3D coordinates, and let N be the number of point clouds. The high-frequency function is as follows:

[0017] HE(x i )=(cos(2 0 πk i ),cos(2 0 πl i ),cos(2 0 πm i ),sin(2 0 πk i ),sin(2 0 πl i ),sin(2 0 πm i ),

[0018] cos(2 1 πk i ),cos(2 1 πl i ),cos(2 1 πm i ),sin(2 1 πk i ),sin(2 1 πl i ),sin(2 1 πm i ),...,

[0019] cos(2 L-1 πk i ),cos(2 L-1 πl i ),cos(2 L-1 πm i ),sin(2 L-1 πk i ),sin(2 L-1 πli ),sin(2 L-1 πm i ));

[0020] Where L is a constant, L = 8-12.

[0021] Preferably, in S1, a point cloud of the scene to be captured is acquired using a depth camera, and the distance z between the point cloud and the projected view in the z-axis direction is... i Corresponding coordinates in the projected view (u) i ,v i ), u i and v i We obtain it from the following formula:

[0022]

[0023] Where f x and f y c represents the focal length of the depth camera in the x-axis and y-axis directions, respectively. x and c y These represent the offsets of the image origin relative to the optical center imaging point along the x-axis and y-axis, respectively. i and y i These represent the coordinates of the original point cloud along the x-axis and y-axis, respectively. i ,v i The value at position () is stored as z. i .

[0024] Preferably, in S2, the feature channel number of the point view features and the projected view features under the point view is converted into 1D using a multilayer perceptron according to the following formula:

[0025]

[0026] Among them G HE and G PP H represents the extended weights of the point view features and the projected view features under the point view, respectively. γ1 and H γ2 For a function with learnable parameters γ1 and γ2, the multilayer perceptron is represented as H. γ1 and H γ2 F v For voxel view features under point view, F HE For point view features, F PP For the projected view features under a point view, F v and F HE For H γ1 The input, F v and F PP For H γ2 Input.

[0027] Preferably, in S3, the extended weights of the point view features are combined with the point view features to perform a Hadamard product operation to obtain the fused point view features.

[0028] The expanded weights of the projected view features under the point view are combined with the projected view features under the point view using the Hadamard product to obtain the fused projected view features under the point view.

[0029] Preferably, S4 uses the Graspness method to determine the grasping geometry, approach direction, and grasping posture of the robotic arm. The specific process is as follows:

[0030] When predicting the grasping geometry region, the point cloud in the scene is first classified into easy grasping, difficult grasping, objects, and irrelevant. Then, the intersection point cloud of easy grasping and objects is found to complete the prediction of the grasping geometry region.

[0031] When predicting the approach direction, the approach directions are first divided into several categories. Then, for each grabbable point, the approach direction with the highest confidence is selected from these categories to complete the approach direction prediction.

[0032] When predicting the grasping posture, cylindrical clusters are performed based on the approach direction to obtain clustered features. Based on the clustered features, the combination of rotation angle, approach depth, and gripper width with the highest confidence is predicted to complete the grasping posture prediction.

[0033] A robotic arm grasping system based on multi-view fusion includes:

[0034] The voxel view feature acquisition module under the point view is used to voxelize the point cloud of the scene to be captured, extract features from the voxel cubes, and map the voxel view features back to the point view.

[0035] The point view feature acquisition module is used to project the point cloud into a high-dimensional space, stitch the point cloud after the encoded data features, and send the stitched features into a multilayer perceptron.

[0036] The projection view feature acquisition module under the point view is used to project the point cloud along the z-axis using a pinhole camera model, extract features from the projection view, and map the projection view features back to the point view.

[0037] The weight processing module is used to combine the voxel view features under the point view, convert the feature channel number of the point view features and the projected view features under the point view into 1 dimension, and obtain the extended weights of the point view features and the projected view features under the point view by copying.

[0038] The feature fusion module is used to multiply the extended weights of the point view features with the point view features, multiply the extended weights of the projected view features under the point view with the projected view features under the point view, and add the corresponding elements in the voxel view features under the point view, the fused point view features, and the fused projected view features under the point view.

[0039] The grasping information prediction module is used to predict the grasping geometry, approach direction and grasping posture of the robotic arm based on the multi-view fusion features.

[0040] A movement module is used to move the robotic arm to the target object based on the predicted grasping geometry, approach direction, and grasping posture.

[0041] A robotic arm, comprising:

[0042] Depth camera, used to acquire point cloud data of the scene to be captured;

[0043] Parallel mechanical grippers are used to grasp randomly placed target objects;

[0044] Memory, used to store computer programs;

[0045] A processor, configured to implement the steps of the multi-view fusion-based robotic arm grasping method described above when executing the computer program.

[0046] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the robotic arm grasping method based on multi-view fusion described above.

[0047] Compared with the prior art, the present invention has the following beneficial technical effects:

[0048] This invention voxelizes the original point cloud of the scene to be grasped, followed by feature extraction. However, voxelization blurs the original geometric features contained in the point cloud. Therefore, geometric modeling can be performed independently for each point cloud based on high-dimensional geometric analysis to ensure enhanced geometric description of each point. Addressing the lack of modeling the relationship between the target object and the background, a pinhole camera model is projected along the z-axis. The resulting projected view can be characterized by feature extraction to represent the relationship between the target object and the background. To achieve feature fusion in the point view, voxel view features need to be mapped back to the point view, and projected view features need to be mapped back to the point view. To selectively and uniformly and adaptively select different view features for fusion, the feature channel counts of the point view features and the projected view features under the point view are converted. Then, the expanded weights of the point view features and the projected view features under the point view are copied. Finally, multi-view fusion features are obtained by adding corresponding elements, facilitating the localization of randomly placed target objects, and enabling the prediction of the robotic arm's grasping geometry, approach direction, and grasping posture, ultimately leading to the grasping of the target object. This invention presents a robotic arm grasping method based on multi-view fusion, deployed on a robot system. It can be used to estimate grasping posture, enabling grasping without a target model in cluttered environments. Its effectiveness was verified on a benchmark (the public dataset Graspnet-1 Billion). After non-maximum suppression of the predicted rotation angles and positions (point coordinates), the top 50 sets with the highest confidence were selected for closed-loop evaluation of grasping force. The average evaluation score of this invention is the highest compared to existing methods, enabling the robot system to achieve high success rate and high reliability in grasping unknown target model objects in cluttered scenarios, and completing sorting tasks in unknown target environments. Attached Figure Description

[0049] Figure 1 This is a flowchart of the 7-DoF grasping method for intelligent robotic arms based on multi-view fusion enhancement as described in this invention.

[0050] Figure 2 This is a schematic diagram of the principle of the multilayer perceptron based on high-frequency geometric analysis of the present invention.

[0051] Figure 3 This is a schematic diagram illustrating the target-background modeling principle based on pinhole camera model projection of the present invention.

[0052] Figure 4 This is a schematic diagram of the attention mechanism principle in the point view of the present invention.

[0053] Figure 5a This is a schematic diagram of a real robot system after the algorithm is deployed.

[0054] Figure 5b It is a series of 7-DoF real robot grasping postures estimated on point clouds.

[0055] Figure 5c It is the grasping posture that the actual robot ultimately chooses to execute.

[0056] In the image: 1 is a depth camera, 2 is a parallel robotic gripper, 3 is a robotic arm, and 4 is a cluttered scene. Detailed Implementation

[0057] The present invention will be further described in detail below with reference to specific embodiments. These descriptions are for explanation purposes only and are not intended to limit the scope of the invention.

[0058] This invention discloses a high-precision 7-DoF grasping method for intelligent robotic arms based on multi-view fusion enhancement, such as... Figure 1 As shown, it includes the following steps:

[0059] Step 1: Voxelize the point cloud of the scene to be captured by the depth camera to obtain a series of voxel cubes. This process can be carried out in any voxelization method, such as conventional voxelization or sparse voxelization.

[0060] The voxel cubes are input into a voxel convolutional neural network that matches the voxel pattern for feature extraction, resulting in voxel view features.

[0061] Step 2: Voxelization blurs the original geometric features contained in the point cloud, which makes it difficult for the optimizer of the voxel convolutional neural network to perform gradient descent in the direction of minimizing the global loss function when learning different pose labels. This can be overcome by using the position information of the point cloud to extract geometric features.

[0062] Obtaining features from a multilayer perceptron based on high-frequency geometric analysis can reduce geometric blurring in the voxel view features in step 1. Specifically, this involves projecting the coordinate data of the point cloud acquired by the depth camera into a high-dimensional space using a high-frequency function. Let the point cloud of the scene to be captured be... This formula expresses a data set in 3D coordinates, x i =(k i l i and m i ), k i l i and m i These represent the coordinates of any point in 3D coordinates, including N points. The high-frequency functions are as follows:

[0063] Where L is related to the highest frequency of the trigonometric function in the high-frequency function, L = 8-12 is chosen, and 10 is chosen in this embodiment, thus obtaining HE(x) i )∈R 60 That is, the projected data is 60-dimensional. Each point in the point cloud individually captures geometric features through a high-frequency function to obtain encoded features.

[0064] For example, substituting the coordinates x1 = (1, 2, 3) into the coordinate values ​​of the three axes and then into each trigonometric function in the formula, we have:

[0065] HE(x1)=(cos(2 0 π×1),cos(2 0 π×2), cos(2 0 π×3),sin(2 0 π×1),sin(2 0 π×2),sin(2 0 π×3),...,cos(2 L-1 π×1),cos(2 L-1 π×2),cos(2 L-1 π×3),sin(2 L-1 π×1),sin(2 L-1 π×2),sin(2 L-1 π×3))

[0066] After using high-frequency function projection, multilayer perceptrons can better learn data containing high-frequency variations, capturing more detailed geometric distributions of the original point cloud. A multilayer perceptron can use a function W with a series of learnable parameters. θ This is represented by θ, where θ represents the learnable parameter. To more comprehensively mine the location information of the data, the original point cloud coordinates are concatenated with the features encoded by the high-frequency function, and then fed into the multilayer perceptron to output the geometric features of the point view, i.e., F. HE (x i ) = W θ ((concatenate(HE(x i ),x i That is, the spliced ​​data is 60+3 dimensions. For example... Figure 2 As shown, the collected point cloud is encoded by a high-frequency function and then processed by a multilayer perceptron based on high-frequency geometric analysis to obtain point view features. The 63 dimensions simply represent that each point carries 63 dimensions of features.

[0067] Step 3: The point cloud source required by the robotic arm is typically a depth camera, a type of depth sensor. Depth cameras are usually mounted on the robotic arm, with their z-axis aligned with the gripper for easy information acquisition. Therefore, the view along the z-axis is aligned with both the target and background objects. To establish the relationship between the target object and the actual background-independent parts, guiding higher-quality grasping, a projected view is obtained using a pinhole camera model. A U-shaped convolutional neural network is then used to establish a relationship between the target object and the actual background (i.e., a mapping relationship between the target object and the actual background) to guide the grasping process. Obtaining the projection of the pinhole camera model along the z-axis facilitates direct modeling of the target object and its irrelevant parts.

[0068] Instead of spherical projection, the original input point cloud is projected along the z-axis using a pinhole camera model. i The coordinates (u) in the projection view corresponding to the distance of the point cloud along the z-axis from the projection view i ,v i The calculation is as follows: z is stored in the projected view. i :

[0069]

[0070] Where f x and f y c represents the focal length of the depth camera in the x-axis and y-axis directions, respectively. x and c y These represent the offsets of the image origin relative to the optical center imaging point along the x-axis and y-axis, respectively. i and y i These represent the coordinates of the original point cloud along the x-axis and y-axis, respectively. i and v i Rounding down (in computer language) means retaining the integer part, (u i ,v i The value at position () is stored as z. i .

[0071] A U-shaped convolutional neural network is used to extract features from the projected view, maintaining the output layer's projected view feature size consistent with the original input. This feature describes the relationship between the target and the irrelevant background. Figure 3 As shown, the collected point cloud is projected onto a pinhole camera model to obtain a projected view. A U-shaped convolutional neural network is used to model the target object to be captured and the parts that are actually unrelated to the background.

[0072] Step 4: By setting parameters, the features obtained in the above three steps are made to have the same number of feature channels C. Not all features are reliable, and it is necessary to filter out those useless information. Therefore, an attention mechanism under point view is introduced.

[0073] To complete feature fusion in the point view, the voxel view features from step 1 and the projected view features from step 3 need to be mapped back to the point view. The former is obtained through voxelization mapping, and the latter is obtained through backprojection of the pinhole camera model, resulting in the voxel view features F under the point view. v ∈R N×C Projected view features F under point view PP ∈R N×C Then add the point view feature F obtained in step 2. HE ∈R N×C This constitutes a multi-view feature.

[0074] To implement the attention mechanism, the feature channel count of the point view features and the projected view features under the point view are converted into 1D using a multilayer perceptron, combining the voxel view features under the point view, for subsequent data fusion.

[0075]

[0076] Among them G HE and G PP H represents the weight of any point in the range [0,1]. γ1 and H γ2 It is a function with learnable parameters γ1 and γ2.

[0077] In order to use the attention mechanism point-by-point, the above weights need to be extended to the input feature F by replication. HE and F PP The same number of feature channels, i.e., the same weight values, are copied several times to obtain C values, which are then concatenated into a group to represent the corresponding weight value for that point. In this way, G... HE and G PP The weights are expanded to G HE ′∈R N×C and G PP '∈R N×C The Hadamard product is used to compute the feature F after the attention mechanism. HE ′ and F PP ′, where ⊙ represents the Hadamardi.

[0078]

[0079] Finally, the multi-view fusion feature F mv The calculation is as follows:

[0080] F mv =F V +F HE ′+F PP ′, which means the element-wise addition of tensors of the same size, F mvIt is characterized by using data as its form of expression.

[0081] like Figure 4 The process involves mapping voxel view features and projected view features to point views. The point view features and voxel view features under the point view are then passed through encoders A and B, both composed of multilayer perceptrons. The voxel view features under the point view and the projected view features under the point view are passed through encoders C and D, also composed of multilayer perceptrons, corresponding to encoders A and B, respectively. The features passed through encoder A are then added to the features passed through encoder B, and the features passed through encoder C are added to the features passed through encoder D. These features are then passed through a ReLU activation function and fed into decoders A and B, respectively. Both decoders are composed of multilayer perceptrons and output feature channels of 1. The outputs are then passed through a Sigmoid function to expand the channel number to C, aligning with the voxel view features, point view features, and projected view features. The expanded output features of encoder A are then multiplied by the point view features using a Hadamard product, and the expanded output features of decoder B are multiplied by the projected view features under the point view using a Hadamard product. The two sets of multiplied features are then directly added to the voxel view features under the point view, achieving fusion enhancement using an attention mechanism to obtain the final multi-view fused features.

[0082] Step 5: After obtaining the multi-view fusion features, the method for estimating the 7-DoF grasping pose is the same as the Graspness method. The grasping pose is decomposed into a classification problem, such as prediction of the graspable geometric region, prediction of the grasping approach direction, and prediction of the grasping pose (i.e., the gripper rotation angle and opening width).

[0083] First, predict the graspable geometric region. The point cloud in the scene can be divided into easy-to-grasp and difficult-to-grasp, and objects and irrelevant points. Use a neural network to find the intersection point cloud of easy-to-grasp and objects.

[0084] After predicting the graspable geometry, the approach direction is predicted for the point cloud of the graspable points. Similar to the previous work Graspness, the approach directions are divided into Vm categories. The neural network needs to select the grasping approach direction with the highest confidence from the Vm categories for each graspable point.

[0085] In this embodiment, a parallel mechanical gripper is selected. After choosing the optimal approach direction, based on the characteristics of the parallel mechanical gripper, cylindrical clustering is performed according to the approach direction. The clustered features are then fed into a neural network to predict the rotation angle, approach depth, and width of the parallel mechanical gripper. Unlike the rotation angle and approach depth, the width estimation is not a classification but a direct regression. Therefore, for a preset set of n rotation angle categories and m approach depth categories, the above process needs to predict the width for each case (n×m possibilities for each rotation angle and approach depth). Finally, the neural network provides the combination of rotation, approach depth, and width with the highest confidence.

[0086] The following are the loss functions needed to train the neural network described above:

[0087]

[0088] in, and There are two functions, if the condition in one of them is satisfied ( or If it is 1, then it is 1; otherwise, it is 0. s p is the label for whether it is an object. g It's a Graspness tag. and These are the predicted values ​​from the training process. For L s Cross-entropy is used, while the remaining tasks for all predictions use the smooth L1 loss function. λ1, λ2, λ3, λ4, λ5 are the weights that balance the multi-task learning loss, and λ1, λ2, λ3, λ4, λ5 and T are constants used for debugging.

[0089] Regarding the different processes described above, when training is a single task, L s The loss function for object segmentation, used to train a neural network to determine whether a point cloud represents an object or is unrelated; L g The loss function for predicting Graspness is used to train a neural network to determine the ease of grasping; L v The loss function for predicting the approach direction is used to train the neural network to predict the optimal approach direction to the target; L rd The loss function for predicting the rotation and approach depth of a parallel manipulator is used to train a neural network to predict the rotation and approach depth of the parallel manipulator; L w The loss function for estimating the width is used to train the neural network to predict the opening and closing width of the parallel manipulator. When handling the aforementioned multi-task tasks, these loss functions need to be combined into a joint training loss function to train the various modules simultaneously. For example, the loss function Loss is used, with training parameters adjusted using λ1, λ2, λ3, λ4, λ5, and T.

[0090] Step 6: Deploy to the robot system, taking the following steps:

[0091] A parallel gripper is mounted at the end of the robotic arm, and a depth camera is mounted on the robotic arm with its z-axis aligned with the parallel gripper. The robotic arm, parallel gripper, and depth camera are then connected to a computer. Hand-eye calibration of the robotic arm and depth camera is required, which can be performed with the eye on the hand or outside the hand. Each time the robot completes a grasping action, the neural network trained in the above steps is used to re-estimate the 7-DoF grasping pose of the current scene and select the most reliable grasping pose. The input point cloud directly outputs the result. The final pose of the robotic arm's end effector is determined using the hand-eye calibration results. The trajectory of the robotic arm is then planned using the ROS system or the robotic arm path planning algorithm provided by the robotic arm manufacturer to complete the grasping action.

[0092] Figure 5a The system on display is a pre-assembled and calibrated grasping robot system. Below the parallel mechanical grippers, some items are haphazardly arranged, such as watermelons, pears, biscuits, and bitter melons. Figure 5b This demonstrates the 7-DoF grasping posture estimated through steps 1-5 in the current scene, which includes parallel mechanical grippers in many different positions. Figure 5c The final gripping posture is shown, with a parallel mechanical gripper in only one position.

[0093] This invention discloses a robotic arm grasping system based on multi-view fusion, comprising the following modules:

[0094] The voxel view feature acquisition module under the point view can be used to voxelize the point cloud of the scene to be captured, extract features from the voxel cubes, and map the voxel view features back to the point view.

[0095] The point view feature acquisition module can be used to project the point cloud into a high-dimensional space, stitch the point cloud after the encoded data features, and send the stitched features into a multilayer perceptron.

[0096] The projection view feature acquisition module under the point view can be used to project the point cloud along the z-axis using a pinhole camera model, extract features from the projection view, and map the projection view features back to the point view.

[0097] The weight processing module can be used to combine voxel view features under point view, convert the feature channel number of point view features and projected view features under point view to 1D, and obtain extended weights of point view features and projected view features under point view by copying.

[0098] The feature fusion module can be used to multiply the extended weights of the point view features with the point view features, multiply the extended weights of the projected view features under the point view with the projected view features under the point view, and add the corresponding elements in the voxel view features under the point view, the fused point view features, and the projected view features under the point view.

[0099] The grasping information prediction module is used to predict the grasping geometry, approach direction and grasping posture of the robotic arm based on the multi-view fusion features. Specifically, it can be divided into a geometry prediction module, an approach direction prediction module and a grasping posture prediction module.

[0100] The geometric region prediction module can be used to classify point clouds in a scene into easy-to-grasp, difficult-to-grasp, objects, and irrelevant, and then find the intersection point cloud of easy-to-grasp and objects;

[0101] The proximity direction prediction module can be used to divide proximity directions into several categories, and then select the proximity direction with the highest confidence from these categories for each grabbable point;

[0102] The gripping posture prediction module can be used to cluster cylinders based on the approach direction, and predict the combination of rotation angle, approach depth and gripper width with the highest confidence based on the features after clustering.

[0103] The movement module can be used to move the robotic arm to the target object based on the predicted grasping geometry, approach direction, and grasping posture.

[0104] The robotic arm of the present invention also includes a memory, a processor, and a computer program stored in the memory that can run on the processor, wherein the processor executes the computer program to implement... Figure 1 The steps shown are steps 1-6 in the above-described robotic arm grasping method. Furthermore, the processor executes the computer program to implement the functions of each of the above modules. The processor can be a central processing unit, or other general-purpose processors, digital signal processors. The memory can be an internal storage unit of the robotic arm, such as the robotic arm's hard drive or RAM. The memory can also be an external storage device of the robotic arm, such as an external hard drive, smart memory card, etc.

[0105] The aforementioned computer program can be divided into multiple modules, which are stored in the aforementioned memory and executed by the aforementioned processor to realize the aforementioned solution of the present invention. The multiple modules can be a series of computer program instruction segments capable of performing the aforementioned functions, and these computer program instruction segments can describe the execution process of the aforementioned computer program in the robotic arm of the present invention.

[0106] When implementing all or part of the above-described solutions, the present invention can also be accomplished by instructing related hardware through a computer program. This computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the processes described in the multi-view fusion-based robotic arm grasping method. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium includes any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory, random access memory, electrical carrier signals, telecommunication signals, and software distribution media, etc.

Claims

1. A robotic arm grasping method based on multi-view fusion, characterized in that, include: S1, the point cloud of the scene to be captured is voxelized to obtain a series of voxel cubes, the voxel cubes are feature extracted to obtain voxel view features, and the voxel view features are mapped back to the point view to obtain voxel view features under the point view. The point cloud is projected into a high-dimensional space to obtain encoded data features. The point cloud is then stitched together with the encoded data features to obtain stitched features. Finally, the stitched features are fed into a multilayer perceptron to output point view features. The point cloud is projected along the z-axis using a pinhole camera model to obtain a projected view. The projected view stores the distance between the point cloud and the projected view in the z-axis direction. Then, features are extracted from the projected view to obtain the projected view features. The projected view features are mapped back to the point view to obtain the projected view features under the point view. S2, the voxel view features, point view features and projection view features under the point view have the same number of feature channels. Combining the voxel view features under the point view, the number of feature channels of the point view features and the projection view features under the point view are converted into 1 dimension to obtain the weights of the point view features and the projection view features under the point view. Then, the extended weights of the point view features and the projection view features under the point view are obtained by copying. The dimension of the extended weights is the number of feature channels of the point view features. S3, multiply the extended weight of the point view feature with the point view feature to obtain the fused point view feature, multiply the extended weight of the projection view feature under the point view with the projection view feature under the point view to obtain the fused projection view feature under the point view, add the corresponding elements in the voxel view feature under the point view, the fused point view feature and the fused projection view feature under the point view to obtain the multi-view fusion feature. S4. Based on the multi-view fusion features, predict the grasping geometry region, approach direction, and grasping posture of the robotic arm, and grasp the target item.

2. The robotic arm grasping method based on multi-view fusion according to claim 1, characterized in that, The voxel view features, point view features, and projection view features described in S1 have the same number of feature channels.

3. The robotic arm grasping method based on multi-view fusion according to claim 1, characterized in that, In S1, the point cloud is projected into a high-dimensional space using a high-frequency function according to the following process: Let the point cloud be x i =(k i l i and m i ), k i l i and m i Let represent the coordinates of any point in 3D coordinates, and let N be the number of point clouds. The high-frequency function is as follows: Where L is a constant, L = 8-12.

4. The robotic arm grasping method based on multi-view fusion according to claim 1, characterized in that, In S1, a point cloud of the scene to be captured is obtained using a depth camera. The distance z between the point cloud and the projected view along the z-axis is... i Corresponding coordinates in the projected view (u) i ,v i ), u i and v i We obtain it from the following formula: Where f x and f y c represents the focal length of the depth camera in the x-axis and y-axis directions, respectively. x and c y These represent the offsets of the image origin relative to the optical center imaging point along the x-axis and y-axis, respectively. i and y i These represent the coordinates of the original point cloud along the x-axis and y-axis, respectively. i ,v i The value at position () is stored as z. i .

5. The robotic arm grasping method based on multi-view fusion according to claim 1, characterized in that, In S2, the feature channel number of the point view features and the projected view features under the point view is converted into 1D using a multilayer perceptron via the following formula: Among them G HE and G PP H represents the extended weights of the point view features and the projected view features under the point view, respectively. γ1 and H γ2 For a function with learnable parameters γ1 and γ2, the multilayer perceptron is represented as H. γ1 and H γ2 F v For voxel view features under point view, F HE For point view features, F PP For the projected view features under a point view, F v and F HE For H γ1 The input, F v and F PP For H γ2 Input.

6. The robotic arm grasping method based on multi-view fusion according to claim 1, characterized in that, In S3, the extended weights of the point view features are combined with the point view features to perform a Hadamard product operation, resulting in the fused point view features. The expanded weights of the projected view features under the point view are combined with the projected view features under the point view using the Hadamard product to obtain the fused projected view features under the point view.

7. The robotic arm grasping method based on multi-view fusion according to claim 1, characterized in that, S4 uses the Graspness method to determine the robotic arm's grasping geometry, approach direction, and grasping posture. The specific process is as follows: When predicting the grasping geometry region, the point cloud in the scene is first classified into easy grasping, difficult grasping, objects, and irrelevant. Then, the intersection point cloud of easy grasping and objects is found to complete the prediction of the grasping geometry region. When predicting the approach direction, the approach directions are first divided into several categories. Then, for each grabbable point, the approach direction with the highest confidence is selected from these categories to complete the approach direction prediction. When predicting the grasping posture, cylindrical clusters are performed based on the approach direction to obtain clustered features. Based on the clustered features, the combination of rotation angle, approach depth, and gripper width with the highest confidence is predicted to complete the grasping posture prediction.

8. A robotic arm grasping system based on multi-view fusion, characterized in that, include: The voxel view feature acquisition module under the point view is used to voxelize the point cloud of the scene to be captured, extract features from the voxel cubes, and map the voxel view features back to the point view. The point view feature acquisition module is used to project the point cloud into a high-dimensional space, stitch the point cloud after the encoded data features, and send the stitched features into a multilayer perceptron. The projection view feature acquisition module under the point view is used to project the point cloud along the z-axis using a pinhole camera model, extract features from the projection view, and map the projection view features back to the point view. The weight processing module is used to combine the voxel view features under the point view, convert the feature channel number of the point view features and the projected view features under the point view into 1 dimension, and obtain the extended weights of the point view features and the projected view features under the point view by copying. The feature fusion module is used to multiply the extended weights of the point view features with the point view features, multiply the extended weights of the projected view features under the point view with the projected view features under the point view, and add the corresponding elements in the voxel view features under the point view, the fused point view features, and the fused projected view features under the point view. The grasping information prediction module is used to predict the grasping geometry, approach direction and grasping posture of the robotic arm based on the multi-view fusion features. A movement module is used to move the robotic arm to the target object based on the predicted grasping geometry, approach direction, and grasping posture.

9. A robotic arm, characterized in that, include: Depth camera, used to acquire point cloud data of the scene to be captured; Parallel mechanical grippers are used to grasp randomly placed target objects; Memory, used to store computer programs; A processor, configured to implement the steps of the multi-view fusion-based robotic arm grasping method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the robotic arm grasping method based on multi-view fusion as described in any one of claims 1 to 7.

Images