A method for estimating the pose of a 3D target grasping robotic arm

Abstract

This invention relates to the field of visual grasping technology and discloses a method for estimating the pose of a three-dimensional target grasping robotic arm. The method involves acquiring RGB and depth images of a target scene, segmenting the RGB images to obtain multiple sub-images, encoding each sub-image, extracting image features from the encoded sub-images, decoding the image features to obtain a heatmap, extracting the grasping bounding box from the heatmap, and using a farthest-point sampling algorithm to extract multiple candidate grasping points from the optimal grasping region. Candidate grasping points with a confidence level of 1 are used as the center point of the spherical region of the object to be grasped. Candidate grasping points with a confidence level greater than a preset point confidence threshold are then selected outward from the center point to form the grasping region points. The grasping region points and grasping posture parameters are then input into a second MLP neural network structure. The optimal grasping region points and optimal grasping posture parameters are extracted from the grasping prediction results, thereby improving the target grasping efficiency for stacked or occluded problems.

Description

Technical Field

[0001] This invention relates to the field of visual grasping technology, and in particular to a method for estimating the pose of a three-dimensional target grasping robotic arm. Background Technology

[0002] For a robotic system project targeting the power industry, achieving fully automated target acquisition requires target localization and grasping. Target detection is a fundamental yet challenging task in robotic operation because it necessitates in-depth reasoning regarding the grasping stability of imperfect or partially observed objects, deriving the required 6D pose of the robotic arm's gripper end effector. The 6D pose refers to the position and angle generated by the robotic arm's end effector. Simultaneously, the impact of the grasping process on the scene must be considered, such as collisions with unrelated objects. Collisions mainly arise because the robot focuses too much on the local scene of the object to be grasped, neglecting the global deployment. This leads to frequent collisions with surrounding objects during subsequent grasping, altering the scene distribution and even causing grasping failure.

[0003] Currently, RGB images captured by a camera are used as input, and a backbone network is used to detect one or more object grasping boxes, including object position and rotation angle information. However, most camera capture methods take scattered objects on a plane as the observation object and are only suitable for three-degree-of-freedom grasping. This is more suitable for the top-down grasping steps of robotic grippers, but it is difficult to solve stacking or occlusion problems. At the same time, due to the lack of other observation results of the surrounding area of ​​the target, collisions are more likely to occur during actual grasping. For example, when the robotic arm grasps the power line clamp, there is a collision problem with the scene. Since collisions will change the working scene, the planning is likely to fail, resulting in a significant change in the working scene. Subsequent grasping requires more time for scene reconstruction, which affects grasping efficiency. Summary of the Invention

[0004] This invention provides a method for estimating the pose of a three-dimensional target grasping robot, which solves the technical problem of low grasping efficiency for targets with stacking or occlusion.

[0005] In view of this, the first aspect of the present invention provides a method for estimating the grasping pose of a three-dimensional target robotic arm, comprising the following steps:

[0006] Acquire the RGB image and depth image of the target scene;

[0007] The RGB image is segmented to obtain multiple sub-images, and each sub-image is encoded.

[0008] Extract image features from the encoded sub-images, decode the image features to obtain a heatmap, and extract capture boxes from the heatmap;

[0009] The depth image is converted into point cloud data through point cloud coordinate transformation. The point cloud data is then encoded using a point cloud encoding network to obtain encoded point cloud features. Finally, the encoded point cloud features are decoded using a point cloud decoding network to obtain pixel-level features.

[0010] The pixel-level features are fed into the first MLP neural network structure to obtain the point cloud confidence of each point cloud data. Based on the point cloud confidence of the point cloud data, the graspable three-dimensional region of each grasping target in the target scene is obtained. The graspable three-dimensional region is merged with the grasping box to obtain the optimal grasping region.

[0011] The farthest point sampling algorithm is used to extract multiple candidate grab points from the optimal grab area. The candidate grab points with a confidence level of 1 are used as the center point of the spherical area of ​​the object to be grabbed. Candidate grab points with a confidence level greater than a preset point confidence threshold are selected outward from the center point to form the grab area points.

[0012] Grasping posture parameters are defined based on the grasping area points. The grasping posture parameters include a three-dimensional direction vector, gripper grasping width, gripper distance from the object, and grasping success confidence.

[0013] The grasping region point and grasping posture parameters are input into the second MLP neural network structure to obtain the grasping prediction result of the gripper in the grasping region point. The optimal grasping region point and optimal grasping posture parameters are extracted from the grasping prediction result.

[0014] Preferably, the step of segmenting the RGB image to obtain multiple sub-images and encoding each sub-image specifically includes:

[0015] The RGB image is segmented to obtain multiple sub-images, wherein the number of sub-images i is N = H × W / P. 2 N represents the number of sub-images, H represents the height of the RGB image, W represents the width of the RGB image, and P represents the area of ​​each sub-image.

[0016] The images are encoded according to their relative positions, forming positionally encoded sub-patterns denoted as x, where x∈R. N ×(P×P×C) Where R represents space and C represents the number of channels in an RGB image.

[0017] Preferably, the step of extracting image features from the encoded sub-image, decoding the image features to obtain a heatmap, and extracting the capture box from the heatmap specifically includes:

[0018] The encoded sub-images are input into the encoder, and the sub-images are fused together through the fusion layer of the encoder. The multi-attention module of the encoder is then used to extract the image features from the fusion layer.

[0019] The image features are input into the decoder for decoding to obtain a heatmap. The heatmap is labeled with the confidence level of each pixel. Based on the confidence level of each pixel, the capture quality, capture angle, and capture width are extracted from the heatmap. The capture angle and the capture width are combined to form a capture box.

[0020] Preferably, the encoder includes a first sub-network module, an output module, and a second sub-network module connected in sequence;

[0021] The first sub-network module includes a first normalization layer and a multi-head attention layer. The first normalization layer is used to normalize the input values. The multi-head attention layer has three input interfaces: a query value, a key value, and a constant value. The multi-head attention layer is used to obtain the attention value using the following formula:

[0022]

[0023] In the formula, B is the scaling factor, Q is the code for the sub-plot, and K, V are the query value, key value, and constant value, respectively.

[0024] The output module comprises a second normalization layer and a first MLP layer. The second normalization layer is connected via residuals and is used to receive the output value of the first sub-network module and normalize the output value. The first MLP layer is used to extract MLP features from the output value after normalization by the second normalization layer.

[0025] The second sub-network module includes a third normalization layer, a sliding window attention block, and a fusion layer. The third normalization layer is used to normalize the MLP features. The residual connections between each sub-module in the sliding window attention block are calculated as follows:

[0026]

[0027]

[0028]

[0029]

[0030] In the formula, LN is the normalized input value, GA represents the fixed multi-attention function, LA is the sliding multi-attention function, and x t-1 This represents the feature input from the previous submodule. x represents the feature extracted by the current submodule. t The features that the current submodule will input to the next submodule. x represents the feature extracted by the next sub-module. t+1 The features that the next-level submodule will input to the next-level submodule;

[0031] The fusion layer is used to fuse the features of multiple sub-modules extracted by the sliding window attention block to obtain fused features.

[0032] Preferably, the image features are input into a decoder for decoding to obtain a heatmap, the heatmap being labeled with the confidence level of each pixel. The step of extracting the grasping quality, grasping angle, and grasping width from the heatmap based on the confidence levels of the pixels, and then assembling the grasping angle and grasping width into a grasping bounding box, specifically includes:

[0033] The image features are input into a decoder for decoding to obtain a heatmap, which is labeled with the confidence level of each pixel.

[0034] The pixels with a confidence level of 0.5 in the heat map are set as the grasping boundary of the object to be grasped. The angle of the object in the world coordinate system is obtained from the heat map as the grasping angle. The area with a confidence level of 0.3 in the heat map is set as the maximum grasping width.

[0035] The area covered by the gripping angle and the gripping width together forms a gripping frame.

[0036] Preferably, the specific steps for converting the depth image into point cloud data through point cloud coordinate transformation include:

[0037] The depth image is converted into point cloud data through point cloud coordinate transformation using the following formula:

[0038]

[0039] In the formula, x0, y0, z0 represent point cloud coordinates, x′, y′ represent image coordinates, D represents the depth value, and f x f y These represent the focal length of the camera in the horizontal and vertical axes, respectively.

[0040] Preferably, the loss function of the first MLP neural network structure is,

[0041]

[0042] In the formula, L1 is the loss function value of the first MLP neural network structure, and N o This represents the number of points in the point cloud data, where m represents any single point in the point cloud, M represents the point cloud data, and c represents the number of points in the point cloud data.m and These represent the actual confidence level and the predicted confidence level of points in the point cloud data, respectively.

[0043] Preferably, the grasping region point and grasping posture parameters are input into the second MLP neural network structure to obtain the grasping prediction result of the gripper in the grasping region point. The step of extracting the optimal grasping region point and optimal grasping posture parameters from the grasping prediction result specifically includes:

[0044] The grasping region points and grasping posture parameters are input into the second MLP neural network structure to obtain the grasping prediction result of the gripper in the grasping region points. The loss function of the second MLP neural network structure is...

[0045]

[0046] In the formula, L2 is the loss function value of the second MLP neural network structure. For the predicted value, i gt The values ​​represent the true values, where p is the gripping area point, r is the three-dimensional direction, w is the gripping width of the gripper, a is the distance the gripper is close to the object, and p is the confidence level of successful gripping.

[0047] The grabbing area points and grabbing posture parameters corresponding to the prediction results are filtered to obtain the grabbing area points and grabbing posture parameters with a grabbing success confidence of greater than 0.6, thus obtaining the optimal grabbing area points and optimal grabbing posture parameters.

[0048] Preferably, the method further includes:

[0049] The optimal grasping area points and the optimal grasping posture parameters are mapped onto the heat map to obtain the corresponding reference grasping box;

[0050] Determine whether the reference grasping box falls completely within the planar projection of the spherical region of the object to be grasped. If the reference grasping box falls completely within the planar projection of the spherical region of the object to be grasped, then the corresponding optimal grasping area point and the optimal grasping posture parameters are used as the final grasping area point and the final grasping posture parameters. If the reference grasping box does not fall completely within the planar projection of the spherical region of the object to be grasped, then the steps of acquiring the RGB image and depth image of the target scene are repeated.

[0051] As can be seen from the above technical solutions, the present invention has the following advantages:

[0052] This invention acquires RGB and depth images of a target scene, segments the RGB image to obtain multiple sub-images, encodes each sub-image, extracts image features from the encoded sub-images, decodes the image features to obtain a heatmap, extracts the capture box from the heatmap, and converts the depth image into point cloud data through point cloud coordinate transformation. The point cloud data is then encoded using a point cloud encoding network to obtain encoded point cloud features, which are then decoded using a point cloud decoding network to obtain pixel-level features. These pixel-level features are fed into a first MLP neural network structure to obtain the point cloud confidence score for each point cloud data point. Based on the point cloud confidence scores, the graspability of each grasping target in the target scene is determined. The three-dimensional region is merged with the grasping box to obtain the optimal grasping region. The farthest point sampling algorithm is used to extract multiple candidate grasping points from the optimal grasping region. The candidate grasping point with a confidence of 1 is used as the center point of the spherical region of the object to be grasped. Candidate grasping points with a confidence greater than a preset point confidence threshold are filtered outward from the center point to form the grasping region points. Grasping posture parameters are defined according to the grasping region points. The grasping region points and grasping posture parameters are input into the second MLP neural network structure to obtain the grasping prediction results of the gripper in the grasping region points. The optimal grasping region point and optimal grasping posture parameters are extracted from the grasping prediction results, thereby improving the target grasping efficiency for stacking or occlusion problems. Attached Figure Description

[0053] Figure 1 A flowchart illustrating a method for estimating the pose of a three-dimensional target grasping robot provided in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram illustrating the first mapping relationship between the reference grasping frame and the planar projection of the spherical region of the object to be grasped;

[0055] Figure 3 A schematic diagram illustrating the second mapping relationship between the reference grasping frame and the planar projection of the spherical region of the object to be grasped, provided in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram illustrating the third mapping relationship between the reference grasping frame and the planar projection of the spherical region of the object to be grasped, as provided in an embodiment of the present invention. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] For easier understanding, please refer to Figure 1 The present invention provides a method for estimating the pose of a three-dimensional target grasping robotic arm, comprising the following steps:

[0059] S1. Obtain the RGB image and depth image of the target scene.

[0060] S2. Segment the RGB image to obtain multiple sub-images, and encode each sub-image.

[0061] S3. Extract the image features of the encoded sub-image, decode the image features to obtain a heatmap, and extract the capture box from the heatmap.

[0062] S4. The depth image is converted into point cloud data through point cloud coordinate transformation. The point cloud data is encoded using a point cloud encoding network to obtain encoded point cloud features. Then, the encoded point cloud features are decoded using a point cloud decoding network to obtain pixel-level features.

[0063] S5. Input the pixel-level features into the first MLP neural network structure to obtain the point cloud confidence of each point cloud data. Based on the point cloud confidence of the point cloud data, obtain the graspable three-dimensional region of each grasping target in the target scene. Merge the graspable three-dimensional region with the grasping box to obtain the best grasping region.

[0064] S6. The farthest point sampling algorithm is used to extract multiple candidate grab points from the best grab area. The candidate grab points with a confidence level of 1 are used as the center point of the spherical area of ​​the object to be grabbed. The candidate grab points with a confidence level greater than the preset point confidence level threshold are selected from the center point to form the grab area points.

[0065] S7. Define the gripping posture parameters based on the gripping area points. The gripping posture parameters include the three-dimensional direction vector, the gripper gripping width, the distance of the gripper from the object, and the gripping success confidence.

[0066] For the 6D pose representation of the grasp, based on the grasping region points as the grasping target points, the five-dimensional grasping parameters are defined as (G t G r G w G a G p ), where G t Indicates the location of the grab point, G r G represents a three-dimensional direction vector, which can be decomposed into a vector along the Z-axis and a rotation about the Z-axis. w G represents the gripper width. a G represents the distance the gripper is close to the object. p This indicates the confidence level of a successful data capture.

[0067] S8. Input the grasping area point and grasping posture parameters into the second MLP neural network structure to obtain the grasping prediction result of the gripper in the grasping area point, and extract the optimal grasping area point and optimal grasping posture parameters from the grasping prediction result.

[0068] It should be noted that the three-dimensional target robotic arm grasping pose estimation method provided in this embodiment acquires RGB and depth images of the target scene, segments the RGB images to obtain multiple sub-images, encodes each sub-image, extracts image features from the encoded sub-images, decodes the image features to obtain a heatmap, extracts the grasping bounding box from the heatmap, and converts the depth image into point cloud data through point cloud coordinate transformation. The point cloud data is then encoded using a point cloud encoding network to obtain encoded point cloud features, which are then decoded using a point cloud decoding network to obtain pixel-level features. These pixel-level features are then fed into a first MLP neural network structure to obtain the point cloud confidence score for each point cloud data. Based on the point cloud confidence score, the target target is obtained. In the target scene, the graspable 3D region of each graspable target is merged with the grasping box to obtain the optimal grasping region. The farthest point sampling algorithm is used to extract multiple candidate grasping points from the optimal grasping region. The candidate grasping point with a confidence of 1 is used as the center point of the spherical region of the object to be grasped. Candidate grasping points with a confidence greater than a preset point confidence threshold are selected outward from the center point to form the grasping region points. Grasping posture parameters are defined according to the grasping region points. The grasping region points and grasping posture parameters are input into the second MLP neural network structure to obtain the grasping prediction result of the gripper in the grasping region points. The optimal grasping region point and optimal grasping posture parameters are extracted from the grasping prediction result, thereby improving the target grasping efficiency for stacking or occlusion problems.

[0069] In one specific embodiment, step S2 specifically includes:

[0070] S201. Segment the RGB image to obtain multiple sub-images, where the number of sub-images i is N = H × W / P. 2 N represents the number of sub-images, H represents the height of the RGB image, W represents the width of the RGB image, and P represents the area of ​​each sub-image.

[0071] S202. Encode each sub-image according to its relative position, dividing it into sub-image patches with positional codes, denoted as x, x∈R. N×(P×P×C) Where R represents space and C represents the number of channels in an RGB image.

[0072] In one example, sub-images at different locations are encoded in a top-to-bottom, left-to-right manner, forming sub-image tiles with location codes.

[0073] In one specific embodiment, step S3 specifically includes:

[0074] S301. Input the encoded sub-images into the encoder, fuse the sub-images through the encoder's fusion layer, and extract the image features from the fusion layer using the encoder's multi-attention module.

[0075] S302. Input the image features into the decoder for decoding to obtain a heatmap. The heatmap is labeled with the confidence level of each pixel. Based on the confidence level of each pixel, extract the grasping quality, grasping angle and grasping width from the heatmap. Combine the grasping angle and grasping width to form a grasping box.

[0076] In one specific embodiment, the encoder includes a first sub-network module, an output module, and a second sub-network module connected in sequence;

[0077] The first sub-network module includes a first normalization layer and a multi-head attention layer. The first normalization layer is used to normalize the input values. The multi-head attention layer has three input interfaces: a query value, a key value, and a constant value. The multi-head attention layer is used to obtain the attention value using the following formula.

[0078]

[0079] In the formula, B is the scaling factor, Q is the code for the sub-plot, and K, V are the query value, key value, and constant value, respectively.

[0080] The output module consists of a second normalization layer and a first MLP layer. The second normalization layer is connected via residuals to receive the output value of the first sub-network module and normalize the output value. The first MLP layer is used to extract MLP features from the output value after normalization by the second normalization layer.

[0081] The second normalization layer receives the input from the first sub-network through residual connections and normalizes it, reducing the resolution and distortion rate of the feature mapping. Then, the first MLP layer extracts the features of the corresponding image.

[0082] The second sub-network module includes a third normalization layer, a sliding window attention block, and a fusion layer. The third normalization layer is used to normalize the MLP features. The residual connections between each sub-module in the sliding window attention block are calculated as follows:

[0083]

[0084]

[0085]

[0086]

[0087] In the formula, LN is the normalized input value, GA represents the fixed multi-attention function, LA is the sliding multi-attention function, and x t-1 This represents the feature input from the previous submodule. x represents the feature extracted by the current submodule. t The features that the current submodule will input to the next submodule. x represents the feature extracted by the next sub-module. t+1 The features that the next-level submodule will input to the next-level submodule;

[0088] The fusion layer is used to fuse the features of multiple sub-modules extracted by the sliding window attention block to obtain fused features.

[0089] In this process, the image features extracted by the multi-attention module are first fixedly divided by a segmentation window, and then the resulting 2×2 fixed sub-features are obtained. At this time, the image features of the object to be grasped only exist in one of the sub-blocks. The MLP process is used to extract the image features, and then the extracted image features are divided by a sliding window to obtain 3×3 sub-features. At this time, the image features of the object to be grasped are distributed in multiple sub-blocks, which extends the features of the region around the object to be grasped.

[0090] Meanwhile, after extracting image features in both fixed and sliding windows, residual connections are added to each sub-module of the attention block, and merging operations are performed in each fusion layer. This allows the regional features of the object to be captured to be continuously fused with the surrounding regional features, so that our attention is no longer focused on the local area, but rather extended to the global area.

[0091] In one specific embodiment, step S302 specifically includes:

[0092] S3021. Input the image features into the decoder for decoding to obtain a heatmap, which is labeled with the confidence level of each pixel.

[0093] The decoder, primarily responsible for generating executable configuration parameters for the crawling end, consists of an attention block and an extension layer. The attention block further extracts features from the fusion layer, uses an MLP layer and a residual network structure to reduce the dimensionality of the feature information, and finally represents the result as a value between 0 and 1 through a normalization process. The extension layer is responsible for transmitting the fused image feature calculation results back to each sub-plot according to the positional encoding.

[0094] The extracted results are output as a heatmap H. Three gripping heads are connected to the output location, including gripping quality Q, gripping angle θ, and gripper width W. Gripping head H1 is connected to gripping quality Q, primarily obtaining the confidence level of each point from heatmap H. Points with a confidence level of 0.5 are set as the gripping boundary of the object to be gripped. Gripping head H2 is connected to the gripping angle θ, primarily obtaining the angle of the object in the world coordinate system from heatmap H. Gripping head H3 is connected to the gripping width W; to set the gripping width margin, the area with a confidence level of 0.3 is set as the maximum gripping width.

[0095] S3022. Set the pixels with a confidence level of 0.5 in the heat map as the grasping boundary of the object to be grasped, obtain the angle of the object in the world coordinate system from the heat map as the grasping angle, and set the area with a confidence level of 0.3 in the heat map as the maximum grasping width.

[0096] S3023. The area covered by the gripping angle and gripping width together constitutes the gripping frame.

[0097] It should be noted that this invention transforms the grasping pose problem into a regression problem, establishing a mapping from the image to the prediction result. I represents the input image. We improve our grasping accuracy by minimizing the difference between the predicted and actual values. The loss function is defined as follows:

[0098]

[0099] Where N is the number of samples, L i The corresponding original image, where l represents one of {Q, W, θ}. The prediction result is represented by Q, where Q is the gripping mass, θ is the gripping angle, and W is the gripper width.

[0100] The optimal crawling region corresponds to the optimal crawling probability. Regions in the heatmap with a confidence level greater than 0.5 are defined as our crawling regions. The optimal crawling probability is defined as:

[0101] G pos A =argmax pos Q

[0102] In the formula, G pos * Let Q be the probability of crawling, and let Q be the optimal confidence level for crawling.

[0103] In one specific embodiment, the specific steps for converting a depth image into point cloud data through point cloud coordinate transformation include:

[0104] The depth image is converted into point cloud data through point cloud coordinate transformation using the following formula:

[0105]

[0106] In the formula, x0, y0, z0 represent point cloud coordinates, x′, y′ represent image coordinates, D represents the depth value, and f x f y These represent the focal length of the camera in the horizontal and vertical axes, respectively.

[0107] In one specific embodiment, the loss function of the first MLP neural network structure is:

[0108]

[0109] In the formula, L1 is the loss function value of the first MLP neural network structure, and N o This represents the number of points in the point cloud data, where m represents any single point in the point cloud, M represents the point cloud data, and c represents the number of points in the point cloud data. m and These represent the actual confidence level and the predicted confidence level of points in the point cloud data, respectively.

[0110] Among them, the threshold c is set. th =0.5, which is the confidence level we set for the object surface to be grasped. If the confidence level is greater than the threshold, it indicates that the point is close to the object surface and is suitable for grasping. The output point cloud confidence level can obtain the graspable area of ​​each object in the working scene from a stereo perspective, which is more accurate than judging from a single image. At the same time, the sampled area is merged with the grasp box to obtain a more accurate grasping area.

[0111] In one specific embodiment, step S8 specifically includes:

[0112] S801. Input the grasping region points and grasping posture parameters into the second MLP neural network structure to obtain the grasping prediction result of the gripper in the grasping region points. The loss function of the second MLP neural network structure is...

[0113]

[0114] In the formula, L2 is the loss function value of the second MLP neural network structure. For the predicted value, i gt The values ​​represent the true values, where p is the gripping area point, r is the three-dimensional direction, w is the gripping width of the gripper, a is the distance the gripper is close to the object, and p is the confidence level of successful gripping.

[0115] S802. Filter the grasping area points and grasping posture parameters corresponding to the prediction results to obtain the grasping area points and grasping posture parameters with a grasping success confidence of greater than 0.6, and obtain the optimal grasping area points and optimal grasping posture parameters.

[0116] In one specific embodiment, the method further includes:

[0117] S11. Map the optimal grab area point and optimal grab posture parameters onto the heat map to obtain the corresponding reference grab box;

[0118] S12. Determine whether the reference grab box falls completely within the planar projection of the spherical region of the object to be grabbed. If the reference grab box falls completely within the planar projection of the spherical region of the object to be grabbed, then use the corresponding optimal grab area point and optimal grab posture parameters as the final grab area point and final grab posture parameters. If the reference grab box does not fall completely within the planar projection of the spherical region of the object to be grabbed, then repeat step S1.

[0119] in, Figures 2-4 This illustrates the mapping relationship between the reference grab box and the planar projection of the spherical region of the object to be grabbed. If the reference grab box falls entirely within the planar projection of the spherical region of the object to be grabbed, such as... Figures 2-3 If the reference grasping frame does not fall entirely within the planar projection of the spherical region of the object to be grasped, then the final grasping posture can be directly used based on the grasping point in this direction; if the reference grasping frame does not fall entirely within the planar projection of the spherical region of the object to be grasped, such as Figure 4 As shown, this indicates that the probability of collision is high in this direction, and therefore this grabbing direction should not be used. The predicted grabbing posture can be used to plan and execute the grabbing path. After execution, the work scene is predicted again, and the remaining items in the work scene are predicted. After obtaining a new executable spherical grabbing area, the grabbing action begins.

[0120] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0121] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0123] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for estimating the pose of a three-dimensional target grasping robotic arm, characterized in that, Includes the following steps: Acquire the RGB image and depth image of the target scene; The RGB image is segmented to obtain multiple sub-images, and each sub-image is encoded. Extract image features from the encoded sub-images, decode the image features to obtain a heatmap, and extract capture boxes from the heatmap; The depth image is converted into point cloud data through point cloud coordinate transformation. The point cloud data is then encoded using a point cloud encoding network to obtain encoded point cloud features. Finally, the encoded point cloud features are decoded using a point cloud decoding network to obtain pixel-level features. The pixel-level features are fed into the first MLP neural network structure to obtain the point cloud confidence of each point cloud data. Based on the point cloud confidence of the point cloud data, the graspable three-dimensional region of each grasping target in the target scene is obtained. The graspable three-dimensional region is merged with the grasping box to obtain the optimal grasping region. The farthest point sampling algorithm is used to extract multiple candidate grab points from the optimal grab area. The candidate grab points with a confidence level of 1 are used as the center point of the spherical area of ​​the object to be grabbed. Candidate grab points with a confidence level greater than a preset point confidence threshold are selected outward from the center point to form the grab area points. Grasping posture parameters are defined based on the grasping area points. The grasping posture parameters include a three-dimensional direction vector, gripper grasping width, gripper distance from the object, and grasping success confidence. The grasping region point and grasping posture parameters are input into the second MLP neural network structure to obtain the grasping prediction result of the gripper in the grasping region point. The optimal grasping region point and optimal grasping posture parameters are extracted from the grasping prediction result.

2. The method for estimating the grasping pose of a three-dimensional target robotic arm according to claim 1, characterized in that, The steps of segmenting the RGB image to obtain multiple sub-images and encoding each sub-image specifically include: The RGB image is segmented to obtain multiple sub-images, wherein the sub-images are... Quantity is N represents the number of sub-images, H represents the height of the RGB image, W represents the width of the RGB image, and P represents the area of ​​each sub-image. The images are encoded according to their relative positions, forming positionally encoded sub-patterns, denoted as x. Where R represents space and C represents the number of channels in an RGB image.

3. The method for estimating the grasping pose of a three-dimensional target robotic arm according to claim 2, characterized in that, Extracting image features from the encoded sub-images, decoding the image features to obtain a heatmap, and extracting capture boxes from the heatmap specifically includes: The encoded sub-images are input into the encoder, and the sub-images are fused together through the fusion layer of the encoder. The multi-attention module of the encoder is then used to extract the image features from the fusion layer. The image features are input into the decoder for decoding to obtain a heatmap. The heatmap is labeled with the confidence level of each pixel. Based on the confidence level of each pixel, the capture quality, capture angle, and capture width are extracted from the heatmap. The capture angle and the capture width are combined to form a capture box.

4. The method for estimating the grasping pose of a three-dimensional target robotic arm according to claim 3, characterized in that, The encoder includes a first sub-network module, an output module, and a second sub-network module connected in sequence. The first sub-network module includes a first normalization layer and a multi-head attention layer. The first normalization layer is used to normalize the input values. The multi-head attention layer has three input interfaces: a query value, a key value, and a constant value. The multi-head attention layer is used to obtain the attention value using the following formula: ； In the formula, B is the scaling factor, Q is the code for the sub-plot, and K, V are the query value, key value, and constant value, respectively. The output module comprises a second normalization layer and a first MLP layer. The second normalization layer is connected via residuals and is used to receive the output value of the first sub-network module and normalize the output value. The first MLP layer is used to extract MLP features from the output value after normalization by the second normalization layer. The second sub-network module includes a third normalization layer, a sliding window attention block, and a fusion layer. The third normalization layer is used to normalize the MLP features. The residual connections between each sub-module in the sliding window attention block are calculated as follows: ； In the formula, LN is the normalized input value, GA represents the fixed multi-attention function, and LA is the sliding multi-attention function. This represents the feature input from the previous submodule. This represents the features extracted by the current submodule. The features that the current submodule will input to the next submodule. This indicates the features extracted by the next sub-module. The features that the next-level submodule will input to the next-level submodule; The fusion layer is used to fuse the features of multiple sub-modules extracted by the sliding window attention block to obtain fused features.

5. The method for estimating the grasping pose of a three-dimensional target robotic arm according to claim 3, characterized in that, The image features are input into a decoder for decoding to obtain a heatmap. The heatmap is labeled with the confidence level of each pixel. Based on the confidence level of each pixel, the grasping quality, grasping angle, and grasping width are extracted from the heatmap. The step of forming a grasping box from the grasping angle and the grasping width specifically includes: The image features are input into a decoder for decoding to obtain a heatmap, which is labeled with the confidence level of each pixel. The pixels with a confidence level of 0.5 in the heat map are set as the grasping boundary of the object to be grasped. The angle of the object in the world coordinate system is obtained from the heat map as the grasping angle. The area with a confidence level of 0.3 in the heat map is set as the maximum grasping width. The area covered by the gripping angle and the gripping width together forms a gripping frame.

6. The method for estimating the grasping pose of a three-dimensional target robotic arm according to claim 3, characterized in that, The specific steps for converting the depth image into point cloud data through point cloud coordinate transformation include: The depth image is converted into point cloud data through point cloud coordinate transformation using the following formula: ； In the formula, Represents point cloud coordinates, This represents the image coordinates, and D represents the depth value. These represent the focal length of the camera in the horizontal and vertical axes, respectively.

7. The method for estimating the grasping pose of a three-dimensional target robotic arm according to claim 3, characterized in that, The loss function for the first MLP neural network architecture is: ； In the formula, L1 is the loss function value of the first MLP neural network structure, and N o This represents the number of points in the point cloud data, where m represents any single point in the point cloud, and M represents the point cloud data. and These represent the actual confidence level and the predicted confidence level of points in the point cloud data, respectively.

8. The method for estimating the grasping pose of a three-dimensional target robotic arm according to claim 3, characterized in that, The grasping region point and grasping posture parameters are input into the second MLP neural network structure to obtain the grasping prediction result of the gripper in the grasping region point. The specific steps of extracting the optimal grasping region point and optimal grasping posture parameters from the grasping prediction result include: The grasping region points and grasping posture parameters are input into the second MLP neural network structure to obtain the grasping prediction result of the gripper in the grasping region points. The loss function of the second MLP neural network structure is: ； In the formula, L2 is the loss function value of the second MLP neural network structure. For predicted values, The values ​​represent the true values, where p is the gripping area point, r is the three-dimensional direction, w is the gripping width of the gripper, a is the distance the gripper is close to the object, and p is the confidence level of successful gripping. The grabbing area points and grabbing posture parameters corresponding to the prediction results are filtered to obtain the grabbing area points and grabbing posture parameters with a grabbing success confidence of greater than 0.6, thus obtaining the optimal grabbing area points and optimal grabbing posture parameters.

9. The method for estimating the grasping pose of a three-dimensional target robotic arm according to claim 8, characterized in that, Also includes: The optimal grasping area points and the optimal grasping posture parameters are mapped onto the heat map to obtain the corresponding reference grasping box; Determine whether the reference grasping box falls completely within the planar projection of the spherical region of the object to be grasped. If the reference grasping box falls completely within the planar projection of the spherical region of the object to be grasped, then the corresponding optimal grasping area point and the optimal grasping posture parameters are used as the final grasping area point and the final grasping posture parameters. If the reference grasping box does not fall completely within the planar projection of the spherical region of the object to be grasped, then the steps of acquiring the RGB image and depth image of the target scene are repeated.

Images