A robotic arm grasping method and system based on calibration constraint point cloud reconstruction

Abstract

This invention discloses a robotic arm grasping method and system based on calibration constraint point cloud reconstruction. The method includes: setting a calibration board at a target location in an industrial scene; performing hand-eye calibration on the robotic arm based on the calibration board to obtain a corresponding hand-eye transformation matrix; capturing images of the industrial scene using an industrial RGB camera mounted on the robotic arm to obtain a target image containing the calibration board; obtaining the calibration camera pose of the industrial RGB camera based on the target image; reconstructing a sparse point cloud based on the target image and the calibration camera pose to obtain a scene sparse point cloud corresponding to the industrial scene; performing path planning on the robotic arm based on the calibration camera pose, the scene sparse point cloud, and the hand-eye transformation matrix to obtain a planned grasping path for the robotic arm; and controlling the robotic arm to grasp the target according to the planned grasping path. This significantly improves the accuracy of sparse point cloud reconstruction, enhances the accuracy and stability of robotic arm grasping, and reduces the cost of image acquisition equipment.

Description

Technical Field

[0001] This invention relates to the field of robotic arm grasping technology, specifically to a robotic arm grasping method and a robotic arm grasping system based on calibration constraint point cloud reconstruction. Background Technology

[0002] In related technologies, in industrial applications, automated robotic arm grasping methods are usually based on RGB-D images. This not only increases the cost of image acquisition equipment, but also results in low precision of the sparse point cloud generated when using RGB-D images for automated robotic arm grasping. This leads to a large deviation in the robotic arm's grasping path, thus greatly reducing the accuracy and stability of the robotic arm's grasping. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a robotic arm grasping method based on calibration-constrained point cloud reconstruction. By constraining the point cloud reconstruction process with calibration information, the accuracy of sparse point cloud reconstruction is greatly improved, thereby enhancing the accuracy and stability of robotic arm grasping. Furthermore, automated robotic arm grasping can be achieved based on RGB images, reducing the cost of image acquisition equipment.

[0004] The technical solution adopted in this invention is as follows:

[0005] A robotic arm grasping method based on calibration constraint point cloud reconstruction includes the following steps: S1, setting a calibration board at the target location in an industrial scene, and performing hand-eye calibration on the robotic arm based on the calibration board to obtain a corresponding hand-eye transformation matrix; S2, taking pictures of the industrial scene using an industrial RGB camera mounted on the robotic arm to obtain a target image containing the calibration board; S3, obtaining the calibration camera pose of the industrial RGB camera based on the target image; S4, performing sparse point cloud reconstruction based on the target image and the calibration camera pose to obtain a scene sparse point cloud corresponding to the industrial scene; S5, performing path planning on the robotic arm based on the calibration camera pose, the scene sparse point cloud, and the hand-eye transformation matrix to obtain a planned grasping path for the robotic arm; S6, controlling the robotic arm to grasp the target according to the planned grasping path.

[0006] In one embodiment of the present invention, step S4 specifically includes the following steps: S41, using a first visual model to extract features from the target image to obtain corresponding visual features; S42, passing the visual features through the part before the output head of a second visual model, and adding camera embedding features according to the same settings as the second visual model to obtain predicted camera features and predicted image features located before the output head of the second visual model; S43, obtaining a pre-trained camera feature encoder, and using the camera feature encoder to encode the calibrated camera pose into calibrated camera features; S44, calculating the relative deviation between the calibrated camera features and the predicted camera features; S45, performing fine-tuning calculations based on the relative deviation and predicted image features using a cross-attention mechanism to obtain fine-tuned image features; S46, obtaining the last-level intermediate features of the fine-tuned image features located before the output head through a DPT model, and performing fine-tuning calculations using a self-attention mechanism after adding calibration position encoding to the intermediate features to obtain the last layer of input features input to the output head of the DPT model; S47, passing the last layer of input features through the last output head of the DPT model to obtain a sparse point cloud of the scene.

[0007] In one embodiment of the present invention, step S43 specifically includes: S431, encoding the calibrated camera pose into the camera feature space using the encoder module based on the VAE autoencoder; S432, reconstructing the calibrated camera pose based on the camera feature space using the camera output head provided by the second vision model to obtain the reconstructed calibrated camera pose; S433, training the encoder module of the VAE autoencoder based on the reconstructed calibrated camera pose and the calibrated camera pose using the reconstruction loss function to obtain the camera feature encoder.

[0008] In one embodiment of the present invention, step S5 specifically includes the following steps: S51, performing target recognition on the target image and extracting the corresponding target region; S52, using a pattern matching algorithm to extract key points from the target region, and filtering out a set of key points with a confidence level greater than a preset confidence level based on the key point matching algorithm; S53, performing point cloud registration based on the key point set and the sparse point cloud of the scene to obtain the world coordinate position of the target; S54, calculating the world coordinate system pose of the robotic arm based on the calibration camera pose and the hand-eye transformation matrix; S55, obtaining the planned grasping path of the robotic arm based on the world coordinate position of the target and the world coordinate system pose of the robotic arm.

[0009] In one embodiment of the present invention, step S55 specifically includes the following steps: S551, obtaining historical path information of the robotic arm in the sparse point cloud of the scene based on the world coordinate position of the grasping target and the world coordinate system pose of the robotic arm, wherein the historical path information includes a set of contact status information between the robotic arm and the industrial scene within a historical time period and a set of pose information of the robotic arm; S552, identifying and quantifying key features from the historical path information to obtain corresponding path features; S553, using a DRL network to obtain the path features through a multi-granularity path adjustment method to generate a target path adjustment method; S554, performing iterative path adjustment on the pose information set of the robotic arm based on the target path adjustment method to obtain a corresponding planned grasping path.

[0010] In one embodiment of the present invention, step S554 specifically includes: S5541, performing fragmented path interpolation processing on the path collision risk points in the robotic arm pose information set according to the path adjustment direction in the target path adjustment method to generate corresponding smooth path segments; S5542, performing dynamic obstacle avoidance path deviation detection on the smooth path segments to generate corresponding avoidance path deviation values; S5543, accumulating the smooth path segments and the avoidance path deviation values ​​to obtain candidate adjustment path segments; S5544, performing inverse kinematic analysis on the candidate adjustment path segments, and when analyzing the joint motion sequence corresponding to the candidate adjustment path segments within the degree of freedom constraint of the robotic arm, replacing the path segment corresponding to the path collision risk points with the candidate adjustment path segments to obtain iteratively adjusted grasping paths, and using them as the corresponding planned grasping paths.

[0011] A robotic arm grasping system based on calibration constraint point cloud reconstruction includes: a first acquisition module, which is used to set a calibration board at a target location in an industrial scene and perform hand-eye calibration on the robotic arm based on the calibration board to obtain a corresponding hand-eye transformation matrix; a second acquisition module, which is used to capture images of the industrial scene using an industrial RGB camera mounted on the robotic arm to obtain a target image containing the calibration board; a third acquisition module, which is used to acquire the calibration camera pose of the industrial RGB camera based on the target image; a fourth acquisition module, which is used to perform sparse point cloud reconstruction based on the target image and the calibration camera pose to obtain a scene sparse point cloud corresponding to the industrial scene; a fifth acquisition module, which is used to perform path planning on the robotic arm based on the calibration camera pose, the scene sparse point cloud, and the hand-eye transformation matrix to obtain a planned grasping path for the robotic arm; and a control module, which is used to control the robotic arm to grasp the target according to the planned grasping path.

[0012] The beneficial effects of this invention are:

[0013] This invention constrains the point cloud reconstruction process by using calibration information, thereby greatly improving the accuracy of sparse point cloud reconstruction, enhancing the accuracy and stability of robotic arm grasping, and enabling automated robotic arm grasping based on RGB images, thus reducing the cost of image acquisition equipment. Attached Figure Description

[0014] Figure 1 This is a flowchart of a robotic arm grasping method based on calibration constraint point cloud reconstruction according to an embodiment of the present invention;

[0015] Figure 2 This is a structural block diagram of sparse point cloud reconstruction according to an embodiment of the present invention;

[0016] Figure 3 This is a training block diagram of a camera feature encoder according to an embodiment of the present invention;

[0017] Figure 4 This is a block diagram of a robotic arm grasping system based on calibration constraint point cloud reconstruction, according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Figure 1 This is a flowchart of a robotic arm grasping method based on calibration constraint point cloud reconstruction, according to an embodiment of the present invention.

[0020] like Figure 1 As shown, the robotic arm grasping method based on calibration constraint point cloud reconstruction according to an embodiment of the present invention may include the following steps:

[0021] S1. Set up a calibration board at the target location in the industrial scene, and perform hand-eye calibration on the robotic arm based on the calibration board to obtain the corresponding hand-eye transformation matrix.

[0022] Specifically, hand-eye calibration of a robotic arm involves determining the fixed spatial transformation relationship between the camera ("eye") and the end effector ("hand") of the robotic arm. This transforms the visually detected target position from the camera coordinate system to the robotic arm coordinate system, enabling precise operation. This transformation relationship can be represented as a 4×4 homogeneous transformation matrix, including rotation and translation components. Depending on the mounting method of the camera and robotic arm, either an eye-on-hand calibration mode or an eye-on-hand external calibration mode can be used. It should be noted that the methods for hand-eye calibration are existing technologies and will not be detailed here.

[0023] S2 uses an industrial RGB camera mounted on a robotic arm to capture images of industrial scenes, including target images of calibration boards.

[0024] Specifically, an industrial RGB camera mounted on a robotic arm can be used to capture images of an industrial scene, and the images captured by the camera can be read. Then, a finely tuned YOLOv8 model is used to perform target recognition on the images, determining whether a calibration board and a grasping target can be simultaneously identified in the image. If neither the calibration board nor the grasping target is simultaneously identified, the robotic arm is controlled to perform a helical observation movement, and the image is captured again, and target recognition is performed again, until both the calibration board and the grasping target are simultaneously identified in the image. The image in which both the calibration board and the grasping target are simultaneously identified is taken as the target image. This process is repeated to acquire N target detection images I1~I N .

[0025] S3, obtain the calibration camera pose of the industrial RGB camera based on the target image.

[0026] Specifically, step S3 may include the following steps:

[0027] S31, the finely tuned YOLOv8 model is used to perform target recognition on the input i-th target detection image to obtain the region coordinates of the calibration board and extract the image of the region where the calibration board is located.

[0028] S32 uses a resolution transformation function to uniformly transform the image of the calibration board area to a preset resolution in order to obtain the image of the target calibration board area.

[0029] The resolution of the image of the calibration board area varies significantly depending on the camera's position. Therefore, the image of the calibration board area needs to be uniformly converted to a preset resolution. This preset resolution can be determined based on the specific circumstances; for example, it could be 768x768.

[0030] S33, the pattern query function is used to locate the calibration points in the calibration board area of ​​the target calibration board area image, and a set of two-dimensional coordinates of the calibration points are obtained.

[0031] S34. Based on the three-dimensional coordinates and two-dimensional coordinates of the calibration points set on the calibration board, perform PnP calculation to obtain the preliminary camera pose for each image calibration.

[0032] S35 uses a joint calibration function to jointly calibrate the camera pose of the multi-view image to obtain the calibrated camera pose.

[0033] S4 reconstructs sparse point clouds based on the target image and the calibrated camera pose to obtain the scene sparse point cloud corresponding to the industrial scene.

[0034] It should be noted that the structural block diagram of sparse point cloud reconstruction in one embodiment of the present invention can be as follows: Figure 2 As shown below, in conjunction with Figure 2 This section details how sparse point cloud reconstruction is performed. Specifically, in one embodiment of the invention, step S4 includes the following steps:

[0035] S41, the first visual model is used to extract features from the target image to obtain the corresponding visual features.

[0036] The first visual module can be a DINO (Distillation with No Labels) model. This involves processing the target image... (Right now, After inputting into the first vision module, it can output the visual features of each frame of the image, i.e. (Right now, ).

[0037] S42, the visual features are output from the part in front of the second visual model, and camera embedding features are added according to the same settings as the second visual model to obtain the predicted camera features and predicted image features located in front of the output head of the second visual model.

[0038] The second visual model can be a VGGT (Visual Geometry Grounded Transformer) model. Specifically, for example... Figure 2 As shown, visual features The front portion of the image is output using the second visual model VGGT, and camera embedding features are added using the same settings as the second visual model. (Right now, ), to obtain the predictive camera features located in front of the output head of the second vision model. (Right now, ) and predicted image features (Right now, ).

[0039] S43, obtain the pre-trained camera feature encoder, and use the camera feature encoder to encode the calibration camera pose into calibration camera features.

[0040] Specifically, a pre-trained camera feature encoder can first be obtained, where the training block diagram of the camera feature encoder can be as follows: Figure 3 As shown below, in conjunction with Figure 3 This section details how the pre-trained camera feature encoder is obtained. Specifically, in one embodiment of the invention, step S43 includes:

[0041] S431, the encoder module based on VAE autoencoder encodes the calibrated camera pose into the camera feature space.

[0042] Specifically, the encoder module of a VAE autoencoder can be used first to calibrate the camera pose. (Right now Encoding into the camera feature space, i.e., calibrating camera features. (Right now, ).

[0043] S432 reconstructs the calibrated camera pose based on the camera feature space using the camera output head provided by the second vision model to obtain the reconstructed calibrated camera pose.

[0044] Specifically, through the camera output head provided by the frozen second visual model. Reconstruct the calibration camera pose to obtain the reconstructed calibration camera pose, i.e., predict the camera pose. ( ).

[0045] S433, based on the reconstruction loss function, trains the encoder module of the VAE autoencoder according to the reconstructed and calibrated camera poses to obtain the camera feature encoder.

[0046] Specifically, it can be based on the reconstruction loss function Train the encoder module of the VAE autoencoder to obtain the camera feature encoder. It should be noted that the reconstruction loss function of the encoder module of the VAE autoencoder can adopt the loss function in the existing technology, and will not be limited to one by one here.

[0047] S44, calculate the relative deviation between the calibrated camera features and the predicted camera features.

[0048] Specifically, the calibration camera features can be calculated using the following formula. and predict camera features relative deviation between :

[0049] .

[0050] S45, based on the cross-attention mechanism, performs fine-tuning calculations based on relative bias and predicted image features to obtain fine-tuned image features.

[0051] Specifically, relative deviation As Q in the cross-attention mechanism, it predicts image features. In the cross-attention mechanism, Q represents the query vector used to find relevance; K represents the keyword vector used for comparison with the query vector; and V represents the value vector containing the actual information. Specifically, fine-tuning of image features can be achieved through the following formula:

[0052] .

[0053] S46: The last intermediate feature of the fine-tuned image features located before the output head is obtained through the DPT model. After adding the calibration position encoding to the intermediate features, a self-attention mechanism is used for fine-tuning calculation to obtain the last layer of input features input to the output head of the DPT model.

[0054] Specifically, fine-tuned image features are obtained through the DPT model. The last intermediate feature located before the output head and in the intermediate features After incorporating calibration location encoding (e.g., calibration location encoding based on 2D-3D point pairs), a self-attention mechanism is used for fine-tuning computation to obtain the last layer of input features input to the DPT model output head. The last layer of input features The calculation formula can still use the fine-tuned calculation formula mentioned above.

[0055] S47, the last layer of input features are passed through the final output head of the DPT model to obtain the sparse point cloud of the scene.

[0056] S5 performs path planning for the robotic arm based on the calibrated camera pose, sparse point cloud of the scene, and hand-eye transformation matrix to obtain the planned grasping path of the robotic arm.

[0057] In one embodiment of the present invention, step S5 specifically includes the following steps:

[0058] S51, perform target recognition on the target image and extract the corresponding target region.

[0059] Specifically, the target image can be identified and the corresponding target region can be extracted by fine-tuning the trained YOLOv8 model.

[0060] S52 uses a pattern matching algorithm to extract key points from the target area and selects a set of key points with a confidence level greater than the preset confidence level based on the key point matching algorithm.

[0061] Specifically, key points can be detected by analyzing image gradients or intensity changes, or key points can be extracted from 3D point cloud data of the target area using descriptors such as point feature histograms or fast point feature histograms.

[0062] After extracting keypoints, matching algorithms can be used to identify correspondences within target regions and filter high-confidence results. Specifically, each keypoint can generate a descriptor (such as a SIFT descriptor or a vector generated by deep learning) to quantify neighborhood features. During matching, corresponding point pairs are found by calculating the similarity between descriptors (such as Euclidean distance). For example, learned descriptors (DeepDesc) can optimize the matching of samples that are difficult to distinguish between categories.

[0063] Then, a confidence assessment is performed, which can be based on the geometric consistency or statistical confidence score of the matching results. Specifically, the coplanarity or consistency of matching point pairs can be checked through homography matrix or affine transformation to remove outliers. For example, the RANSAC algorithm can estimate the optimal transformation parameters and retain high-confidence matches with consistent patterns. Alternatively, the confidence scores of key points can be directly output based on deep learning methods (such as LIFT) for filtering. The higher the score, the stronger the matching reliability. In addition, threshold filtering of descriptor similarity (such as retaining only matches with similarity below a certain value) can also effectively remove low-quality matches.

[0064] S53 performs point cloud registration based on key point set and sparse point cloud of scene to obtain the world coordinate position of the target to be captured.

[0065] Specifically, point cloud registration involves finding the optimal rigid body transformation (rotation and translation) between one set of point clouds (source point cloud) and another set of point clouds (target point cloud) to align them in space. Here, the source point cloud is the set of keypoints, and the target point cloud is the sparse point cloud of the scene. The transformation matrix from the source point cloud to the target point cloud is calculated by matching descriptor clouds. Specifically, as one possible implementation, the transformation matrix can be calculated using coarse matching, for example, by quickly estimating the initial transformation using algorithms such as RANSAC or 4PCS. As another possible implementation, the transformation matrix can be calculated using fine registration, for example, by optimizing transformation parameters based on methods such as ICP (Iterative Nearest Point) or NDT (Normal Distribution Transformation) to improve registration accuracy. After obtaining the transformation matrix, the keypoint set and the sparse point cloud of the scene are aligned using the transformation matrix to determine the world coordinate position of the target object.

[0066] S54 calculates the world coordinate system pose of the robotic arm based on the calibrated camera pose and the hand-eye transformation matrix.

[0067] Specifically, the camera pose is first transformed from the camera coordinate system to the camera pose corresponding to the coordinate system of the robotic arm's end effector.

[0068] ,

[0069] in, This indicates the camera pose corresponding to the coordinate system of the robotic arm's end effector. This represents the hand-eye transformation matrix.

[0070] Secondly, the camera pose corresponding to the coordinate system of the robotic arm's end effector is transformed to the camera pose corresponding to the world coordinate system, i.e., the world coordinate system pose.

[0071] ,

[0072] in, Represents the pose in the world coordinate system. It indicates the pose of the end effector of the robotic arm, which can be obtained in real time from the status of the robotic arm.

[0073] Therefore, the world coordinate system pose of the robotic arm can be calculated using the following formula:

[0074] .

[0075] S55 obtains the planned grasping path of the robotic arm based on the world coordinate position of the grasped target and the world coordinate system pose of the robotic arm.

[0076] In one embodiment of the present invention, step S55 specifically includes the following steps:

[0077] S551 obtains the historical path information of the robotic arm in the sparse point cloud of the scene based on the world coordinate position of the grasping target and the world coordinate system pose of the robotic arm. The historical path information includes a set of information on the contact status between the robotic arm and the industrial scene within a historical time period, as well as a set of pose information of the robotic arm.

[0078] Specifically, based on the world coordinate position of the target and the world coordinate system pose of the robotic arm, the historical path information of the robotic arm grasping the target in the sparse point cloud of the scene can be extracted from the database.

[0079] The contact status information set in the historical path information can include the contact status information between the robotic arm and the industrial scene at each historical moment within the historical time period. Specifically, it can include relevant information about the obstruction at a certain moment, such as the relative position of the obstruction and the robotic arm, the appearance of the obstruction, and the current state of the obstruction. The pose information set of the robotic arm in the historical path information can include the pose information of the robotic arm at each historical moment within the historical time period.

[0080] S552 identifies and quantifies key features from historical path information to obtain corresponding path features.

[0081] In one embodiment of the present invention, the path features may include: the arrangement of obstacles, the degree of freedom restriction features, and the path connection status.

[0082] Specifically, the arrangement of obstructions can be extracted from the contact status information. The arrangement of obstructions can include: the edge contour of the area where the obstructions are concentrated, the displacement tendency of the obstructions, and the safe clearance of the obstructions.

[0083] Freedom constraint features can be extracted from the pose information set of the robotic arm. These features may include: the allowable range of rotation angles of the robotic arm joints, the allowable range of curvature of the end effector's movement path, and the speed smooth transition control range.

[0084] Path disconnection detection is performed on the temporal matching relationship between the contact condition information set and the pose information set to obtain the path connection status. Specifically, the contact condition information set and the pose information set can be matched according to each historical moment to generate a time-registered path information set, and multiple time-continuous time-registered path information sets are integrated to obtain path segments. Then, path disconnection detection is performed based on changes in the contact condition information and changes in the robot arm's pose information, and path collision risk features are obtained to determine the path connection status, i.e., to determine whether there is a risk of path disconnection in the path segment.

[0085] S553 uses a DRL network to obtain path features through a multi-granularity path adjustment method to generate the target path adjustment method.

[0086] In one embodiment of the present invention, step S553 specifically includes the following steps:

[0087] S5531 inputs path features into the DRL network and generates a collaborative adjustment vector by integrating multi-granularity feature information.

[0088] Specifically, when inputting path features into the DRL network, each feature is first preprocessed. For obstacle arrangement, information such as the edge contours of the obstacle concentration area, the displacement tendency of the obstacles, and the safe clearance of the obstacles is quantified and standardized. For degree-of-freedom constraint features, the allowable range of rotation angles of the robotic arm joints, the allowable range of curvature of the end effector's movement path, and the speed smooth transition control range are quantified and standardized. For path connection, information such as path disconnection risk and path collision risk is adaptively transformed and standardized. Then, the preprocessed path features are input into the multi-granularity feature information integration module of the DRL network. In the multi-granularity feature information integration module, features of different granularities are gradually integrated. For example, low-level features are integrated first, and then the integrated information result is integrated with high-level features to obtain a collaborative adjustment mode vector.

[0089] S5532, Generate the priority order of path adjustment based on the collaborative adjustment weight configuration relationship in the collaborative adjustment method vector.

[0090] Specifically, firstly, the path adjustment elements represented by each element in the collaborative adjustment method vector can be analyzed. For example, one element in the vector can represent the weight of obstacle avoidance, another element can represent the weight of path smoothness, and so on. Then, these weights are sorted according to their magnitude. The larger the weight, the higher the priority of the corresponding path adjustment element in the path adjustment task.

[0091] S5533, based on the priority ranking of path adjustment, performs path replanning on the path collision risk features in the path features to obtain a set of alternative path adjustment strategies.

[0092] Specifically, for path collision risks with high priority, the specific reasons for the collision are analyzed. Based on the reasons for the collision, combined with the robotic arm's motion capabilities and scene information, different path adjustment strategies are used to adjust the path, thereby generating a set of alternative path adjustment strategies.

[0093] S5534 performs dynamic scenario adaptability testing on the set of alternative path adjustment strategies and selects the target path adjustment method.

[0094] S554 performs iterative path adjustment on the pose information set of the robotic arm based on the target path adjustment method to obtain the corresponding planned grasping path.

[0095] In one embodiment of the present invention, step S554 specifically includes the following steps:

[0096] S5541, based on the path adjustment direction in the target path adjustment method, performs fragmented path interpolation processing on the path collision risk points in the robot arm pose information set to generate corresponding smooth path segments.

[0097] Specifically, the location of path collision risk points is first determined. Then, based on the path adjustment direction guidance, multiple reference points are selected within a preset range of path collision risk points. Next, a piecewise cubic polynomial interpolation method is used to generate path segments that smoothly connect to the path collision risk points, i.e., smooth path segments, based on the pose information of the reference points. It should be noted that the obtained path segments must satisfy the degree-of-freedom constraints of the robotic arm.

[0098] S5542 performs dynamic obstacle avoidance path deviation detection on smooth path segments to generate corresponding avoidance path deviation values.

[0099] Specifically, when performing dynamic obstacle avoidance path deviation detection on smooth path segments, relevant information about obstacles in the industrial scenario can be incorporated. Specifically, an obstacle can be pre-assigned along the direction of travel on the smooth path segment. If present, the required angle and displacement of the robotic arm—the avoidance path deviation value—can be calculated based on the obstacle's current state and the robotic arm's safe clearance.

[0100] S5543, accumulate the smooth path segment and the avoidance path deviation value to obtain the candidate adjustment path segment.

[0101] Specifically, when accumulating the deviation values ​​of the smooth path segment and the avoidance path, the deviation values ​​of the avoidance path can be adjusted based on the smooth path segment according to the angle and displacement.

[0102] S5544 performs inverse kinematics analysis on the candidate adjustment path segments. When the analysis shows that the joint motion sequence corresponding to the candidate adjustment path segment is within the degree of freedom limit of the robotic arm, the candidate adjustment path segment replaces the path segment corresponding to the path collision risk point to obtain the iteratively adjusted grasping path, which is then used as the corresponding planned grasping path.

[0103] S6 controls the robotic arm to grasp the target according to the planned grasping path.

[0104] Therefore, this invention constrains the point cloud reconstruction process by using calibration information, thereby greatly improving the accuracy of sparse point cloud reconstruction, enhancing the accuracy and stability of robotic arm grasping, and enabling automated robotic arm grasping based on RGB images, thus reducing the cost of image acquisition equipment; furthermore, this invention can accurately plan the grasping path, thereby enabling accurate target grasping.

[0105] In summary, the robotic arm grasping method based on calibration constraint point cloud reconstruction according to embodiments of the present invention involves setting a calibration board at the target location in an industrial scene, performing hand-eye calibration on the robotic arm based on the calibration board to obtain the corresponding hand-eye transformation matrix, capturing images of the industrial scene using an industrial RGB camera mounted on the robotic arm to obtain a target image including the calibration board, obtaining the calibration camera pose of the industrial RGB camera based on the target image, reconstructing a sparse point cloud based on the target image and the calibration camera pose to obtain a scene sparse point cloud corresponding to the industrial scene, and planning a path for the robotic arm based on the calibration camera pose, the scene sparse point cloud, and the hand-eye transformation matrix to obtain a planned grasping path for the robotic arm, and controlling the robotic arm to grasp the target according to the planned grasping path. This not only increases the cost of the image acquisition equipment, but also, when implementing automated robotic arm grasping based on RGB-D images, the generated sparse point cloud has low accuracy, resulting in a large deviation in the robotic arm grasping path, thus significantly reducing the accuracy and stability of the robotic arm grasping.

[0106] Corresponding to the robotic arm grasping method based on calibration constraint point cloud reconstruction in the above embodiments, the present invention also proposes a robotic arm grasping system based on calibration constraint point cloud reconstruction.

[0107] like Figure 4 As shown, the robotic arm grasping system based on calibration constraint point cloud reconstruction in this embodiment of the invention may include: a first acquisition module 100, a second acquisition module 200, a third acquisition module 300, a fourth acquisition module 400, a fifth acquisition module 500, and a control module 600.

[0108] The system comprises the following modules: a first acquisition module 100, which sets a calibration board at the target location in the industrial scene and performs hand-eye calibration on the robotic arm based on the calibration board to obtain the corresponding hand-eye transformation matrix; a second acquisition module 200, which takes a picture of the industrial scene using an industrial RGB camera mounted on the robotic arm to obtain a target image containing the calibration board; a third acquisition module 300, which obtains the calibration camera pose of the industrial RGB camera based on the target image; a fourth acquisition module 400, which performs sparse point cloud reconstruction based on the target image and the calibration camera pose to obtain the scene sparse point cloud corresponding to the industrial scene; a fifth acquisition module 500, which performs path planning on the robotic arm based on the calibration camera pose, the scene sparse point cloud, and the hand-eye transformation matrix to obtain the planned grasping path of the robotic arm; and a control module 600, which controls the robotic arm to grasp the target according to the planned grasping path.

[0109] In one embodiment of the present invention, the fourth acquisition module 400 is specifically used for: extracting features from the target image using a first visual model to obtain corresponding visual features; passing the visual features through the part before the output head of the second visual model, and adding camera embedding features according to the same settings as the second visual model to obtain predicted camera features and predicted image features located before the output head of the second visual model; acquiring a pre-trained camera feature encoder, and using the camera feature encoder to encode the calibrated camera pose into calibrated camera features; calculating the relative deviation between the calibrated camera features and the predicted camera features; performing fine-tuning calculations based on the relative deviation and predicted image features using a cross-attention mechanism to obtain fine-tuned image features; obtaining the last intermediate feature of the fine-tuned image features located before the output head through the DPT model, and performing fine-tuning calculations using a self-attention mechanism after adding calibrated position encoding to the intermediate features to obtain the last layer of input features input to the output head of the DPT model; passing the last layer of input features through the last output head of the DPT model to obtain a sparse point cloud of the scene.

[0110] In one embodiment of the present invention, the fourth acquisition module 400 is specifically used for: encoding the calibrated camera pose into the camera feature space using the encoder module based on the VAE autoencoder; reconstructing the calibrated camera pose based on the camera feature space using the camera output head provided by the second vision model to obtain the reconstructed calibrated camera pose; and training the encoder module of the VAE autoencoder based on the reconstructed calibrated camera pose and the calibrated camera pose using the reconstruction loss function to obtain the camera feature encoder.

[0111] In one embodiment of the present invention, the fifth acquisition module 500 is specifically used for: identifying the target image and extracting the corresponding target region; extracting key points in the target region using a pattern matching algorithm, and selecting a set of key points with a confidence level greater than a preset confidence level based on the key point matching algorithm; performing point cloud registration based on the key point set and the sparse point cloud of the scene to obtain the world coordinate position of the target; calculating the world coordinate system pose of the robotic arm based on the calibrated camera pose and the hand-eye transformation matrix; and obtaining the planned grasping path of the robotic arm based on the world coordinate position of the target and the world coordinate system pose of the robotic arm.

[0112] In one embodiment of the present invention, the fifth acquisition module 500 is specifically used to: acquire historical path information of the robotic arm in the sparse point cloud of the scene based on the world coordinate position of the grasping target and the world coordinate system pose of the robotic arm, wherein the historical path information includes a set of contact status information between the robotic arm and the industrial scene within a historical time period and a set of pose information of the robotic arm; identify and quantify key features from the historical path information to obtain corresponding path features; use a DRL network to acquire the path features in a multi-granularity path adjustment manner to generate a target path adjustment method; and perform iterative path adjustment on the pose information set of the robotic arm based on the target path adjustment method to obtain a corresponding planned grasping path.

[0113] In one embodiment of the present invention, the fifth acquisition module 500 is specifically used to: perform fragmented path interpolation processing on the path collision risk points in the robotic arm pose information set according to the path adjustment direction in the target path adjustment method, so as to generate corresponding smooth path segments; perform dynamic obstacle avoidance path deviation detection on the smooth path segments to generate corresponding avoidance path deviation values; accumulate the smooth path segments and the avoidance path deviation values ​​to obtain candidate adjustment path segments; perform inverse kinematic analysis on the candidate adjustment path segments, and when the analysis shows that the joint motion sequence corresponding to the candidate adjustment path segment is within the degree of freedom limit of the robotic arm, replace the path segment corresponding to the path collision risk point with the candidate adjustment path segment to obtain the iteratively adjusted grasping path, and use it as the corresponding planned grasping path.

[0114] It should be noted that for details not disclosed in the robotic arm grasping system based on calibration constraint point cloud reconstruction in this embodiment of the invention, please refer to the details disclosed in the robotic arm grasping method based on calibration constraint point cloud reconstruction described above, which will not be elaborated here.

[0115] According to an embodiment of the present invention, a robotic arm grasping system based on calibration-constrained point cloud reconstruction includes the following steps: A first acquisition module sets a calibration board at a target location in an industrial scene and performs hand-eye calibration on the robotic arm based on the calibration board to obtain a corresponding hand-eye transformation matrix. A second acquisition module captures an image of the industrial scene using an industrial RGB camera mounted on the robotic arm to obtain a target image including the calibration board. A third acquisition module acquires the calibration camera pose of the industrial RGB camera based on the target image. A fourth acquisition module performs sparse point cloud reconstruction based on the target image and the calibration camera pose to obtain a scene sparse point cloud corresponding to the industrial scene. A fifth acquisition module performs path planning for the robotic arm based on the calibration camera pose, the scene sparse point cloud, and the hand-eye transformation matrix to obtain a planned grasping path. A control module controls the robotic arm to grasp the target according to the planned grasping path. Therefore, by constraining the point cloud reconstruction process with calibration information, the accuracy of sparse point cloud reconstruction is greatly improved, enhancing the accuracy and stability of the robotic arm grasping. Furthermore, automated robotic arm grasping can be achieved based on RGB images, reducing the cost of image acquisition equipment.

[0116] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. "A plurality of" means two or more, unless otherwise explicitly specified.

[0117] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0118] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0119] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0120] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0121] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for robot arm grasping based on calibrated constraint point cloud reconstruction, characterized in that, The method comprises the following steps: S1, setting a calibration board at a target position of an industrial scene, and calibrating a hand-eye calibration of the robot arm based on the calibration board to obtain a corresponding hand-eye transformation matrix; S2, capturing the industrial scene by an industrial RGB camera arranged on the robot arm to obtain a target image containing the calibration board; S3, obtaining a calibration camera pose of the industrial RGB camera according to the target image; S4, performing sparse point cloud reconstruction according to the target image and the calibration camera pose to obtain a corresponding scene sparse point cloud of the industrial scene; wherein step S4 specifically comprises the following steps: S41, extracting features of the target image by using a first visual model to obtain corresponding visual features; S42, outputting the visual features through a part before the output head of a second visual model, and adding camera embedding features according to the same setting of the second visual model to obtain predicted camera features and predicted image features located before the output head of the second visual model; S43, obtaining a pre-trained camera feature encoder, and encoding the calibration camera pose into calibration camera features by using the camera feature encoder; S44, calculating the relative deviation between the calibration camera features and the predicted camera features; wherein the relative deviation between the calibration camera features and the predicted camera features is calculated by the following formula: ， wherein, is the calibrated camera feature, is the predicted camera feature, is the relative deviation; S45, performing fine-tuning calculation according to the relative deviation and the predicted image features based on a cross-attention mechanism to obtain fine-tuned image features; wherein the relative deviation is taken as a query vector Q in the cross-attention mechanism, and the predicted image features are taken as a key vector K and a value vector V in the cross-attention mechanism, and the fine-tuning calculation is performed by the following formula to obtain the fine-tuned image features: ， wherein, denotes the predicted image feature, denotes the fine-tuned image feature; S46, obtaining the last level intermediate features of the fine-tuned image features located before the output head by a DPT model, and performing fine-tuning calculation by using a self-attention mechanism after adding calibration position encoding to the intermediate features to obtain the last layer input features input to the output head of the DPT model; S47, obtaining the scene sparse point cloud by passing the last layer input features through the last output head of the DPT model; S5, performing path planning of the robot arm based on the calibration camera pose, the scene sparse point cloud and the hand-eye transformation matrix to obtain a planned grabbing path of the robot arm; S6, controlling the robot arm to perform target grabbing according to the planned grabbing path.

2. The method of claim 1, wherein, Step S43 specifically comprises: S431, encoding the calibration camera pose into a camera feature space based on an encoder module of a VAE autoencoder; S432, reconstructing the calibration camera pose based on the camera feature space through a camera output head provided by the second visual model to obtain a reconstructed calibration camera pose; S433, training the encoder module of the VAE autoencoder according to the reconstructed calibration camera pose and the calibration camera pose based on a reconstruction loss function to obtain the camera feature encoder.

3. The method of claim 1, wherein, Step S5 specifically comprises the following steps: S51, target recognition is performed on the target image, and a corresponding target area for picking is extracted; S52, a pattern matching algorithm is used to extract key points from the target area for picking, and a key point set with a confidence greater than a preset confidence is screened out based on a key point matching algorithm; S53, point cloud registration is performed based on the key point set and the scene sparse point cloud to obtain a world coordinate position of the target for picking; S54, a world coordinate system pose of the robot arm is calculated based on the calibrated camera pose and the hand-eye transformation matrix; S55, a planned picking path of the robot arm is obtained based on the world coordinate position of the target for picking and the world coordinate system pose of the robot arm.

4. The robot arm grasping method based on calibration constraint point cloud reconstruction according to claim 3, characterized in that, Step S55 specifically includes the following steps: S551, the corresponding historical path information of the robot arm in the scene sparse point cloud is obtained based on the world coordinate position of the target for picking and the world coordinate system pose of the robot arm, wherein the historical path information includes a set of contact state information of the robot arm with the industrial scene and a set of pose information of the robot arm in a historical time period; S552, key features are identified and quantified from the historical path information to obtain corresponding path features; S553, a DRL network is used to obtain a multi-granularity path adjustment mode of the path features to generate a target path adjustment mode; S554, the set of pose information of the robot arm is iteratively adjusted based on the target path adjustment mode to obtain a corresponding planned picking path.

5. The robot arm grasping method based on calibration constraint point cloud reconstruction according to claim 4, characterized in that, Step S554 specifically includes: S5541, according to the path adjustment direction in the target path adjustment mode, a segment path interpolation process is performed on the path collision risk point in the set of pose information of the robot arm to generate a corresponding smooth path segment; S5542, a dynamic obstacle avoidance path deviation detection is performed on the smooth path segment to generate a corresponding avoidance path deviation value; S5543, the smooth path segment and the avoidance path deviation value are accumulated to obtain a candidate adjustment path segment; S5544, inverse kinematics analysis is performed on the candidate adjustment path segment, and when the joint motion sequence corresponding to the candidate adjustment path segment is within the degree of freedom limit of the robot arm, the candidate adjustment path segment replaces the path segment corresponding to the path collision risk point to obtain an iteratively adjusted picking path as the corresponding planned picking path.

6. A robotic arm grasping system based on calibrated constraint point cloud reconstruction, characterized in that, Comprise: A first acquisition module, the first acquisition module is used for setting a calibration board at a target position of an industrial scene, and performing hand-eye calibration measurement on a robot arm based on the calibration board to obtain a corresponding hand-eye transformation matrix; A second acquisition module, the second acquisition module is used for shooting the industrial scene through an industrial RGB camera arranged on the robot arm to obtain a target image containing the calibration board; A third acquisition module, the third acquisition module is used for obtaining a calibrated camera pose of the industrial RGB camera according to the target image; a fourth acquisition module, configured to perform sparse point cloud reconstruction according to the target image and the calibrated camera pose to acquire a scene sparse point cloud corresponding to the industrial scene; the fourth acquisition module is specifically configured to: perform feature extraction on the target image by using a first visual model to acquire corresponding visual features; output a part before a head of the second visual model through the visual features, and add camera embedding features according to the same setting of the second visual model to acquire predicted camera features and predicted image features located before the head of the second visual model; acquire a pre-trained camera feature encoder, and encode the calibrated camera pose into a calibrated camera feature by using the camera feature encoder; calculate a relative deviation between the calibrated camera feature and the predicted camera feature; wherein the relative deviation between the calibrated camera feature and the predicted camera feature is calculated by the following formula: ， wherein, is the calibrated camera feature, is the predicted camera feature, is the relative deviation; perform fine-tuning calculation according to the relative deviation and the predicted image features based on a cross-attention mechanism to acquire fine-tuned image features; wherein the relative deviation is taken as a query vector Q in the cross-attention mechanism, and the predicted image features are taken as a key vector K and a value vector V in the cross-attention mechanism, and the fine-tuning calculation is performed by the following formula to acquire the fine-tuned image features: ， wherein, denotes the predicted image feature, denotes the fine-tuned image feature; acquire last-stage intermediate features of the fine-tuned image features located before the head of the output by using a DPT model, and perform fine-tuning calculation by using a self-attention mechanism after adding calibrated position encoding to the intermediate features to acquire last-layer input features input to the head of the output of the DPT model; acquire the scene sparse point cloud by passing the last-layer input features through the last output head of the DPT model; a fifth acquisition module, configured to perform path planning on the mechanical arm based on the calibrated camera pose, the scene sparse point cloud and the hand-eye transformation matrix to acquire a planned grabbing path of the mechanical arm; a control module, configured to control the mechanical arm to perform target grabbing according to the planned grabbing path.

Images