Man-machine cooperation mobile mechanical arm grabbing method and system based on deep learning

Abstract

The invention discloses a man-machine cooperation mobile mechanical arm grabbing method and system based on deep learning, and the method comprises the steps: obtaining a color image and a depth image of a target object, and converting the depth image into point cloud data; a grabbed object is detected and segmented through an Attention-Mask network, a mask and a region of interest of the grabbed object are generated, and then a seven-degree-of-freedom grabbing posture of the grabbed object is calculated based on the point cloud data through the AnyGrasp network; a confidence-driven man-machine cooperation mechanism is introduced to adjust the grabbing posture; through image-to-point cloud conversion and hand-eye calibration, the mechanical arm can accurately grab and assemble the target object without repeated calibration after displacement. According to the method, the success rate of target recognition and grabbing is remarkably increased, autonomy and safety are enhanced, reliable operation under the low confidence coefficient condition is guaranteed through a man-machine cooperation mode, and the flexibility and efficiency of robot operation under the construction scene are improved through the movable mechanical arm.

Description

Technical Field

[0001] This invention relates to the field of robot vision recognition and autonomous grasping technology, and in particular to a human-machine collaborative mobile robotic arm grasping method and system based on deep learning. Background Technology

[0002] With the increasing demand for intelligent robots in prefabricated building construction, the development of mobile robot systems capable of automatically identifying targets and completing grasping and assembly tasks on construction sites is of great significance. Construction site environments are complex and ever-changing, with a wide variety of target objects placed haphazardly. Traditional fixed grasping solutions for industrial robots are ill-suited to these environments. On one hand, many existing robot grasping methods operate primarily in controlled environments (such as factory assembly lines), typically relying on pre-calibrated fixed cameras or known target models. When the position of the robotic arm or camera changes, the coordinate relationships need to be recalibrated, a time-consuming process that reduces efficiency and hinders deployment in large-scale mobile scenarios such as construction sites. On the other hand, fully autonomous visual grasping algorithms may introduce uncertainties in complex environments. For example, sensor noise, occlusion, or novel object shapes can all lead to a decrease in the confidence level of recognition or grasping decisions. If the robot blindly performs grasping under low confidence conditions, grasping failures may occur, potentially even causing danger. Therefore, current technologies lack a mechanism to introduce human correction when grasping is uncertain, thus balancing autonomy and safety.

[0003] In summary, existing technologies still face many challenges in achieving highly robust and precise universal grasping in the complex and dynamic environment of construction sites. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a human-machine collaborative mobile robotic arm grasping method and system based on deep learning. This invention is applicable to prefabricated building construction sites and can achieve human-machine collaborative mobile robotic arm grasping based on a cascaded neural network of deep learning.

[0005] The objective of this invention is achieved through the following technical solution: The first aspect of this invention provides a human-machine collaborative mobile robotic arm grasping method based on deep learning, characterized by comprising the following steps: (1) Use an RGB-D camera installed at the end of a mobile robotic arm to acquire color and depth images of the target object at the construction site, and convert the depth image into point cloud data; (2) Input the color image and point cloud data into the deep learning cascaded neural network for processing. After the operator selects the prompt box of the object to be grasped from the camera view through the mouse of the human-computer interaction interface, the object to be grasped is first detected and segmented by the Attention-Mask network to generate the mask of the object to be grasped and the region of interest. Then the AnyGrasp network calculates the seven degrees of freedom grasping posture of the object to be grasped based on the point cloud data. (3) Obtain the confidence evaluation value of the grasping posture and compare it with the preset confidence threshold: If the confidence evaluation value is less than the confidence threshold, enter the manual collaboration mode, and the operator corrects and adjusts the grasping posture until the grasping posture is confirmed to be valid; if the confidence evaluation value is greater than or equal to the confidence threshold, the grasping posture is confirmed to be valid, and grasping is performed according to the grasping posture. (4) The final confirmed valid grasping posture is mapped from the camera coordinate system to the world coordinate system of the robotic arm through the hand-eye calibration algorithm. The collision-free motion trajectory is planned using the motion planning framework in the robot operating system. The robotic arm performs the grasping action according to the collision-free motion trajectory and grasping posture, grasps the target object and completes the assembly and placement.

[0006] Furthermore, in step (1), the image format of the RGB-D camera is an H×W×4 matrix, where H represents the image height, W represents the image width, and 4 represents that the image has a total of 4 channels: R, G, B, and D.

[0007] Furthermore, in step (1), the target object includes precast concrete assembly components and precast steel structure assembly components, which are displayed through a unified building model.

[0008] Further, in step (1), the conversion of the depth image into point cloud data specifically includes: The depth image and corresponding timestamp captured by the RGB-D camera installed at the end of the mobile robotic arm are read. The camera's intrinsic parameter matrix and distortion coefficients are loaded, and the depth scale coefficients and the effective range of the depth camera are read. Then, the depth image is distorted according to the distortion coefficients, and invalid values ​​are filtered according to the effective range to obtain the depth image after invalid value filtering. Spatial smoothing and temporal filtering are applied to the depth image after filtering invalid values, and hole filling is performed on small areas of invalid depth using neighborhood propagation or interpolation methods to obtain the depth image after hole filling. Based on the depth image after the holes are filled, for each effective pixel, its three-dimensional point in the camera coordinate system is calculated according to the depth scale coefficient and the camera's intrinsic parameter matrix, thereby obtaining an organized three-dimensional point cloud. The standard point cloud processing steps of voxel raster downsampling, outlier removal, and normal estimation are performed on the organized 3D point cloud, while preserving the mapping relationship from pixel index to 3D point, thus completing the conversion from depth image to point cloud data.

[0009] Furthermore, the cascaded neural network of deep learning includes an Attention-Mask network and an AnyGrasp network. The Attention-Mask network includes a cascaded VIPT subnetwork and a SAM2 subnetwork. The AnyGrasp network includes a geometric processing module and a temporal correlation module. After the operator selects the object to be grasped from the camera's perspective using a mouse bounding box on the human-computer interaction interface, the color image and point cloud data are input into the Attention-Mask network. The VIPT sub-network maps the object to the labeled RGB-D feature channels and iteratively generates time-based prompts while continuously calculating the spatial position of the target. When the target's spatial position falls within the robotic arm's workspace, the SAM2 sub-network is automatically activated. It uses the image encoding and prompt encoding from the VIPT sub-network as input and the mask in the memory bank as a feature reference. Through the attention memory mechanism, it compares the features related to the target and then iteratively outputs the pixel-level mask of the current target within the local attention domain through mask decoding, which serves as the generated mask. The generated mask is stored in the memory bank as a feature reference, and the prompt encoding is obtained by encoding the time-based prompts. The camera model backprojects the masked color image generated by the Attention-Mask network into a point cloud, which is then used as input to the AnyGrasp network. Discrete parameters are derived through the geometry processing module. Simultaneously, the center and edges of the grasped object are estimated from the point clouds of each grasping pose to determine the object's centroid. A stability score is calculated based on the distance from the end effector plane to the object's centroid, and this score is linearly adjusted using weights to obtain the final score. The top K grasping pose sequences with the highest final scores are selected, and all their corresponding discrete parameters are converted into seven-DOF grasping poses with their confidence intervals. Based on the seven-DOF grasping poses, a time correlation module is used to construct a seven-DOF grasping pose feature vector for each consecutive predicted grasping pose, combining geometric and color features, to calculate the cosine similarity between two adjacent consecutive seven-DOF grasping poses. This time correlation module selects the seven-DOF grasping pose corresponding to the maximum cosine similarity to generate a time correlation matrix to link poses across time, and uses this time correlation matrix to create a continuous grasping pose sequence.

[0010] Furthermore, the discrete parameters include the grab point M, the view V, the in-plane rotation angle φ, the approach depth deep, and the end effector opening w; The discrete parameters are converted into a seven-DOF grasping posture with its confidence interval. Specifically, this includes: taking view V as the z-axis of the end effector coordinate system, rotating around the z-axis by an in-plane rotation angle φ to obtain the x-axis of the end effector, and setting y=z×x to obtain the y-axis, so as to obtain the spatial coordinates (x,y,z) of the end effector grasping position; taking the grasping point M as the contact midpoint and moving backward or forward along the approach direction by a approach depth (deep), and adding the fixed offset otcp of the end effector on each axis in geometric space, to obtain the rotational posture (α,β,γ) of the end effector relative to each axis of the world coordinate system; and obtaining the seven-DOF grasping posture G={x,y,z,α,β,γ,w} based on (x,y,z), (α,β,γ) and the opening w of the end effector. The end effector is a two-finger parallel gripper.

[0011] Furthermore, the confidence score is obtained by a weighted linear combination of the force closure mass and the collision probability estimate; The preset confidence threshold adopts an adaptive threshold. ,in and , respectively, are the mean and standard deviation of the confidence evaluation for recent task windows, and k is the safety factor.

[0012] Furthermore, the step of the operator correcting and adjusting the grasping posture specifically includes: In the human collaboration mode, K grasping poses from the AnyGrasp network output are overlaid on the color image and point cloud data view respectively, and the grasping pose corresponding to the highest confidence evaluation value among the K grasping poses in the current frame is highlighted by default. The operator uses 3D interactive markers to translate the target pose of the end effector in the x, y, and z directions and rotate it around each axis, and fine-tunes the opening of the end effector using sliders or numerical values ​​to obtain the adjusted grasping posture. After each manual correction and adjustment, collision filtering and path pre-planning are performed immediately. The point cloud is converted into an octree map as the calculation basis for the robotic arm's path planning and collision filtering detection. Based on this, the confidence evaluation value of the grasping posture is recalculated. The execution button is unlocked when the updated confidence evaluation value is greater than or equal to the confidence threshold. After confirming that there is no collision on the grasping path, the planned motion trajectory is displayed. The operator finally confirmed that the grasping posture was valid, wrote the adjusted grasping posture back to the execution pipeline, and then performed the grasping according to the confirmed grasping path without collision, displaying the planned motion trajectory.

[0013] Furthermore, step (4) specifically includes: The camera-to-endpoint extrinsic parameter matrix is ​​obtained using a hand-eye calibration algorithm. Before the capture process, the pose of the target object in the world coordinate system is obtained using the following coordinate transformation chain:

[0014] In the formula, This represents the pose of the target object in the world coordinate system. This represents the calibration transformation pose from the world coordinate system to the base coordinate system. This indicates the pose of the robotic arm's end effector in the base coordinate system. This represents the hand-eye extrinsic parameters from the robotic arm's end effector to the camera coordinate system. This indicates the pose of the target object in the camera coordinate system; By utilizing the planning pipeline of the motion planning framework in the robot operating system and the interface of the open-source motion planning library, a collision-free motion trajectory of the current state is generated in the joint / Cartesian hybrid space. At the same time, the path segments of the planned collision-free motion trajectory are fitted and resampled to generate a smooth and continuous velocity curve to meet the maximum velocity constraint of the robotic arm joint. Then, the robotic arm is controlled to perform the grasping action according to the collision-free motion trajectory and grasping posture. After grasping the target object, the grasping stability is quickly determined based on the end torque and joint current: if slippage, lack of clamping or collision risk is detected, a safety stop is immediately executed and the pose is finely adjusted, and then the grasping is attempted again; if the grasping stability is determined, the object is placed along the planned assembly path and the end constraint is released.

[0015] A second aspect of this invention provides a system for the aforementioned deep learning-based human-machine collaborative mobile robotic arm grasping method, comprising: a six-degree-of-freedom robotic arm, a deformable tracked vehicle, an RGB-D camera, an RGB-D time-of-flight camera, and an industrial control computer. The six-degree-of-freedom robotic arm is mounted on the deformable tracked vehicle, the RGB-D camera is mounted at the end of the six-degree-of-freedom robotic arm, the RGB-D time-of-flight camera is mounted on the deformable tracked vehicle, and the industrial control computer is located inside the deformable tracked vehicle. The industrial control computer contains a grasping control system, which includes a visual perception module, a mask segmentation module, a grasping posture generation module, a human-machine collaboration module, a hand-eye calibration module, and a motion planning and execution control module. The system comprises the following modules: a visual perception module, an RGB-D camera and an RGB-D time-of-flight camera, and a human-machine collaboration module. The visual perception module acquires color and depth images of target objects at the construction site, along with corresponding timestamps, and converts the depth images into point cloud data. The mask segmentation module is equipped with an Attention-Mask network, which uses streaming memory to continuously detect and segment target objects, and outputs a pixel-level mask of the grasped object and its confidence score. The grasping pose generation module inputs the point cloud data after pixel-level mask cropping into an AnyGrasp network to generate and score the seven-DOF grasping poses of the grasped object. The human-machine collaboration module performs stable score calculations, confidence threshold determination, and visualization interaction on the K seven-DOF grasping poses. When the confidence score is less than the confidence threshold, a GUI interface is provided for operators to fine-tune and confirm the position, posture, and end effector opening. Collision filtering and path pre-planning are performed in real time after each adjustment. The hand-eye calibration module is used to obtain the extrinsic parameters from the camera coordinate system to the robot's world coordinate system based on the hand-eye calibration algorithm, and accurately maps the confirmed valid grasping posture from the camera coordinate system to the robot's world coordinate system. The motion planning and execution control module is used to plan a collision-free motion trajectory using the motion planning framework in the robot operating system, so as to generate a collision-free motion trajectory in the obstacle scene and perform time parameterization, so that the robot arm can perform grasping action according to the collision-free motion trajectory and grasping posture, grasp the target object, and complete the assembly and placement.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention combines semantic segmentation of color images (Attention-Mask network) with grasping pose generation of deep point clouds (AnyGrasp network) through cascaded neural networks, making full use of the complementary advantages of color images and depth information. Compared with solutions based solely on point clouds, this invention can more accurately locate target objects in complex backgrounds and generate targeted grasping poses, significantly improving the grasping success rate. In addition, the Attention-Mask network can identify specific types of target objects as needed, and the AnyGrasp grasping network has the ability to grasp any unknown object. Therefore, the system described in this invention is suitable for picking up components, tools, and even randomly piled debris at construction sites, and has strong versatility.

[0017] (2) The present invention introduces a confidence-driven human-machine collaboration mechanism. When the algorithm lacks sufficient confidence in the grasping decision, it automatically triggers manual review and adjustment. This closed-loop human supervision is particularly important in complex construction sites and can effectively avoid grasping failure or safety accidents caused by the robot acting rashly under uncertain circumstances. Therefore, the present invention greatly improves the reliability of the system in unknown environments while maintaining a high degree of autonomy.

[0018] (3) The mobile robotic arm system of the present invention integrates the hand-eye calibration relationship between the end-effector camera and the robotic arm. Calibration only needs to be completed during the initial deployment. Afterwards, when the robotic arm is carried by the mobile platform to move freely to a new position, it can directly complete the grasping task through visual perception without manual recalibration. This solves the problem that traditional mobile robots need to be re-aligned and positioned after each movement, reducing downtime for calibration. Combined with the flexibility and mobility of the mobile platform, the robot can cover different areas of the construction site and quickly perform multi-point grasping and assembly tasks, thereby improving the overall efficiency of automated construction operations. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the framework of the deep learning-based human-machine collaborative mobile robotic arm grasping method of the present invention. Figure 2 This is a flowchart illustrating the framework of the cascaded neural network for deep learning in this invention. Figure 3 This is a schematic diagram of the structure of the deep learning-based human-machine collaborative mobile robotic arm grasping system of the present invention. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit this application.

[0021] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to determination," or "includes." Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process or method. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0023] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0024] See Figure 1 The present invention provides a deep learning-based human-machine collaborative mobile robotic arm grasping method applicable to prefabricated building construction sites, which specifically includes the following steps: (1) Use an RGB-D camera installed at the end of a mobile robotic arm to acquire color and depth images of the target objects at the construction site, and convert the depth images into point cloud data.

[0025] Furthermore, the target objects include precast concrete assembly components and precast steel structure assembly components, which are represented through a unified building model (IFC).

[0026] Furthermore, an RGB-D camera installed at the end of the mobile robotic arm is used to acquire color and depth images of the target object at the construction site, specifically including the following sub-steps: (1.1) The operator controls the deformable tracked vehicle to approach the potential gripping position by redirection. Specifically, for the assembly construction site, the deformable tracked vehicle is first manually oriented near the gripping position, and then approaches the potential gripping position by redirection when the gripping target is determined.

[0027] (1.2) The target object is positioned in the 40%-60% area of ​​the RGB-D camera's field of view by fine-tuning the alignment at the end; the angular velocity within the exposure window is limited to not exceed the threshold to suppress motion blur; the region of interest (ROI) is selected according to the task and acquired using a static-acquisition-motion rhythm to obtain synchronized color image I(u,v) and depth image D(u,v), where u and v represent the horizontal and vertical coordinates of the image, respectively.

[0028] (1.3) Perform a fast quality assessment (sharpness, saturation ratio, histogram equalization, etc.) on the newly acquired color and depth images. If the quality is not up to standard, adaptive re-sampling (automatic exposure adjustment) is performed. The color image I(u,v), depth image D(u,v) and metadata are cached together as subsequent inputs. The metadata includes timestamps, exposure, temperature, etc.

[0029] Furthermore, the image format of an RGB-D camera is an H×W×4 matrix, where H represents the image height, W represents the image width, and 4 indicates that the image has four channels: R, G, B, and D. This matrix can be converted into realistic 3D size and position information through a geometric model. This geometric model describes the relationship between the 3D spatial position of a point on the surface of a target object and the corresponding point in the image. The geometric model is based on the camera's intrinsic and extrinsic parameter matrices, with the intrinsic parameter matrix containing the focal length in the x-direction. focal length in the y direction The position of the center point in the x-direction and the position of the center point in the y direction The extrinsic parameter matrix contains a rotation matrix R that describes the rotation relationship of the camera coordinate system relative to the world coordinate system and a translation vector t that describes the translation relationship of the camera coordinate system relative to the world coordinate system.

[0030] Further, the depth image is converted into point cloud data, specifically including the following sub-steps: (1.4) Read the depth image D(u,v) and corresponding timestamp acquired by the RGB-D camera installed at the end of the mobile robotic arm, and load the camera's intrinsic parameter matrix. With distortion coefficients, read depth scale coefficients With the effective range of the depth camera ,in These represent the focal lengths in the x and y directions, respectively. These represent the coordinates of the center point in the x and y directions, respectively. These represent the closest measurable distance and the farthest usable distance from the camera's optical center along the line of sight to the measured point, respectively. Then, the depth image is distorted according to the distortion coefficients. Specifically, OpenCV is used to calculate the remapping table, and then the depth image is remapped to generate a distorted depth image. Subsequently, invalid value filtering is performed on the distorted depth image according to the effective range to obtain the invalid value filtered depth image.

[0031] (1.5) Apply spatial smoothing and temporal filtering (such as inter-frame exponential smoothing) to the depth image after filtering invalid values, and use neighborhood propagation or interpolation methods to fill holes in small areas of invalid depth to obtain the depth image after hole filling.

[0032] (1.6) Based on the obtained depth image after hole filling, for each effective pixel (u,v), according to the depth scale coefficient and the camera's intrinsic parameter matrix Calculate the 3D points in its camera coordinate system The calculation formula is as follows:

[0033]

[0034]

[0035] In the formula, These represent the coordinates of the 3D point corresponding to pixel (u, v) in the depth image along the x, y, and z directions, respectively. This leads to an organized 3D point cloud. .

[0036] (1.7) Organized 3D point clouds Perform standard point cloud processing steps such as voxel raster downsampling, outlier removal, and normal estimation, while preserving the mapping relationship between pixel indices and 3D points, to complete the conversion of depth image to point cloud data.

[0037] (2) Input the color image and point cloud data into the deep learning cascaded neural network for processing. The operator selects the object to be captured from the camera's perspective using the mouse on the human-computer interaction interface. Next, the Attention-Mask network is used to detect and segment the object to be grasped, generating a mask for the object and a region of interest. Then, the AnyGrasp network calculates the seven-DOF grasping pose of the object based on point cloud data. The cascaded neural network of deep learning includes the Attention-Mask network and the AnyGrasp network.

[0038] It should be understood that the prompt box It is the area of ​​interest selected by the user.

[0039] In this embodiment, the structure of the cascaded neural network for deep learning is as follows: Figure 2 As shown, this includes Attention-Mask network and AnyGrasp network.

[0040] The Attention-Mask network comprises cascaded VIPT and SAM2 sub-networks. The operator uses a mouse cursor on the user interface to select a tooltip indicating the object to be captured from the camera's perspective. Then, the color image and point cloud data are input into the Attention-Mask network, and the VIPT subnetwork will display the cue box. Mapping to labeled RGB-D feature channels and iteratively generating time prompts. Simultaneously, the spatial position of the grasping target (i.e., the object to be grasped) is continuously calculated. The inverse kinematics of the robotic arm is used to calculate whether there is an intersection between the spatial position of the grasping target and the robotic arm's workspace. When the spatial position of the grasping target falls within the robotic arm's workspace, the SAM2 sub-network is automatically activated. The SAM2 sub-network uses image encoding and cue encoding from the VIPT sub-network as input and a mask in the memory bank as a feature reference. It compares features related to the grasping target through an attention memory mechanism, and then iteratively outputs a pixel-level mask of the current grasping target within the local attention domain through mask decoding. This is used as the generated mask, thus continuously filtering out noise in the perception process and achieving computational resource efficiency and a closed-loop perception control in highly disturbed environments. The generated mask is stored in the memory bank as a feature reference, and the cue encoding uses time-based cues... Encode and obtain.

[0041] It should be noted that by using the inverse kinematics of the robotic arm to solve for whether there exists a joint solution q that allows the end effector to reach the pose of the grasping target, and all joints fall within the workspace limits of the robotic arm, it can be determined whether the spatial position of the grasping target falls within the workspace of the robotic arm.

[0042] The AnyGrasp network comprises a geometry processing module and a temporal correlation module. After the operator confirms the mask segmentation is correct, the mask generated by the Attention-Mask network (i.e., the masked color image) is back-projected into a point cloud using the camera model as input to the AnyGrasp network. The geometry processing module derives discrete parameters, including the grasping point M, the view V, the in-plane rotation angle φ (i.e., the rotation angle of the gripper around the grasping approach direction), the approach depth deep (i.e., the distance from the grasping point to the origin of the gripper coordinate system along the approach direction), and the end effector opening w. Simultaneously, the center and edges of the grasped object are estimated using the point clouds of each grasping posture to determine the object's center of gravity. A stability score is calculated based on the distance from the end effector plane to the object's center of gravity, and this score is further optimized by linearly adjusting the weights to obtain the final score. Subsequently, the grasping postures are sorted in descending order according to the final score, and the top K grasping posture sequences with the highest final scores are selected. Next, based on the top K grasping posture sequences, all their corresponding discrete parameters are converted into seven-degree-of-freedom grasping postures with their confidence intervals. Based on the seven-degree-of-freedom (DOF) grasping posture, a seven-DOF grasping posture feature vector is constructed for each consecutive predicted grasping posture using a temporal correlation module, combining geometric and color features, to calculate the cosine similarity between two adjacent consecutive seven-DOF grasping postures. ,in, These represent two consecutive seven-degree-of-freedom grasping postures. They represent The corresponding seven-degree-of-freedom grasping posture feature vector; the time association module is based on cosine similarity, selects the seven-degree-of-freedom grasping posture corresponding to the maximum cosine similarity to generate a time association matrix to link postures across time, and uses the time association matrix to create a continuous grasping posture sequence. That is, by obtaining a series of time association matrices, a continuous grasping posture sequence can be obtained by iterative calculation.

[0043] Furthermore, when calculating the stability score, the pre-grabbing posture is set by sampling, and the stability score of each posture is calculated. Stability score = posture suitability × depth suitability × position effectiveness × grasping stability.

[0044] Furthermore, all discrete parameters are converted into a seven-DOF grasping posture with its confidence interval. Specifically, this includes: using view V as the z-axis of the end effector coordinate system, rotating around the z-axis by an in-plane rotation angle φ to obtain the x-axis of the end effector, and setting y = z × x to obtain the y-axis, thus obtaining the spatial coordinates (x, y, z) of the end effector's grasping position; moving backward or forward by a depth (deep) along the approach direction with the grasping point M as the contact midpoint, and adding the fixed offset (otcp) of the end effector along each axis in geometric space, which can be calculated using the formula... Obtain the rotational attitude (α,β,γ) of the end effector relative to each axis of the world coordinate system; obtain the seven-degree-of-freedom grasping attitude G={x,y,z,α,β,γ,w} based on (x,y,z), (α,β,γ) and the opening w of the end effector.

[0045] Furthermore, the seven-DOF grasping posture consists of the position, orientation, and opening of the end effector in the world coordinate system, denoted as G={x,y,z,α,β,γ,w}, where (x,y,z) are the spatial coordinates of the end effector's grasping position, (α,β,γ) are the rotational orientations of the end effector relative to the axes of the world coordinate system, and w is the opening of the end effector. The end effector is a two-finger parallel gripper.

[0046] (3) Obtain the confidence evaluation value S of the grasping posture, and compare the confidence evaluation value S with the preset confidence threshold. Comparison: If the confidence score S is less than the confidence threshold If the grasping posture is not confirmed to be valid, the system will enter manual collaboration mode, where operators will correct and adjust the grasping posture until it is confirmed to be valid. If the confidence score S is greater than or equal to the confidence threshold... If the grabbing posture is valid, then the grabbing will proceed according to the grabbing posture.

[0047] Furthermore, the confidence score S is determined by the force closure mass. Collision probability estimation The force is obtained by a linear combination of weights. For the force-closing mass: the contact normal, the lever arm from the contact point to the center of mass, the friction coefficient μ, and the contact model (hard finger / soft finger) are used to construct a discrete friction cone vector for each contact. The corresponding 6D (force and torque) is calculated for each vector, and these are aggregated into a point set. The convex hull is then calculated, and the radius ε of the largest sphere centered at the origin and inscribed within this convex hull is taken as the force-grabbing mass (the larger the better; ε>0 indicates force closure). For collision probability estimation: discrete and continuous collision detection is performed on the entire trajectory using the FCL underlying library to determine whether there is penetration with the environment / self / target geometry. This is the basis for probability estimation.

[0048] Furthermore, the preset confidence threshold Adaptive threshold ,in and , respectively, are the mean and standard deviation of the confidence evaluation for recent task windows, and k is the safety factor.

[0049] Furthermore, the operator corrects and adjusts the grasping posture, specifically including the following sub-steps: (3.1) In the manual collaboration mode, K grasping postures output from AnyGrasp network are overlaid on the color image and point cloud data view respectively. By default, the grasping posture corresponding to the highest confidence evaluation value S among the K grasping postures in the current frame is highlighted for the operator to use as a benchmark.

[0050] (3.2) The operator translates the target pose of the end effector in the x, y, and z directions and rotates it around each axis by using three-dimensional interactive markers, and fine-tunes the opening w of the end effector by using a slider or numerical method to obtain the adjusted grasping posture.

[0051] (3.3) After each manual correction and adjustment, collision filtering and path pre-planning are performed immediately. The point cloud is converted into an octree map as the calculation basis for the robotic arm's path planning and collision filtering detection. Based on this, the confidence evaluation value S of the grasping posture is recalculated until the updated confidence evaluation value S is greater than or equal to the confidence threshold. Unlock the execution button when the target path is cleared and the planned motion trajectory is displayed after confirming that there is no collision in the grab path.

[0052] It should be noted that collision filtering detection involves using an FCL collision detector to perform a rapid collision check after the operator fine-tunes the seven-DOF attitude. Candidates that are "clearly likely to collide / cross the boundary" are immediately eliminated, retaining only those without collision and with a safe distance. Path pre-planning uses the MoveIt default planning library, MoveIt Planning Pipeline, to first plan a feasible short trajectory from "current → pre-grab → grab" and then uses TOTG for time parameterization to obtain the planned motion trajectory.

[0053] (3.4) The operator finally confirms that the grasping posture is valid, writes the adjusted grasping posture back to the execution pipeline, and then performs the grasping according to the confirmed grasping path without collision, displaying the planned motion trajectory.

[0054] (4) The final confirmed valid grasping posture is mapped from the camera coordinate system to the world coordinate system of the robotic arm through the hand-eye calibration algorithm. The collision-free motion trajectory is planned using the Motion Planning Framework (MoveIt) in the Robot Operating System (ROS). The robotic arm performs the grasping action according to the collision-free motion trajectory and grasping posture, grasps the target object and completes the assembly and placement.

[0055] (4.1) Use the hand-eye calibration algorithm to obtain the extrinsic parameter matrix from the camera to the end point. Before the capture is executed, obtain the pose of the target object in the world coordinate system according to the following coordinate transformation chain:

[0056] In the formula, This represents the pose of the target object in the world coordinate system. This represents the calibration transformation pose from the world coordinate system to the base coordinate system. This indicates the pose of the robotic arm's end effector in the base coordinate system. This represents the hand-eye extrinsic parameters from the robotic arm's end effector to the camera coordinate system (specifically obtained through a hand-eye calibration algorithm). This represents the pose of the target object in the camera coordinate system. The hand-eye calibration algorithm uses the AX=XB model, where A is the hand, i.e., the end effector of the robotic arm, and its relative motion between two shots; B is the eye, i.e., the camera's viewpoint base point, and its relative motion relative to the calibration target between two shots.

[0057] (4.2) The planning pipeline of MoveIt in ROS and the interface of the open motion planning library (OMPL) are used to generate the collision-free motion trajectory of the current state in the joint / Cartesian hybrid space. At the same time, the path segments of the planned collision-free motion trajectory are fitted and resampled to generate a smooth and continuous velocity curve to meet the maximum velocity constraint of the robotic arm joint. Then, the robotic arm is controlled to perform the grasping action according to the collision-free motion trajectory and grasping posture.

[0058] (4.3) After grasping the target object, the grasping stability is quickly determined based on the end torque and joint current: if slippage, lack of clamping or collision risk is detected, a safety stop is immediately executed and the pose is finely adjusted, and then the grasping is attempted again; if the grasping stability is determined, the object is placed along the planned assembly path and the end constraint is released.

[0059] In the process of quickly determining the gripping stability based on the end torque, let the normal gripping force of the two fingers be Fn, the static friction coefficient be μ, the object's weight be Gravity, the equivalent half distance from the inner side of the two fingers to the object's center of mass be r, and the external torque Fw about the line connecting the grippers. Then, the necessary and sufficient approximate conditions for no slippage are: 2μFn≥Gravity and 2μFnr≥Fw. If either condition is not met, there is a risk of slippage. When the estimated external torque exceeds the upper limit of "model + variance", there is a risk of collision.

[0060] When quickly determining gripping stability based on joint current, an increase in tangential force while the normal force / gripper current remains unchanged indicates a risk of slippage; if the gripper closes to the target width but the current does not increase significantly, there is a risk of inadequate gripping.

[0061] It is worth mentioning that this invention also provides a system for implementing the deep learning-based human-machine collaborative mobile robotic arm grasping method described in the above embodiments. For example... Figure 3As shown, the system includes a six-degree-of-freedom robotic arm, a deformable tracked vehicle, an RGB-D camera, an RGB-D time-of-flight camera, and an industrial control computer. The six-degree-of-freedom robotic arm is mounted on the deformable tracked vehicle, the RGB-D camera is mounted at the end of the six-degree-of-freedom robotic arm, the RGB-D time-of-flight camera is mounted on the deformable tracked vehicle, and the industrial control computer is located inside the deformable tracked vehicle. The industrial control computer contains a grasping control system, which includes a visual perception module, a mask segmentation module, a grasping posture generation module, a human-machine collaboration module, a hand-eye calibration module, and a motion planning and execution control module.

[0062] The system comprises several modules: a visual perception module and a mask segmentation module. The visual perception module uses an RGB-D camera and an RGB-D time-of-flight camera to acquire color and depth images of target objects at the construction site, along with corresponding timestamps, and converts the depth images into point cloud data. The mask segmentation module is equipped with an Attention-Mask network, a video-level instance segmentation network that provides prompting. Based on streaming memory, it continuously detects and segments target objects, outputting pixel-level masks of the grasped object and their confidence scores. The grasping pose generation module inputs the point cloud data, cropped from the pixel-level masks, into the AnyGrasp network to generate and score the seven-DOF grasping poses of the grasped object. The human-machine collaboration module performs stable score calculations, confidence threshold determination, and visual interaction on the K seven-DOF grasping poses. When the confidence score is less than the confidence threshold, a GUI interface is provided for operators to fine-tune the position, pose, and end effector opening. Collision filtering and path pre-planning are performed in real-time after each adjustment. The hand-eye calibration module is used to obtain the extrinsic parameters from the camera coordinate system to the robot arm's world coordinate system based on the hand-eye calibration algorithm, and accurately maps the confirmed and valid grasping posture from the camera coordinate system to the robot arm's world coordinate system. The motion planning and execution control module is used to plan a collision-free motion trajectory using the motion planning framework in the robot operating system, to generate a collision-free motion trajectory in obstacle scenarios and parameterize it over time, so that the robot arm can perform the grasping action according to the collision-free motion trajectory and grasping posture, grasp the target object and complete the assembly and placement.

[0063] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A human-machine collaborative mobile robotic arm grasping method based on deep learning, characterized in that, Includes the following steps: (1) Use an RGB-D camera installed at the end of a mobile robotic arm to acquire color and depth images of the target object at the construction site, and convert the depth image into point cloud data; (2) Input the color image and point cloud data into the deep learning cascaded neural network for processing. After the operator selects the prompt box of the object to be grasped from the camera view through the mouse of the human-computer interaction interface, the object to be grasped is first detected and segmented by the Attention-Mask network to generate the mask of the object to be grasped and the region of interest. Then the AnyGrasp network calculates the seven degrees of freedom grasping posture of the object to be grasped based on the point cloud data. (3) Obtain the confidence evaluation value of the grasping posture and compare it with the preset confidence threshold: If the confidence evaluation value is less than the confidence threshold, enter the manual collaboration mode, and the operator corrects and adjusts the grasping posture until the grasping posture is confirmed to be valid; if the confidence evaluation value is greater than or equal to the confidence threshold, the grasping posture is confirmed to be valid, and grasping is performed according to the grasping posture. (4) The final confirmed valid grasping posture is mapped from the camera coordinate system to the world coordinate system of the robotic arm through the hand-eye calibration algorithm. The collision-free motion trajectory is planned using the motion planning framework in the robot operating system. The robotic arm performs the grasping action according to the collision-free motion trajectory and grasping posture, grasps the target object and completes the assembly and placement.

2. The deep learning-based human-machine collaborative mobile robotic arm grasping method according to claim 1, characterized in that, In step (1), the image format of the RGB-D camera is an H×W×4 matrix, where H represents the image height, W represents the image width, and 4 represents that the image has a total of 4 channels: R, G, B, and D.

3. The deep learning-based human-machine collaborative mobile robotic arm grasping method according to claim 1, characterized in that, In step (1), the target object includes precast concrete assembly components and precast steel structure assembly components, which are displayed through a unified building model.

4. The deep learning-based human-machine collaborative mobile robotic arm grasping method according to claim 1, characterized in that, In step (1), converting the depth image into point cloud data specifically includes: The depth image and corresponding timestamp captured by the RGB-D camera installed at the end of the mobile robotic arm are read. The camera's intrinsic parameter matrix and distortion coefficients are loaded, and the depth scale coefficients and the effective range of the depth camera are read. Then, the depth image is distorted according to the distortion coefficients, and invalid values ​​are filtered according to the effective range to obtain the depth image after invalid value filtering. Spatial smoothing and temporal filtering are applied to the depth image after filtering invalid values, and hole filling is performed on small areas of invalid depth using neighborhood propagation or interpolation methods to obtain the depth image after hole filling. Based on the depth image after the holes are filled, for each effective pixel, its three-dimensional point in the camera coordinate system is calculated according to the depth scale coefficient and the camera's intrinsic parameter matrix, thereby obtaining an organized three-dimensional point cloud. The standard point cloud processing steps of voxel raster downsampling, outlier removal, and normal estimation are performed on the organized 3D point cloud, while preserving the mapping relationship from pixel index to 3D point, thus completing the conversion from depth image to point cloud data.

5. The deep learning-based human-machine collaborative mobile robotic arm grasping method according to claim 1, characterized in that, The deep learning cascaded neural network includes an Attention-Mask network and an AnyGrasp network. The Attention-Mask network includes a cascaded VIPT subnetwork and a SAM2 subnetwork. The AnyGrasp network includes a geometric processing module and a temporal correlation module. After the operator selects the object to be grasped from the camera's perspective using a mouse bounding box on the human-computer interaction interface, the color image and point cloud data are input into the Attention-Mask network. The VIPT sub-network maps the object to the labeled RGB-D feature channels and iteratively generates time-based prompts while continuously calculating the spatial position of the target. When the target's spatial position falls within the robotic arm's workspace, the SAM2 sub-network is automatically activated. It uses the image encoding and prompt encoding from the VIPT sub-network as input and the mask in the memory bank as a feature reference. Through the attention memory mechanism, it compares the features related to the target and then iteratively outputs the pixel-level mask of the current target within the local attention domain through mask decoding, which serves as the generated mask. The generated mask is stored in the memory bank as a feature reference, and the prompt encoding is obtained by encoding the time-based prompts. The camera model backprojects the masked color image generated by the Attention-Mask network into a point cloud, which is then used as input to the AnyGrasp network. Discrete parameters are derived through the geometry processing module. Simultaneously, the center and edges of the grasped object are estimated from the point clouds of each grasping pose to determine the object's centroid. A stability score is calculated based on the distance from the end effector plane to the object's centroid, and this score is linearly adjusted using weights to obtain the final score. The top K grasping pose sequences with the highest final scores are selected, and all their corresponding discrete parameters are converted into seven-DOF grasping poses with their confidence intervals. Based on the seven-DOF grasping poses, a time correlation module is used to construct a seven-DOF grasping pose feature vector for each consecutive predicted grasping pose, combining geometric and color features, to calculate the cosine similarity between two adjacent consecutive seven-DOF grasping poses. This time correlation module selects the seven-DOF grasping pose corresponding to the maximum cosine similarity to generate a time correlation matrix to link poses across time, and uses this time correlation matrix to create a continuous grasping pose sequence.

6. The deep learning-based human-machine collaborative mobile robotic arm grasping method according to claim 5, characterized in that, The discrete parameters include the grab point M, the view V, the in-plane rotation angle φ, the approach depth deep, and the end effector opening w. The discrete parameters are converted into a seven-DOF grasping posture with its confidence interval. Specifically, this includes: taking view V as the z-axis of the end effector coordinate system, rotating around the z-axis by an in-plane rotation angle φ to obtain the x-axis of the end effector, and setting y=z×x to obtain the y-axis, so as to obtain the spatial coordinates (x,y,z) of the end effector grasping position; taking the grasping point M as the contact midpoint and moving backward or forward along the approach direction by a approach depth (deep), and adding the fixed offset otcp of the end effector on each axis in geometric space, to obtain the rotational posture (α,β,γ) of the end effector relative to each axis of the world coordinate system; and obtaining the seven-DOF grasping posture G={x,y,z,α,β,γ,w} based on (x,y,z), (α,β,γ) and the opening w of the end effector. The end effector is a two-finger parallel gripper.

7. The deep learning-based human-machine collaborative mobile robotic arm grasping method according to claim 1, characterized in that, The confidence score is obtained by a weighted linear combination of the force closure mass and the collision probability estimate. The preset confidence threshold adopts an adaptive threshold. , of which and , respectively, are the mean and standard deviation of the confidence evaluation for recent task windows, and k is the safety factor.

8. The deep learning-based human-machine collaborative mobile robotic arm grasping method according to claim 1, characterized in that, The process of adjusting the grasping posture by the operator specifically includes: In the human collaboration mode, K grasping poses from the AnyGrasp network output are overlaid on the color image and point cloud data view respectively, and the grasping pose corresponding to the highest confidence evaluation value among the K grasping poses in the current frame is highlighted by default. The operator uses 3D interactive markers to translate the target pose of the end effector in the x, y, and z directions and rotate it around each axis, and fine-tunes the opening of the end effector using sliders or numerical values ​​to obtain the adjusted grasping posture. After each manual correction and adjustment, collision filtering and path pre-planning are performed immediately. The point cloud is converted into an octree map as the calculation basis for the robotic arm's path planning and collision filtering detection. Based on this, the confidence evaluation value of the grasping posture is recalculated. The execution button is unlocked when the updated confidence evaluation value is greater than or equal to the confidence threshold. After confirming that there is no collision on the grasping path, the planned motion trajectory is displayed. The operator finally confirmed that the grasping posture was valid, wrote the adjusted grasping posture back to the execution pipeline, and then performed the grasping according to the confirmed grasping path without collision, displaying the planned motion trajectory.

9. The deep learning-based human-machine collaborative mobile robotic arm grasping method according to claim 1, characterized in that, Step (4) specifically includes: The camera-to-endpoint extrinsic parameter matrix is ​​obtained using a hand-eye calibration algorithm. Before the capture process, the pose of the target object in the world coordinate system is obtained using the following coordinate transformation chain:

10. In the formula, This represents the pose of the target object in the world coordinate system. This represents the calibration transformation pose from the world coordinate system to the base coordinate system. This indicates the pose of the robotic arm's end effector in the base coordinate system. This represents the hand-eye extrinsic parameters from the robotic arm's end effector to the camera coordinate system. This indicates the pose of the target object in the camera coordinate system; By utilizing the planning pipeline of the motion planning framework in the robot operating system and the interface of the open-source motion planning library, a collision-free motion trajectory of the current state is generated in the joint / Cartesian hybrid space. At the same time, the path segments of the planned collision-free motion trajectory are fitted and resampled to generate a smooth and continuous velocity curve to meet the maximum velocity constraint of the robotic arm joint. Then, the robotic arm is controlled to perform the grasping action according to the collision-free motion trajectory and grasping posture. After grasping the target object, the grasping stability is quickly determined based on the end torque and joint current: if slippage, lack of clamping or collision risk is detected, a safety stop is immediately executed and the pose is finely adjusted, and then the grasping is attempted again; If the stability check is passed, the device will be placed along the planned assembly path and the end constraint will be released.

11. A system for implementing the deep learning-based human-machine collaborative mobile robotic arm grasping method according to any one of claims 1-9, characterized in that, include: It includes a six-degree-of-freedom robotic arm, a deformable tracked vehicle, an RGB-D camera, an RGB-D time-of-flight camera, and an industrial control computer. The six-degree-of-freedom robotic arm is mounted on the deformable tracked vehicle, the RGB-D camera is mounted at the end of the six-degree-of-freedom robotic arm, the RGB-D time-of-flight camera is mounted on the deformable tracked vehicle, and the industrial control computer is located inside the deformable tracked vehicle. The industrial control computer has a grasping control system, which includes a vision perception module, a mask segmentation module, a grasping posture generation module, a human-machine collaboration module, a hand-eye calibration module, and a motion planning and execution control module. The system comprises the following modules: a visual perception module, an RGB-D camera and an RGB-D time-of-flight camera, and a human-machine collaboration module. The visual perception module acquires color and depth images of target objects at the construction site, along with corresponding timestamps, and converts the depth images into point cloud data. The mask segmentation module is equipped with an Attention-Mask network, which uses streaming memory to continuously detect and segment target objects, and outputs a pixel-level mask of the grasped object and its confidence score. The grasping pose generation module inputs the point cloud data after pixel-level mask cropping into an AnyGrasp network to generate and score the seven-DOF grasping poses of the grasped object. The human-machine collaboration module performs stable score calculations, confidence threshold determination, and visualization interaction on the K seven-DOF grasping poses. When the confidence score is less than the confidence threshold, a GUI interface is provided for operators to fine-tune and confirm the position, posture, and end effector opening. Collision filtering and path pre-planning are performed in real time after each adjustment. The hand-eye calibration module is used to obtain the extrinsic parameters from the camera coordinate system to the robot's world coordinate system based on the hand-eye calibration algorithm, and accurately maps the confirmed valid grasping posture from the camera coordinate system to the robot's world coordinate system. The motion planning and execution control module is used to plan a collision-free motion trajectory using the motion planning framework in the robot operating system, so as to generate a collision-free motion trajectory in the obstacle scene and perform time parameterization, so that the robot arm can perform grasping action according to the collision-free motion trajectory and grasping posture, grasp the target object, and complete the assembly and placement.

Images