Mechanical arm door handle grabbing and door opening control method and device

Abstract

The invention discloses a mechanical arm door handle grabbing and door opening control method and device, and the method comprises the steps: constructing an end-to-end closed-loop system from instance segmentation, grabbing point prediction, pose settlement and coordinate transformation, and execution control through combining a door handle mask obtained by fusing an upstream recognition detection task and combining a GGCNN algorithm. Combined prediction of the position and direction of the grabbing point is achieved, a closed-loop integration mechanism of visual perception and motion control is constructed, and the success rate of door handle grabbing and the robustness of door opening operation are remarkably improved. Meanwhile, the grabbing target pose is published through TF2 broadcast, so that nodes such as a downstream motion planner and a visualization tool can dynamically query coordinate transformation, point-to-point communication is not needed, and the modularization and expandability of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of automation control technology, and in particular to a method and device for controlling the gripping and opening of a door handle by a robotic arm based on mask guidance. Background Technology

[0002] Door handle grasping and opening operations are typical application scenarios in service robots, home assistant robots, and mobile operation platforms. Traditional robotic arm door opening methods usually rely on manual teaching or preset grasping points, that is, the door handle position and grasping posture are manually calibrated, and the system only performs repetitive movements. This method lacks generalization ability when faced with door handles of different shapes, sizes, installation heights, and orientations, and cannot adapt to autonomous operation tasks in unknown environments.

[0003] In recent years, vision-based grasping methods have gradually become a research hotspot. Existing technologies typically use object detection networks to locate the door handle region, then extract precise contours through instance segmentation, and finally generate grasping points based on geometric analysis or template matching. However, these methods suffer from problems such as decoupling prediction of grasping points from grasping pose and insufficient utilization of prior information from the mask.

[0004] The patent "Method and System for Controlling Door Opening with a Robotic Arm" (Patent No.: CN202211274233.1) proposes a method for controlling door opening with a robotic arm based on preset door information. Its core idea is to obtain "door opening information" (including door position, number of door panels, handle position, opening direction, lock position, etc.) in advance through manual intervention or sensors, determine the door type (hinged door / sliding door) based on this information, and select the appropriate grasping strategy. For example... Figure 5 As shown, its workflow includes: ① Obtain door opening information: Acquire basic door attributes through external input or simple sensors (such as cameras); ② Determine the type of door: If the number of door panels is a single leaf, it is determined to be a hinged door; if there are multiple leaves and they are not on the same straight line, it is determined to be a sliding door. ③ Perform differentiated grasping: For hinged doors, control the robotic arm to grasp the door handle by clamping it and rotate the handle to unlock it; for sliding doors, control the robotic arm to grasp the door surface by adhering to it and pull the door panel in the opening direction. ④ Motion Coordination Control: During the door opening process, the relative speed is calculated based on the end torque of the robotic arm and the speed of the wheelchair, and then converted into joint motion speed through the Jacobian matrix to achieve coordinated movement between the wheelchair and the robotic arm.

[0005] The aforementioned patent achieves basic automatic door opening functionality, but its technical solution has the following drawbacks: ① Heavily reliant on prior door information and lacking autonomous perception capabilities: This method requires the system to know structured information such as the number of door panels, handle positions, and opening directions in advance. In real open environments, robots cannot obtain such information in advance and must autonomously determine door attributes from raw images through vision. However, this invention does not provide any ability to automatically detect the presence and geometry of door handles from images, resulting in its applicability only to highly structured known environments and severely limited generalization capabilities.

[0006] ② The grasping strategy is coarse-grained and cannot adapt to the geometric details of the handle: For hinged doors, it only uses the general instruction of "gripping the door handle", without solving the problem of fine-grained generation of the grasping point position, grasping direction and gripper width. This invention does not establish a mapping mechanism from visual features to six-degree-of-freedom grasping pose, which makes it very easy for slippage or collision to occur due to grasping deviation. Summary of the Invention

[0007] In view of the above problems, the present invention provides a robotic arm door handle grasping and opening control method and device for overcoming or at least partially solving the above problems. The method realizes the joint prediction of the position and direction of the grasping point and constructs a closed-loop integrated mechanism of visual perception and motion control, which significantly improves the success rate of door handle grasping and the robustness of door opening operation.

[0008] This invention provides the following solution: A method for controlling the gripping and opening of a door handle by a robotic arm, comprising: Acquire the RGB image and depth map of the scene, perform target detection and instance segmentation on the RGB image, and generate a pixel-level binary mask for the door handle; The pixel-level binary mask is fused with the depth map and then input into a generative grasping convolutional neural network to predict and obtain the grasping quality map, grasping corner map and grasping width map end-to-end. The pixel coordinates of the optimal grab point are determined on the grab quality map, and the optimal grab angle and optimal grab width corresponding to the optimal grab point are extracted from the grab angle map and the grab width map. Based on the pixel coordinates of the optimal gripping point and the intrinsic parameter matrix of the depth camera, back projection is performed to calculate the three-dimensional coordinates of the optimal gripping point in the camera coordinate system. The three-dimensional coordinates are transformed to the coordinate system of the robotic arm base using the camera extrinsic parameters to obtain the target position; based on the optimal gripping angle and the local surface normal vector of the door handle, the target pose matrix of the robotic arm end effector is constructed, thereby generating a six-degree-of-freedom target pose; The six-degree-of-freedom target pose is broadcast using the TF2 coordinate transformation library of the robot operating system ROS2; The robotic arm control system receives and analyzes the six-degree-of-freedom target pose, performs motion planning and trajectory tracking, controls the robotic arm to move to the grasping pose, and controls the gripper to close according to the optimal grasping width to perform grasping and opening actions.

[0009] Preferably, the grasping angle diagram includes a cosine component diagram of the grasping angle and a sine component diagram of the grasping angle; The generative grasping convolutional neural network adopts an encoder-decoder symmetric architecture. The encoder performs downsampling through multiple convolutions to extract depth features, and the decoder performs upsampling through transposed convolutions to restore resolution. Finally, it outputs four feature maps with the same resolution as the input image, which correspond to the grasping quality map, the cosine component map of the grasping angle, the sine component map of the grasping angle, and the grasping width map, respectively.

[0010] Preferably, the method for determining the optimal grab point includes: Gaussian filtering is applied to the captured quality image to suppress noise; The grasping angle is calculated based on the cosine component graph and the sine component graph, and the result is mapped to... The interval is then filtered using a Gaussian filter. Multiply the captured width image by a scale factor to restore it to pixel unit width, and then apply a Gaussian filter. The local peak detection algorithm is used to extract the Top-K local maxima points on the crawling quality map and return the optimal crawling point; For each peak point, extract the corresponding angle and width to generate a grab object, which includes the grab center, grab angle, grab length and grab width; During the inference phase, the crawling quality heatmap is traversed, and the location corresponding to the global maximum value is selected as the optimal crawling center: Read the grab angle and grab width at the corresponding positions from the grab angle map and the grab width map.

[0011] Preferably, the gripping width is converted into the physical opening distance of the gripper through a calibration relationship, which is used to control the degree of closure of the end effector.

[0012] Preferably, the step of constructing the target pose matrix includes: The first axis of the end effector is a unit vector parallel to the gripping direction defined by the optimal gripping angle; The normal vector perpendicular to the local plane of the door handle is taken as the second axis; The third axis is calculated by multiplying the first axis vector with the second axis vector, and each axis is orthogonalized to form a right-handed coordinate system, thereby determining the target attitude matrix.

[0013] Preferably: Broadcasting the six-DOF target pose via the TF2 coordinate transformation library includes: The six-degree-of-freedom target pose is defined as a transformation relationship between a sub-coordinate system and the robot base coordinate system, and this transformation is continuously broadcast through the ROS2 TF2 broadcaster so that any node subscribing to TF2 can dynamically obtain the pose information.

[0014] Preferably, the action of grasping and opening the door further includes: Collision-free path planning can be performed using the fast extended random tree star algorithm or the probabilistic route graph method. The proportional-integral-derivative control algorithm is used to drive the joints of the robotic arm to achieve precise tracking of the planned trajectory; When the gripper contacts the door handle, an impedance control strategy is used to adaptively adjust the gripping force. The impedance control strategy is based on the position error, stiffness coefficient, and damping coefficient to construct a force control model.

[0015] Preferably, the impedance control strategy is expressed by the following formula:

[0016] in, For positional error, , These are the stiffness and damping coefficients, respectively.

[0017] Preferably, the target detection and instance segmentation are implemented using the YOLOv8 or YOLOv11 model.

[0018] A robotic arm door handle gripping and opening control device is provided for executing the aforementioned robotic arm door handle gripping and opening control method. The device includes: The input data acquisition unit is used to acquire the RGB image and depth map of the scene, perform target detection and instance segmentation on the RGB image, and generate a pixel-level binary mask for the door handle; The fusion prediction unit is used to fuse the pixel-level binary mask with the depth map and input it into the generative grasping convolutional neural network to predict and obtain the grasping quality map, grasping corner map and grasping width map end-to-end. The optimal gripping point determination unit is used to determine the pixel coordinates of the optimal gripping point on the gripping quality map, and extract the optimal gripping angle and optimal gripping width corresponding to the optimal gripping point from the gripping angle map and the gripping width map. The projection unit is used to perform back projection based on the pixel coordinates of the optimal grasping point and the intrinsic parameter matrix of the depth camera, and to calculate the three-dimensional coordinates of the optimal grasping point in the camera coordinate system. The six-degree-of-freedom target pose calculation unit is used to transform the three-dimensional coordinates to the robot arm base coordinate system using camera extrinsic parameters to obtain the target position; and to construct the target pose matrix of the robot arm end effector based on the optimal gripping angle and the local surface normal vector of the door handle, thereby generating the six-degree-of-freedom target pose. The broadcasting unit is used to broadcast the six-degree-of-freedom target pose through the TF2 coordinate transformation library of the robot operating system ROS2; The execution unit is used by the robotic arm control system to receive and analyze the six-degree-of-freedom target pose, perform motion planning and trajectory tracking, control the robotic arm to move to the grasping pose, and control the gripper to close according to the optimal grasping width to perform grasping and opening actions.

[0019] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This application provides a robotic arm door handle grasping and opening control method and apparatus. By fusing the door handle mask obtained from upstream recognition and detection tasks and combining it with the GGCNN algorithm, an end-to-end closed-loop system is constructed, encompassing instance segmentation, grasping point prediction, pose calculation and coordinate transformation, and execution control. This achieves joint prediction of the grasping point position and orientation, and establishes a closed-loop integration mechanism for visual perception and motion control, significantly improving the success rate of door handle grasping and the robustness of door opening operations. Simultaneously, the grasping target pose is broadcast via TF2, allowing downstream nodes such as motion planners and visualization tools to dynamically query coordinate transformations without point-to-point communication, thus enhancing system modularity and scalability.

[0020] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0022] Figure 1 This is a flowchart of a robotic arm door handle grasping and opening control method provided in an embodiment of the present invention; Figure 2 This is a joint prediction map of the position and orientation of the grabbing point provided in the embodiment of the present invention; Figure 3 This is a schematic diagram of a robotic arm door handle gripping and opening control device provided in an embodiment of the present invention; Figure 4This is a schematic diagram of a robotic arm door handle gripping and opening control device provided in an embodiment of the present invention; Figure 5 This is a flowchart of an existing technical solution. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0024] See Figure 1 This invention provides a method for controlling the gripping and opening of a door handle using a robotic arm, as shown in the embodiment of the invention. Figure 1 As shown, by fusing the door handle mask obtained from the upstream recognition and detection task and combining it with the GGCNN algorithm, an end-to-end closed-loop system is constructed, encompassing instance segmentation, grasp point prediction, pose calculation and coordinate transformation, and execution control. This method may include: S101: Acquire the RGB image and depth map of the scene, perform target detection and instance segmentation on the RGB image, and generate a pixel-level binary mask for the door handle; as the input front-end of the entire system, it is responsible for acquiring high-quality visual information from the environment and outputting a pixel-level door handle mask for subsequent grasping decisions. This module does not involve the core innovation of this invention, but rather the invocation and integration of upstream visual perception technologies, and its functions can be implemented by existing mature solutions.

[0025] The specific implementation is as follows: ① Image Acquisition: Simultaneously acquire RGB images and depth maps of the scene using an RGB-D camera. To ensure the consistency of spatial information; ②Target detection: Using efficient target detection models such as YOLOv8 and YOLOv11, the RGB image is processed to quickly locate the door handle area and output the bounding box; ③ Instance segmentation: Using the bounding boxes output by YOLOv8 as cues, a lightweight segmentation model is driven to perform instance segmentation, generating high-precision pixel-level binary masks. Precisely depict the outline of the door handle; The output of this module, the pixel-level mask of the door handle and its corresponding depth information, is the basic input for all subsequent control decisions. Its quality directly determines the accuracy of the grasp point prediction and the reliability of the pose calculation.

[0026] S102: The pixel-level binary mask is fused with the depth map and then input into a generative grasping convolutional neural network to predict and obtain a grasping quality map, a grasping angle map, and a grasping width map end-to-end; In specific implementation, the embodiments of this application may provide that the grasping angle map includes a cosine component map of the grasping angle and a sine component map of the grasping angle. The generative grasping convolutional neural network adopts an encoder-decoder symmetric architecture. The encoder performs downsampling through multiple convolutions to extract depth features, and the decoder performs upsampling through transposed convolutions to restore resolution. Finally, it outputs four feature maps with the same resolution as the input image, which correspond to the grasping quality map, the cosine component map of the grasping angle, the sine component map of the grasping angle, and the grasping width map, respectively.

[0027] After accurately segmenting the door handle, the system further employs the GGCNN model to predict the optimal grasping pose end-to-end from the depth image, seamlessly connecting with the upstream detection and segmentation module to form a complete visual perception closed loop of "detection → segmentation → grasping".

[0028] like Figure 2 As shown, the input to GGCNN is a depth image. To improve the targeting of data capture, the binary mask output by the segmentation task is integrated. As the second input channel, the guided model predicts the grab point only within the door handle area. The input image is normalized to... The range is then scaled to a fixed size of 300×300 to fit network input requirements.

[0029] GGCNN employs a symmetrical encoder-decoder architecture. The encoder consists of three convolutional downsampling layers to extract spatial structure and texture features from depth images. The decoder consists of three transposed convolutional upsampling layers to progressively restore the feature map resolution to the input size. The network output consists of four feature maps with the same resolution as the input, corresponding to: ① Capture quality image : Represents the confidence level of each pixel location as the capture center; ② Capture the cosine component of the angle. Plot with sine component Joint decoding is used to capture corner images. ; ③ Capture the width image : Indicates the gripper opening width (normalized value) required for gripping at this position.

[0030] S103: Determine the pixel coordinates of the optimal grasping point on the grasping quality map, and extract the optimal grasping angle and optimal grasping width corresponding to the optimal grasping point from the grasping angle map and the grasping width map; In specific implementation, the method for determining the optimal grasping point in this application embodiment includes: Gaussian filtering is applied to the captured quality image to suppress noise; The grasping angle is calculated based on the cosine component graph and the sine component graph, and the result is mapped to... The interval is then filtered using a Gaussian filter. Multiply the captured width image by a scale factor to restore it to pixel unit width, and then apply a Gaussian filter. The local peak detection algorithm is used to extract the Top-K local maxima points on the crawling quality map and return the optimal crawling point; For each peak point, extract the corresponding angle and width to generate a grab object, which includes the grab center, grab angle, grab length and grab width; During the inference phase, the crawling quality heatmap is traversed, and the location corresponding to the global maximum value is selected as the optimal crawling center: Read the grab angle and grab width at the corresponding positions from the grab angle map and the grab width map.

[0031] The gripping width is converted into the physical opening distance of the gripper through a calibration relationship, which is used to control the degree of closure of the end effector.

[0032] The network weights are initialized using Xavier, and the activation function is ReLU to ensure training stability and feature representation capability. The original network output is post-processed to convert it into executable grabbing parameters, as follows: ① Quality Map Smoothing: Improves the quality map of the captured image. Apply Gaussian filter ( ), suppress noise; ② Angle Decoding: Calculate the grasping angle based on the cosine and sine components. The result is mapped to The interval is then subjected to Gaussian filtering. )

[0033] ③ Width Restoration: Restore the width of the captured image. Multiply by a scale factor (default 150.0), restore to pixel unit width, and apply a Gaussian filter. ); ④ Crawling point selection: The local peak detection algorithm is used to extract the top-K local maxima points on the quality map. The parameters are set to a minimum distance of 20 pixels and a confidence threshold of 0.2. The optimal crawling point is returned by default. ⑤ Parameter capture and encapsulation: For each peak point Extract the corresponding angle With width Generate a crawling object, including the crawling center, crawling angle, crawling length, and crawling width.

[0034] During the inference phase, the system traverses and captures quality heatmaps. Select the position corresponding to the global maximum value. As the optimal crawling center:

[0035] Subsequently, from the grab corner map and capture width map Read the grab angle at the corresponding position and crawl width .

[0036] In this solution, the crawling point containing the maximum response value is selected as the optimal crawling point, which is parameterized as follows:

[0037] S104: Based on the pixel coordinates of the optimal grasping point and the intrinsic parameter matrix of the depth camera, perform back projection to calculate the three-dimensional coordinates of the optimal grasping point in the camera coordinate system; specifically, in this application embodiment, the steps for constructing the target pose matrix include: Let the unit vector parallel to the gripping direction defined by the optimal gripping angle be taken as the first axis of the end effector (assuming...). (The unit vector along the grab direction).

[0038] The normal vector perpendicular to the local plane of the door handle is taken as the second axis. (The normal vector is perpendicular to the door panel). The third axis is obtained by multiplying the first axis vector by the second axis vector. The axes are orthogonalized to form a right-handed coordinate system, thereby determining the target attitude matrix.

[0039] S105: Transform the three-dimensional coordinates to the robot arm base coordinate system using camera extrinsic parameters to obtain the target position; construct the target pose matrix of the robot arm end effector based on the optimal gripping angle and the local surface normal vector of the door handle, thereby generating a six-degree-of-freedom target pose; After predicting the gripping point of the door handle, the system needs to convert the coordinates of the gripping center in the two-dimensional image plane into a three-dimensional spatial position in the camera coordinate system, and further calculate the target pose that the robotic arm can execute. This process is completed by the pose calculation module, the core of which lies in utilizing the intrinsic parameter matrix of the depth camera. Perform 3D back projection on pixel coordinates and combine it with the extrinsic parameter matrix. A coordinate system transformation is achieved, ultimately generating the six-DOF target pose.

[0040] Specifically, the system first obtains the pixel coordinates of the optimal grab point from the grab prediction module. and its corresponding original depth value (Unit: 16-bit integer). This depth value is not the actual physical distance and needs to be converted using the depth scale of the depth camera. The specific formula is as follows:

[0041] in, These are the physical calibration coefficients for the camera's depth sensor, typically obtained through calibration board experiments, used to convert depth-encoded values ​​into millimeter-level true distances. Subsequently, the camera's intrinsic parameter matrix is ​​used... Perform back projection, camera intrinsic parameter matrix for:

[0042] in, Focal length The coordinates of the master point are all from the depth camera calibration results. Based on this, the three-dimensional coordinates of the capture point in the camera coordinate system are... It can be accurately calculated using the inverse transform formula of the pinhole camera model:

[0043]

[0044]

[0045] Further combining the extrinsic parameter matrix Transform the points to the coordinate system of the robotic arm base:

[0046] At the same time, based on the orientation angle output by GGCNN Calculate the grab normal vector By combining the door handle surface normal obtained through local plane fitting, the attitude matrix of the robotic arm end effector can be determined. Specifically, let's assume... Let be the unit vector along the grab direction. Let be the normal vector perpendicular to the door panel, then This forms a right-handed coordinate system. Ultimately, this constitutes a complete six-degree-of-freedom target pose:

[0047] in, It is a translation vector.

[0048] In addition, the grab width output by GGCNN Converted into the physical opening distance of the grippers after calibration. It is used to control the degree of closure of the end effector to ensure that the gripping force is moderate and does not damage the handle.

[0049] S106: Broadcast the six-degree-of-freedom target pose through the TF2 coordinate transformation library of the robot operating system ROS2; specifically, in this application embodiment, broadcasting the six-degree-of-freedom target pose through the TF2 coordinate transformation library can include: The six-degree-of-freedom target pose is defined as a transformation relationship between a sub-coordinate system and the robot base coordinate system, and this transformation is continuously broadcast through the ROS2 TF2 broadcaster so that any node subscribing to TF2 can dynamically obtain the pose information.

[0050] A distributed communication architecture based on ROS 2 is adopted to achieve efficient and low-latency data interaction between the vision perception system and the robotic arm control system. Specifically, the system publishes the pose of the grasped target through ROS 2's TF2 broadcast mechanism, instead of traditional custom message topics. Any downstream node (such as the robotic arm motion planner, visualization tool RViz2, etc.) can dynamically query the pose of the sub-coordinate system relative to the robot base or other arbitrary coordinate systems through the standard TF2 interface, without the need to establish a point-to-point message subscription relationship. This fully utilizes the mature coordinate transformation management capabilities in the ROS ecosystem, greatly improving the modularity and scalability of the system.

[0051] S107: The robotic arm control system receives and analyzes the six-degree-of-freedom target pose, performs motion planning and trajectory tracking, controls the robotic arm to move to the grasping pose, and controls the gripper to close according to the optimal grasping width to perform grasping and opening actions.

[0052] In a specific implementation, the embodiments of this application may further include the following: Collision-free path planning can be performed using the fast extended random tree star algorithm or the probabilistic route graph method. The proportional-integral-derivative control algorithm is used to drive the joints of the robotic arm to achieve precise tracking of the planned trajectory; When the gripper contacts the door handle, an impedance control strategy is used to adaptively adjust the gripping force. The impedance control strategy is based on the position error, stiffness coefficient, and damping coefficient to construct a force control model.

[0053] The impedance control strategy is expressed by the following equation:

[0054] in, For positional error, , These are the stiffness and damping coefficients, respectively.

[0055] The execution module consists of a six-degree-of-freedom robotic arm, a three-finger gripper, and its motion control system. It is responsible for converting the door handle pose information provided by the upstream visual perception system into a series of precise and safe physical actions to ultimately complete the door opening task.

[0056] After receiving the target pose, the robotic arm control system executes the following steps in sequence: ① Motion planning: Collision-free path planning is performed using RRT* or PRM algorithms to generate a smooth trajectory from the current pose to the grasping pose; ②Trajectory tracking: The underlying controller uses a PID control algorithm to drive each joint of the robotic arm to track the planned trajectory, with a position control accuracy better than ±1mm; ③ Grasping Execution: After the robotic arm reaches the grasping pose, the three-finger gripper closes according to the predicted width L, and the grasping force is adaptively adjusted using an impedance control strategy.

[0057] in, For positional error, , These are the stiffness and damping coefficients. This strategy enables the grippers to automatically adjust the applied force during contact, avoiding impact or damage caused by hard contact.

[0058] For lever-type or spherical door handles, GGCNN provides the gripping point location. Grab Angle and crawl width The system first bases its detection on the angle of capture. Determine the main axis direction of the door handle and calculate the required rotational torque around that axis accordingly; grip width. Helps determine the optimal opening width of the robotic arm gripper to ensure a secure and precise grip on the door handle; gripping point The system guides the robotic arm to move to the accurate position for grasping; after successfully grasping the door handle, the system controls the robotic arm to apply a constant torque around the door axis and perform a rotational motion at a preset angle to open the door.

[0059] It is evident that this method solves the problem of inaccurate prediction of the grasping pose of traditional door handles. Traditional methods rely on manual teaching or rough estimation based on bounding boxes, which cannot accurately obtain the position and direction of the grasping point, and have a low success rate, especially when facing various door handles such as handle-type and spherical ones.

[0060] It also solves the problem of the disconnect between visual perception and motion control in existing technologies. Existing systems often process detection and segmentation, grasping and planning separately, and lack an end-to-end mapping mechanism from pixel-level masks to six-degree-of-freedom end poses.

[0061] In addition, it solves the problems of low integration and complex deployment in traditional systems, where modules rely on customized message passing, making it difficult to achieve efficient collaboration within standard robot frameworks such as ROS.

[0062] This method uses the pixel-level binary mask of the door handle output by instance segmentation as a guide, integrates the depth image input into the GGCNN network, and predicts the grasping quality heatmap, grasping angle (cosine / sine components) and grasping width end-to-end to achieve joint optimization of grasping center and direction; uses the camera intrinsic parameters to perform 3D back projection on the optimal grasping point, and combines the extrinsic parameter matrix to transform to the coordinate system of the robotic arm base; based on the grasping angle output by GGCNN and the local plane fitting normal, a right-handed coordinate system is constructed to generate the complete SE(3) target pose; the grasping target pose is broadcast through TF2, so that downstream motion planners, visualization tools and other nodes can dynamically query coordinate transformation without point-to-point communication, thereby improving the modularity and scalability of the system.

[0063] In summary, the robotic arm door handle grasping and opening control method provided in this application, by fusing the door handle mask obtained from the upstream recognition and detection task and combining it with the GGCNN algorithm, constructs an end-to-end closed-loop system encompassing instance segmentation, grasping point prediction, pose calculation and coordinate transformation, and execution control. It achieves joint prediction of the grasping point position and orientation, and establishes a closed-loop integration mechanism for visual perception and motion control, significantly improving the success rate of door handle grasping and the robustness of door opening operations.

[0064] See Figure 3 This application embodiment can also provide a robotic arm door handle gripping and opening control device, such as... Figure 3 As shown, the device used to perform the above-described robotic arm door handle gripping and opening control method may include: The input data acquisition unit 301 is used to acquire the RGB image and depth map of the scene, perform target detection and instance segmentation on the RGB image, and generate a pixel-level binary mask of the door handle; The fusion prediction unit 302 is used to fuse the pixel-level binary mask with the depth map and input it into the generative grasping convolutional neural network to predict and obtain the grasping quality map, grasping corner map and grasping width map end-to-end. The optimal gripping point determination unit 303 is used to determine the pixel coordinates of the optimal gripping point on the gripping quality map, and extract the optimal gripping angle and optimal gripping width corresponding to the optimal gripping point from the gripping angle map and the gripping width map. Projection unit 304 is used to perform back projection based on the pixel coordinates of the optimal grasping point and the intrinsic parameter matrix of the depth camera, and calculate the three-dimensional coordinates of the optimal grasping point in the camera coordinate system. The six-degree-of-freedom target pose calculation unit 305 is used to transform the three-dimensional coordinates to the robot arm base coordinate system using camera extrinsic parameters to obtain the target position; and to construct the target pose matrix of the robot arm end effector based on the optimal gripping angle and the local surface normal vector of the door handle, thereby generating the six-degree-of-freedom target pose. Broadcast unit 306 is used to broadcast the six-degree-of-freedom target pose through the TF2 coordinate transformation library of the robot operating system ROS2; The execution unit 307 is used by the robotic arm control system to receive and analyze the six-degree-of-freedom target pose, perform motion planning and trajectory tracking, control the robotic arm to move to the grasping pose, and control the gripper to close according to the optimal grasping width to perform grasping and opening actions.

[0065] This application embodiment can also provide a robotic arm door handle grasping and opening control device, the device including a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the steps of the above-described robotic arm door handle grasping and opening control method according to the instructions in the program code.

[0066] like Figure 4 As shown in the figure, the robotic arm door handle grasping and opening control device provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0067] In this embodiment, the processor 10 may be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0068] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiment of the robotic arm door handle grasping and opening control method.

[0069] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions: Acquire the RGB image and depth map of the scene, perform target detection and instance segmentation on the RGB image, and generate a pixel-level binary mask for the door handle; The pixel-level binary mask is fused with the depth map and then input into a generative grasping convolutional neural network to predict and obtain the grasping quality map, grasping corner map and grasping width map end-to-end. The pixel coordinates of the optimal grab point are determined on the grab quality map, and the optimal grab angle and optimal grab width corresponding to the optimal grab point are extracted from the grab angle map and the grab width map. Based on the pixel coordinates of the optimal gripping point and the intrinsic parameter matrix of the depth camera, back projection is performed to calculate the three-dimensional coordinates of the optimal gripping point in the camera coordinate system. The three-dimensional coordinates are transformed to the coordinate system of the robotic arm base using the camera extrinsic parameters to obtain the target position; based on the optimal gripping angle and the local surface normal vector of the door handle, the target pose matrix of the robotic arm end effector is constructed, thereby generating a six-degree-of-freedom target pose; The six-degree-of-freedom target pose is broadcast using the TF2 coordinate transformation library of the robot operating system ROS2; The robotic arm control system receives and analyzes the six-degree-of-freedom target pose, performs motion planning and trajectory tracking, controls the robotic arm to move to the grasping pose, and controls the gripper to close according to the optimal grasping width to perform grasping and opening actions.

[0070] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0071] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0072] Communication interface 12 can be an interface for a communication model, used to connect with other devices or systems.

[0073] Of course, it should be noted that, Figure 4 The structure shown does not constitute a limitation on the robotic arm door handle gripping and opening control device in the embodiments of this application. In practical applications, the robotic arm door handle gripping and opening control device may include more than Figure 4More or fewer components as shown, or combinations of certain components.

[0074] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the above-described robotic arm door handle grasping and opening control method.

[0075] GGCNN (Generative Grasping Convolutional Neural Network): A generative grasping convolutional neural network used to predict grasping poses.

[0076] RGB-D (Red Green Blue - Depth): Image data containing color and depth information.

[0077] ROS 2 (Robot Operating System 2): The second generation version of the robot operating system.

[0078] TF2 (Transform Library 2): A coordinate transformation library in ROS 2.

[0079] ReLU (Rectified Linear Unit): The activation function of the modified linear unit.

[0080] RRT* (Rapidly-exploring Random Tree Star): A fast-exploring random tree star algorithm.

[0081] PRM (Probabilistic Roadmap): a method of probabilistic roadmaps.

[0082] PID (Proportional-Integral-Derivative): Proportional-Integral-Derivative control algorithm.

[0083] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0084] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0085] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for controlling the gripping and opening of a door handle using a robotic arm, characterized in that, The method includes: Acquire the RGB image and depth map of the scene, perform target detection and instance segmentation on the RGB image, and generate a pixel-level binary mask for the door handle; The pixel-level binary mask is fused with the depth map and then input into a generative grasping convolutional neural network to predict the grasping quality map, grasping corner map, and grasping width map end-to-end. The pixel coordinates of the optimal grab point are determined on the grab quality map, and the optimal grab angle and optimal grab width corresponding to the optimal grab point are extracted from the grab angle map and the grab width map. Based on the pixel coordinates of the optimal gripping point and the intrinsic parameter matrix of the depth camera, back projection is performed to calculate the three-dimensional coordinates of the optimal gripping point in the camera coordinate system. The three-dimensional coordinates are transformed to the coordinate system of the robotic arm base using the camera extrinsic parameters to obtain the target position; based on the optimal gripping angle and the local surface normal vector of the door handle, the target pose matrix of the robotic arm end effector is constructed, thereby generating a six-degree-of-freedom target pose; The six-degree-of-freedom target pose is broadcast using the TF2 coordinate transformation library of the robot operating system ROS2; The robotic arm control system receives and analyzes the six-degree-of-freedom target pose, performs motion planning and trajectory tracking, controls the robotic arm to move to the grasping pose, and controls the gripper to close according to the optimal grasping width to perform grasping and opening actions.

2. The robotic arm door handle gripping and opening control method according to claim 1, characterized in that, The grasping angle diagram includes a cosine component diagram of the grasping angle and a sine component diagram of the grasping angle; The generative grasping convolutional neural network adopts an encoder-decoder symmetric architecture. The encoder performs downsampling through multiple convolutions to extract depth features, and the decoder performs upsampling through transposed convolutions to restore resolution. Finally, it outputs four feature maps with the same resolution as the input image, which correspond to the grasping quality map, the cosine component map of the grasping angle, the sine component map of the grasping angle, and the grasping width map, respectively.

3. The robotic arm door handle gripping and opening control method according to claim 2, characterized in that, Methods for determining the optimal crawling point include: Gaussian filtering is applied to the captured quality image to suppress noise; The grasping angle is calculated based on the cosine component graph and the sine component graph, and the result is mapped to... The interval is then filtered using a Gaussian filter. Multiply the captured width image by a scale factor to restore it to pixel unit width, and then apply a Gaussian filter. The local peak detection algorithm is used to extract the Top-K local maxima points on the crawling quality map and return the optimal crawling point; For each peak point, extract the corresponding angle and width to generate a grab object, which includes the grab center, grab angle, grab length and grab width; During the inference phase, the crawling quality heatmap is traversed, and the location corresponding to the global maximum value is selected as the optimal crawling center: Read the grab angle and grab width at the corresponding positions from the grab angle map and the grab width map.

4. The robotic arm door handle gripping and opening control method according to claim 3, characterized in that, The gripping width is converted into the physical opening distance of the gripper through a calibration relationship, which is used to control the degree of closure of the end effector.

5. The robotic arm door handle gripping and opening control method according to claim 1, characterized in that, The steps for constructing the target pose matrix include: The first axis of the end effector is a unit vector parallel to the gripping direction defined by the optimal gripping angle; The normal vector perpendicular to the local plane of the door handle is taken as the second axis; The third axis is calculated by multiplying the first axis vector with the second axis vector, and each axis is orthogonalized to form a right-handed coordinate system, thereby determining the target attitude matrix.

6. The robotic arm door handle gripping and opening control method according to claim 1, characterized in that, The six-degree-of-freedom target pose broadcast via the TF2 coordinate transformation library includes: The six-degree-of-freedom target pose is defined as a transformation relationship between a sub-coordinate system and the robot base coordinate system, and this transformation is continuously broadcast through the ROS2 TF2 broadcaster so that any node subscribing to TF2 can dynamically obtain the pose information.

7. The robotic arm door handle gripping and opening control method according to claim 1, characterized in that, The execution of the grabbing and opening actions further includes: Collision-free path planning can be performed using the fast extended random tree star algorithm or the probabilistic route graph method. The proportional-integral-derivative control algorithm is used to drive the joints of the robotic arm to achieve precise tracking of the planned trajectory; When the gripper contacts the door handle, an impedance control strategy is used to adaptively adjust the gripping force. The impedance control strategy is based on the position error, stiffness coefficient, and damping coefficient to construct a force control model.

8. The robotic arm door handle gripping and opening control method according to claim 7, characterized in that, The impedance control strategy is expressed by the following equation: in, For positional error, , These are the stiffness and damping coefficients, respectively.

9. The robotic arm door handle gripping and opening control method according to claim 1, characterized in that, The target detection and instance segmentation are implemented using the YOLOv8 or YOLOv11 model.

10. A robotic arm door handle gripping and opening control device, characterized in that, The apparatus for performing the robotic arm door handle grasping and opening control method according to any one of claims 1-9, the apparatus comprising: The input data acquisition unit is used to acquire the RGB image and depth map of the scene, perform target detection and instance segmentation on the RGB image, and generate a pixel-level binary mask for the door handle; The fusion prediction unit is used to fuse the pixel-level binary mask with the depth map and input it into the generative grasping convolutional neural network to predict and obtain the grasping quality map, grasping corner map and grasping width map end-to-end. The optimal gripping point determination unit is used to determine the pixel coordinates of the optimal gripping point on the gripping quality map, and extract the optimal gripping angle and optimal gripping width corresponding to the optimal gripping point from the gripping angle map and the gripping width map; The projection unit is used to perform back projection based on the pixel coordinates of the optimal grasping point and the intrinsic parameter matrix of the depth camera, and to calculate the three-dimensional coordinates of the optimal grasping point in the camera coordinate system. The six-degree-of-freedom target pose calculation unit is used to transform the three-dimensional coordinates to the robot arm base coordinate system using camera extrinsic parameters to obtain the target position; and to construct the target pose matrix of the robot arm end effector based on the optimal gripping angle and the local surface normal vector of the door handle, thereby generating the six-degree-of-freedom target pose. The broadcast unit is used to broadcast the six-degree-of-freedom target pose through the TF2 coordinate transformation library of the robot operating system ROS2; The execution unit is used by the robotic arm control system to receive and analyze the six-degree-of-freedom target pose, perform motion planning and trajectory tracking, control the robotic arm to move to the grasping pose, and control the gripper to close according to the optimal grasping width to perform grasping and opening actions.

Images