Bolt disassembly method, system, device and medium based on visual servo and point cloud plane pose estimation

Abstract

This invention relates to the field of power line inspection technology, and discloses a bolt dismantling method, system, device, and medium based on visual servoing and point cloud planar pose estimation. The method involves acquiring RGB images of the target metal plane using a camera mounted on the end effector of a robotic arm and performing bolt alignment, outputting the converged seed points of the bolt centers for each bolt, and then acquiring a depth image of the target metal plane after bolt alignment. The point cloud is extracted and an initial plane equation is obtained by combining the seed points of each bolt center. The planar point cloud is then optimized using the depth image and the initial plane equation to obtain a set of effective planar points. Pose estimation is performed based on the effective planar point cloud, and a homogeneous planar pose transformation matrix is ​​constructed. This matrix is ​​then used to fine-tune the posture of the robotic arm's end effector, outputting a vertical pose reference pose. Finally, the robotic arm is driven to perform bolt dismantling operations based on the vertical pose reference pose, solving the technical problem in existing technologies where the tool axis cannot be strictly perpendicular to the target plane.

Description

Technical Field

[0001] This invention relates to the field of power line inspection technology, and in particular to a bolt disassembly method, system, device and medium based on visual servoing and point cloud planar pose estimation. Background Technology

[0002] With the development of automation technology, automated disassembly of bolts and nuts on metal components is a core element in ensuring equipment maintenance efficiency and operational safety in scenarios such as power line inspection, aerospace, and high-end manufacturing. Traditional operation modes rely heavily on manual or remote operation, which is not only inefficient and labor-intensive, but also difficult to adapt to complex and high-risk operating environments such as high-voltage towers and deep-space capsules. This fails to meet the core demands of high-end equipment maintenance for automated and intelligent operations, becoming a key bottleneck restricting the intelligent upgrading of related fields.

[0003] To overcome the limitations of traditional operations and enhance the autonomy of robotic arm bolt disassembly, visual servo technology has been widely adopted as an autonomous guidance solution for end-effector tightening tools. This technology extracts target bolt features through visual perception and drives the robotic arm to align the bolt, becoming the mainstream technical approach for automated bolt disassembly. However, due to calibration errors in the vision system, uncertainties in depth perception, and the inherent limitations of traditional visual servo control, existing visual guidance solutions can only achieve feature convergence within the image domain and cannot guarantee strict perpendicularity between the tightening tool axis and the target metal plane in three-dimensional space. In high-contact bolt disassembly operations, even a slight deviation in posture can easily cause problems such as sleeve jamming and thread stripping, directly leading to operational failure or even damage to components. This has become a core technical bottleneck restricting the accuracy and reliability of automated bolt disassembly. Summary of the Invention

[0004] This invention provides a bolt disassembly method, system, device, and medium based on visual servoing and point cloud planar pose estimation. It solves the technical problems in the prior art where pure visual guidance is limited by calibration error and depth uncertainty, making it difficult to ensure that the axis of the tightening gun is completely perpendicular to the target plane, which in turn leads to sleeve jamming, thread stripping, and insufficient operational reliability.

[0005] The first aspect of this invention provides a bolt disassembly method based on visual servoing and point cloud planar pose estimation, comprising:

[0006] The camera mounted on the end of the robotic arm acquires RGB images of the target metal plane, performs bolt alignment, and outputs the bolt center seed point after each bolt converges.

[0007] The depth image of the target metal plane after the bolts are aligned is obtained, and point cloud extraction and plane fitting are performed by combining the seed points of each bolt center to obtain the initial plane equation.

[0008] The planar point cloud is optimized using the depth image and the initial plane equation to obtain an effective planar point cloud set.

[0009] Pose estimation is performed using the aforementioned effective point cloud in the plane, and a homogeneous planar pose transformation matrix is ​​constructed.

[0010] The planar pose homogeneous transformation matrix is ​​used to fine-tune the end effector posture of the robotic arm and output a vertical pose reference posture.

[0011] Based on the vertical attitude reference pose, the robotic arm is driven to perform bolt removal operations.

[0012] Optionally, the step of acquiring RGB images of the target metal plane using a camera mounted on the end effector of the robotic arm, performing bolt alignment, and outputting the converged bolt center seed points includes:

[0013] The RGB image of the target metal plane is acquired by a camera mounted on the end of the robotic arm;

[0014] Bolt detection is performed on the RGB image using a preset target detection model to obtain the detection box coordinates of each bolt;

[0015] The center feature points of each bolt are extracted from the coordinates of each detection frame;

[0016] Combine all the central feature points to construct the current state vector and set the desired vector;

[0017] Calculate the difference between the current state vector and the expected vector to obtain the error vector;

[0018] By taking the time derivative of the error vector, an equivalent relationship is established between the error rate of change and the motion speed of image feature points;

[0019] Substituting the equivalent relation into the preset first-order decoupling control law, we obtain the image feature point motion speed control equation.

[0020] An image Jacobian matrix is ​​introduced, and the generalized inverse of the image Jacobian matrix is ​​solved.

[0021] The camera control speed is obtained by substituting the generalized inverse solution result and the error vector into the image feature point motion velocity control equation;

[0022] Introduce the Jacobian matrix of the robotic arm and solve for the generalized inverse of the Jacobian matrix;

[0023] The generalized inverse kinematics solution result and the camera control speed are used to convert the robot arm joint angular velocity control command and execute bolt alignment.

[0024] Jump to execute the step of using a preset target detection model to detect bolts in the RGB image and obtain the detection box coordinates of each bolt until the magnitude of the error vector is less than the preset alignment tolerance threshold, and output the converged bolt center seed point of each bolt.

[0025] Optionally, the step of acquiring a depth image of the target metal plane after bolt alignment, and combining it with point cloud extraction and plane fitting of each bolt center seed point to obtain an initial plane equation, includes:

[0026] Obtain a depth image of the target metal plane after the bolts are aligned, delineate a preset neighborhood rectangle local region centered on the seed point of each bolt, and extract the corresponding local pixel coordinate set;

[0027] Merge all the aforementioned local pixel coordinate sets to obtain the total local pixel coordinate set;

[0028] Extract the real-time depth value from the depth image, and perform pixel back projection on the total local pixel coordinate set in combination with the camera intrinsic parameter matrix to obtain a local 3D point cloud;

[0029] Fit all the local 3D point clouds and denoise them to obtain the initial plane equation.

[0030] Optionally, the step of optimizing the planar point cloud using the depth image and the initial plane equation to obtain a valid planar point cloud set includes:

[0031] Pixel back projection is performed on the depth image to obtain a global point cloud;

[0032] Based on the initial plane equation, calculate the Euclidean distance between each global point cloud in the global point cloud set and the initial plane;

[0033] Remove global point clouds whose Euclidean distance is greater than or equal to a preset distance threshold from the global point cloud set to obtain a coarsely filtered planar point cloud set;

[0034] Euclidean clustering is performed on the coarsely selected point set in the plane to obtain multiple clusters;

[0035] The average center is obtained by averaging all the seed points of the bolt centers.

[0036] Calculate the geometric center of each of the said clusters;

[0037] Calculate the target distance between each geometric center and the average center, and select the cluster corresponding to the minimum target distance as the effective point cloud set in the plane.

[0038] Optionally, the step of using the planar effective point cloud for pose estimation and constructing a planar pose homogeneous transformation matrix includes:

[0039] The point cloud is centered using the aforementioned effective point cloud set to obtain a centralized point cloud set;

[0040] Construct a covariance matrix using the centralized point set;

[0041] Perform eigenvalue decomposition on the covariance matrix to extract the initial normal vector;

[0042] The initial normal vector is corrected to obtain the plane normal vector;

[0043] The effective point cloud set in the plane is projected onto the point cloud to obtain the projected point cloud set in the plane;

[0044] Calculate the convex hull of the set of projection points on the plane, and solve for the minimum bounding rectangle of the convex hull;

[0045] Extract the direction vector corresponding to the longest side of the minimum bounding rectangle as the main direction vector;

[0046] The cross product of the principal direction vector and the plane normal vector is used to obtain the X-axis of the local plane coordinate system after normalization.

[0047] The cross product of the plane normal vector and the X-axis of the local plane coordinate system is used to obtain the Y-axis of the local plane coordinate system.

[0048] Obtain the centroid of the point cloud of the effective point cloud set in the plane, and construct the homogeneous transformation matrix of the plane pose by combining the plane normal vector, the X-axis of the plane local coordinate system and the Y-axis of the plane local coordinate system.

[0049] Optionally, the step of using the planar pose homogeneous transformation matrix to fine-tune the end effector posture of the robotic arm and outputting a vertical pose reference posture includes:

[0050] Extract the plane normal vector from the homogeneous transformation matrix of the plane pose as the unit normal vector of the target plane;

[0051] Obtain the unit direction vector along the camera's Z-axis;

[0052] The rotation axis is obtained by performing a cross product operation between the unit normal vector of the target plane and the unit direction vector of the camera's Z-axis.

[0053] The rotation axis is normalized to obtain a unit rotation axis;

[0054] Calculate the attitude deviation angle between the unit normal vector of the target plane and the unit direction vector of the camera Z-axis;

[0055] Construct the antisymmetric matrix of the unit rotation axis;

[0056] The attitude correction rotation matrix is ​​calculated using the antisymmetric matrix and the attitude deviation angle;

[0057] The attitude of the robotic arm end effector is fine-tuned according to the attitude correction rotation matrix, and a vertical attitude reference pose is output.

[0058] Optionally, driving the robotic arm to perform bolt removal operations based on the vertical attitude reference pose includes:

[0059] The robotic arm is driven to move to the safe avoidance position corresponding to the first bolt according to the preset diagonal disassembly sequence, and the first bolt is marked as the bolt to be disassembled;

[0060] The contact force and torque information corresponding to the bolt to be disassembled, which is fed back in real time by the force sensor installed at the end of the robotic arm, is obtained and converted to obtain the contact force and torque vector;

[0061] The pose correction is solved by inputting the contact force and torque vectors into the preset admittance model.

[0062] The vertical attitude reference pose and the pose correction amount are used to convert the robotic arm command pose;

[0063] According to the commanded posture of the robotic arm, drive the sleeve installed at the end of the robotic arm to fit into the bolt to be removed to perform a loosening operation;

[0064] Drive the robotic arm to move to a safe avoidance position for the next bolt to be removed, following the preset diagonal disassembly sequence;

[0065] Jump to the step of obtaining and converting the contact force and torque information corresponding to the bolt to be disassembled, which is fed back in real time by the force sensor installed at the end of the robotic arm, to obtain the contact force and torque vector, until all bolts are disassembled.

[0066] A second aspect of the present invention provides a bolt disassembly system based on visual servoing and point cloud planar pose estimation, comprising:

[0067] The bolt alignment module is used to acquire RGB images of the target metal plane through a camera mounted on the end of the robotic arm, perform bolt alignment, and output the bolt center seed point after convergence of each bolt.

[0068] The point cloud processing module is used to acquire the depth image of the target metal plane after the bolts are aligned, and to perform point cloud extraction and plane fitting by combining the seed points at the center of each bolt to obtain the initial plane equation.

[0069] The point cloud optimization module is used to optimize the planar point cloud using the depth image and the initial plane equation to obtain an effective planar point cloud set.

[0070] The pose estimation module is used to perform pose estimation using the plane effective point cloud set and construct a plane pose homogeneous transformation matrix.

[0071] The attitude fine-tuning module is used to fine-tune the attitude of the robotic arm end effector using the planar pose homogeneous transformation matrix and output the vertical attitude reference pose.

[0072] The bolt removal module is used to drive the robotic arm to perform bolt removal operations based on the vertical attitude reference pose.

[0073] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the bolt disassembly method based on visual servoing and point cloud planar pose estimation as described above.

[0074] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed, it implements the bolt disassembly method based on visual servoing and point cloud planar pose estimation as described above.

[0075] As can be seen from the above technical solutions, the present invention has the following advantages:

[0076] This invention provides a bolt dismantling method, system, device, and medium based on visual servoing and point cloud planar pose estimation. It acquires RGB images of the target metal plane using a camera mounted on the end effector of a robotic arm and performs bolt alignment, outputting the converged seed points of the bolt centers. Then, it acquires a depth image of the target metal plane after bolt alignment, and combines the seed points of each bolt center to perform point cloud extraction and initial plane fitting to obtain an initial plane equation. Subsequently, it uses the depth image and the initial plane equation to optimize the planar point cloud, obtaining a set of effective planar point clouds. Based on the effective planar point clouds, it performs pose estimation and constructs a homogeneous planar pose transformation matrix. Then, it uses this homogeneous planar pose transformation matrix to perform fine-tuning of the robotic arm's end effector posture, outputting a vertical pose reference pose. Finally, it drives the robotic arm to perform bolt dismantling operations based on the vertical pose reference pose. This invention is based on high-precision pose estimation and posture fine-tuning using a combined visual-point cloud approach. By integrating visual servo initial alignment with point cloud "local to global" processing, the bolt is first precisely initially aligned using visual servo. Then, a precise planar pose homogeneous orthogonal pose matrix is ​​constructed through point cloud extraction, fitting, and optimization, representing the true 3D pose of the target metal plane. Based on the planar pose homogeneous orthogonal pose matrix, the deviation between the tool axis of the robotic arm and the Z-axis of the target plane is accurately calculated using the Rodriguez formula, driving the robotic arm to perform attitude fine-tuning. This solves the technical problem in existing technologies where the tool axis cannot be strictly perpendicular to the target plane, effectively avoiding operational failures such as sleeve insertion jamming and thread stripping caused by non-perpendicularity. This significantly improves the accuracy and reliability of bolt disassembly operations. At the same time, the point cloud "local to global" processing method also ensures the real-time performance and robustness of the operation process, efficiently adapting to the high-precision bolt disassembly requirements of complex industrial sites. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 The flowchart illustrates the steps of a bolt disassembly method based on visual servoing and point cloud planar pose estimation, as provided in Embodiment 1 of the present invention.

[0079] Figure 2 This is a flowchart illustrating the steps of a bolt disassembly method based on visual servoing and point cloud planar pose estimation, as provided in Embodiment 2 of the present invention.

[0080] Figure 3This is a structural block diagram of a bolt disassembly system based on visual servoing and point cloud planar pose estimation provided in Embodiment 3 of the present invention;

[0081] Figure 4 This is a structural block diagram of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0082] This invention provides a bolt disassembly method, system, device, and medium based on visual servoing and point cloud planar pose estimation. It addresses the technical problems in the prior art where pure visual guidance is limited by calibration errors and depth uncertainties, making it difficult to ensure that the tightening gun axis is completely perpendicular to the target plane, which in turn leads to sleeve insertion jamming, thread stripping, and insufficient operational reliability.

[0083] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0084] Visual servoing technology has been widely adopted as a mainstream solution for autonomous guidance of robotic arm end-effectors. This method typically utilizes a camera to extract feature points of the target bolt, driving the robotic arm to move until the feature points reach the desired position on the image plane. However, existing research indicates that, limited by camera internal calibration errors, uncertainties in monocular / binocular visual depth, and the weak decoupling capability of traditional visual servoing control laws, when the visual algorithm converges in the image domain, it often cannot guarantee that the axis of the tightening gun (i.e., the camera's Z-axis) is strictly perpendicular to the target metal plane in three dimensions. In high-contact tightening operations, such minute attitude deviations are extremely critical, easily leading to jamming of the tightening gun sleeve or stripping of the threads.

[0085] To address the bottleneck of non-perpendicular orientation, the approach of using 3D point cloud scanning and global fitting of the target metal plane to correct the normal vector faces significant limitations: on the one hand, the global point cloud data is massive and computationally time-consuming, making it difficult to meet the real-time requirements of industrial sites; on the other hand, in unstructured and complex industrial sites, point clouds are easily affected by noise interference from cluttered backgrounds, irrelevant metal parts, and light reflections, resulting in extremely poor robustness of plane normal vector estimation and making the algorithm prone to failure.

[0086] Furthermore, during the physical contact engagement phase after vision or point cloud guidance (i.e., the instant the gun sleeve is tightened into the bolt), due to the unavoidable slight residual positioning errors in the robotic arm's kinematic calibration and the vision system, if the robotic arm still employs a traditional rigid position control strategy, the sleeve cannot make any compliant yielding or adaptive attitude adjustment when subjected to contact force. In this situation, the system not only fails to complete the assembly but also faces an extremely high risk of hard contact impact and the classic "jamming" and "wedging" phenomena in shaft-hole assembly.

[0087] In summary, existing methods such as single-vision guidance, global point cloud fitting, or rigid position control are insufficient to meet the requirements of high-precision and high-reliability automated bolt disassembly. The purpose of this invention is to propose a comprehensive disassembly method that combines rapid visual alignment, interference-resistant precise 3D posture correction, and flexible physical contact, thereby completely resolving the technical bottlenecks of end-effector non-parallelism and hard contact jamming.

[0088] Please see Figure 1 , Figure 1 This is a flowchart illustrating the steps of a bolt disassembly method based on visual servoing and point cloud planar pose estimation, as provided in Embodiment 1 of the present invention.

[0089] This invention provides a bolt disassembly method based on visual servoing and point cloud planar pose estimation, comprising:

[0090] Step 101: Acquire RGB images of the target metal plane using a camera mounted on the end of the robotic arm, perform bolt alignment, and output the bolt center seed point after each bolt converges.

[0091] In this embodiment of the invention, the camera adopts an integrated configuration of "eye on hand," fixed to the end effector of the robotic arm, and pre-calibrated in three dimensions coaxially with the end tightening tool. After acquiring the RGB image of the target metal plane, the center pixel coordinates of each bolt in the image are extracted by the target detection algorithm. Based on the visual servo control strategy, the end effector of the robotic arm is driven to move until the center pixel coordinates of each bolt converge to the desired working position preset in the image, thus completing the initial alignment of the bolts. After the visual servo convergence stabilizes, the center pixel coordinates of each bolt in the image coordinate system are output as the center seed point of the corresponding bolt.

[0092] Step 102: Obtain the depth image of the target metal plane after the bolts are aligned, and combine it with the seed points at the center of each bolt to perform point cloud extraction and plane fitting to obtain the initial plane equation.

[0093] In this embodiment of the invention, after the bolts are aligned, a camera mounted on the end of the robotic arm acquires a depth image of the target metal plane after bolt alignment, generating point cloud data corresponding to the depth image. Using the output seed points at the center of each bolt as spatial constraints, a point cloud extraction operation is performed on the point cloud data to extract local point cloud data around each bolt area on the target metal plane, while removing irrelevant background clutter points and light reflection interference points. A plane fitting operation is performed on the extracted local point cloud data, and the least squares plane parameters are iteratively solved using a random sampling consensus algorithm to finally obtain the initial plane equation representing the spatial pose of the target metal plane.

[0094] Step 103: Optimize the planar point cloud using the depth image and the initial plane equation to obtain the effective point cloud set of the plane.

[0095] In this embodiment of the invention, the depth point cloud corresponding to the acquired depth image of the target metal plane is filtered point by point, the distance from each point to the initial plane is calculated, and outlier noise points and background interference points whose distance exceeds the threshold are removed; at the same time, effective point cloud data that are located in the neighborhood of the initial plane and conform to the planar distribution characteristics are retained. After denoising and filtering, a set of effective point clouds representing the true shape of the target metal plane is obtained.

[0096] Step 104: Use the effective point cloud set of the plane to perform pose estimation and construct the homogeneous transformation matrix of the planar pose.

[0097] In this embodiment of the invention, a three-dimensional spatial pose calculation is performed on the effective point set of the plane to accurately fit the actual spatial position and normal attitude information of the target metal plane and establish the plane reference coordinate relationship. Then, combined with the calibration correlation parameters of the camera coordinate system and the end-effector coordinate system, the translation and rotation attitude of the plane are integrated, and the parameters are encapsulated according to the homogeneous coordinate transformation rules to finally generate a planar pose homogeneous transformation matrix that can represent the true three-dimensional pose of the target plane.

[0098] Step 105: Use a planar pose homogeneous transformation matrix to fine-tune the end effector posture of the robotic arm and output the vertical pose reference posture.

[0099] In this embodiment of the invention, the normal attitude parameters of the target metal plane are extracted from the homogeneous transformation matrix of the planar pose and correlated with the spatial pose of the coordinate system of the robotic arm end effector. Based on the extracted normal attitude parameters, the Rodriguez formula is used to accurately calculate the Z-axis attitude deviation between the axis of the tightening tool at the end effector of the robotic arm and the normal of the target plane. Then, the corresponding attitude fine-tuning command is generated to drive the end effector of the robotic arm to perform attitude adjustment operation. After the attitude is adjusted to the point that the Z-axis deviation is eliminated and the tool axis is completely perpendicular to the target plane, the vertical attitude reference pose representing the precise attitude is output.

[0100] Step 106: Based on the vertical attitude reference pose, drive the robotic arm to perform bolt removal operation.

[0101] In this embodiment of the invention, using the vertical attitude reference pose as a reference, the robotic arm end effector is driven to move to the safe avoidance position corresponding to the first bolt according to a preset diagonal disassembly sequence, and the first bolt is marked as the bolt to be disassembled. Subsequently, the contact force and torque information corresponding to the bolt to be disassembled, which is fed back in real time by the force sensor installed at the end effector of the robotic arm, is acquired and subjected to coordinate transformation to obtain a standardized contact force and torque vector. The contact force and torque vector is input into a preset admittance model to solve for the pose correction amount of the robotic arm end effector. The vertical attitude reference pose and the pose correction amount are fused and transformed to generate a command pose that the robotic arm can recognize and execute. According to the command pose of the robotic arm, the robotic arm end effector is driven to move so that the sleeve installed at the end effector of the robotic arm is accurately fitted into the bolt to be disassembled, and the electric tightening gun is controlled to start the loosening operation. After the loosening is completed, the robotic arm is driven to move to the safe avoidance position of the next bolt to be disassembled according to the preset diagonal disassembly sequence. Then, the step of acquiring and transforming the contact force and torque information fed back in real time by the force sensor is executed, and the process is repeated until all bolts are disassembled.

[0102] In this invention, a camera mounted on the end effector of a robotic arm acquires RGB images of the target metal plane and performs bolt alignment, outputting the converged seed points of the bolt centers. Then, a depth image of the target metal plane after bolt alignment is obtained. Combined with the seed points of each bolt center, point cloud extraction and initial plane fitting are performed to obtain an initial plane equation. Subsequently, the depth image and the initial plane equation are used to optimize the planar point cloud to obtain a set of effective planar points. Based on this set of effective planar points, pose estimation is performed, and a homogeneous planar pose transformation matrix is ​​constructed. This homogeneous planar pose transformation matrix is ​​then used to fine-tune the posture of the robotic arm's end effector, outputting a vertical pose reference pose. Finally, based on the vertical pose reference pose, the robotic arm is driven to perform bolt disassembly. This invention is based on visual servo initial alignment and integrates a point cloud "local to global" processing flow to construct an orthogonal pose matrix. This addresses the limitations of existing pure visual guidance, which is constrained by camera internal calibration errors and monocular / binocular visual depth uncertainty, only achieving feature convergence within the image domain but lacking... To address the core technical deficiency of ensuring strict three-dimensional perpendicularity between the tightening gun axis and the target metal plane, this method utilizes visual servoing to perform initial bolt alignment followed by the introduction of depth point clouds for planar pose estimation. This eliminates the reliance on depth perception and calibration accuracy in pure visual guidance. By employing a technique of "local seed point guidance extraction - initial plane fitting - point cloud optimization," a high-precision effective planar point cloud set is obtained, significantly improving the accuracy and robustness of planar normal vector estimation. Furthermore, a homogeneous planar pose transformation matrix is ​​constructed to accurately represent the three-dimensional pose of the target plane. Combined with end-effector attitude fine-tuning, this achieves complete perpendicularity between the tightening gun axis and the target plane, thereby eliminating operational failures such as sleeve insertion jamming and thread stripping caused by non-perpendicular axis. Simultaneously, the local point cloud processing method avoids the drawbacks of massive global point cloud data and time-consuming computation, balancing real-time performance and accuracy. This effectively improves the reliability and stability of automated bolt disassembly operations and can adapt to the high-precision and high-reliability operational requirements of complex industrial sites.

[0103] Camera mounted at the end effector of the robotic arm: An RGB-D camera installed on the end effector of the robotic arm, capable of simultaneously acquiring RGB and depth images; Target metal plane: The metal workpiece plane on which the bolts to be disassembled are mounted; RGB image: A three-channel color two-dimensional image acquired by the camera, used for bolt detection and visual servoing; Bolt alignment: A precise alignment process based on visual servoing driving the movement of the end effector camera to bring the bolt center feature point close to the desired position; Bolt center seed point: The coordinates of the center feature point of each bolt in the image pixel plane after visual servoing convergence; Depth image: A two-dimensional image acquired by the camera containing pixel depth information, used for three-dimensional point cloud reconstruction; Point cloud extraction and plane fitting: The process of extracting local three-dimensional point clouds based on the bolt center seed point and fitting plane mathematical equations; Initial plane equation: The initial plane equation obtained by fitting the local three-dimensional point cloud. Initial mathematical expression for the target metal plane; Planar point cloud optimization: the process of filtering and clustering the global point cloud to remove interference and extract effective point clouds based on the initial plane equation; Planar effective point cloud set: the optimized set of three-dimensional point clouds containing only the target metal plane; Pose estimation: the process of calculating the position and orientation of the target metal plane in the camera coordinate system based on the planar effective point cloud; Planar pose homogeneous transformation matrix: a 4×4 homogeneous transformation matrix representing the transformation from the camera coordinate system to the local planar coordinate system; Attitude fine-tuning: the process of adjusting the attitude of the robotic arm end effector based on the planar pose matrix to make the tool axis perpendicular to the plane; Vertical attitude reference pose: the reference pose of the robotic arm end effector after attitude fine-tuning, at which point the sleeve axis is strictly perpendicular to the target metal plane; Bolt removal operation: the process of the robotic arm driving the sleeve to fit the bolt and complete the loosening operation based on the vertical attitude reference pose.

[0104] It should be noted that the robotic arm end effector is equipped with a mounting flange, on which a six-dimensional force sensor, an electric tightening gun, and a disassembly sleeve are fixedly connected sequentially from top to bottom. The industrial camera adopts an integrated "eye-on-hand" configuration, fixedly mounted on the periphery of the robotic arm end effector and moving synchronously with the end effector as a whole. The camera optical axis, the output spindle of the electric tightening gun, and the central axis of the sleeve are pre-calibrated in three dimensions at the end effector, ensuring that the axes of the three remain strictly collinear and coincident throughout the entire process. The entire system of camera, tightening gun, sleeve, and force sensor is rigidly integrated into the robotic arm end effector to form an integrated disassembly module; among which, the six-dimensional force sensor... The device collects three-dimensional force and torque data in real time during the contact process between the sleeve and the bolt, providing feedback for admittance compliance control (i.e. bolt disassembly operation). The electric tightening gun relies on the output spindle to transmit torque to drive the sleeve to loosen the bolt. The pre-completed coaxial calibration ensures that the plane normal vector obtained by the camera vision calculation and the vertical attitude reference position output after the robot arm attitude fine adjustment can directly match the working axis of the sleeve. This allows the camera attitude correction result to be accurately equivalent to the sleeve attitude adjustment, structurally ensuring that the sleeve can be vertically aligned with the bolt to complete the fitting and disassembly operation without offset or jamming.

[0105] Please see Figure 2 , Figure 2 This is a flowchart illustrating the steps of a bolt disassembly method based on visual servoing and point cloud planar pose estimation, as provided in Embodiment 2 of the present invention.

[0106] This invention provides a bolt disassembly method based on visual servoing and point cloud planar pose estimation, comprising:

[0107] Step 201: Acquire RGB images of the target metal plane using a camera mounted on the end of the robotic arm, perform bolt alignment, and output the bolt center seed point after each bolt converges.

[0108] Further, step 201 may include the following sub-steps:

[0109] S11. Acquire RGB images of the target metal plane using a camera mounted on the end of the robotic arm.

[0110] In this embodiment of the invention, an RGB-D camera installed at the end of a robotic arm is used to simultaneously acquire RGB color images and depth images of the target metal plane. The RGB images are used for subsequent bolt target detection and visual servo control, while the depth images are used to assist in the calibration of the camera pose and spatial scale. It should be noted that in step 201, the core control logic is implemented based on the RGB images.

[0111] S12. Use a preset target detection model to detect bolts in the RGB image and obtain the detection box coordinates of each bolt.

[0112] In this embodiment of the invention, the acquired RGB image is input into a pre-trained YOLO target detection model to automatically detect and locate four target bolts. Let the set of detection box coordinates for the i-th bolt (i∈{1,2,3,4}, corresponding to the four bolts to be aligned) output by the YOLO model be:

[0113]

[0114] In the formula, Bolt serial number Let be the horizontal pixel coordinates of the top-left corner of the i-th bolt detection box. Let be the vertical pixel coordinates of the top-left corner of the i-th bolt detection box. Let be the horizontal pixel coordinates of the bottom right corner of the i-th bolt detection frame. Let be the vertical pixel coordinates of the bottom right corner of the i-th bolt detection box. Let be the coordinates of the detection frame for the i-th bolt.

[0115] S13. Extract the center feature points of each bolt from the coordinates of each detection frame.

[0116] In this embodiment of the invention, the center feature point of each bolt (i.e., the coordinates of the bolt center in the image pixel plane) is calculated based on the detection box coordinates. The calculation formula is as follows:

[0117]

[0118] In the formula, Let be the horizontal pixel coordinates of the center feature point of the i-th bolt. Let be the vertical pixel coordinates of the center feature point of the i-th bolt.

[0119] S14. Combine all central feature points to construct the current state vector and set the expected vector.

[0120] In this embodiment of the invention, the coordinates of the current center feature points of the four bolts are combined into a visual servo current state vector, and the desired state vector corresponding to the desired feature point set is manually set, with the following expressions:

[0121]

[0122]

[0123] In the formula, This is the current state vector of the bolt's center feature point. Let these be the coordinates of the current center feature point of the i-th bolt. Let be the expected vector of the bolt's center feature point.

[0124] S15. Calculate the difference between the current state vector and the expected vector to obtain the error vector.

[0125] In this embodiment of the invention, the core error function of the visual servoing system is the difference between the current state vector and the desired state vector, expressed as:

[0126]

[0127] In the formula, The error vector for bolt alignment represents the visual servoing error vector.

[0128] S16. Take the time derivative of the error vector and establish an equivalent relationship between the rate of change of error and the motion speed of image feature points.

[0129] In this embodiment of the invention, due to the desired state vector Assuming the error vector is a constant (does not change with time), taking the time derivative of the error vector yields an equivalent relationship between the rate of change of the error and the velocity of the image feature points:

[0130]

[0131] In the formula, The time rate of change of the error vector. This represents the motion velocity of image feature points.

[0132] S17. Substitute the equivalent relation into the preset first-order decoupling control law to obtain the image feature point motion velocity control equation.

[0133] In this embodiment of the invention, to achieve stable convergence of the visual servoing system, a first-order decoupling control law is preset, with the expression as follows:

[0134]

[0135] In the formula, is the gain coefficient of the first-order decoupling control law.

[0136] Combining the equivalent relation obtained in S16, the governing equation for the motion velocity of image feature points can be derived:

[0137]

[0138] S18. Introduce the image Jacobian matrix and solve for the generalized inverse of the image Jacobian matrix.

[0139] In this embodiment of the invention, there is a mapping relationship between the motion velocity of image feature points and the six-degree-of-freedom motion velocity of the end-effector camera in three-dimensional space, expressed as:

[0140]

[0141] right Find the generalized inverse of Moore-Penrose. .

[0142] In the formula, The velocity of the image feature points. Let Jacobian matrix be the image. To control the speed of the camera, Let be the generalized inverse of the image Jacobian matrix.

[0143] S19. The generalized inverse solution result and the error vector are substituted into the image feature point motion velocity control equation to obtain the camera control velocity.

[0144] In this embodiment of the invention, by combining the control equation of S17 and the mapping relationship of S18, and substituting the generalized inverse matrix, the motion control velocity law of the robotic arm's end-effector camera can be derived:

[0145]

[0146] S110. Introduce the Jacobian matrix of the robotic arm and solve the generalized inverse of the Jacobian matrix of the robotic arm.

[0147] In this embodiment of the invention, to convert camera control speed into joint control commands executable by the robotic arm, a robotic arm Jacobian matrix is ​​introduced to establish a mapping relationship between joint space motion and camera task space motion:

[0148]

[0149] Jacobian matrix for robotic arm By performing the Moore-Penrose generalized inverse, the pseudo-inverse matrix is ​​obtained. , used for solving joint velocities;

[0150] In the formula, For the joint angle of the robotic arm, This refers to the angular velocity of the robotic arm joints.

[0151] S111. The generalized inverse solution result and the camera control speed are converted into joint angular velocity control commands for the robotic arm and bolt alignment is executed.

[0152] In this embodiment of the invention, by combining the camera control speed obtained in S19 and the pseudo-inverse of the robotic arm Jacobian obtained in S110, the angular velocity control commands for each joint of the robotic arm can be obtained:

[0153]

[0154] The joint angular velocity command is sent to the bottom controller of the robotic arm to drive the movement of the end-effector camera, thereby achieving bolt alignment control.

[0155] S112, Jump to execute the step of using a preset target detection model to detect bolts in the RGB image and obtain the detection box coordinates of each bolt until the magnitude of the error vector is less than the preset alignment tolerance threshold, and output the bolt center seed point after convergence for each bolt.

[0156] In this embodiment of the invention, the control system iteratively executes the entire process of S12~S111 with a fixed control cycle, continuously updating the bolt feature points, errors, and control quantities, so that the current bolt center feature point continuously approaches the desired position. An alignment tolerance threshold is set, and when the following convergence conditions are met, the vision servo system is determined to have converged, and the robotic arm is controlled to stop moving:

[0157]

[0158] In the formula, Let be the magnitude of the error vector. This is a preset alignment tolerance threshold.

[0159] After the system converges, it outputs the coordinates of the center feature points of the current four bolts, i.e. the bolt center seed points, completing the initial alignment of the bolts on the target metal plane.

[0160] Pre-set target detection model: A pre-trained YOLO-based target detection model for automatic bolt detection and localization; Detection box coordinates: Pixel coordinates of the bolt localization rectangle output by the target detection model; Center feature point: Pixel coordinates of the bolt center point calculated from the bolt detection box coordinates; Current state vector: Visual servo state vector composed of the coordinates of all current center feature points of the bolts; Expected vector: Visual servo state vector corresponding to the target position after bolt alignment; Error vector: The difference between the current state vector and the expected vector, representing the bolt position deviation; Error change rate: The derivative of the error vector with respect to time, representing the dynamic trend of error change; Image feature point motion speed: The motion speed of the bolt center feature point in the image pixel plane; Equivalent relation: Error change rate and image feature point motion speed Equivalent relationship; First-order decoupling control law: control law used for visual servoing stable convergence; Image feature point motion velocity control equation: feature point velocity solution equation obtained by substituting the equivalent relationship and control law; Image Jacobian matrix: matrix representing the mapping relationship between image feature point motion velocity and camera 3D motion velocity; Generalized inverse: Moore-Penrose pseudo-inverse method used for solving non-square / singular matrices; Camera control velocity: camera six-degree-of-freedom 3D motion velocity obtained by visual servoing solution; Robotic arm Jacobian matrix: matrix representing the mapping relationship between robotic arm joint motion and end-effector task space motion; Robotic arm joint angular velocity control command: robotic arm executable joint angular velocity command converted from camera control velocity; Preset alignment tolerance threshold: preset threshold for visual servoing convergence determination.

[0161] Step 202: Obtain the depth image of the target metal plane after the bolts are aligned. Delineate a local area of ​​a preset neighborhood rectangle centered on the seed point of each bolt and extract the corresponding local pixel coordinate set.

[0162] In this embodiment of the invention, the current frame depth image after visual servo convergence is first obtained, and the four bolt center seed points output in step 201 are used as reference points for local region extraction. The center seed point of the i-th bolt is denoted as:

[0163]

[0164] In the formula, Let the pixel coordinates be the center seed point of the i-th bolt; Let be the horizontal pixel coordinate of the seed point at the center of the i-th bolt; Let be the vertical pixel coordinates of the seed point at the center of the i-th bolt.

[0165] Using the seed point at the center of each bolt as the center, a predefined neighborhood rectangle (ROI region) is defined, and the set of pixel coordinates in this region satisfies:

[0166]

[0167] In the formula, This is the set of local pixel coordinates (ROI region pixel set) corresponding to the i-th bolt. The preset ROI region width; The height of the preset ROI region; The horizontal pixel coordinates of the pixel; These are the vertical pixel coordinates of the pixel.

[0168] Using the above method, the local pixel coordinate sets corresponding to the four bolts are extracted respectively. This completes the delineation of local regions and pixel extraction.

[0169] Step 203: Merge all local pixel coordinate sets to obtain the total local pixel coordinate set.

[0170] In this embodiment of the invention, the local pixel coordinate sets corresponding to the four bolts extracted in step 202 are merged, and duplicate pixels are removed to obtain a total local pixel coordinate set, which is used to characterize the effective pixel area around the bolts on the target metal plane, providing pixel range constraints for subsequent three-dimensional point cloud extraction and avoiding the extraction of pixels in irrelevant background areas.

[0171] Step 204: Extract the real-time depth value from the depth image and perform pixel back projection on the total local pixel coordinate set in combination with the camera intrinsic parameter matrix to obtain the local 3D point cloud.

[0172] In this embodiment of the invention, each pixel in the total local pixel coordinate set is first traversed. Extract the real-time depth value corresponding to the pixel from the depth image. (i.e., the point in the camera coordinate system) Towards depth).

[0173] pixels Convert to homogeneous coordinate form:

[0174]

[0175] In the formula, 1 represents the homogeneous coordinates of the pixel; 1 represents the homogeneity term of the homogeneous coordinates.

[0176] Based on the camera pinhole projection model, back projection calculation is performed on the pixels to obtain the 3D point coordinates in the camera coordinate system. The calculation formula is as follows:

[0177]

[0178] In the formula, These are the 3D point coordinates of the pixel in the camera coordinate system. This is the real-time depth value of the pixel in the depth image; This is the camera intrinsic parameter matrix; These are the homogeneous coordinates of the pixel.

[0179] The standard form of the camera intrinsic parameter matrix K is:

[0180]

[0181] In the formula, , These are the camera's focal lengths in the horizontal and vertical directions, respectively. , These are the horizontal and vertical pixel coordinates of the camera's optical center in the image plane, respectively.

[0182] By traversing all pixels in the local pixel coordinate set, the back projection calculation of all pixels is completed, resulting in a local 3D point cloud set around the target metal planar bolt. ,in Let j be the camera coordinates of the j-th 3D point. This represents the total number of points in the local 3D point cloud.

[0183] Step 205: Fit all local 3D point clouds and denoise them to obtain the initial plane equation.

[0184] In this embodiment of the invention, the RANSAC (Random Sample Consensus) algorithm is used to perform planar fitting and noise point removal on the local 3D point cloud obtained in step 204. The specific implementation process is as follows:

[0185] Three non-collinear points are randomly selected from a local 3D point cloud to construct the initial plane of the target metal plane;

[0186] Calculate the Euclidean distance from all 3D points to the initial plane, and mark points with a distance less than a preset distance threshold as interior points (valid plane points), and points with a distance greater than the threshold as exterior points (noise points).

[0187] Repeat the above "sampling-fitting-interior point statistics" process until the preset number of iterations, and select the initial plane with the most interior points as the optimal initial plane for the target metal plane;

[0188] By performing least-squares optimization on the optimal initial plane using all interior points, the final initial plane equation is obtained, expressed as:

[0189]

[0190] In the formula, The initial plane equation corresponding to the target metal plane; The normal vector component of the plane (i.e., the initial Z-axis direction); This is the intercept term of the initial plane; These are the spatial coordinates of a three-dimensional point.

[0191] After noise point removal and plane fitting are performed using the RANSAC algorithm, the initial plane equation of the target metal plane is obtained, and the initial fitting of the metal plane is completed, providing a plane normal reference for subsequent robot arm pose adjustment.

[0192] Initial Plane: The preliminary fitted plane obtained by fitting the local point cloud extracted from the target metal plane. The initial plane is not an independent new plane, but essentially the preliminary spatial fitting shape of the target metal plane. It is a preliminary representation plane obtained by fitting the image and point cloud data collected from the target metal plane. The mathematical expression corresponding to this plane is the initial plane equation. Preset Neighborhood Rectangular Local Region: A rectangular region of interest defined with the seed point of the bolt center as the center. Local Pixel Coordinate Set: The set of pixel coordinates within a single region of interest. Total Local Pixel Coordinate Set: The merged set of local pixel coordinate sets corresponding to all bolts. Real-time Depth Value: The distance value from the camera to the target corresponding to the pixel in the depth image. Camera Intrinsic Matrix: A 3×3 matrix representing the camera's optical parameters. Pixel Back Projection: The process of converting two-dimensional pixels and depth values ​​into three-dimensional coordinates based on the camera pinhole model. Local 3D Point Cloud: The set of 3D point clouds around the bolt obtained by back projecting the total local pixels. Plane Fitting: The process of solving the plane mathematical equation for the point cloud using the RANSAC algorithm. Denoising: The operation of removing noise points and outliers from the point cloud using the RANSAC algorithm.

[0193] Step 206: Optimize the planar point cloud using the depth image and the initial plane equation to obtain the effective point cloud set of the plane.

[0194] Furthermore, step 206 may include the following sub-steps:

[0195] S21. Perform pixel back projection on the depth image to obtain the global point cloud.

[0196] In this embodiment of the invention, this step is the same as the pixel back projection in step 204, and will not be described again here.

[0197] S22. Based on the initial plane equation, calculate the Euclidean distance between each global point cloud in the global point cloud set and the initial plane.

[0198] In this embodiment of the invention, based on the initial plane equation obtained in step 205, the Euclidean distance from each global point cloud in the global point cloud set to the initial plane is calculated. The initial plane equation is:

[0199]

[0200] For any global point cloud in the global point cloud set The formula for calculating its Euclidean distance to the initial plane is:

[0201]

[0202] In the formula, Let be the Euclidean distance from the i-th global point cloud to the initial plane; The coefficients of the initial plane equation; Let be the three-dimensional coordinates of the i-th global point cloud.

[0203] S23. Remove global point clouds whose Euclidean distance is greater than or equal to a preset distance threshold from the global point cloud set to obtain a coarsely filtered planar point cloud set.

[0204] In this embodiment of the invention, a preset distance threshold is set, and only global point clouds with Euclidean distances less than the preset distance threshold are retained, while noise points and non-planar points with distances greater than or equal to the threshold are removed, resulting in a coarsely filtered planar point cloud set, which is defined as follows:

[0205]

[0206] In the formula, For coarse screening of points in a plane; Let i be the i-th global point cloud in the global point cloud set; A preset distance threshold is used to filter valid point clouds belonging to the initial plane.

[0207] S24. Perform Euclidean clustering on the coarsely selected point cloud in the plane to obtain multiple clusters.

[0208] In this embodiment of the invention, Euclidean clustering is performed on the coarsely selected point cloud set to remove background point clouds and irrelevant object interference. For any two global point clouds in the set... and If the Euclidean distance between two points satisfies the preset clustering distance threshold, then the two points are assigned to the same cluster. The Euclidean distance calculation formula is:

[0209]

[0210] In the formula, For point With point The Euclidean distance between them; , These are the three-dimensional coordinates of two global point clouds to be calculated.

[0211] when At that time, and They are grouped into the same cluster, among which This is a preset clustering distance threshold.

[0212] After traversing all point clouds, multiple clusters are obtained. The expression for the cluster set is:

[0213]

[0214] In the formula, The set of clusters obtained by Euclidean clustering; This represents the k-th cluster. This represents the total number of clusters.

[0215] S25. The average center is obtained by averaging all the seed points of the bolt centers.

[0216] In this embodiment of the invention, the three-dimensional coordinates of the four bolt center seed points output in step 201 are extracted, averaged, and the average center is obtained. The calculation formula is as follows:

[0217]

[0218] In the formula, The average center (target reference center) of all bolt center seed points.

[0219] S26. Calculate the geometric center of each cluster.

[0220] In this embodiment of the invention, for each cluster Calculate its geometric center (i.e., the average of the three-dimensional coordinates of all point clouds within the cluster), using the following formula:

[0221]

[0222] In the formula, Let be the geometric center of the k-th cluster; The number of global point clouds contained in the k-th cluster; The coordinates of the j-th global point cloud within the k-th cluster are given.

[0223] S27. Calculate the target distance between each geometric center and the average center, and select the cluster corresponding to the minimum target distance as the effective point cloud set in the plane.

[0224] In this embodiment of the invention, the target distance (i.e., Euclidean distance) between the geometric center and the average center of each cluster is calculated. By minimizing the target distance, the cluster with the smallest target distance is selected as the effective point cloud in the plane. The selection criterion is as follows:

[0225]

[0226] In the formula, For a set of valid points in a plane; The independent variable corresponding to the minimum value (i.e., the cluster with the smallest target distance).

[0227] By minimizing the spatial distance between the cluster center and the seed point through the objective function, the background point cloud and irrelevant object interference are eliminated, and only the effective point cloud corresponding to the target metal plane is retained, thus completing the planar point cloud optimization.

[0228] Global point cloud set: A set of 3D point clouds of the entire scene obtained by back-projecting all pixels of the complete depth image; Euclidean distance: The straight-line distance between a point and a plane, and between points in 3D space; Preset distance threshold: A preset distance threshold for filtering planar point clouds; Planar coarse-filtered point cloud set: A set of point clouds after initial planar distance filtering; Euclidean clustering: A point cloud clustering algorithm based on Euclidean distance; Cluster: A set of point clouds with similar spatial positions after Euclidean clustering; Average center: The average of the 3D coordinates of all bolt center seed points; Geometric center: The average of the 3D coordinates of all point clouds within a single cluster; Target distance: The Euclidean distance between the geometric center and the average center of the cluster.

[0229] Step 207: Use the effective point cloud set in the plane to perform pose estimation and construct the homogeneous transformation matrix of the planar pose.

[0230] Furthermore, step 207 may include the following sub-steps:

[0231] S31. Point cloud centering is performed using a planar effective point cloud set to obtain a centralized point cloud set.

[0232] In this embodiment of the invention, the effective point cloud of the plane output in step 206 is first obtained. This point cloud contains N three-dimensional points, and the three-dimensional coordinates of the i-th point are denoted as . .

[0233] First, calculate the geometric center (centroid of the point cloud) of the effective point cloud set in the plane. The calculation formula is as follows:

[0234]

[0235] In the formula, It is the geometric center (centroid of the point cloud) of the effective point cloud set in the plane. This represents the total number of valid points in the plane.

[0236] Based on the geometric center, the effective point cloud of the plane is centralized to obtain a centralized point cloud, where the calculation formula for each centralized point is:

[0237]

[0238] In the formula, Let be the centered 3D coordinates of the i-th point.

[0239] S32. Construct a covariance matrix using a centralized point cloud.

[0240] In this embodiment of the invention, a covariance matrix is ​​constructed based on a centralized point cloud for principal component analysis (PCA). The formula for calculating the covariance matrix is ​​as follows:

[0241]

[0242] In the formula, Let be the covariance matrix of the effective point set in the plane, with a dimension of 3×3.

[0243] S33. Perform eigenvalue decomposition on the covariance matrix and extract the initial normal vector.

[0244] In this embodiment of the invention, the covariance matrix is ​​decomposed into eigenvalues ​​to obtain three eigenvalues ​​and their corresponding eigenvectors. The eigenvalue decomposition formula is as follows:

[0245]

[0246] In the formula, Let j be the j-th eigenvalue of the covariance matrix; Eigenvalues The corresponding eigenvector; j is the eigenvalue index, which takes the value {1,2,3}.

[0247] Since the discreteness of a planar point cloud is minimal along the plane normal, corresponding to the minimum eigenvalue of its covariance matrix, the minimum eigenvalue is extracted. Corresponding feature vector , as the initial normal vector of the target plane, is the smallest eigenvalue of the covariance matrix; This is the initial normal vector corresponding to the smallest eigenvalue.

[0248] S34. Correct the initial normal vector to obtain the plane normal vector.

[0249] In this embodiment of the invention, to ensure that the plane normal vector points in the opposite direction to the camera center (i.e., away from the camera and pointing towards the target metal plane), which conforms to the control logic of the robotic arm's end-effector advancing along the positive Z-axis, the initial normal vector is corrected for direction consistency using the geometric center of the point cloud (the vector pointing from the camera origin to the plane). The correction formula is as follows:

[0250]

[0251] In the formula, This is the corrected plane normal vector (Z-axis of the local plane coordinate system); It is the dot product of the initial normal vector and the geometric center, used to determine the consistency of direction.

[0252] The corrected Z is the plane normal vector of the target metal plane, and its direction conforms to the control logic of the robotic arm.

[0253] S35. Project the effective point cloud set of the plane into a point cloud to obtain the projected point cloud set of the plane.

[0254] In this embodiment of the invention, the modified plane normal vector is used as the normal direction, and all three-dimensional points in the effective point cloud of the plane are projected onto the target metal plane perpendicular to the Z-axis to obtain the plane projection point cloud.

[0255] S36. Calculate the convex hull of the set of projection points on the plane, and solve for the minimum bounding rectangle of the convex hull.

[0256] In this embodiment of the invention, firstly, a two-dimensional convex hull calculation is performed on the cloud of planar projection points to extract the convex hull profile of the planar projection point set, which is used to characterize the outer boundary of the target metal plane; then, the minimum bounding rectangle is solved based on the convex hull profile. The minimum bounding rectangle is the rectangle with the smallest area that can completely enclose the convex hull profile, which is used to determine the main direction of the plane. This is a conventional convex hull calculation, which will not be described in detail here.

[0257] S37. Extract the direction vector corresponding to the longest side of the smallest bounding rectangle as the main direction vector.

[0258] In this embodiment of the invention, the direction vector corresponding to the longest side of the smallest bounding rectangle is extracted, and after normalization, it is used as the initial principal direction vector of the target plane.

[0259] S38. The cross product of the principal direction vector and the plane normal vector is used to obtain the X-axis of the local plane coordinate system after normalization.

[0260] In this embodiment of the invention, to construct a right-handed orthogonal coordinate system, the cross product of the principal direction vector and the plane normal vector Z is performed, and after normalization, the X-axis of the local planar coordinate system is obtained. The calculation formula is as follows:

[0261]

[0262] In the formula, is the unit vector of the X-axis in the local planar coordinate system; It is the unit principal direction vector corresponding to the longest side of the smallest bounding rectangle.

[0263] S39. The cross product of the plane normal vector and the X-axis of the local plane coordinate system is used to obtain the Y-axis of the local plane coordinate system.

[0264] In this embodiment of the invention, the Y-axis of the local plane coordinate system is obtained by performing a cross product operation between the plane normal vector and the X-axis of the local plane coordinate system, ensuring that the coordinate system satisfies the right-hand orthogonality. The calculation formula is as follows:

[0265]

[0266] In the formula, y is the unit vector of the Y-axis in the local planar coordinate system.

[0267] at this time, A right-handed orthogonal local coordinate system constitutes the target metal plane, wherein... It is a plane normal vector. , It is an orthogonal direction vector in the plane.

[0268] S310. Obtain the centroid of the point cloud of the effective point cloud set in the plane, and construct the homogeneous transformation matrix of the plane pose by combining the plane normal vector, the X-axis of the plane local coordinate system and the Y-axis of the plane local coordinate system.

[0269] In this embodiment of the invention, the geometric center (centroid of the point cloud) of the effective point cloud set in the plane is used as the origin of the local plane coordinate system, combined with the local plane coordinate system. Using the unit vectors of the axes, construct a homogeneous planar pose transformation matrix to characterize the pose transformation from the camera coordinate system to the local planar coordinate system. The matrix expression is:

[0270]

[0271] In the formula, This is a planar pose homogeneous transformation matrix with a dimension of 4×4; 0 represents the zero vector, and 1 represents the homogeneous term.

[0272] This homogeneous transformation matrix fully characterizes the pose of the target metal plane in the camera coordinate system, and can be used for subsequent pose adjustment and operation control of the robotic arm.

[0273] Point cloud centering: a preprocessing step of subtracting the centroid of the point cloud from the effective point cloud coordinates in a plane; Centered point cloud set: the set of point clouds after centering; Covariance matrix: a 3×3 discreteness representation matrix constructed based on the centered point cloud; Eigenvalue decomposition: the process of solving for eigenvalues ​​and eigenvectors of the covariance matrix; Initial normal vector: the eigenvector corresponding to the smallest eigenvalue of the covariance matrix; Plane normal vector: the unit normal vector of the target plane after direction correction, which is the Z-axis of the local coordinate system of the plane; Point cloud projection: the operation of projecting the effective point cloud in a plane onto the target plane; Plane projected point cloud set: the set of point clouds located in the target plane after projection. A set of two-dimensional point clouds in the plane; convex hull: the smallest convex polygon boundary of the projected point cloud; minimum bounding rectangle: the rectangle with the smallest area enclosing the convex hull; principal direction vector: the unit direction vector corresponding to the longest side of the minimum bounding rectangle; cross product operation: the cross product operation of three-dimensional vectors, used to construct an orthogonal coordinate system; local plane coordinate system X-axis: the normalized unit vector obtained by the cross product of the principal direction vector and the plane normal vector; local plane coordinate system Y-axis: the unit vector obtained by the cross product of the plane normal vector and the X-axis; centroid of the point cloud: the average of the three-dimensional coordinates of all points in the effective point cloud of the plane, which is the origin of the local plane coordinate system.

[0274] Step 208: Use a planar pose homogeneous transformation matrix to fine-tune the end effector posture of the robotic arm and output the vertical pose reference posture.

[0275] Furthermore, step 208 may include the following sub-steps:

[0276] S41. Extract the plane normal vector from the homogeneous transformation matrix of the plane pose as the unit normal vector of the target plane.

[0277] In this embodiment of the invention, the plane normal vector in the homogeneous planar pose transformation matrix constructed in step 207 is extracted as the target plane unit normal vector. .

[0278] S42. Obtain the unit direction vector of the camera's Z-axis.

[0279] In this embodiment of the invention, the unit direction vector of the Z-axis in the current coordinate system of the camera is obtained. This vector corresponds to the axial direction of the tightening gun at the end of the robotic arm and is a unit vector, expressed as:

[0280]

[0281] In the formula, The unit direction vector of the Z-axis in the current coordinate system of the camera (direction of the tightening gun axis). These are the unit vectors of the camera's Z-axis in the camera coordinate system. Quantity.

[0282] S43. The rotation axis is obtained by performing a cross product operation between the target plane unit normal vector and the camera Z-axis unit direction vector.

[0283] In this embodiment of the invention, to determine the rotation reference between the camera's Z-axis and the target plane's Z-axis, the cross product of the camera's Z-axis unit direction vector and the target plane's unit normal vector is performed to obtain the rotation axis. The calculation formula is as follows:

[0284]

[0285] In the formula, Let be the rotation axis vector, which is perpendicular to... and The plane that is formed.

[0286] S44. Normalize the rotation axis to obtain a unit rotation axis.

[0287] In this embodiment of the invention, the rotation axis k is normalized to obtain the unit rotation axis vector, and the calculation formula is as follows:

[0288]

[0289] In the formula, The unit rotation axis vector; These are the unit rotation axis vectors in the camera coordinate system. Quantity.

[0290] S45. Calculate the attitude deviation angle between the unit normal vector of the target plane and the unit direction vector of the camera's Z-axis.

[0291] In this embodiment of the invention, the angle (i.e., attitude deviation angle) between the camera's Z-axis and the target plane's Z-axis is calculated using a vector dot product. The calculation formula is as follows:

[0292]

[0293] In the formula, This is the attitude deviation angle between the camera's Z-axis and the target plane's Z-axis (the angle that needs fine-tuning).

[0294] S46. Construct the antisymmetric matrix of the unit rotation axis.

[0295] In this embodiment of the invention, a corresponding antisymmetric matrix K is constructed based on the unit rotation axis vector u, which is used for subsequent calculations of the Rodrigues rotation formula. The expression for the antisymmetric matrix is:

[0296]

[0297] In the formula, Unit rotation axis vector The corresponding antisymmetric matrix; These are the unit rotational axial quantities. Quantity.

[0298] S47. Calculate the attitude correction rotation matrix using the antisymmetric matrix and the attitude deviation angle.

[0299] In this embodiment of the invention, the Rodriguez rotation formula is used to convert the antisymmetric matrix corresponding to the attitude deviation angle and the unit rotation axis into an attitude correction rotation matrix executable by the robotic arm control system. The calculation formula is as follows:

[0300]

[0301] In the formula, For attitude correction rotation matrix (3×3 dimension); It is a 3×3 identity matrix.

[0302] S48. Fine-tune the attitude of the robotic arm end effector according to the attitude correction rotation matrix and output the vertical attitude reference pose.

[0303] In this embodiment of the invention, the calculated attitude correction rotation matrix (or converted into the corresponding quaternion or Euler angle correction) is sent to the robotic arm controller to drive the end effector of the robotic arm around the unit rotation axis. Rotation The angle ensures that the axis of the tightening gun is strictly perpendicular to the target metal plane.

[0304] After the attitude fine-tuning is completed, the current end-effector pose is used as the vertical attitude reference pose for subsequent robotic arm operation control along the plane normal (such as bolt tightening, plane detection, etc.), thus completing the output of camera attitude fine-tuning and vertical attitude reference pose.

[0305] Target plane unit normal vector: The plane unit normal vector extracted from the homogeneous transformation matrix of the plane pose; Camera Z-axis unit direction vector: The Z-axis unit vector corresponding to the sleeve axis in the camera coordinate system; Rotation axis: The attitude rotation reference vector obtained by the cross product of the camera Z-axis and the target plane normal vector; Unit rotation axis: The vector of the rotation axis after normalization; Attitude deviation angle: The angle between the camera Z-axis and the target plane normal vector; Antisymmetric matrix: A 3×3 antisymmetric matrix constructed from the unit rotation axis; Attitude correction rotation matrix: The attitude correction matrix of the robotic arm end effector calculated based on the Rodriguez formula.

[0306] Step 209: Based on the vertical attitude reference pose, drive the robotic arm to perform bolt removal operation.

[0307] Furthermore, step 209 may include the following sub-steps:

[0308] S51. Drive the robotic arm to move to the safe avoidance position corresponding to the first bolt according to the preset diagonal disassembly sequence, and mark the first bolt as the bolt to be disassembled.

[0309] In this embodiment of the invention, a preset diagonal disassembly sequence is first executed. This sequence involves alternately disassembling bolts along the diagonal direction of the target metal plane to balance the disassembly force and prevent deformation of the metal plane. Specifically, this includes:

[0310] 1. Using the vertical posture reference position as a reference, the end of the drive arm moves to the safe avoidance position of the first bolt to be removed. This position is located directly above the bolt and maintains a preset vertical distance (such as 50~100mm) from the center of the bolt to ensure no risk of collision.

[0311] 2. Mark the bolt as the bolt to be disassembled, and denote it as the i-th bolt to be disassembled (i=1), and record its reference pose information.

[0312] S52. Obtain the contact force and torque information corresponding to the bolt to be disassembled, which is fed back in real time by the force sensor installed at the end of the robotic arm, and convert it to obtain the contact force and torque vector.

[0313] In this embodiment of the invention, a six-dimensional force sensor embedded in the end effector (sleeve) of the robotic arm is used to collect real-time contact force and torque information between the end effector of the robotic arm and the bolt to be disassembled. The collected calibrated quantities are converted into contact force and torque vectors, expressed as:

[0314]

[0315] In the formula, The contact force and torque vector at the end of the robotic arm has a dimension of 6×1; These are the lower edges of the robotic arm's end effector in the camera coordinate system. Component of contact force in the axial direction; They are respectively around Torque components of the shaft.

[0316] S53. Using the contact force and torque vector input to the preset admittance model, solve for the pose correction.

[0317] In this embodiment of the invention, a preset admittance control strategy is activated based on the contact force and torque vector, and the position and attitude correction of the robotic arm end is calculated in real time through the set admittance model.

[0318] The admittance model uses a second-order linear dynamic equation, the formula of which is:

[0319]

[0320] In the formula, The acceleration vector for pose correction; The velocity vector is the pose correction amount; This is the pose correction vector; These are the set 6×6 positive definite diagonal mass matrix, damping matrix, and virtual stiffness matrix, respectively.

[0321] Furthermore, the pose correction vector The expression is:

[0322]

[0323] In the formula, The end of the robotic arm Shaft position correction amount; They are respectively around The attitude correction amount of the axis.

[0324] By solving the above second-order linear dynamic equations, the pose correction amount of the robotic arm end effector is obtained, which is used to compensate for the position and attitude deviations caused by the contact force.

[0325] S54. The vertical attitude reference pose and pose correction amount are converted into the robot arm command pose.

[0326] In this embodiment of the invention, the vertical attitude reference pose obtained in step 208 is superimposed with the pose correction amount solved in step S53 to convert it into the desired pose that the robotic arm controller can execute. The calculation formula is as follows:

[0327]

[0328] In the formula, This is the command pose vector at the end effector of the robotic arm; The initial reference trajectory pose set for visual servoing and pose estimation (i.e., the initial trajectory corresponding to the vertical attitude reference pose). This is the pose correction value obtained by solving the admittance model.

[0329] To ensure the robotic arm can flexibly adjust in accordance with the direction of the contact force, the parameters are set using a virtual stiffness matrix K:

[0330]

[0331] In the formula, The plane perpendicular to the Z-axis ( Radial stiffness of the plane; For yaw / pitch stiffness; This provides greater stiffness along the Z-axis and roll angle. By setting smaller radial and yaw / pitch stiffness, the robotic arm can flexibly correct its course in accordance with the direction of the contact force.

[0332] S55. According to the robot arm's commanded position, drive the sleeve installed at the end of the robot arm to fit onto the bolt to be removed and perform a loosening operation.

[0333] In this embodiment of the invention, the end effector of the driven arm is positioned according to a commanded posture. Engage in exercise:

[0334] 1. Move vertically downwards along the Z-axis to precisely fit the end sleeve onto the head of the bolt to be removed;

[0335] 2. Based on contact force and torque vector When detected (Axial contact force) reaches the preset sleeve insertion threshold, and When the (loosening torque) reaches the preset start threshold, the loosening operation is triggered;

[0336] 3. The end effector of the drive arm rotates counterclockwise in a pre-set diagonal disassembly sequence to loosen the bolts. During this process, the admittance model continuously calculates the pose correction. This is to compensate for the contact force deviation during the loosening process and ensure that the sleeve is fully inserted into the bolt without any jamming.

[0337] S56. Drive the robotic arm to move to the safe avoidance position of the next bolt to be removed according to the preset diagonal disassembly sequence.

[0338] In this embodiment of the invention, after the current bolt to be removed is loosened (e.g., the torque feedback reaches a preset completion threshold), the end of the driven arm is activated:

[0339] 1. Lift along the Z-axis to a safe height, away from the current bolt position;

[0340] 2. Following the preset diagonal disassembly sequence, move to a safe avoidance position for the next bolt to be disassembled;

[0341] 3. Mark the next bolt as a new bolt to be removed and proceed to the next removal step.

[0342] S57. Jump to execute the step of obtaining the contact force and torque information corresponding to the bolts to be disassembled in real time from the force sensor installed at the end of the robotic arm and converting it to obtain the contact force and torque vector, until all bolts are disassembled.

[0343] In this embodiment of the invention, steps S52 to S56 are executed iteratively until all bolts (e.g., 4 bolts) have been loosened. The specific determination logic is as follows: iterate through the disassembly states of all bolts to be removed, and determine the loosening torque of each bolt. All reached the preset completion threshold, and the contact force When the load drops to the no-load threshold, it is determined that all bolts have been removed, and the bolt removal process ends.

[0344] Preset diagonal disassembly sequence: The preset sequence for alternating disassembly of bolts along the diagonal of the target metal plane; Safe avoidance position: A preset starting position directly above the bolt to be disassembled, with no risk of collision; Bolt to be disassembled: The target bolt for which the disassembly operation is currently being performed; Force sensor: A six-dimensional force / torque sensor installed at the end of the robotic arm; Contact force and torque information: End-effector contact force and torque data collected by the force sensor; Contact force and torque vector: The converted six-dimensional contact force and torque vector; Preset admittance model: A force-controlled admittance model based on the second-order linear dynamic equation; Pose correction: The end-effector pose compensation amount obtained from the admittance model; Robotic arm command pose: The executable pose obtained by superimposing the vertical attitude reference pose and the pose correction amount; Sleeve: An actuator at the end of the robotic arm used to insert the bolt; Loosening operation: A bolt disassembly operation in which the bolt is disassembled by rotating the sleeve counterclockwise after it is inserted into the bolt.

[0345] This invention has the following advantages:

[0346] High-precision pose estimation and attitude fine-tuning based on vision-point cloud joint: By integrating visual servo bolt initial alignment and point cloud local to global optimization processing, a homogeneous transformation matrix (orthogonal pose matrix) of the planar pose of the target metal plane is constructed; then, based on the Rodriguez rotation formula, the attitude deviation between the camera Z-axis (corresponding to the tightening gun axis direction) and the plane normal Z-axis is calculated, and the end of the arm is driven to perform attitude fine-tuning, completely solving the problem that the tightening gun (sleeve) axis is not perpendicular to the target metal plane.

[0347] The flexible sleeve insertion control strategy based on force admittance: During the contact insertion stage between the sleeve and the bolt, the virtual stiffness is dynamically adjusted through the admittance control model, which gives the end of the robotic arm compliant yielding and correction capabilities, eliminates the risk of sleeve jamming and bolt collision caused by hard contact, ensures that the sleeve can be inserted into the bolt without jamming, and smoothly completes the bolt disassembly operation.

[0348] Please see Figure 3 , Figure 3 This is a structural block diagram of a bolt disassembly system based on visual servoing and point cloud planar pose estimation, provided in Embodiment 3 of the present invention.

[0349] This invention provides a bolt disassembly system based on visual servoing and point cloud planar pose estimation, comprising:

[0350] The bolt alignment module 301 is used to acquire RGB images of the target metal plane through a camera mounted on the end of the robotic arm, perform bolt alignment, and output the bolt center seed point after each bolt has converged.

[0351] The point cloud processing module 302 is used to acquire the depth image of the target metal plane after the bolts are aligned, and to perform point cloud extraction and plane fitting by combining the seed points at the center of each bolt to obtain the initial plane equation.

[0352] Point cloud optimization module 303 is used to optimize the planar point cloud using the depth image and the initial plane equation to obtain an effective planar point cloud set;

[0353] The pose estimation module 304 is used to perform pose estimation using a set of effective points in the plane and to construct a homogeneous planar pose transformation matrix.

[0354] The attitude fine-tuning module 305 is used to fine-tune the attitude of the robotic arm end effector using a planar pose homogeneous transformation matrix and output a vertical attitude reference pose.

[0355] Bolt removal module 306 is used to drive a robotic arm to perform bolt removal operations based on a vertical attitude reference pose.

[0356] Since the above is a system corresponding to a bolt disassembly method based on visual servoing and point cloud planar pose estimation, its implementation principle is the same as that of a bolt disassembly method based on visual servoing and point cloud planar pose estimation. For the sake of convenience and brevity, those skilled in the art can clearly understand that the specific working process of the system and modules described above can be referred to the corresponding process in the aforementioned method embodiments, and will not be repeated here.

[0357] Please see Figure 4 , Figure 4 This is a structural block diagram of an electronic device provided in Embodiment 4 of the present invention.

[0358] An electronic device according to an embodiment of the present invention includes: a memory 401 and a processor 402. The memory 401 stores a computer program. When the computer program is executed by the processor 402, the processor 402 performs a bolt disassembly method based on visual servoing and point cloud planar pose estimation as described in the above embodiment.

[0359] Memory 401 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Memory 401 has storage space 403 for program code 413 for performing any of the method steps described above. For example, storage space 403 for program code may include individual program codes 413 for implementing the various steps in the methods described above. This program code may be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. The program code may be compressed, for example, in a suitable form. When run by a computing processing device, this code causes the computing processing device to perform the various steps in the methods described above. This program code may be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. The program code may be compressed, for example, in a suitable form. When this code is run by a computing device, it causes the computing device to perform the various steps in the bolt removal method based on visual servoing and point cloud planar pose estimation described above.

[0360] Embodiment 5 of the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the bolt disassembly method based on visual servoing and point cloud planar pose estimation as described in the above embodiments.

[0361] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0362] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0363] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0364] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0365] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0366] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A bolt disassembly method based on visual servoing and point cloud planar pose estimation, characterized in that, include: The camera mounted on the end of the robotic arm acquires RGB images of the target metal plane, performs bolt alignment, and outputs the bolt center seed point after each bolt converges. The depth image of the target metal plane after the bolts are aligned is obtained, and point cloud extraction and plane fitting are performed by combining the seed points of each bolt center to obtain the initial plane equation. The planar point cloud is optimized using the depth image and the initial plane equation to obtain an effective planar point cloud set. Pose estimation is performed using the aforementioned effective point cloud in the plane, and a homogeneous planar pose transformation matrix is ​​constructed. The planar pose homogeneous transformation matrix is ​​used to fine-tune the end effector posture of the robotic arm and output a vertical pose reference posture. Based on the vertical attitude reference pose, the robotic arm is driven to perform bolt removal operations.

2. The bolt disassembly method based on visual servoing and point cloud planar pose estimation according to claim 1, characterized in that, The process involves acquiring RGB images of the target metal plane using a camera mounted on the end effector of the robotic arm, performing bolt alignment, and outputting the converged bolt center seed points, including: The RGB image of the target metal plane is acquired by a camera mounted on the end of the robotic arm; Bolt detection is performed on the RGB image using a preset target detection model to obtain the detection box coordinates of each bolt; The center feature points of each bolt are extracted from the coordinates of each detection frame; Combine all the central feature points to construct the current state vector and set the desired vector; Calculate the difference between the current state vector and the expected vector to obtain the error vector; By taking the time derivative of the error vector, an equivalent relationship is established between the error rate of change and the motion speed of image feature points; Substituting the equivalent relation into the preset first-order decoupling control law, we obtain the image feature point motion speed control equation. An image Jacobian matrix is ​​introduced, and the generalized inverse of the image Jacobian matrix is ​​solved. The camera control speed is obtained by substituting the generalized inverse solution result and the error vector into the image feature point motion velocity control equation; Introduce the Jacobian matrix of the robotic arm and solve for the generalized inverse of the Jacobian matrix; The generalized inverse kinematics solution result and the camera control speed are used to convert the robot arm joint angular velocity control command and execute bolt alignment. Jump to execute the step of using a preset target detection model to detect bolts in the RGB image and obtain the detection box coordinates of each bolt until the magnitude of the error vector is less than the preset alignment tolerance threshold, and output the converged bolt center seed point of each bolt.

3. The bolt disassembly method based on visual servoing and point cloud planar pose estimation according to claim 1, characterized in that, The process involves acquiring a depth image of the target metal plane after bolt alignment, and combining this image with point cloud extraction and plane fitting using seed points at the center of each bolt to obtain an initial plane equation, including: Obtain a depth image of the target metal plane after the bolts are aligned, delineate a preset neighborhood rectangle local region centered on the seed point of each bolt, and extract the corresponding local pixel coordinate set; Merge all the aforementioned local pixel coordinate sets to obtain the total local pixel coordinate set; Extract the real-time depth value from the depth image, and perform pixel back projection on the total local pixel coordinate set in combination with the camera intrinsic parameter matrix to obtain a local 3D point cloud; Fit all the local 3D point clouds and denoise them to obtain the initial plane equation.

4. The bolt disassembly method based on visual servoing and point cloud planar pose estimation according to claim 1, characterized in that, The step of optimizing the planar point cloud using the depth image and the initial plane equation to obtain an effective planar point cloud set includes: Pixel back projection is performed on the depth image to obtain a global point cloud; Based on the initial plane equation, calculate the Euclidean distance between each global point cloud in the global point cloud set and the initial plane; Remove global point clouds whose Euclidean distance is greater than or equal to a preset distance threshold from the global point cloud set to obtain a coarsely filtered planar point cloud set; Euclidean clustering is performed on the coarsely selected point set in the plane to obtain multiple clusters; The average center is obtained by averaging all the seed points of the bolt centers. Calculate the geometric center of each of the said clusters; Calculate the target distance between each geometric center and the average center, and select the cluster corresponding to the minimum target distance as the effective point cloud set in the plane.

5. The bolt disassembly method based on visual servoing and point cloud planar pose estimation according to claim 1, characterized in that, The step of using the plane effective point cloud for pose estimation and constructing a plane pose homogeneous transformation matrix includes: The point cloud is centered using the aforementioned effective point cloud set to obtain a centralized point cloud set; Construct a covariance matrix using the centralized point set; Perform eigenvalue decomposition on the covariance matrix to extract the initial normal vector; The initial normal vector is corrected to obtain the plane normal vector; The effective point cloud set in the plane is projected onto the point cloud to obtain the projected point cloud set in the plane; Calculate the convex hull of the set of projection points on the plane, and solve for the minimum bounding rectangle of the convex hull; Extract the direction vector corresponding to the longest side of the minimum bounding rectangle as the main direction vector; The cross product of the principal direction vector and the plane normal vector is used to obtain the X-axis of the local plane coordinate system after normalization. The cross product of the plane normal vector and the X-axis of the local plane coordinate system is used to obtain the Y-axis of the local plane coordinate system. Obtain the centroid of the point cloud of the effective point cloud set in the plane, and construct the homogeneous transformation matrix of the plane pose by combining the plane normal vector, the X-axis of the plane local coordinate system and the Y-axis of the plane local coordinate system.

6. The bolt disassembly method based on visual servoing and point cloud planar pose estimation according to claim 1, characterized in that, The step of fine-tuning the end effector posture of the robotic arm using the homogeneous planar pose transformation matrix and outputting a vertical pose reference includes: Extract the plane normal vector from the homogeneous transformation matrix of the plane pose as the unit normal vector of the target plane; Obtain the unit direction vector along the camera's Z-axis; The rotation axis is obtained by performing a cross product operation between the unit normal vector of the target plane and the unit direction vector of the camera's Z-axis. The rotation axis is normalized to obtain a unit rotation axis; Calculate the attitude deviation angle between the unit normal vector of the target plane and the unit direction vector of the camera Z-axis; Construct the antisymmetric matrix of the unit rotation axis; The attitude correction rotation matrix is ​​calculated using the antisymmetric matrix and the attitude deviation angle; The attitude of the robotic arm end effector is fine-tuned according to the attitude correction rotation matrix, and a vertical attitude reference pose is output.

7. The bolt disassembly method based on visual servoing and point cloud planar pose estimation according to any one of claims 1-6, characterized in that, The step of driving the robotic arm to perform bolt removal operations based on the vertical attitude reference pose includes: The robotic arm is driven to move to the safe avoidance position corresponding to the first bolt according to the preset diagonal disassembly sequence, and the first bolt is marked as the bolt to be disassembled; The contact force and torque information corresponding to the bolt to be disassembled, which is fed back in real time by the force sensor installed at the end of the robotic arm, is obtained and converted to obtain the contact force and torque vector; The pose correction is solved by inputting the contact force and torque vectors into the preset admittance model. The vertical attitude reference pose and the pose correction amount are used to convert the robotic arm command pose; According to the commanded posture of the robotic arm, drive the sleeve installed at the end of the robotic arm to fit into the bolt to be removed to perform a loosening operation; Drive the robotic arm to move to a safe avoidance position for the next bolt to be removed, following the preset diagonal disassembly sequence; Jump to the step of obtaining and converting the contact force and torque information corresponding to the bolt to be disassembled, which is fed back in real time by the force sensor installed at the end of the robotic arm, to obtain the contact force and torque vector, until all bolts are disassembled.

8. A bolt disassembly system based on visual servoing and point cloud planar pose estimation, characterized in that, include: The bolt alignment module is used to acquire RGB images of the target metal plane through a camera mounted on the end of the robotic arm, perform bolt alignment, and output the bolt center seed point after convergence of each bolt. The point cloud processing module is used to acquire the depth image of the target metal plane after the bolts are aligned, and to perform point cloud extraction and plane fitting by combining the seed points at the center of each bolt to obtain the initial plane equation. The point cloud optimization module is used to optimize the planar point cloud using the depth image and the initial plane equation to obtain an effective planar point cloud set. The pose estimation module is used to perform pose estimation using the plane effective point cloud set and construct a plane pose homogeneous transformation matrix. The attitude fine-tuning module is used to fine-tune the attitude of the robotic arm end effector using the planar pose homogeneous transformation matrix and output the vertical attitude reference pose. The bolt removal module is used to drive the robotic arm to perform bolt removal operations based on the vertical attitude reference pose.

9. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the bolt removal method based on visual servoing and point cloud planar pose estimation as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the bolt disassembly method based on visual servoing and point cloud planar pose estimation as described in any one of claims 1-7.

Images