A bolt anti-loosening blind pin automatic installation device and method based on visual space positioning

Abstract

The application provides a bolt anti-loosening closed pin automatic installation device and method based on visual space positioning, which comprises a control module and a mechanical arm, and the tail end of the mechanical arm is provided with a gripper component and an image acquisition unit; the image acquisition unit is used for acquiring an RGB image and a depth image of the front area of the mechanical arm; the gripper component comprises two clamps for clamping the two sides of the closed pin, a push rod is arranged between the two clamps, and one end of the closed pin to be installed is connected with the front end of the push rod; a linkage mechanism is arranged between the push rod and the two clamps to realize the forward movement of the push rod and the closed pin during the opening of the two clamps; and a six-dimensional torque sensor is arranged between the tail end of the mechanical arm and the gripper component. The application completely replaces manual tower climbing live working, avoids the fatal safety hazards such as high-voltage electric shock and high-altitude falling faced by the workers, does not require the workers to enter the high-voltage dangerous area, the whole installation process can be automatically completed, and the skill requirement for the workers is reduced.

Description

Technical Field

[0001] This invention relates to the field of power system automation operation and maintenance technology, and in particular to an automatic installation device and method for bolt anti-loosening locking pins based on visual spatial positioning. Background Technology

[0002] The bolts on the spacers of high-voltage transmission lines are the core fasteners of the split conductor spacers, responsible for fixing the clamps, maintaining the spacing between the sub-conductors, and providing vibration and anti-galling effects. For example... Figure 4 and Figure 5 As shown, the bolt 5 on the spacer bar is provided with a pin hole 16, into which a closed pin 6 is inserted to loosen the bolt 5. The structure of the closed pin 6 is as follows: Figure 5 As shown.

[0003] However, the closed pins 6 connected to the bolts 5 of the spacer are prone to falling off due to long-term wind vibration. Once they fall off, the closed pins need to be replaced in time. Currently, the main methods for addressing the problem of closed pins falling off are manual installation on the tower under power and remote installation from the ground using an insulated operating rod.

[0004] Manual installation requires operators to climb the tower for live installation, which exposes them to the dual risks of high voltage and high drop, resulting in poor safety and high labor costs. On the other hand, using an insulated operating rod for remote ground-based installation is limited by the operating angle and tool flexibility, making it difficult to guarantee the alignment accuracy of the closed pin. This requires a high level of skill from the operator and often requires multiple attempts to accurately insert the closed pin, placing extremely high demands on the operator's skills and patience. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by providing an automatic installation device and method for bolt anti-loosening locking pins based on visual spatial positioning.

[0006] The objective of this invention is achieved through the following technical solution: an automatic bolt anti-loosening closed pin installation device based on visual spatial positioning, comprising a control module and a robotic arm. The end of the robotic arm is equipped with a gripper component and an image acquisition unit. The image acquisition unit acquires RGB and depth images of the area in front of the robotic arm. The gripper component includes two grippers for clamping both sides of the closed pin, with a push rod positioned between the two grippers. One end of the closed pin to be installed is connected to the front end of the push rod. A linkage mechanism is provided between the push rod and the two grippers to enable the two grippers to drive the push rod and the closed pin forward during the opening process. A six-dimensional torque sensor is provided between the end of the robotic arm and the gripper component. The robotic arm moves the closed pin to a pre-insertion position directly in front of the pin hole on the bolt. The clamping power component drives the two grippers to open, causing the push rod to move the closed pin forward, thereby achieving complete insertion of the closed pin.

[0007] Preferably, the linkage mechanism includes a guide rail fixedly mounted on the gripper component, a push rod connecting block slidably connected to the guide rail, and a connecting rod between the two grippers and the push rod connecting block. The two ends of the connecting rod are respectively hinged to the push rod connecting block and the grippers. The gripper component drives the two grippers to move relative to each other in a direction perpendicular to the guide rail.

[0008] Preferably, the gripper component is an electric gripper, and the image acquisition unit is a depth camera.

[0009] An automatic installation method for bolt anti-loosening locking pins based on visual spatial positioning includes the following specific steps: Step 1: The image acquisition unit acquires and obtains RGB-D image frames of the scene in front in real time. Each RGB-D image frame contains an RGB image and a depth image that is pixel-aligned with it.

[0010] Step 2: Extract the two-dimensional bounding box of the target bolt that needs to be replaced based on the RGB image; Step 3: Extract the pixel-level mask of the target bolt based on the RGB image, the two-dimensional bounding box coordinates of the pin hole, and the two-dimensional bounding box coordinates of the target bolt, so as to accurately identify each pixel in the image that belongs to the target bolt; Step 4: Based on the depth image, the pixel-level mask of the target bolt, and the pre-built 3D model of the target bolt, output the spatial pose of the pin hole on the target bolt in the coordinate system of the image acquisition unit. Step 5: Based on the spatial pose in the coordinate system of the image acquisition unit and the transformation relationship between the coordinate system of the image acquisition unit and the coordinate system of the end flange of the robotic arm, output the spatial pose of the pin hole in the coordinate system of the end flange of the robotic arm. Step 6: Based on the spatial pose of the pin hole in the flange coordinate system at the end of the robotic arm and the point cloud data collected by the depth camera, generate the motion path of the robotic arm. The robotic arm moves the closed pin on the gripper component to the pre-insertion position, and then the two grippers open and push the closed pin completely into the pin hole through the push rod.

[0011] As a preferred option, the specific method for step two is as follows: S21. Based on the application scenario characteristics of bolts, construct a set of scenario-based text prompts; S22. Input the RGB image and the set of text prompts into the open vocabulary object detection model. The model searches for regions in the RGB image that match the semantics of the text description through a visual-language alignment mechanism, and outputs the two-dimensional bounding boxes of all candidate bolts and their corresponding confidence scores. S23. Depth Range Filtering: For each candidate 2D bounding box, extract the average depth value of its corresponding region on the depth image, remove 2D bounding boxes with obviously unreasonable depths, and retain 2D bounding boxes within a reasonable working distance. S24. Automatic determination of missing pin status: For each candidate bolt retained after filtering, the depth map information is used to determine whether the closed pin on it has fallen off; if the closed pin on the current candidate bolt has fallen off, the bolt is marked as the target bolt that needs to be replaced with a pin. S25. From the marked target bolts to be replaced, select the target bolt closest to the robotic arm as the current working target, and output the two-dimensional bounding box of the target bolt.

[0012] Preferably, in step S24, when using depth map information to determine whether the closed pin on the bolt has fallen off, the specific method is as follows: based on the two-dimensional bounding box and geometric prior of the bolt, locate the image sub-region where the pin hole is located, and extract the depth pixel value corresponding to the image sub-region; if the depth change value of the image sub-region exceeds the set threshold, it is determined that the closed pin of the bolt has fallen off and is marked as the target bolt that needs to be re-pinned; otherwise, no marking is required.

[0013] As a preferred method, in step three, a prompt-based image segmentation large model is used to extract the pixel-level mask of the target bolt. The specific method is as follows: S31. Input the RGB image into the image encoder of the large image segmentation model to generate high-dimensional image feature embeddings; S32. Use the two-dimensional bounding box as the bounding box cue input image segmentation model's cue encoder to generate cue position embeddings; S33. The high-dimensional image features and the prompt location are embedded together into the mask decoder of the large image segmentation model. The mask decoder fuses the two types of information through a cross-attention mechanism and outputs the candidate pixel-level mask of the target bolt. The pixel-level mask with the highest confidence is selected from the candidate pixel-level masks as the final result.

[0014] Preferably, in step six, when generating the movement path of the robotic arm, the 3D point cloud data collected in real time by the depth camera is converted into a 3D occupancy grid map, and an obstacle envelope model within the workspace of the robotic arm is constructed in real time to ensure that the planned movement path does not collide with surrounding obstacles.

[0015] As a preferred method, in step six, when fully pushing the closed pin into the pin hole, the specific method is as follows: N1: The six-dimensional torque sensor at the end of the robotic arm reads the contact force and torque data in each direction in real time during the insertion process; The collected raw sensor data is then preprocessed. N2: Compare the force and torque data in each direction with preset safety thresholds; when the force and torque in all directions do not exceed the safety thresholds, the system will steadily advance the closing pin along the insertion direction at the set feed speed; when the force or torque in any direction exceeds the safety threshold, the system determines that a contact collision has occurred and triggers compliance compensation; during compliance compensation, the system first calculates the displacement compensation X and velocity compensation amount that the end effector should produce based on the contact force F_ext detected by the sensor that exceeds the safety threshold. The specific calculation formula is as follows: ; In the above equation, M, B, and K represent the desired mass matrix, damping matrix, and stiffness matrix of the system, respectively. The positional deviation is output as the displacement compensation amount. ； The first derivative of the position deviation is used as the output velocity compensation quantity. This is the second derivative of the positional deviation; The calculated displacement and velocity compensation values ​​are then superimposed onto the original motion command, and the displacement and velocity compensation are executed by the robotic arm.

[0016] Preferably, in step six, if the contact force in a certain direction is detected to still exceed the safety threshold after compliance compensation, the robotic arm automatically retracts in the opposite direction to the pre-insertion position. Then, the system re-acquires the image and updates the spatial pose of the pin hole. The robotic arm then re-executes the pin hole insertion process based on the updated spatial pose of the pin hole.

[0017] The beneficial effects of this invention are: 1. This invention completely replaces manual tower climbing for live-line work, avoiding fatal safety hazards such as high-voltage electric shock and falls from heights faced by workers. It eliminates the need for personnel to enter high-voltage danger zones, and the entire installation process can be completed automatically. It also reduces the skill requirements for workers and improves work efficiency.

[0018] 2. This invention employs algorithms such as 2D bounding box extraction, pixel-level mask segmentation, and 3D model matching to achieve accurate calculation of the spatial pose of the pin hole, with alignment accuracy far exceeding that of manual visual operation. Pixel-level mask segmentation effectively eliminates interference from wires, spacers, other components, complex lighting, and background, enabling stable target identification even in partially obscured outdoor environments. The use of pixel-level aligned RGB-D images ensures accurate depth information for each pixel in the RGB image, providing a reliable data foundation for 3D pose calculation. Ultimately, this enables high-precision installation of the closed pin, significantly increasing the probability of successful installation on the first attempt. This effectively solves the problem of traditional ground-based remote operation requiring multiple installations for successful insertion, significantly improving the installation efficiency of the closed pin.

[0019] 3. The installation process incorporates open-vocabulary object detection algorithms and prompt-based image segmentation large models, breaking the bottleneck of traditional machine vision's heavy reliance on large amounts of on-site data collection, annotation, and model retraining. This enables the installation scheme of this invention to have excellent zero-shot or few-shot generalization capabilities, and can easily cope with complex unstructured environments such as different tower types, different lighting conditions, and aging and rusting bolts, greatly reducing the cost of algorithm deployment and maintenance.

[0020] 4. During the closed pin insertion control process, the detection feedback data of a six-dimensional torque sensor and a compliance compensation mechanism are introduced to upgrade the traditional "rigid position control" to "force-position hybrid control". This perfectly simulates the tactile fine-tuning ability of a human hand during precision assembly and can still ensure an extremely high success rate of closed pin installation under dynamic interference such as slight wind deviation. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the installation device of the present invention.

[0022] Figure 2 This is a schematic diagram of the gripper component.

[0023] Figure 3 This is a flowchart of the installation method of the present invention.

[0024] Figure 4 This is a schematic diagram of the bolt structure of the present invention.

[0025] Figure 5 This is a schematic diagram of a closed-loop pin.

[0026] In the diagram: 1. Robotic arm, 2. End flange, 3. Image acquisition unit, 4. Gripper assembly, 5. Bolt, 6. Closing pin, 7. Gripper, 8. Guide rail, 9. Slider, 10. Guide rail mounting plate, 11. Push rod connecting block, 12. First connecting shaft, 13. Second connecting shaft, 14. Push rod, 15. Connecting rod, 16. Pin hole. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0028] Those skilled in the art should understand that, in the disclosure of this invention, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting this invention.

[0029] It is understood that the term "a" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.

[0030] like Figures 1 to 2 As shown, an automatic bolt anti-loosening closed pin installation device based on visual spatial positioning includes a control module and a robotic arm 1. The end of the robotic arm 1 is provided with a gripper component 4 and an image acquisition unit 3. The image acquisition unit 3 is used to acquire RGB images and depth images of the area in front of the robotic arm 1. The gripper component 4 includes two grippers 7 for clamping both sides of the closed pin 6. A push rod 14 is provided between the two grippers 7. One end of the closed pin 6 to be installed is connected to the front end of the push rod 14. A linkage mechanism is provided between the push rod 14 and the two grippers 7 to realize that the two grippers 7 drive the push rod 14 and the closed pin to move forward during the opening process. A six-dimensional torque sensor is provided between the end of the robotic arm 1 and the gripper component 4. The robotic arm 1 drives the closed pin 6 to the pre-insertion position directly in front of the pin hole 16 on the bolt 5. The clamping power component drives the two grippers 7 to open so that the push rod 14 drives the closed pin 6 to move forward, thereby realizing the complete insertion of the closed pin 6.

[0031] Both the gripper component 4 and the image acquisition unit 3 are mounted on the end effector of the robotic arm 1. The gripper component 4 is mounted on the end flange 2 of the robotic arm 1. Both the gripper component 4 and the image acquisition unit 3 move synchronously with the end effector of the robotic arm. The image acquisition unit 3 is mounted on top of the end effector of the robotic arm 1 in an eye-in-hand configuration, acquiring RGB color images and aligned depth images of the scene in front of it in real time to form RGB-D image frames. The image acquisition unit 3 transmits the acquired image stream to a remote server or edge computing node via a high-bandwidth wired connection.

[0032] Specifically, the linkage mechanism includes a guide rail 8 fixedly mounted on the gripper component 4, a push rod connecting block 11 slidably connected to the guide rail 8, and a connecting rod 15 between the two grippers 7 and the push rod connecting block 11. The two ends of the connecting rod 15 are hinged to the push rod connecting block 11 and the grippers 7, respectively. The gripper component 4 drives the two grippers 7 to move relative to each other in a direction perpendicular to the guide rail 8. The push rod connecting block 11 is slidably mounted on the guide rail 8, and the guide rail 8 provides a constraint guide for the push rod connecting block 11, restricting it to linear motion only along the axial direction of the guide rail 8. When the closed pin 6 is in the clamped state, one end of the closed pin 6 abuts against the front end of the push rod 14, and the front ends of the two grippers 7 clamp the two sides of the closed pin 6, respectively. When the two grippers 7 open, they release the closed pin 6. During the opening process, the grippers 7 drive the push rod connecting block 11 forward along the guide rail 8 via the connecting rod 15, thereby pushing the closed pin 6 into the pin hole 16 of the bolt 5.

[0033] The gripper component 4 is equipped with a guide rail mounting plate 10, and the guide rail 8 is fixedly mounted on the guide rail mounting plate 10. A slider 9 is mounted on the guide rail 8, and a push rod connecting block 11 is mounted on the slider 9, achieving a sliding connection between the push rod connecting block 11 and the guide rail 8 via the slider 9. One end of the connecting rod 15 is rotatably connected to the gripper 7 via a first connecting shaft 12, and the other end of the connecting rod 15 is rotatably connected to the push rod connecting block 11 via a second connecting shaft 13.

[0034] The front end of the push rod 14 is provided with a slot for connecting one end of the closed pin 6. When the closed pin 6 is clamped, one end of the closed pin 6 is inserted into the slot.

[0035] In this application, the gripper component 4 is an electric gripper 7, and the image acquisition unit 3 is a depth camera.

[0036] In actual use, the robotic arm 1 is mounted on a mobile device that is mounted on a high-voltage power line and can travel along the high-voltage power line. The robotic arm 1 is transported to the vicinity of the work area by the mobile device.

[0037] like Figure 3As shown, the present invention also provides an automatic installation method for bolt anti-loosening locking pins based on visual spatial positioning, including the following specific steps: Step 1: Image acquisition unit 3 acquires and obtains RGB-D image frames of the scene in front in real time. The RGB-D image frame contains an RGB image and a depth image that is pixel-aligned with it.

[0038] Because the working environment of high-voltage transmission lines has visual characteristics that are completely different from traditional industrial scenes, namely complex and variable outdoor lighting conditions, including strong direct sunlight at noon, large areas of shadow cast by the tower, backlighting and glare; the surface of bolt 5 often corrodes due to long-term outdoor exposure, resulting in irregular local high-gloss reflection points; and multi-split conductors and spacer hardware form a dense and interfering background in the image. In view of the special characteristics of the actual working scene, in this step, the automatic exposure function of the depth camera is used for preliminary brightness adjustment, and adaptive histogram equalization (CLAHE) is performed on the RGB image at the software level to improve local contrast; and hole filling based on neighborhood interpolation is performed on invalid pixels in the depth image caused by strong reflections.

[0039] Furthermore, in the Eye-in-Hand configuration, the camera moves with the robotic arm, resulting in significant limitations on the viewing angle in multi-split conductor environments. This method employs a two-stage observation strategy: first, the robotic arm moves to a preset global observation pose, acquiring a global image of the spacer region with a large field of view for coarse bolt detection; then, the robotic arm adjusts its end effector pose based on the coarse detection results, positioning the camera directly facing the target bolt and moving it to an appropriate distance to acquire a high-resolution local image for subsequent precise segmentation and pose estimation.

[0040] Step 2: Extract the two-dimensional bounding box of the target bolt 5 that needs to be replaced based on the RGB image.

[0041] In this step, the YOLO-World open vocabulary object detection algorithm is used to extract the two-dimensional bounding boxes. This algorithm was chosen because bolts 5 on spacers from different tower types, voltage levels, and manufacturers exhibit significant differences in dimensions, installation angles, and corrosion levels. Traditional supervised object detection methods would require collecting a large number of images for each new scene, manually labeling them, and retraining the model, resulting in extremely high deployment and maintenance costs. The YOLO-World open vocabulary object detection algorithm can achieve zero-shot detection using text prompts, fundamentally solving the problem of insufficient generalization ability.

[0042] The specific method for step two is as follows: S21. Based on the application scenario characteristics of bolts, construct a set of scenario-based text prompts. Text prompts include phrases such as "spacer bolt with pin hole" and "hexagonal bolt on spacer clamp." The prompts incorporate the contextual constraints of the spacer bar scenario (i.e., explicitly specifying the spacer bolt rather than the generalized bolt) to guide the detector to focus on the bolts on the spacer bar rather than bolts on other hardware in the image, reducing false detections caused by complex hardware backgrounds (clips, equalizing rings, insulator strings, conductors, etc.).

[0043] S22. Input the RGB image and the set of text prompts into the open vocabulary target detection model. The model searches for regions in the RGB image that match the semantics of the text description (too far or too close, i.e. not within the effective working distance of the robotic arm 1) through a visual-language alignment mechanism, and outputs the two-dimensional bounding boxes of all candidate bolts 5 and their corresponding confidence scores. S23. Depth Range Filtering: For each candidate 2D bounding box, extract the average depth value of its corresponding region on the depth image, remove 2D bounding boxes with obviously unreasonable depths, and retain 2D bounding boxes within a reasonable working distance. S24. Automatic determination of missing pin status: For each candidate bolt 5 retained after filtering, the depth map information is used to determine whether the closed pin 6 on it has fallen off; if the closed pin 6 on the current candidate bolt 5 has fallen off, then the bolt 5 is marked as the target bolt 5 that needs to be replaced with a pin. S25. From the marked target bolts 5 to be replaced, select the target bolt 5 closest to the robotic arm 1 as the current working target, and output the two-dimensional bounding box of the target bolt 5. The output two-dimensional bounding box is used for subsequent steps.

[0044] In step S24, when using depth map information to determine whether the closed pin 6 has fallen off, the specific method is as follows: Based on the two-dimensional bounding box and geometric prior of the bolt 5, locate the image sub-region where the pin hole 16 is located, and extract the depth pixel value corresponding to the image sub-region; if the depth change value of the image sub-region exceeds the set threshold (i.e., the pin hole 16 is in a through-hole state, the depth value is significantly greater than the surface depth value of the bolt 5, and the depth difference exceeds the set threshold), then it is determined that the closed pin 6 of the bolt 5 has fallen off and is marked as the target bolt 5 that needs to be re-pinned; otherwise, if the depth value of the pin hole 16 region is consistent with the surface depth of the bolt 5 (indicating that the pin hole 16 is occupied by the closed pin 6), then no marking is required.

[0045] This method effectively utilizes the three-dimensional features of depth information, avoiding misjudgments caused by corrosion interference and lighting changes when relying solely on the appearance of RGB images for binary classification of the presence / absence of closed pins.

[0046] Step 3: Extract the pixel-level mask of the target bolt 5 based on the RGB image, the two-dimensional bounding box coordinates of the pin hole 16, and the two-dimensional bounding box coordinates of the target bolt 5, so as to accurately identify each pixel in the image belonging to the target bolt 5.

[0047] In this step, the Segment Anything Model (SAM_b version) based on prompts is used to extract the pixel-level mask of the target bolt. The specific mask extraction process is as follows: S31. Input the RGB image into the image encoder of the large-scale image segmentation model to generate a high-dimensional image feature embedding. This image encoder has been pre-trained on a large number of diverse images and has a powerful general representation capability for the edge, texture, and shape features of various objects. Specifically, the image encoder performs deep processing on the entire image, extracting all the edges, contours, textures, shapes, object structures, and semantic information from the image and converting them into a set of very long digital vectors. This set of digital vectors, which contains all the abstract features of the image and has a high dimension, is the high-dimensional image feature embedding.

[0048] S32. Using the 2D bounding box as the input bounding box cue, the cue encoder of the large image segmentation model generates a cue location embedding; this cue informs the model of the approximate region where the target object to be segmented is located. Specifically, after this bounding box is fed into the cue encoder, the encoder converts the box's location, size, and spatial region information into a set of high-dimensional numerical vectors. This set of high-dimensional feature vectors, specifically used to express the spatial location information of the bounding box, is the cue location embedding.

[0049] S33. The high-dimensional image features and the prompt location are embedded together into the mask decoder of the large image segmentation model. The mask decoder fuses the two types of information through a cross-attention mechanism and outputs the candidate pixel-level mask of the target bolt. The pixel-level mask with the highest confidence is selected from the candidate pixel-level masks as the final result.

[0050] In this step, the pose estimation algorithm in the subsequent fourth step requires an accurate object mask to distinguish the foreground target from the background. In this application scenario, the bounding box inevitably contains a large number of background pixels such as wires, spacers, and sky. If these are directly input into the pose estimation algorithm, it will seriously interfere with model matching, leading to pose estimation deviations or even complete failure.

[0051] In the application scenario of this invention, the spacer bolts are regular geometric shapes (hexagonal nuts, cylindrical bolts) with clear metal edge contours. Based on the prompt-based image segmentation model, pre-trained on massive amounts of images, the segmentation accuracy for such regular geometric edges (e.g., hexagonal edges, cylindrical contours) can reach sub-pixel level. This is crucial for precision positioning scenarios where the mounting hole diameter is only a few millimeters. If the segmentation accuracy is insufficient, pixel-level errors at the mask edges will be amplified through subsequent pose estimation, resulting in the final closed pin failing to be properly installed.

[0052] Step 4: Based on the depth image, the pixel-level mask of the target bolt, and the pre-built 3D model of the target bolt, output the spatial pose of the pin hole on the target bolt in the coordinate system of the image acquisition unit.

[0053] The 6D spatial pose of the pin hole in the coordinate system of the image acquisition unit includes the position in the three directions of x / y / z axes, as well as the roll angle, pitch angle, and yaw angle.

[0054] In this step, the position of the closed pin mounting hole has been accurately marked in the 3D model (OBJ format) of the bolt: a local coordinate system based on the pin hole is established with the center of the pin hole as the origin and the direction of the pin hole axis as the Z-axis, and its rigid body transformation relationship relative to the overall coordinate system of the bolt model is recorded.

[0055] This step is based on the FoundationPose pose estimation algorithm and has been adapted and improved in the following ways to suit the specific application scenario: The basic principle of the FoundationPose algorithm is the render-and-compare mechanism: the 3D model of the bolt is virtually rendered in different candidate poses to generate corresponding rendered images, and then compared with the actual observed bolt image (obtained by cropping the pixel-level mask of the target bolt) for feature comparison. The spatial pose that best matches the rendered image and the observed image is solved through iterative optimization.

[0056] In this step, the spatial pose of the target bolt as a whole in the image acquisition unit coordinate system (camera coordinate system) is first obtained based on the depth image. Then, by combining the rigid body transformation relationship between the pre-calibrated local coordinate system of the pin hole and the overall coordinate system of the bolt in the 3D model of the target bolt, the precise spatial pose of the pin hole in the image acquisition unit coordinate system (camera coordinate system) can be directly calculated through pure geometric coordinate transformation.

[0057] Based on real-world application scenarios, this step has made the following targeted adaptations: 1. Bolt Symmetry Disambiguation: Hexagonal bolts possess 60-degree rotational symmetry. The standard FoundationPose algorithm, when handling such symmetrical objects, easily generates multiple equivalent pose solutions (i.e., various equivalent orientations of the 60-degree rotation), leading to inter-frame jumps in pose estimation. This method introduces a closed-pin mounting hole as a symmetry disambiguation feature. Since the position of the mounting hole breaks the bolt's rotational symmetry, the system pre-marks the mounting hole position in the CAD model and adds matching weights to the mounting hole region during pose optimization, enabling the algorithm to lock in the unique correct orientation from multiple equivalent pose solutions.

[0058] 2. Outdoor Depth Noise Adaptation: The standard FoundationPose algorithm was developed and validated in a controlled indoor environment, which places high demands on the quality of depth data. This method addresses the depth noise and hole issues generated by outdoor depth cameras under metallic reflective surfaces and strong sunlight by adjusting the weighting strategy of depth data in the rendering-matching loss function: regions with low depth confidence (pixels with depth values ​​close to zero or exhibiting abrupt changes) are automatically reduced in their contribution to the matching loss, ensuring that pose optimization primarily relies on a subset of pixels with reliable depth quality, thus improving the algorithm's robustness in outdoor environments.

[0059] 3. Pose Timing Filtering and Smoothing: The 6D pose data output frame by frame by the FoundationPose algorithm is processed sequentially using both moving average filtering and Kalman filtering. Moving average filtering eliminates high-frequency noise spikes, while Kalman filtering outputs a smooth pose trajectory through state prediction and observation correction. The filter parameters are optimized based on the robotic arm's motion control frequency (typically 125Hz) and the on-site wind vibration characteristics to achieve a balance between pose tracking response speed and smoothness.

[0060] Step 5: Based on the spatial pose in the coordinate system of the image acquisition unit and the transformation relationship between the coordinate system of the image acquisition unit and the coordinate system of the end flange of the robotic arm, output the spatial pose of the pin hole in the coordinate system of the end flange of the robotic arm.

[0061] The coordinate transformation process follows the classic rigid body transformation chain: T_B_target = T_B_E × T_E_C × T_C_target; Where T_B_E is the end flange pose calculated in real time by the positive kinematics of the robotic arm, T_E_C is the hand-eye calibration matrix, T_C_target is the spatial pose of the pin hole in the coordinate system of the image acquisition unit, and T_B_target is the spatial pose of the pin hole in the coordinate system of the end flange of the robotic arm.

[0062] This step employs the Tsai-Lenz hand-eye calibration algorithm, which obtains the precise rigid body transformation relationship between the image acquisition unit coordinate system (camera coordinate system) and the robotic arm end flange coordinate system by solving the homogeneous transformation equation system AX=XB. Considering that the mechanical body and camera mounting bracket may undergo minor structural deformations due to factors such as vibration and temperature changes, this system is designed with a periodic recalibration mechanism: before each reassembly operation or after a periodic interval, the system automatically executes a fast hand-eye calibration program to update the hand-eye calibration matrix and eliminate accumulated drift errors.

[0063] Step 6: Based on the spatial pose of the pin hole 16 in the flange coordinate system at the end of the robotic arm and the point cloud data collected by the depth camera, generate the motion path of the robotic arm 1. The robotic arm 1 makes the closed pin 6 on the gripper component 4 reach the pre-insertion position. Then, the two grippers 7 open and push the closed pin 6 completely into the pin hole 16 through the push rod 14.

[0064] In this step, when generating the robotic arm's movement path, the 3D point cloud data acquired in real time by the depth camera is converted into a 3D occupancy grid map, and an obstacle envelope model within the robotic arm's workspace is constructed in real time to ensure that the planned movement path does not collide with surrounding obstacles. Based on the MoveIt motion planning library in the ROS framework (Robot Operating System), RRT is used. It uses sampling-based planning algorithms such as PRM to search for collision-free optimal paths in the joint configuration space of the robotic arm.

[0065] This step, when installing the closed pin, includes two stages: The first stage is the coarse approach stage, in which the robotic arm moves rapidly in position control mode to the pre-insertion position directly in front of the mounting hole (about 20-30mm from the pin hole); the second stage is the fine insertion stage, in which the control mode seamlessly switches to admittance impedance control mode (force-position hybrid control), and the closed pin is slowly inserted into the pin hole in a compliant manner by the opening of the grippers and the pushing action of the push rod.

[0066] The specific method for fully pushing the closed pin into the pin hole is as follows: N1: The six-dimensional torque sensor at the end of the robotic arm reads the contact force and torque data in each direction during insertion in real time, and preprocesses the collected raw sensor data. Specifically, the six-dimensional torque sensor at the end of the arm reads the contact force and torque data in each direction (x / y / z translational directions and torque directions around the three axes) during insertion at a sampling frequency of 500Hz. The collected raw force signals are low-pass filtered to remove high-frequency measurement noise, and the gravity compensation value (i.e., deducting the components of the end effector and the closing pin's own weight in each direction) is subtracted to obtain the pure external contact force / torque.

[0067] N2: Compare the force and torque data in each direction with preset safety thresholds; when the force and torque in all directions do not exceed the safety thresholds, the system will steadily advance the closing pin along the insertion direction at the set feed speed; when the force or torque in any direction exceeds the safety threshold (e.g., due to wind deviation, minor visual errors, or burrs on the inner wall of the bolt pin hole causing the closing pin to be squeezed against the hole wall), the system determines that a contact collision has occurred and triggers compliance compensation; during compliance compensation, the system first calculates the displacement compensation amount X and velocity compensation amount that the end of the robotic arm should produce based on the contact force F_ext detected by the sensor that exceeds the safety threshold. The specific calculation formula is as follows: ; In the above equation, M, B, and K represent the desired mass matrix, damping matrix, and stiffness matrix of the system, respectively. The positional deviation is output as the displacement compensation amount. ； This is the first derivative of the position deviation, i.e., the velocity deviation, and is output as the velocity compensation quantity. This is the second derivative of the position deviation, i.e., the acceleration deviation; The matrix of parameters M, B, and K is determined by offline calibration experiments based on the specific dimensions of the closed pin and the mounting hole (maximum diameter of the closed pin, diameter of the pin hole, and fit clearance) and material properties (elastic modulus of steel, coefficient of friction, etc.). Specifically, a lower stiffness K_z value is set in the insertion direction (Z-axis) to allow smooth feed motion, and moderate stiffness K_x and K_y values ​​are set in the plane perpendicular to the insertion direction (X / Y-axis) to achieve compliant clearance during collision. The damping parameter B is determined based on the critical damping condition to avoid oscillation.

[0068] The calculated displacement and velocity compensation values ​​are then superimposed onto the original motion command, and the displacement and velocity compensation are executed by the robotic arm.

[0069] Furthermore, if the contact force in a certain direction is detected to still exceed the safety threshold after compliance compensation, the robotic arm automatically retracts in the opposite direction to the pre-insertion position. Then, the system re-acquires the image and updates the spatial pose of the pin hole. The robotic arm then re-executes the pin hole insertion process based on the updated spatial pose of the pin hole.

[0070] If the system fails to insert the pin successfully after retrying more than the preset number of times (e.g., 3 times), it will pause the operation and issue an alarm, awaiting manual intervention. This mechanism effectively prevents the closed pin from bending or the bolt from being damaged due to forced insertion.

[0071] Compared with existing technologies, the present invention achieves the following advantages: 1. This invention completely replaces manual tower climbing for live-line work, avoiding fatal safety hazards such as high-voltage electric shock and falls from heights faced by workers. It eliminates the need for personnel to enter high-voltage danger zones, and the entire installation process can be completed automatically. It also reduces the skill requirements for workers and improves work efficiency.

[0072] 2. This invention employs algorithms such as 2D bounding box extraction, pixel-level mask segmentation, and 3D model matching to achieve accurate calculation of the spatial pose of the pin hole 16, with alignment accuracy far exceeding that of manual visual operation. Pixel-level mask segmentation effectively eliminates interference from wires, spacers, other components, complex lighting, and background, enabling stable target identification even in partially obscured outdoor environments. The use of pixel-level aligned RGB-D images ensures accurate depth information for each pixel in the RGB image, providing a reliable data foundation for 3D pose calculation. Ultimately, this enables high-precision installation of the closed pin 6, significantly increasing the probability of successful installation on the first attempt. This effectively solves the problem of traditional ground-based remote operation requiring multiple installations for successful insertion, significantly improving the installation efficiency of the closed pin 6.

[0073] 3. The installation process incorporates open vocabulary object detection algorithms and prompt-based image segmentation large models, breaking the bottleneck of traditional machine vision's heavy reliance on large amounts of on-site data collection, annotation, and model retraining. This enables the installation scheme of this invention to have excellent zero-shot or few-shot generalization capabilities, and can easily cope with complex unstructured environments such as different tower types, different lighting conditions, and aging and rusting of bolts, greatly reducing the cost of algorithm deployment and maintenance.

[0074] 4. During the insertion control of the closed pin 6, the detection feedback data of the six-dimensional torque sensor and the compliance compensation mechanism are introduced to upgrade the traditional "rigid position control" to "force-position hybrid control", which perfectly simulates the tactile fine adjustment ability of human hands during precision assembly. Even under dynamic interference such as slight wind deviation, it can still ensure an extremely high success rate of closed pin 6 installation.

[0075] This invention is not limited to the preferred embodiments described above. Anyone can derive other products in various forms under the guidance of this invention. However, regardless of any changes in shape or structure, any technical solution that is the same as or similar to this application falls within the protection scope of this invention.

Claims

1. An automatic installation device for bolt anti-loosening locking pins based on visual spatial positioning, characterized in that, The system includes a control module and a robotic arm. The end effector of the robotic arm is equipped with a gripper component and an image acquisition unit. The image acquisition unit is used to acquire RGB and depth images of the area in front of the robotic arm. The gripper component includes two grippers for clamping the two sides of a closing pin. A push rod is set between the two grippers, and one end of the closing pin to be installed is connected to the front end of the push rod. A linkage mechanism is set between the push rod and the two grippers to enable the two grippers to drive the push rod and the closing pin forward during the opening process. A six-dimensional torque sensor is set between the end effector of the robotic arm and the gripper component. The robotic arm moves the closed pin to the pre-insertion position directly in front of the pin hole on the bolt. The clamping power unit drives the two jaws to open, so that the push rod moves the closed pin forward, thereby achieving the complete insertion of the closed pin. The linkage mechanism includes a guide rail fixedly mounted on the gripper component and a push rod connecting block slidably connected to the guide rail. A connecting rod is provided between the two grippers and the push rod connecting block, and the two ends of the connecting rod are respectively hinged to the push rod connecting block and the grippers. The gripper component drives the two grippers to move relative to each other in a direction perpendicular to the guide rail.

2. The automatic installation device for bolt anti-loosening locking pins based on visual spatial positioning according to claim 1, characterized in that, The gripper component is an electric gripper, and the image acquisition unit is a depth camera.

3. An automatic installation method for bolt anti-loosening locking pins based on visual spatial positioning, based on the automatic installation device for bolt anti-loosening locking pins based on visual spatial positioning as described in any one of claims 1-2, characterized in that, The specific steps include the following: Step 1: The image acquisition unit acquires and obtains RGB-D image frames of the scene in front in real time. Each RGB-D image frame contains an RGB image and a depth image that is pixel-aligned with it. Step 2: Extract the two-dimensional bounding box of the target bolt that needs to be replaced based on the RGB image; Step 3: Extract the pixel-level mask of the target bolt based on the RGB image, the two-dimensional bounding box coordinates of the pin hole, and the two-dimensional bounding box coordinates of the target bolt, so as to accurately identify each pixel in the image that belongs to the target bolt; Step 4: Based on the depth image, the pixel-level mask of the target bolt, and the pre-built 3D model of the target bolt, output the spatial pose of the pin hole on the target bolt in the coordinate system of the image acquisition unit. Step 5: Based on the spatial pose in the coordinate system of the image acquisition unit and the transformation relationship between the coordinate system of the image acquisition unit and the coordinate system of the end flange of the robotic arm, output the spatial pose of the pin hole in the coordinate system of the end flange of the robotic arm. Step 6: Based on the spatial pose of the pin hole in the flange coordinate system at the end of the robotic arm and the point cloud data collected by the depth camera, generate the motion path of the robotic arm. The robotic arm moves the closed pin on the gripper component to the pre-insertion position, and then the two grippers open and push the closed pin completely into the pin hole through the push rod.

4. The automatic installation method for bolt anti-loosening locking pins based on visual spatial positioning according to claim 3, characterized in that, The specific method for step two is as follows: S21. Based on the application scenario characteristics of bolts, construct a set of scenario-based text prompts; S22. Input the RGB image and the set of text prompts into the open vocabulary object detection model. The model searches for regions in the RGB image that match the semantics of the text description through a visual-language alignment mechanism, and outputs the two-dimensional bounding boxes of all candidate bolts and their corresponding confidence scores. S23. Depth Range Filtering: For each candidate 2D bounding box, extract the average depth value of its corresponding region on the depth image, remove 2D bounding boxes with obviously unreasonable depths, and retain 2D bounding boxes within a reasonable working distance. S24. Automatic determination of missing pin status: For each candidate bolt retained after filtering, the depth map information is used to determine whether the closed pin on it has fallen off. If the closed pin on the current candidate bolt has fallen off, then the bolt is marked as the target bolt that needs to be replaced with a new pin. S25. From the marked target bolts to be replaced, select the target bolt closest to the robotic arm as the current working target, and output the two-dimensional bounding box of the target bolt.

5. The automatic installation method for bolt anti-loosening locking pins based on visual spatial positioning according to claim 4, characterized in that, In step S24, when using depth map information to determine whether the closed pin on the bolt has fallen off, the specific method is as follows: based on the two-dimensional bounding box and geometric prior of the bolt, locate the image sub-region where the pin hole is located, and extract the depth pixel value corresponding to the image sub-region; if the depth change value of the image sub-region exceeds the set threshold, it is determined that the closed pin of the bolt has fallen off and is marked as the target bolt that needs to be re-pinned; otherwise, no marking is required.

6. The automatic installation method for bolt anti-loosening locking pins based on visual spatial positioning according to claim 3, characterized in that, In step three, a prompt-based image segmentation model is used to extract the pixel-level mask of the target bolt. The specific method is as follows: S31. Input the RGB image into the image encoder of the large image segmentation model to generate high-dimensional image feature embeddings; S32. Use the two-dimensional bounding box as the bounding box cue input image segmentation model's cue encoder to generate cue position embeddings; S33. The high-dimensional image features and the prompt location are jointly input into the mask decoder of the large image segmentation model. The mask decoder fuses the two types of information through a cross-attention mechanism and outputs a candidate pixel-level mask of the target bolt. The pixel-level mask with the highest confidence level is selected from the candidate pixel-level masks as the final result.

7. The automatic installation method for bolt anti-loosening locking pins based on visual spatial positioning according to claim 3, characterized in that, In step six, when generating the movement path of the robotic arm, the 3D point cloud data collected in real time by the depth camera is converted into a 3D occupancy grid map, and an obstacle envelope model in the workspace of the robotic arm is constructed in real time to ensure that the planned movement path does not collide with surrounding obstacles.

8. The automatic installation method for bolt anti-loosening locking pins based on visual spatial positioning according to claim 3, characterized in that, In step six, when fully pushing the closed pin into the pin hole, the specific method is as follows: N1: The six-dimensional torque sensor at the end of the robotic arm reads the contact force and torque data in each direction in real time during the insertion process; The collected raw sensor data is then preprocessed. N2: Compare the force and torque data in each direction with the preset safety thresholds; when the force and torque in all directions do not exceed the safety thresholds, the system will steadily advance the closing pin along the insertion direction at the set feed speed; when the force or torque in any direction exceeds the safety threshold, the system determines that a contact collision has occurred and triggers compliance compensation; during compliance compensation, the system first calculates the position and velocity deviations that the robotic arm end should produce based on the contact force F_ext detected by the sensor that exceeds the safety threshold. The specific calculation formula is as follows: ； In the above equation, M, B, and K represent the desired mass matrix, damping matrix, and stiffness matrix of the system, respectively. The positional deviation is output as the displacement compensation amount. ； The first derivative of the position deviation is used as the output velocity compensation quantity. The second derivative of the positional deviation; The calculated displacement and velocity compensation values ​​are then superimposed onto the original motion command, and the displacement and velocity compensation are executed by the robotic arm.

9. The automatic installation method for bolt anti-loosening locking pins based on visual spatial positioning according to claim 8, characterized in that, In step six, if the contact force in a certain direction is detected to still exceed the safety threshold after compliance compensation, the robotic arm automatically retracts in the opposite direction to the pre-insertion position. Then, the system re-acquires the image and updates the spatial pose of the pin hole. The robotic arm then re-executes the pin hole insertion process based on the updated spatial pose of the pin hole.

Images