Double-arm collaborative bolt alignment method and system based on hybrid visual servo

Abstract

The invention discloses a double-arm cooperative bolt alignment method and system based on hybrid visual servo, and relates to the field of robot visual servo and cooperative control, and the method comprises a visual perception module which is used for obtaining part space pose information based on first image information of a part where a target bolt is located; according to the second image information of the target bolt, obtaining a two-dimensional image coordinate and space depth information of a bolt angular point of the target bolt; the hybrid visual servo control module is used for generating a motion control instruction of the main mechanical arm by utilizing a staged visual alignment control strategy based on the spatial pose information of the part and the two-dimensional image coordinates and spatial depth information of the angular points of the bolts; and the double-arm cooperative control module is used for generating a cooperative motion instruction of the slave mechanical arm based on a cooperative control algorithm in the process that the master mechanical arm carries out visual servo alignment according to the motion control instruction. According to the method, the bolt alignment task under the complex working condition can be completed in a self-adaptive mode, and the intelligent level of overhauling operation is improved.

Description

Technical Field

[0001] This application relates to the field of robot vision servoing and cooperative control, and in particular to a dual-arm cooperative bolt alignment method and system based on hybrid vision servoing. Background Technology

[0002] Currently, robotic systems face challenges such as high environmental uncertainty and inherent perception errors when performing precision operations like bolt tightening in unstructured environments like industrial manufacturing and equipment maintenance. Their operational accuracy and environmental adaptability urgently need improvement. While visual servoing technology is widely used in such precision operations, existing mainstream methods—position-based visual servoing (PBVS) and image-based visual servoing (IBVS)—have significant limitations in practical bolt alignment tasks. For the PBVS method, the control error is defined in three-dimensional Cartesian space, resulting in faster convergence. However, its control performance heavily relies on the accuracy of the target's three-dimensional pose estimation. In small-scale target scenarios like bolts, effective visual features for estimation are scarce, leading to poor pose estimation reliability, amplified errors, and difficulty in achieving stable convergence at large initial distances. For the IBVS method, the control error is directly defined in the image plane, resulting in higher control accuracy. However, it is extremely sensitive to noise and bias in feature extraction, easily leading to decreased control performance or even servo failure in complex environments, and is also unsuitable for long-distance alignment. Furthermore, for dual-arm collaborative operation modes that improve operational stability and expand operating space, existing technologies lack an effective and universal solution for achieving coordinated movement, synchronous alignment, and stable following of the two robotic arms under visual servo guidance.

[0003] Therefore, how to design a visual servoing method that can integrate the advantages of PBVS and IBVS and be effectively applied to bolt alignment control systems for dual-arm collaborative operations is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the present invention provides a dual-arm collaborative bolt alignment method and system based on hybrid vision servoing, which overcomes the above-mentioned defects.

[0005] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a dual-arm collaborative bolt alignment system based on hybrid vision servoing, comprising: The visual perception module is used to acquire first image information of the component where the target bolt is located, and to identify and extract features from the first image information to obtain spatial pose information of the component; acquire second image information of the target bolt, and to extract features from the second image information to obtain two-dimensional image coordinates and spatial depth information of the bolt corner point of the target bolt; A hybrid vision servo control module is used to generate motion control commands for the main robotic arm based on the spatial pose information of the component and the two-dimensional image coordinates and spatial depth information of the bolt corner point, using a phased vision alignment control strategy. The phased vision alignment control strategy includes a coarse alignment control strategy, which generates a first motion control command based on the pose error of the component where the target bolt is located using the PBVS method; and a precise alignment control strategy, which generates a second motion control command based on the image feature error of the target bolt using the IBVS method. The dual-arm collaborative control module is used to generate collaborative motion commands for the slave arm based on the real-time spatial pose information of the master robotic arm during the visual servo alignment process of the master robotic arm according to the first motion control command and the second motion control command. The control execution module is used to manipulate the main robotic arm according to the first motion control command and the second motion control command, and to manipulate the slave robotic arm according to the cooperative motion command, so as to complete the visual servo alignment operation of the two arms in coordination.

[0006] Optionally, the visual perception module includes: The 3D model extraction unit is used to extract the 3D geometric model of the component from a pre-built component model library based on the maintenance operation requirements. The spatial pose localization and tracking unit is used to perform visual recognition and feature extraction on the component based on the three-dimensional geometric model of the component, use the PnP method for initial localization, and use a model tracking-based pose estimation method to continuously visually track the component, update the spatial pose information of the component, and obtain the continuous spatial pose information of the component relative to the camera coordinate system. The bolt corner detection unit is used to perform target detection on the second image information, identify and locate the target bolt, and extract the two-dimensional image coordinates of the bolt corner. The bolt corner depth estimation unit is used to obtain the spatial depth information of the bolt corner point through a depth estimation algorithm based on the two-dimensional image coordinates of the bolt corner point.

[0007] Optionally, the spatial pose localization and tracking unit includes: The initial pose estimation subunit is used to perform visual recognition and feature extraction on the component based on the three-dimensional geometric model of the component, and to obtain the initial spatial pose of the component through the PnP method. The continuous pose tracking subunit is used to continuously visually track the component based on the first image information, the three-dimensional geometric model of the component, and the pose estimation result of the previous frame, using a model-based pose estimation method, thereby obtaining the continuous spatial pose information of the component relative to the camera coordinate system.

[0008] Optionally, the bolt corner detection unit includes: The target detection subunit is used to perform target detection on the second image information and obtain the region of interest of the target bolt in the second image information; A mask extraction subunit is used to perform image segmentation in the region of interest to obtain a pixel-level mask of the target bolt region; The feature point extraction subunit is used to perform geometric structure analysis on the target bolt under the constraint of a pixel-level mask in the target bolt region, and extract the two-dimensional image coordinates of the bolt corner points of the target bolt.

[0009] Optionally, the bolt corner depth estimation unit includes: The second initial pose estimation subunit is used to construct a homography mapping relationship based on the two-dimensional image coordinates of the bolt corner points and the bolt size to obtain the initial spatial pose of the target bolt relative to the camera coordinate system, i.e., the second initial pose. The depth information extraction subunit is used to optimize the second initial pose using VVS, and to calculate the spatial depth information of each bolt corner point in the camera coordinate system based on the optimized second initial pose and the bolt size.

[0010] Optionally, the hybrid vision servo control module includes: The first vision alignment control unit is used to generate the first motion control command by adopting the PBVS method and using the difference between the spatial pose information of the component and the preset desired alignment pose as feedback, so as to guide the main robotic arm to perform coarse vision alignment. The second vision alignment control unit is used to generate the second motion control command by employing the IBVS method and using the difference between the expected two-dimensional image coordinates of the bolt corner point and the two-dimensional image coordinates of the bolt corner point, as well as the spatial depth information, to guide the main robotic arm to perform precise vision alignment. The control mode switching unit is used to switch the PBVS method to the IBVS method when a preset switching condition is met; the switching condition is based on a threshold setting for coarse visual alignment.

[0011] Optionally, the dual-arm collaborative control module includes: The base calibration unit is used to fix the end effector of the main robotic arm and the end effector of the slave robotic arm through a dedicated calibration connector. Based on the dedicated calibration connector, it obtains the pose information of the TCP of the main robotic arm end relative to the TCP of the slave robotic arm end. Based on the RTDE protocol, it obtains the pose information of the TCP of the main robotic arm end relative to the main robotic arm base and the pose information of the TCP of the slave robotic arm end relative to the slave robotic arm base, and calculates the relative pose relationship between the main robotic arm base and the slave robotic arm base. An end-effector constraint unit is used to store the end-effector TCP pose constraint relationship, which includes attitude constraint and position constraint. The attitude constraint is that the desired attitude of the slave end-effector TCP is rotated 180 degrees relative to the master end-effector TCP about the y-axis of the master end-effector TCP coordinate system. The position constraint is that the master end-effector TCP and the slave end-effector TCP move symmetrically along the z-axis of the master end-effector TCP coordinate system and the slave end-effector TCP coordinate system, respectively, during the visual servo alignment process. The slave arm pose generation unit is used to collect real-time spatial pose information of the end effector TCP of the main robotic arm relative to the main robotic arm base in real time based on the RTDE protocol during the visual servo alignment process of the main robotic arm according to the first motion control command and the second motion control command; calculate the pose information of the slave robotic arm end effector TCP relative to the main robotic arm base based on the pose constraint relationship of the end effector TCP; and calculate the pose information of the slave robotic arm end effector TCP relative to the slave robotic arm base through the relative pose relationship between the main robotic arm base and the slave robotic arm base, that is, the desired pose of the slave robotic arm end effector TCP, and generate the cooperative motion command of the slave robotic arm.

[0012] Secondly, this application provides a dual-arm collaborative bolt alignment method based on hybrid vision servoing, the specific steps of which are as follows: S1. Obtain the first image information of the component where the target bolt is located, and perform recognition and feature extraction on the first image information to obtain the spatial pose information of the component; S2. Using the PBVS method, the difference between the spatial pose information of the component and the preset desired alignment pose is used as feedback to generate a first motion control command to guide the main robotic arm to perform coarse visual alignment. S3. During the coarse visual alignment process, the first cooperative motion command of the slave robot is generated based on the real-time spatial pose information of the master robot based on the cooperative control algorithm, so as to guide the slave robot to perform the first cooperative operation. S4. Based on preset switching conditions, after determining that coarse visual alignment is completed, the second image information of the target bolt is obtained, and feature extraction is performed on the second image information to obtain the two-dimensional image coordinates and spatial depth information of the bolt corner point of the target bolt; the switching conditions are based on the threshold setting of coarse visual alignment. S5. Using the IBVS method, and taking the difference between the expected two-dimensional image coordinates of the bolt corner point and the two-dimensional image coordinates of the bolt corner point, as well as the spatial depth information, as feedback, a second motion control command is generated to guide the main robotic arm to perform precise visual alignment. S6. During the precise visual alignment process, a second collaborative motion command for the slave robotic arm is generated based on the real-time spatial pose information of the master robotic arm according to the collaborative control algorithm, guiding the slave robotic arm to perform a second collaborative operation and completing the visual servo alignment operation of the two arms.

[0013] Optionally, the step of obtaining the spatial pose information of the components in S1 is as follows: S11. Search the component model library according to the maintenance operation requirements and extract the three-dimensional geometric model of the component; S12. Based on the three-dimensional geometric model of the component, perform visual recognition and feature extraction on the component, and obtain the initial spatial pose of the component through the PnP method; S13. Based on the first image information, the three-dimensional geometric model of the component, and the pose estimation result of the previous frame, the component is continuously visually tracked using a model tracking-based pose estimation method to obtain continuous spatial pose information of the component relative to the camera coordinate system.

[0014] Optionally, the specific steps of the first collaborative operation in S3 are as follows: S31. The end effector of the main robotic arm and the end effector of the slave robotic arm are fixedly connected by a dedicated calibration connector. Based on the dedicated calibration connector, the pose information of the TCP of the main robotic arm end relative to the TCP of the slave robotic arm end is obtained. Based on the RTDE protocol, the pose information of the TCP of the main robotic arm end relative to the main robotic arm base and the pose information of the TCP of the slave robotic arm end relative to the slave robotic arm base are obtained. The relative pose relationship between the main robotic arm base and the slave robotic arm base is calculated. S32. During the process of the main robotic arm performing visual servo alignment according to the first motion control command, the real-time spatial pose information of the TCP end of the main robotic arm relative to the main robotic arm base is collected in real time based on the RTDE protocol. S33. Based on the real-time spatial pose information of the master robot arm's end-effector TCP relative to the master robot arm base and the preset end-effector TCP pose constraint relationship, calculate the pose information of the slave robot arm's end-effector TCP relative to the master robot arm base, and calculate the pose information of the slave robot arm's end-effector TCP relative to the slave robot arm base through the relative pose relationship between the master robot arm base and the slave robot arm base, i.e., the desired pose of the slave robot arm's end-effector TCP, and generate the first cooperative motion command of the slave robot arm; wherein, the end-effector TCP pose constraint relationship includes attitude constraint and position constraint, the attitude constraint is that the desired pose of the slave robot arm's end-effector TCP is rotated 180 degrees relative to the master robot arm's end-effector TCP around the y-axis of the master robot arm's end-effector TCP coordinate system, and the position constraint is that the master robot arm's end-effector TCP and the slave robot arm's end-effector TCP move symmetrically along the z-axis of the master robot arm's end-effector TCP coordinate system and the slave robot arm's end-effector TCP coordinate system, respectively, during the visual servo alignment process; S34. Guide the robotic arm to perform the first collaborative operation according to the first collaborative motion command.

[0015] Optionally, the specific steps of S4 are as follows: S41. Acquire the second image information and perform target detection on the second image information to obtain the region of interest of the target bolt in the second image information; S42. Perform image segmentation on the region of interest to obtain a pixel-level mask of the target bolt region; S43. Under the constraint of the pixel-level mask of the target bolt area, perform geometric structure analysis on the target bolt and extract the two-dimensional image coordinates of the bolt corner points of the target bolt; S44. Based on the two-dimensional image coordinates of the bolt corner points and the homography mapping relationship constructed by the bolt size, the initial spatial pose of the target bolt relative to the camera coordinate system is obtained, namely the second initial pose. S45. The second initial pose is optimized using VVS, and the spatial depth information of each bolt corner point in the camera coordinate system is calculated based on the optimized second initial pose and the bolt size.

[0016] According to the specific embodiments provided in this application, this application has the following technical effects: 1) This invention constructs a hybrid vision servo control system and applies the PBVS method and IBVS method in stages. At the system level, it achieves a smooth transition from large-scale guidance to high-precision alignment. This ensures the system's rapid convergence capability in complex unstructured environments while also taking into account the alignment accuracy under close-range working conditions, effectively improving the overall efficiency and stability of the bolt vision alignment process.

[0017] 2) By designing a dual-arm collaborative control system, the present invention enables the slave robotic arm to synchronously follow the motion state of the master robotic arm in real time under the condition of satisfying the preset spatial constraints. Without increasing the additional perception burden, the invention achieves coordinated and consistent movement of the two arms during the bolt alignment process, effectively improving the stability and alignment consistency of the dual-arm collaborative operation.

[0018] 3) The system structure of this invention is clear and the modules are well-defined. It can be deployed and expanded on different models of multi-degree-of-freedom robotic arms and camera platforms. It has good versatility and engineering applicability and is suitable for overhead contact line maintenance, industrial assembly and other operation scenarios that require high-precision dual-arm collaborative alignment. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram of a three-dimensional geometric model of a component provided in an embodiment of this application; Figure 2 A schematic diagram of manually selected component feature points in an image plane, provided as an embodiment of this application; Figure 3 A schematic diagram showing the initial spatial pose estimation result of a component provided in an embodiment of this application; Figure 4 This is a schematic diagram of the total position error variation curve between the camera and components during the PBVS servo process, provided in an embodiment of this application. Figure 5 This is a schematic diagram of the total attitude error variation curve between the camera and components during the PBVS servo process, provided in an embodiment of this application. Figure 6 (a) is a camera field of view image before PBVS servoing starts according to an embodiment of this application; (b) is a camera field of view image during PBVS servoing according to an embodiment of this application; (c) is a camera field of view image after PBVS servoing is completed according to an embodiment of this application. Figure 7(a) is a schematic diagram of second image data provided in an embodiment of this application; (b) is a schematic diagram of the region of interest of the target bolt in the image provided in an embodiment of this application; (c) is a schematic diagram of the pixel-level mask of the target bolt region provided in an embodiment of this application; (d) is a schematic diagram of bolt mask geometric analysis provided in an embodiment of this application; (e) is a schematic diagram of bolt corner points on the image plane provided in an embodiment of this application. Figure 8 This is a diagram showing the spatial depth estimation result of the bolt corner point in an IBVS servo process according to an embodiment of this application. Figure 9 A schematic diagram of the curve showing the change of total system error over time during the IBVS servo process, provided as an embodiment of this application; Figure 10 (a) is a camera view before IBVS servoing starts according to an embodiment of this application; (b) is a camera view during IBVS servoing according to an embodiment of this application; (c) is a camera view after IBVS servoing is completed according to an embodiment of this application. Figure 11 A schematic diagram illustrating the spatial pose relationship between the end effector TCPs of the master and slave robotic arms and their respective bases in a dual-arm robot system provided in an embodiment of this application; Figure 12 This is an overall block diagram of a dual-arm cooperative tracking control algorithm provided in an embodiment of this application; Figure 13 A schematic diagram illustrating the TCP pose constraint relationship of the dual-arm end effectors throughout the entire collaborative operation process, provided in an embodiment of this application. Figure 14 This application provides a TCP spatial motion trajectory diagram of the end effectors of the master and slave robotic arms during a collaborative operation process, as shown in one embodiment. Figure 15 (a) is a schematic diagram of the initial relative state between the master and slave robotic arms and the component before the start of the PBVS phase according to an embodiment of this application; (b) is a schematic diagram of the process state during the PBVS phase where the master robotic arm performs coarse visual alignment of the component under visual servo guidance and the slave robotic arm performs collaborative tracking; (c) is a schematic diagram of the state when the system is about to switch to the IBVS phase after the PBVS phase is completed, where the ends of the master and slave robotic arms have completed spatial alignment of the component and entered the close-range working area of ​​the bolt; (d) is a schematic diagram of the process state during the IBVS phase where the camera mounted on the end of the master robotic arm performs precise visual alignment of the target bolt and the slave robotic arm performs collaborative tracking; (e) is a schematic diagram of the state after the IBVS phase is completed where the ends of the master and slave robotic arms achieve precise visual alignment with the target bolt; (f) is a schematic diagram of the axial alignment of the end effector with the target bolt. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] This embodiment discloses a dual-arm collaborative bolt alignment system based on hybrid vision servoing, including: The visual perception module is used to acquire the first image information of the component where the target bolt is located, and to identify and extract features from the first image information to obtain the spatial pose information of the component; to acquire the second image information of the target bolt, and to extract features from the second image information to obtain the two-dimensional image coordinates and spatial depth information of the bolt corner point of the target bolt; The hybrid vision servo control module is used to generate motion control commands for the main robotic arm based on the spatial pose information of the parts and the two-dimensional image coordinates and spatial depth information of the bolt corner points, using a phased vision alignment control strategy. The phased vision alignment control strategy includes a coarse alignment control strategy, which uses position-based visual servoing (PBVS) to generate the first motion control command based on the pose error of the part where the target bolt is located; and a precise alignment control strategy, which uses image-based visual servoing (IBVS) to generate the second motion control command based on the image feature error of the target bolt. The dual-arm collaborative control module is used to generate collaborative motion commands for the slave arm based on the real-time spatial pose information of the master arm during the visual servo alignment process of the master arm according to the first motion control command and the second motion control command. The control execution module is used to manipulate the main robotic arm according to the first motion control command and the second motion control command, and to manipulate the slave robotic arm according to the cooperative motion command, so as to complete the visual servo alignment operation of the two arms in coordination.

[0024] Furthermore, the dual-arm collaborative bolt alignment system based on hybrid visual servoing disclosed in this embodiment includes a visual perception module, a hybrid visual servo control module, a dual-arm collaborative control module, and a control execution module. The visual perception module acquires visual information (including first and second image information) of the target bolt and its associated components. The hybrid visual servo control module, based on the acquired visual information, utilizes the control characteristics of both the PBVS and IBVS methods to achieve phased visual alignment control of the target bolt. During system operation, the hybrid visual servo control module first guides the relative pose between the camera mounted on the end effector of the main robotic arm and the component containing the target bolt using the PBVS method, achieving rapid and stable coarse visual alignment. Then, it switches to the IBVS method for precise visual alignment of the target bolt to ensure final operational accuracy. In addition, the dual-arm collaborative control module enables coordinated motion control between the master and slave robotic arms. The master robotic arm carries a camera and performs visual servo alignment tasks, while the slave robotic arm synchronously tracks the master robotic arm based on its real-time spatial pose information, thereby achieving stable bolt alignment under dual-arm collaborative conditions.

[0025] In one embodiment, the visual perception module includes: The 3D model extraction unit is used to extract the 3D geometric model of a component from a pre-built component model library based on maintenance operation requirements. The spatial pose localization and tracking unit is used to perform visual recognition and feature extraction of parts based on the three-dimensional geometric model of the parts. It adopts the Perspective-n-Point (PnP) method for initial localization and uses a model tracking-based pose estimation method to continuously visually track the parts, update the spatial pose information of the parts, and obtain the continuous spatial pose information of the parts relative to the camera coordinate system. The bolt corner detection unit is used to perform target detection on the second image information, identify and locate the target bolt, and extract the two-dimensional image coordinates of the bolt corner. The bolt corner depth estimation unit is used to obtain the spatial depth information of the bolt corner point through a depth estimation algorithm based on the two-dimensional image coordinates of the bolt corner point.

[0026] In one embodiment, the spatial pose localization and tracking unit includes: The initial pose estimation subunit is used to perform visual recognition and feature extraction of the parts based on the three-dimensional geometric model of the parts, and to obtain the initial spatial pose of the parts through the PnP method. The continuous pose tracking subunit is used to continuously visually track the component based on the first image information, the three-dimensional geometric model of the component, and the pose estimation result of the previous frame, using a model-based pose estimation method, thereby obtaining the continuous spatial pose information of the component relative to the camera coordinate system.

[0027] Furthermore, in the component alignment stage, the first image information of the component where the target bolt is located is acquired through the visual perception module, and the component spatial pose information for the PBVS method is extracted.

[0028] During system operation, the vision perception module performs real-time visual perception of the working environment, continuously acquiring image information (i.e., the first image information) containing the target bolt and its associated components. Given the fixed structural rules and geometric shapes of the components containing the target bolt, and the fact that their 3D models can be pre-acquired, a component model library containing 3D geometric models of various components is constructed using 3D modeling software (such as SolidWorks) before system deployment.

[0029] Furthermore, during the component alignment stage, the system extracts the 3D geometric model of the component containing the target bolt from a pre-built component model library based on maintenance requirements, such as... Figure 1 As shown, the three-dimensional geometric model of the parts and their structural parameters are stored in the system as geometric prior information.

[0030] The visual perception module, based on the extracted 3D geometric model of the components, performs visual recognition and feature extraction on the components. It uses the PnP method for initial localization and obtains continuous spatial pose information of the components relative to the camera coordinate system through model tracking-based pose estimation. Specifically, the system selects several feature point positions (i.e., points with obvious geometric features, such as corner points) on the components in the image plane through human-computer interaction. Figure 2 As shown, based on the corresponding spatial coordinates of image feature points in the 3D geometric model of the component, a correspondence between image feature points and model spatial points is established, thereby completing the estimation of the initial spatial pose of the component, as follows. Figure 3 As shown, after obtaining the initial spatial pose estimation result, the system further introduces a pose optimization mechanism based on error minimization to optimize and correct the initial spatial pose. This optimization mechanism uses the reprojection error of image feature points as the optimization objective. By iteratively adjusting the camera pose parameters, the error between the model's projected features and the observed features of the image is gradually reduced, thereby obtaining spatial pose estimation results with higher consistency and accuracy. Specifically, the Levenberg-Marquardt algorithm or gradient descent method can be used for implementation. This improves the accuracy and consistency of the initial spatial pose estimation, thus providing stable and reliable initial pose conditions for the subsequent pose tracking process.

[0031] Furthermore, the visual perception module employs a model-tracking-based pose estimation method for continuous visual tracking of components. Specifically, the system uses the pose estimation result of the previous frame as the initial value. By projecting the 3D geometric model of the component onto the first image information and establishing a correspondence between the model and the observed features of the first image information, the pose is updated using the Virtual Visual Servoing (VVS) method. The VVS method transforms the pose estimation process into a virtual visual servoing process. By constructing a virtual camera and minimizing the error between the model projection features and the observed features, the virtual camera is driven to generate corresponding motion, thereby achieving iterative updates of the component's spatial pose information and completing continuous and smooth tracking of the component. Based on the virtual visual servoing concept, the pose update mechanism establishes a stable error feedback relationship between the model projection features and the observed features, giving the pose update process good convergence and stability, thus effectively reducing the impact of illumination changes, local occlusion, and noise on the continuous pose tracking process. Finally, the obtained six-dimensional spatial pose information of the parts relative to the camera coordinate system (i.e., the spatial pose information of the parts) is output to the hybrid vision servo control module as input to the PBVS method, providing stable and reliable pose information for the subsequent alignment between the main robotic arm and the parts.

[0032] In one embodiment, the bolt corner detection unit includes: The target detection subunit is used to perform target detection on the second image information and obtain the region of interest of the target bolt in the second image information; The mask extraction subunit is used to perform image segmentation in the region of interest to obtain a pixel-level mask of the target bolt region; The feature point extraction subunit is used to perform geometric structure analysis on the target bolt under the constraint of a pixel-level mask in the target bolt region, and extract the two-dimensional image coordinates of the bolt corner points of the target bolt.

[0033] In one embodiment, the bolt corner depth estimation unit includes: The second initial pose estimation subunit is used to construct a homography mapping relationship based on the two-dimensional image coordinates of the bolt corner points and the bolt size to obtain the initial spatial pose of the target bolt relative to the camera coordinate system, i.e., the second initial pose. The depth information extraction subunit is used to optimize the second initial pose using VVS, and to calculate the spatial depth information of each bolt corner point in the camera coordinate system based on the optimized second initial pose and the bolt size.

[0034] Furthermore, after the coarse visual alignment is completed, the system enters the precise visual alignment. The visual perception module acquires the image information of the target bolt (i.e., the second image data) and extracts the two-dimensional image coordinates of the bolt corner points and their corresponding spatial depth information for the IBVS method.

[0035] Furthermore, after PBVS guidance is completed, the camera mounted on the end of the main robotic arm has moved to the close working area of ​​the target bolt. At this time, the relative distance between the camera and the target bolt is within the range suitable for fine visual perception, and the scale of the target bolt in the image plane is significantly increased, which is beneficial for subsequent high-precision feature extraction and alignment control.

[0036] like Figure 7 As shown, in the bolt visual perception stage, the visual perception module first processes the acquired second image data as follows: Figure 7 As shown in (a), target detection is performed using a Yolov7 deep learning model. First, the target bolt in the second image data is identified and located, thus obtaining the region of interest (ROI) of the target bolt in the second image data, as shown in (a). Figure 7 As shown in (b). Based on this, the system further extracts a pixel-level mask of the target bolt region using image segmentation within the region of interest, as shown in... Figure 7 As shown in (c), it is used to precisely define the effective sensing range of the bolt.

[0037] Subsequently, under the constraint of a pixel-level mask in the target bolt area, a line segment detector (LSD) is used to analyze the geometric features of the target bolt head. Figure 7 As shown in (d), the two-dimensional image coordinates of the bolt corner point in the image plane are determined as follows: Figure 7 As shown in (e). Through the above processing, the system can achieve stable and reliable extraction of bolt corner features of target bolts under complex background conditions.

[0038] Furthermore, after obtaining the two-dimensional image coordinates of the bolt corner points, the visual perception module needs to further acquire the spatial depth information corresponding to the bolt corner points. Considering that the bolt head structure can be approximated as a planar structure within a local range, the spatial depth information of the bolt corner points can be obtained by establishing a homography mapping relationship between the image plane and the bolt plane. Specifically, the system uses the extracted two-dimensional image coordinates of the bolt corner points, combined with the actual size coordinates of the bolt corner points in the bolt plane coordinate system (i.e., bolt size), and uses the Direct Linear Transformation (DLT) method to establish multiple sets of correspondence equations between image pixels and bolt plane points. Solving the equation set determines the homography matrix between the image plane and the bolt plane. This homography matrix, combined with the camera perspective projection principle and camera intrinsic parameters, can obtain the initial spatial pose of the target bolt relative to the camera coordinate system, i.e., the second initial pose; including rotation and translation parameters. After obtaining the second initial pose, the system also introduces a VVS pose optimization mechanism. By constructing a virtual camera and minimizing the error between the projected feature points and the observed feature points in the image, the virtual camera is driven to generate corresponding motion, thereby iteratively updating the second initial pose to improve the accuracy of the second initial pose estimation result. Finally, based on the optimized second initial pose and combined with the actual size coordinates of the bolt corner points in the bolt plane coordinate system, the system determines the spatial coordinates of the bolt corner points in the camera coordinate system through matrix operations, thus obtaining the spatial depth information corresponding to each bolt corner point. Figure 8 The image shown is a spatial depth estimation result of the bolt corner point in a specific embodiment of the IBVS servo process. Analysis of the spatial depth estimation results of the bolt corner point in the initial and converged states reveals that the depth estimation error is controlled within approximately 2 cm, which meets the depth information accuracy requirements of the subsequent IBVS method.

[0039] Through the above processing, the visual perception module obtains the two-dimensional image coordinates of the bolt corner points and their corresponding spatial depth information for the IBVS method, and outputs them as visual feedback quantities to the hybrid visual servo control module, providing stable and reliable perception input for the subsequent IBVS method.

[0040] In one embodiment, the hybrid vision servo control module includes: The first vision alignment control unit is used to generate a first motion control command by using the PBVS method and the difference between the spatial pose information of the component and the preset desired alignment pose as feedback, so as to guide the main robotic arm to perform coarse vision alignment. The second vision alignment control unit is used to generate a second motion control command by using the IBVS method and the difference between the expected two-dimensional image coordinates of the bolt corner and the two-dimensional image coordinates of the bolt corner, as well as spatial depth information, to guide the main robotic arm to perform precise vision alignment. The control mode switching unit is used to switch the PBVS method to the IBVS method when a preset switching condition is met; the switching condition is based on a threshold setting for coarse visual alignment.

[0041] Furthermore, the hybrid vision servo control module employs the PBVS method, using the pose error of the component containing the target bolt as feedback to generate the first motion control command. This command guides the relative pose between the camera mounted at the end of the main robotic arm and the component containing the target bolt, achieving rapid and stable coarse visual alignment. Specifically: The hybrid vision servo control module receives the spatial pose information of the target bolt component relative to the camera coordinate system from the vision perception module. This component spatial pose information is six-dimensional, and it is compared with a pre-set desired alignment pose to determine the pose error between the camera mounted on the end effector of the main robotic arm and the component. Based on this pose error, the hybrid vision servo control module uses the PBVS method to calculate the desired camera velocity according to a pre-designed velocity servo rate, thereby guiding the overall movement of the end effector of the main robotic arm in three-dimensional space. By progressively reducing the pose error between the camera and the component, the end effector of the main robotic arm can achieve rapid approach and alignment in spatial scale.

[0042] Furthermore, during the PBVS phase, the initial distance between the camera and the component is approximately 55 cm, and the working distance upon completion of the servo operation is approximately 35 cm. When the total positional error between the camera and the component is less than 1 mm and the total attitude error is less than 1 degree, the system determines that the PBVS phase convergence is complete, with a servo convergence time of approximately 8-10 seconds. The curve showing the change in the total positional error between the camera and the component during the PBVS servo process is shown below. Figure 4 As shown, the total attitude error variation curve between the camera and components during the PBVS servo process is as follows: Figure 5 As shown in the figure, under the control of PBVS, both the position error and attitude error gradually decrease as the servo process progresses and eventually converge to the preset threshold range, verifying the stability and effectiveness of the system in the initial alignment stage of the components. The camera field-of-view images before, during, and after the PBVS servo process are shown in the figure. Figure 6As shown in (a)-(c), the projected contours displayed in the figures are the 3D geometric models of the components obtained based on 3D reconstruction, used to characterize the system's real-time estimation results of the component's spatial pose. It can be seen that as the PBVS process progresses, the relative pose between the camera and the component gradually converges and tends to stabilize. When the pose deviation decreases to within a preset threshold range, the system determines that the PBVS stage has converged.

[0043] Furthermore, the hybrid vision servo control module switches to the IBVS method, using image feature error as feedback to achieve precise visual alignment between the camera mounted at the end of the main robotic arm and the target bolt; the specific processing steps are as follows: After acquiring the two-dimensional image coordinates and corresponding spatial depth information of the bolt corner points, the hybrid vision servo control module switches from the PBVS method to the IBVS method and receives the two-dimensional image coordinates and corresponding spatial depth information of the bolt corner points output by the vision perception module. The system uses the expected two-dimensional image coordinates of the target bolt in the desired alignment state as a reference feature, compares them with the two-dimensional image coordinates of the bolt corner points extracted at the current moment, and calculates the feature error of the bolt corner points in the image plane, i.e., the image feature error.

[0044] The hybrid vision servo control module constructs a control relationship with image feature errors as feedback, generating a second motion control command to drive the end effector of the main robotic arm. This command guides the camera mounted on the end effector to gradually reduce the bolt corner feature error within the image plane, thereby achieving precise alignment control between the camera and the target bolt. In this way, the system can directly adjust the bolt alignment error in the image feature space, effectively improving alignment accuracy under close-range working conditions.

[0045] Furthermore, during the IBVS bolt alignment stage, the initial distance between the camera and the target bolt is approximately 35 cm, the distance at servo completion is approximately 20 cm, and the overall servo distance is approximately 15 cm. When the total feature error of the bolt corner point in the image plane decreases to within a preset threshold range, the system determines that the IBVS stage has converged, with a servo convergence time of approximately 8-10 seconds. The curve showing the change of the total system error over time during the IBVS servo process is shown below. Figure 9 As shown in the diagram, under IBVS control, the total error of the bolt corner feature gradually decreases during the servo process and eventually converges to a stable state, verifying the stability and effectiveness of the system in the bolt precision alignment stage. A schematic diagram of the camera field of view change during the IBVS servo process is shown below. Figure 10As shown in (a)-(c), the figures depict the motion of four feature points on the bolt head in the image plane. It can be seen that as the IBVS servo process proceeds, each feature point gradually converges towards its desired position along the image plane, intuitively reflecting the stable tracking and convergence process of the bolt corner features by the system during the fine alignment stage.

[0046] Through the aforementioned precise alignment control process based on IBVS, the camera mounted at the end of the main robotic arm can achieve high-precision alignment with the target bolt, providing reliable positional conditions for subsequent bolt tightening and other operations.

[0047] In one embodiment, the dual-arm collaborative control module includes: The base calibration unit is used to fix the end effector of the main robotic arm and the end effector of the slave robotic arm through a dedicated calibration connector. Based on the dedicated calibration connector, it obtains the pose information of the TCP of the main robotic arm end relative to the TCP of the slave robotic arm end. Based on the Real-Time Data Exchange (RTDE) protocol, it obtains the pose information of the TCP of the main robotic arm end relative to the main robotic arm base and the pose information of the TCP of the slave robotic arm end relative to the slave robotic arm base, and calculates the relative pose relationship between the main robotic arm base and the slave robotic arm base. The end-effector constraint unit is used to store the end-effector TCP pose constraint relationship. The end-effector TCP pose constraint relationship includes attitude constraint and position constraint. The attitude constraint is that the desired attitude of the end-effector TCP of the robot arm rotates 180 degrees relative to the end-effector TCP of the main robot arm around the y-axis of the coordinate system of the end-effector TCP of the main robot arm. The position constraint is that the end-effector TCP of the main robot arm and the end-effector TCP of the robot arm move symmetrically along the z-axis of the coordinate system of the end-effector TCP of the main robot arm and the end-effector TCP of the robot arm respectively during the visual servo alignment process. The slave arm pose generation unit is used to collect real-time spatial pose information of the end effector TCP relative to the master arm base in real time based on the RTDE protocol during the visual servo alignment process of the master arm according to the first motion control command and the second motion control command; calculate the pose information of the slave arm end effector TCP relative to the master arm base based on the pose constraint relationship of the end effector TCP; and calculate the pose information of the slave arm end effector TCP relative to the slave arm base through the relative pose relationship between the master arm base and the slave arm base, that is, the desired pose of the slave arm end effector TCP, and generate the cooperative motion command of the slave arm.

[0048] Furthermore, during the visual servo alignment process performed by the main robotic arm, the dual-arm collaborative control module generates collaborative motion commands for the slave robotic arm based on the real-time spatial pose information of the main robotic arm, enabling the slave robotic arm to synchronously track the motion state of the main robotic arm. Specifically: During the hybrid vision servo alignment process performed by the master robotic arm, the dual-arm collaborative control module performs unified modeling and management of the spatial pose relationship between the master and slave robotic arms. A schematic diagram of the spatial pose relationship between the end effector TCPs of the master and slave robotic arms and their respective bases in the dual-arm robot system is shown below. Figure 11 As shown, this is used to describe the overall motion relationship of the two arms during collaborative work.

[0049] During the system deployment phase, to determine the relative pose relationship between the master and slave robotic arms' bases, the system uses pre-fabricated dedicated calibration connectors to achieve deterministic alignment calibration of the master and slave robotic arm end effectors within a physical connection structure. In this calibration process, the spatial pose relationship between the master and slave robotic arm end effectors' TCPs is determined by the pre-designed calibration connectors. Simultaneously, the system can obtain the pose information of the master and slave robotic arm end effectors' TCPs relative to their respective bases via the RTDE protocol. Based on the known pose relationships of the master robotic arm end effector's TCP relative to its base, the slave robotic arm end effector's TCP relative to its base, and the master robotic arm end effector's TCP relative to its base, the relative pose relationship between the master and slave robotic arm bases can be calculated through matrix operations using robot forward kinematics. This establishes a unified spatial reference coordinate system for subsequent dual-arm cooperative control. After determining the pose relationship between the two arm bases, the dual-arm cooperative control module receives the spatial pose information of the end effector TCP of the master robotic arm in real time during system operation. Based on the pose constraints of the end effector TCP, it generates the desired pose of the end effector TCP of the slave robotic arm, enabling the slave robotic arm to synchronously follow the movement of the master robotic arm in terms of spatial position and attitude. The overall block diagram of the dual-arm cooperative tracking control algorithm is as follows: Figure 12 As shown, firstly, during the visual servo alignment process of the main robotic arm according to motion control commands, the real-time spatial pose information of the end effector TCP of the main robotic arm relative to the main robotic arm base is acquired in real time through the RTDE protocol. Then, based on the pose constraints of the end effector TCP, the pose information of the slave robotic arm's end effector TCP relative to the main robotic arm base is calculated. Finally, the pose information of the slave robotic arm's end effector TCP relative to the slave robotic arm base, i.e., the desired pose of the slave robotic arm's end effector TCP, is calculated through the relative pose relationship between the main robotic arm base and the slave robotic arm base, generating cooperative motion commands for the slave robotic arm for cooperative tracking. It should be noted that, as... Figure 13The illustrated end-effector TCP pose constraints are applied throughout the entire collaborative operation process, serving as a continuous constraint condition for the dual-arm collaborative control. These end-effector TCP pose constraints are pre-set manually during the system design phase based on specific operational requirements. Their purpose is to ensure that the end-effector TCPs of the master and slave robotic arms always meet the operational requirements of axial alignment and symmetrical collaborative motion throughout the bolt alignment process. Specifically, the pose constraints include both attitude and position constraints. Regarding attitude constraints, the desired attitude of the slave robotic arm's end-effector TCP is rotated 180 degrees relative to the master robotic arm's end-effector TCP around the y-axis of the master robotic arm's end-effector TCP coordinate system to ensure symmetrical arrangement of the dual-arm end-effectors in the axial direction. Regarding position constraints, the master and slave robotic arm end-effector TCPs move symmetrically along the z-axis of their respective end-effector TCP coordinate systems (i.e., the master robotic arm's end-effector TCP coordinate system and the slave robotic arm's end-effector TCP coordinate system) during collaborative motion, ensuring that the dual-arm end-effector TCPs are always symmetrically distributed about the target bolt axis in space.

[0050] like Figure 14 The figure shows the spatial motion trajectory diagram of the TCP at the end effectors of the master and slave robotic arms during collaborative operation in one embodiment. As can be seen from the figure, the motion trajectories of the TCP at the end effectors of the master and slave robotic arms are centrally symmetrically distributed in space. This indicates that under the action of the dual-arm collaborative control module, the slave robotic arm can stably and continuously synchronize the motion state of the master robotic arm, demonstrating the stability and consistency of the dual-arm collaborative control module in actual operation.

[0051] Through the aforementioned dual-arm collaborative control process, the slave robotic arm can synchronously follow the motion state of the master robotic arm in real time and continuously during the master robotic arm's visual servo alignment task, thereby achieving stable bolt alignment operations under dual-arm collaborative conditions and providing a reliable motion coordination basis for subsequent dual-arm collaborative operations such as bolt tightening.

[0052] In one embodiment, the control execution module performs motion control on the master robotic arm and the slave robotic arm respectively, based on the motion control commands output by the hybrid vision servo control module and the collaborative motion commands generated by the dual-arm collaborative control module, to complete the bolt visual alignment operation under dual-arm collaborative conditions, specifically as follows: During system operation, the control execution module receives motion control commands from the master robotic arm output by the hybrid vision servo control module and collaborative motion commands from the slave robotic arm generated by the dual-arm collaborative control module. It then converts the motion control commands into corresponding robotic arm motion control signals to drive the master and slave robotic arms to perform corresponding pose adjustment actions. Specifically, the master robotic arm, guided by the switching between PBVS and IBVS methods, completes continuous motion from coarse visual alignment to precise visual alignment based on the output of the hybrid vision servo control module. The slave robotic arm, based on the collaborative motion commands generated by the dual-arm collaborative control module, performs real-time, synchronous collaborative following of the master robotic arm's motion state, while meeting pre-set end-effector TCP pose constraints.

[0053] During the dual-arm collaborative execution, the system continuously monitors the visual servo error status between the camera mounted on the end effector of the main robotic arm and the target bolt. When the hybrid visual servo control module determines that the image feature error of the bolt corner point converges within a preset threshold range, the system determines that the bolt visual alignment process is complete and sends an alignment completion signal to the control execution module. Upon receiving the alignment completion signal, the control execution module stops the current visual servo control process, keeping the main and slave robotic arms in a stable working posture that satisfies the collaborative constraint relationship. This completes the bolt visual alignment operation under dual-arm collaborative conditions and provides reliable initial pose conditions for subsequent bolt tightening and other operation steps.

[0054] A physical diagram illustrating the dual-arm coordinated bolt alignment process is shown below. Figure 15 As shown, where, Figure 15 (a) represents the initial relative state between the master and slave robotic arms and components before the start of the PBVS phase; Figure 15 (b) represents the process state of the main robotic arm performing rough visual alignment of the parts under the guidance of visual servoing and the collaborative tracking of the robotic arm during the PBVS stage; Figure 15 (c) indicates the state when the PBVS phase is completed and the system is about to switch to the IBVS phase. At this time, the ends of the master and slave robotic arms have completed the spatial alignment of the parts and entered the bolt close-range operation area. Figure 15 (d) represents the process state during the IBVS phase where the camera mounted on the end of the main robotic arm performs precise visual alignment of the target bolt and performs collaborative tracking with the robotic arm. Figure 15 (e) indicates the state in which the end of the main robotic arm and the end of the slave robotic arm achieve precise visual alignment with the target bolt after the IBVS phase is completed; Figure 15 In the diagram (f), it is indicated that after visual alignment is completed, the system transforms the bolt alignment pose in the camera coordinate system to the end tool coordinate system based on the pre-calibrated hand-eye calibration matrix, thereby achieving axial alignment of the target bolt by the end effector such as the sleeve.

[0055] Through the above process, the system, under the combined effect of hybrid vision servo control and dual-arm collaborative control, achieves a continuous and stable transition from coarse vision alignment to precise vision alignment and execution tool alignment, ensuring the coordinated movement of the two arms throughout the alignment process and providing reliable system-level support for subsequent dual-arm collaborative operations such as bolt tightening.

[0056] This embodiment also discloses a dual-arm collaborative bolt alignment method based on hybrid vision servoing, the specific steps of which are as follows: S1. Obtain the first image information of the component where the target bolt is located, and perform recognition and feature extraction on the first image information to obtain the spatial pose information of the component; S2. Using the PBVS method, the difference between the spatial pose information of the component and the preset desired alignment pose is used as feedback to generate the first motion control command, which guides the main robotic arm to perform coarse visual alignment. S3. During the coarse visual alignment process, the first collaborative motion command of the slave robot is generated based on the real-time spatial pose information of the master robot based on the collaborative control algorithm, guiding the slave robot to perform the first collaborative operation. S4. Based on the preset switching conditions, after the coarse visual alignment is completed, the second image information of the target bolt is obtained, and the second image information is used to extract features to obtain the two-dimensional image coordinates and spatial depth information of the bolt corner point of the target bolt; the switching conditions are based on the threshold setting of the coarse visual alignment. S5. Using the IBVS method, and taking the difference between the expected two-dimensional image coordinates of the bolt corner point and the two-dimensional image coordinates of the bolt corner point, as well as the spatial depth information, as feedback, a second motion control command is generated to guide the main robotic arm to perform precise visual alignment. S6. During the precise visual alignment process, the collaborative control algorithm generates a second collaborative motion command for the slave robot based on the real-time spatial pose information of the master robot, guiding the slave robot to perform a second collaborative operation and completing the visual servo alignment operation of the two arms.

[0057] In one embodiment, the step of obtaining the spatial pose information of the components in S1 is as follows: S11. Search the parts model library according to the maintenance operation requirements and extract the three-dimensional geometric model of the parts; S12. Based on the three-dimensional geometric model of the parts, perform visual recognition and feature extraction on the parts, and obtain the initial spatial pose of the parts through the PnP method. S13. Based on the first image information, the three-dimensional geometric model of the component, and the pose estimation result of the previous frame, a pose estimation method based on model tracking is used to continuously visually track the component, thereby obtaining continuous spatial pose information of the component relative to the camera coordinate system.

[0058] In one embodiment, the specific steps of the first cooperative operation in S3 are as follows: S31. The end effector of the main robotic arm and the end effector of the slave robotic arm are fixedly connected by a dedicated calibration connector. Based on the dedicated calibration connector, the pose information of the TCP of the main robotic arm end relative to the TCP of the slave robotic arm end is obtained. Based on the RTDE protocol, the pose information of the TCP of the main robotic arm end relative to the main robotic arm base and the pose information of the TCP of the slave robotic arm end relative to the slave robotic arm base are obtained. The relative pose relationship between the main robotic arm base and the slave robotic arm base is calculated. S32. During the process of visual servo alignment of the main robotic arm according to the first motion control command, the real-time spatial pose information of the TCP end of the main robotic arm relative to the base of the main robotic arm is collected in real time based on the RTDE protocol. S33. Based on the real-time spatial pose information of the end-effector TCP relative to the main robot arm base and the preset pose constraint relationship of the end-effector TCP, calculate the pose information of the end-effector TCP relative to the main robot arm base, and calculate the pose information of the end-effector TCP relative to the end-effector base through the relative pose relationship between the main robot arm base and the end-effector base, that is, the desired pose of the end-effector TCP, and generate the first cooperative motion command of the end-effector; wherein, the pose constraint relationship of the end-effector TCP includes attitude constraint and position constraint. The attitude constraint is that the desired pose of the end-effector TCP of the end-effector is rotated 180 degrees relative to the end-effector TCP of the main robot arm around the y-axis of the coordinate system of the end-effector TCP of the main robot arm. The position constraint is that the end-effector TCP of the main robot arm and the end-effector TCP of the end-effector move symmetrically along the z-axis of the coordinate system of the end-effector TCP of the main robot arm and the end-effector TCP of the end-effector respectively during the visual servo alignment process. S34. Guide the robotic arm to perform the first collaborative operation according to the first collaborative motion command.

[0059] In one embodiment, the second cooperative operation is implemented using the same method steps as the first cooperative operation described above.

[0060] In one embodiment, the specific steps of S4 are as follows: S41. Acquire the second image information and perform target detection on the second image information to obtain the region of interest of the target bolt in the second image information; S42. Perform image segmentation in the region of interest to obtain a pixel-level mask of the target bolt region; S43. Under the constraint of the pixel-level mask of the target bolt area, perform geometric structure analysis on the target bolt and extract the two-dimensional image coordinates of the bolt corner points of the target bolt; S44. Based on the two-dimensional image coordinates of the bolt corner points and the homography mapping relationship constructed by the bolt size, the initial spatial pose of the target bolt relative to the camera coordinate system is obtained, namely the second initial pose. S45. The second initial pose is optimized using VVS, and the spatial depth information of each bolt corner point in the camera coordinate system is calculated based on the optimized second initial pose and the bolt size.

[0061] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0062] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. In summary, the content of this specification should not be construed as a limitation of this application.

Claims

1. A dual-arm collaborative bolt alignment system based on hybrid vision servoing, characterized in that, include: The visual perception module is used to acquire first image information of the component where the target bolt is located, and to identify and extract features from the first image information to obtain spatial pose information of the component; acquire second image information of the target bolt, and to extract features from the second image information to obtain two-dimensional image coordinates and spatial depth information of the bolt corner point of the target bolt; A hybrid vision servo control module is used to generate motion control commands for the main robotic arm based on the spatial pose information of the component and the two-dimensional image coordinates and spatial depth information of the bolt corner point, using a phased vision alignment control strategy. The phased vision alignment control strategy includes a coarse alignment control strategy, which generates a first motion control command based on the pose error of the component where the target bolt is located using the PBVS method; and a precise alignment control strategy, which generates a second motion control command based on the image feature error of the target bolt using the IBVS method. The dual-arm collaborative control module is used to generate collaborative motion commands for the slave arm based on the real-time spatial pose information of the master robotic arm during the visual servo alignment process of the master robotic arm according to the first motion control command and the second motion control command. The control execution module is used to manipulate the main robotic arm according to the first motion control command and the second motion control command, and to manipulate the slave robotic arm according to the cooperative motion command, so as to complete the visual servo alignment operation of the two arms in coordination.

2. The dual-arm collaborative bolt alignment system based on hybrid vision servoing according to claim 1, characterized in that, The visual perception module includes: The 3D model extraction unit is used to extract the 3D geometric model of the component from a pre-built component model library based on the maintenance operation requirements. The spatial pose localization and tracking unit is used to perform visual recognition and feature extraction on the component based on the three-dimensional geometric model of the component, use the PnP method for initial localization, and use a model tracking-based pose estimation method to continuously visually track the component, update the spatial pose information of the component, and obtain the continuous spatial pose information of the component relative to the camera coordinate system. The bolt corner detection unit is used to perform target detection on the second image information, identify and locate the target bolt, and extract the two-dimensional image coordinates of the bolt corner. The bolt corner depth estimation unit is used to obtain the spatial depth information of the bolt corner point through a depth estimation algorithm based on the two-dimensional image coordinates of the bolt corner point.

3. The dual-arm collaborative bolt alignment system based on hybrid vision servoing according to claim 2, characterized in that, The spatial pose localization and tracking unit includes: The initial pose estimation subunit is used to perform visual recognition and feature extraction on the component based on the three-dimensional geometric model of the component, and to obtain the initial spatial pose of the component through the PnP method. The continuous pose tracking subunit is used to continuously visually track the component based on the first image information, the three-dimensional geometric model of the component, and the pose estimation result of the previous frame, using a model-based pose estimation method, thereby obtaining the continuous spatial pose information of the component relative to the camera coordinate system.

4. The dual-arm collaborative bolt alignment system based on hybrid vision servoing according to claim 2, characterized in that, The bolt corner detection unit includes: The target detection subunit is used to perform target detection on the second image information and obtain the region of interest of the target bolt in the second image information; A mask extraction subunit is used to perform image segmentation in the region of interest to obtain a pixel-level mask of the target bolt region; The feature point extraction subunit is used to perform geometric structure analysis on the target bolt under the constraint of a pixel-level mask in the target bolt area, and extract the two-dimensional image coordinates of the bolt corner points of the target bolt.

5. A dual-arm collaborative bolt alignment system based on hybrid vision servoing according to claim 2, characterized in that, The bolt corner depth estimation unit includes: The second initial pose estimation subunit is used to construct a homography mapping relationship based on the two-dimensional image coordinates of the bolt corner points and the bolt size to obtain the initial spatial pose of the target bolt relative to the camera coordinate system, i.e., the second initial pose. The depth information extraction subunit is used to optimize the second initial pose using VVS, and to calculate the spatial depth information of each bolt corner point in the camera coordinate system based on the optimized second initial pose and the bolt size.

6. The dual-arm collaborative bolt alignment system based on hybrid vision servoing according to claim 1, characterized in that, The hybrid vision servo control module includes: The first vision alignment control unit is used to generate the first motion control command by adopting the PBVS method and using the difference between the spatial pose information of the component and the preset desired alignment pose as feedback, so as to guide the main robotic arm to perform coarse vision alignment. The second vision alignment control unit is used to generate the second motion control command by employing the IBVS method and using the difference between the expected two-dimensional image coordinates of the bolt corner point and the two-dimensional image coordinates of the bolt corner point, as well as the spatial depth information, as feedback, to guide the main robotic arm to perform precise vision alignment. The control mode switching unit is used to switch the PBVS method to the IBVS method when a preset switching condition is met; the switching condition is based on a threshold setting for coarse visual alignment.

7. A dual-arm collaborative bolt alignment system based on hybrid vision servoing according to claim 1, characterized in that, The dual-arm collaborative control module includes: The base calibration unit is used to fix the end effector of the main robotic arm and the end effector of the slave robotic arm through a dedicated calibration connector. Based on the dedicated calibration connector, it obtains the pose information of the TCP of the main robotic arm end relative to the TCP of the slave robotic arm end. Based on the RTDE protocol, it obtains the pose information of the TCP of the main robotic arm end relative to the main robotic arm base and the pose information of the TCP of the slave robotic arm end relative to the slave robotic arm base, and calculates the relative pose relationship between the main robotic arm base and the slave robotic arm base. An end-effector constraint unit is used to store the end-effector TCP pose constraint relationship, which includes attitude constraint and position constraint. The attitude constraint is that the desired attitude of the slave end-effector TCP is rotated 180 degrees relative to the master end-effector TCP about the y-axis of the master end-effector TCP coordinate system. The position constraint is that the master end-effector TCP and the slave end-effector TCP move symmetrically along the z-axis of the master end-effector TCP coordinate system and the slave end-effector TCP coordinate system, respectively, during the visual servo alignment process. The slave arm pose generation unit is used to collect real-time spatial pose information of the end effector TCP of the main robotic arm relative to the main robotic arm base in real time based on the RTDE protocol during the visual servo alignment process of the main robotic arm according to the first motion control command and the second motion control command; calculate the pose information of the slave robotic arm end effector TCP relative to the main robotic arm base based on the pose constraint relationship of the end effector TCP; and calculate the pose information of the slave robotic arm end effector TCP relative to the slave robotic arm base through the relative pose relationship between the main robotic arm base and the slave robotic arm base, that is, the desired pose of the slave robotic arm end effector TCP, and generate the cooperative motion command of the slave robotic arm.

8. A dual-arm collaborative bolt alignment method based on hybrid vision servoing, characterized in that, The specific steps are as follows: S1. Obtain the first image information of the component where the target bolt is located, and perform recognition and feature extraction on the first image information to obtain the spatial pose information of the component; S2. Using the PBVS method, the difference between the spatial pose information of the component and the preset desired alignment pose is used as feedback to generate a first motion control command to guide the main robotic arm to perform coarse visual alignment. S3. During the coarse visual alignment process, the first cooperative motion command of the slave robot is generated based on the real-time spatial pose information of the master robot based on the cooperative control algorithm, so as to guide the slave robot to perform the first cooperative operation. S4. Based on preset switching conditions, after determining that coarse visual alignment is completed, the second image information of the target bolt is obtained, and feature extraction is performed on the second image information to obtain the two-dimensional image coordinates and spatial depth information of the bolt corner point of the target bolt; the switching conditions are based on the threshold setting of coarse visual alignment. S5. Using the IBVS method, and taking the difference between the expected two-dimensional image coordinates of the bolt corner point and the two-dimensional image coordinates of the bolt corner point, as well as the spatial depth information, as feedback, a second motion control command is generated to guide the main robotic arm to perform precise visual alignment. S6. During the precise visual alignment process, a second collaborative motion command for the slave robotic arm is generated based on the real-time spatial pose information of the master robotic arm according to the collaborative control algorithm, guiding the slave robotic arm to perform a second collaborative operation and complete the visual servo alignment operation of the two arms.

9. A dual-arm collaborative bolt alignment method based on hybrid vision servoing according to claim 8, characterized in that, The steps for obtaining the spatial pose information of the components mentioned in S1 are as follows: S11. Search the component model library according to the maintenance operation requirements and extract the three-dimensional geometric model of the component; S12. Based on the three-dimensional geometric model of the component, perform visual recognition and feature extraction on the component, and obtain the initial spatial pose of the component through the PnP method; S13. Based on the first image information, the three-dimensional geometric model of the component, and the pose estimation result of the previous frame, the component is continuously visually tracked using a model tracking-based pose estimation method to obtain continuous spatial pose information of the component relative to the camera coordinate system.

10. A dual-arm collaborative bolt alignment method based on hybrid vision servoing according to claim 8, characterized in that, The specific steps for the first collaborative operation in S3 are as follows: S31. The end effector of the main robotic arm and the end effector of the slave robotic arm are fixedly connected by a dedicated calibration connector. Based on the dedicated calibration connector, the pose information of the TCP of the main robotic arm end relative to the TCP of the slave robotic arm end is obtained. Based on the RTDE protocol, the pose information of the TCP of the main robotic arm end relative to the main robotic arm base and the pose information of the TCP of the slave robotic arm end relative to the slave robotic arm base are obtained. The relative pose relationship between the main robotic arm base and the slave robotic arm base is calculated. S32. During the process of the main robotic arm performing visual servo alignment according to the first motion control command, the real-time spatial pose information of the TCP end of the main robotic arm relative to the main robotic arm base is collected in real time based on the RTDE protocol. S33. Based on the real-time spatial pose information of the master robot arm's end-effector TCP relative to the master robot arm base and the preset end-effector TCP pose constraint relationship, calculate the pose information of the slave robot arm's end-effector TCP relative to the master robot arm base, and calculate the pose information of the slave robot arm's end-effector TCP relative to the slave robot arm base through the relative pose relationship between the master robot arm base and the slave robot arm base, i.e., the desired pose of the slave robot arm's end-effector TCP, and generate the first cooperative motion command of the slave robot arm; wherein, the end-effector TCP pose constraint relationship includes attitude constraint and position constraint, the attitude constraint is that the desired pose of the slave robot arm's end-effector TCP is rotated 180 degrees relative to the master robot arm's end-effector TCP around the y-axis of the master robot arm's end-effector TCP coordinate system, and the position constraint is that the master robot arm's end-effector TCP and the slave robot arm's end-effector TCP move symmetrically along the z-axis of the master robot arm's end-effector TCP coordinate system and the slave robot arm's end-effector TCP coordinate system, respectively, during the visual servo alignment process; S34. Guide the robotic arm to perform the first collaborative operation according to the first collaborative motion command.

11. A dual-arm collaborative bolt alignment method based on hybrid vision servoing according to any one of claims 7-10, characterized in that, The specific steps of S4 are as follows: S41. Acquire the second image information and perform target detection on the second image information to obtain the region of interest of the target bolt in the second image information; S42. Perform image segmentation on the region of interest to obtain a pixel-level mask of the target bolt region; S43. Under the constraint of the pixel-level mask of the target bolt area, perform geometric structure analysis on the target bolt and extract the two-dimensional image coordinates of the bolt corner points of the target bolt; S44. Based on the two-dimensional image coordinates of the bolt corner points and the homography mapping relationship constructed by the bolt size, the initial spatial pose of the target bolt relative to the camera coordinate system is obtained, namely the second initial pose. S45. The second initial pose is optimized using VVS, and the spatial depth information of each bolt corner point in the camera coordinate system is calculated based on the optimized second initial pose and the bolt size.

Images