A 3D vision-guided robotic bolt tightening method and system

Abstract

This invention discloses a 3D vision-guided robotic bolt tightening method and system, relating to the field of intelligent manufacturing technology. The method includes the following steps: Step S1. Constructing a minimum number of image capture models based on known camera and chassis data, and solving for the optimal image capture path; Step S2. Acquiring images and point cloud information of the chassis; Step S3. Inputting the images into a bolt intelligent detection model to obtain bolt size and position data; Step S4. Cropping bolt surface point cloud information based on the bolt data, and fitting it to calculate the bolt's posture and dimensions; Step S5. Calculating the transformation relationship between the image coordinate system and the tightening gun posture in the chassis coordinate system; Step S6. Guiding the robot to perform tightening operations based on the transformation relationship and the bolt's posture and dimensions.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, specifically to a 3D vision-guided robotic bolt tightening method and system. Background Technology

[0002] Bolts, as a basic mechanical fastener, consist of a head and a threaded shank. They reliably connect components by mating with a nut or a pre-threaded hole, playing a vital role in industrial manufacturing, construction engineering, and transportation. Traditional bolt tightening operations have long relied on manual labor, which not only suffers from inherent drawbacks such as low efficiency and high labor intensity for workers, but also makes it difficult to guarantee consistent assembly quality due to human factors, failing to meet the demands of modern manufacturing for high-speed, high-precision assembly. With the popularization of industrial robot technology, automated tightening solutions based on teach-programmable logic controllers (TLCs) have emerged. While this solution addresses the efficiency issue, it reveals limitations in actual production: influenced by machining errors in the chassis longitudinal beams, transportation errors in electric flatbed carts, and cumulative assembly errors, a systematic deviation occurs between the preset teach-programmable path and the actual posture of the longitudinal beams. This causes the robot's end effector tightening gun to fail to accurately match the bolt's true spatial coordinates, severely restricting assembly accuracy and process stability under complex working conditions.

[0003] Chinese patent CN119260374A (A 3D Vision-Guided Automotive Door Follow-up Assembly System) proposes a 3D vision-guided automotive door follow-up assembly system. In this system, when the first robotic arm grasps the door to be assembled and moves it to the teaching position for secondary positioning and correction, the first camera acquires a grayscale image and a depth image of the door. When the second robotic arm and the linear conveyor belt are in follow-up motion, the second camera acquires a grayscale image and a depth image of the vehicle body. The processor constructs a hinge hole pose matrix based on the vehicle body grayscale and depth images and determines the real-time installation pose of the hinge holes. A threaded hole pose matrix is ​​constructed based on the door grayscale and depth images, and the real-time installation pose of the door is determined based on the threaded hole pose matrix and the hinge hole pose matrix. The first robotic arm aligns the door to be assembled with the vehicle body based on the real-time installation pose of the door. The second robotic arm tightens the bolts based on the real-time installation pose of the hinge holes, thus installing the door onto the vehicle body. While this solution can automate bolt tightening operations, the multi-robot, multi-camera configuration not only increases the complexity of the system, such as the difficulty of camera calibration, synchronization, and data integration, but may also lead to image inconsistencies due to differences in camera parameters and imaging characteristics, affecting the subsequent construction of the pose matrix and the determination of the installation pose. It also increases hardware costs and installation and debugging difficulties.

[0004] Chinese patent CN116958083A (A Method for Detecting Loose Bolts on the Bottom of a High-Speed ​​Train Based on Robot Adaptive Pose Adjustment) proposes a method for detecting loose bolts on the bottom of a high-speed train based on robot adaptive pose adjustment. This method involves collecting data from multiple runs covering all loose bolt points to build and generate a bolt target detection model. The trained model is used to detect bolt regions in 2D images, and bolt point clouds are obtained using pixel coordinate mapping. A base plane is extracted from the bolt point cloud, and the camera rotation matrix and the displacement between the bolt position and the origin are calculated. Combining this with the pose transformation relationship between the camera and the robotic arm, the robotic arm is controlled to move the camera to the target pose, and bolt region detection is repeated until the confidence level meets the preset expectation. While this scheme effectively detects the bolt state and extracts point cloud information using an intelligent detection method, it does not utilize the camera's field of view or relevant data about the target object. Relying on a confidence level to determine the effectiveness of the detection requires multiple checks and judgments, resulting in low execution efficiency and failing to meet the requirements for operational efficiency and production cycle time.

[0005] Chinese patent CN118379354A (A Bolt Head Pose Estimation Method Based on 3D Vision Guidance and Adaptive Pass-Through Filtering) proposes a bolt head pose estimation method. This method uses an industrial robot to drive a 3D camera to scan the area covering the bolt head for the first point cloud acquisition. Point cloud processing is then performed to segment the bolt mounting plane and calculate its normal vector. The industrial robot is then repositioned so that the 3D camera's scanning surface is parallel to the normal vector of the bolt mounting plane. The industrial robot then drives the 3D camera to scan the area covering the bolt head for a second point cloud acquisition. Mode-based adaptive pass-through filtering is used for point cloud processing to segment the point cloud near the top circle of the bolt head. Edge extraction and spatial circle fitting algorithms are used to extract the features of the top circle of the bolt head to obtain the bolt head pose. While this scheme uses a 3D camera to extract the bolt head pose, the implementation involves two scans with the 3D camera and then adaptive pass-through filtering to segment the bolt head. This results in high computational cost, low detection efficiency, and an inability to effectively distinguish targets similar to bolts (such as rivets), making it unsuitable for assembly requirements in complex environments. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a 3D vision-guided robotic bolt tightening method and system to meet the stringent requirements of industrial assembly scenarios.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a robot bolt tightening method based on 3D vision guidance, comprising the following steps: Step S1. Construct a model for the minimum number of photos based on the known camera and chassis data, and solve for the optimal photo path; Step S2. Acquire images and point cloud information of the vehicle frame; Step S3. Input the image into the intelligent bolt detection model to obtain the bolt size and position data; Step S4. Based on the size and position data of the bolt, cut out the point cloud information of the bolt surface, and fit it to calculate the bolt's posture and specifications. Step S5. Calculate the transformation relationship between the image coordinate system and the vehicle frame coordinate system for the position of the tightening gun; Step S6. Guide the robot to perform the tightening operation according to the transformation relationship and the posture and specifications of the bolt.

[0008] Based on the above technical solution, in step S1, the minimum number of photos required is defined as follows: Problem Description: Given a set of bolts on a vehicle frame The position of each bolt The camera's field of view is Solve to minimize the number of photos taken, ensuring that each bolt is photographed at least once; Decision variables: : Indicates whether to select bolts As the center of the camera's field of view ; : Indicates the camera's field of view Did you take a picture of the bolts? , ; Objective function:

[0009] Constraints: Each bolt was tapped at least once: ; Field of view When not selected, bolt Not counting being photographed: ; Field of view It can only cover bolts within its range:

[0010]

[0011] in, and Representing the field of view respectively The coordinates of the center point, and These represent the width and height of the field of view, respectively.

[0012] Based on the above technical solution, in step S3, the intelligent bolt detection model is a convolutional neural network, which is established by training bolt images:

[0013] in, This indicates the center coordinates of the bolt in the image. This represents the radius of the bolt in the image. Represents a convolutional neural network. This indicates the input bolt image.

[0014] Based on the above technical solution, in step S4, the orientation of the bolt is calculated using a plane fitting method based on the point cloud information of the bolt surface. Due to the camera Z Axis and frame coordinate system Z Since the axes coincide, the plane fitting equation can be expressed as:

[0015] in, and In camera coordinate system shaft and The coordinate components of the normal vector of the axis. For the lower plane of the camera coordinate system The position of the axis.

[0016] Based on the above technical solution, in step S4, the physical dimensions of the bolt head are calculated using a circle fitting method based on the point cloud information of the bolt surface, and the bolt specifications are classified according to the preset classification standards. The equation of a standard circle is expressed as:

[0017] in, Indicates the position of the bolt's center. Represents point cloud coordinates, This indicates the physical dimensions of the bolt head.

[0018] Based on the above technical solution, in step S5, the transformation relationship of the tightening gun pose from the image coordinate system to the chassis coordinate system includes the transformation of the flange pose between the image coordinate system and the robot base coordinate system, the transformation of the flange and tightening gun poses between the robot base coordinate system and the chassis coordinate system, and the transformation of the tightening gun pose from the robot base coordinate system to the chassis coordinate system. The transformation between the image coordinate system and the robot base coordinate system of the flange pose includes acquiring the calibration plate image and recording the robot flange pose, calculating the pose of the calibration plate relative to the camera through camera calibration, and solving the transformation relationship between the image coordinate system and the flange pose of the robot base coordinate system using hand-eye calibration; the transformation relationship is to convert a point in the image coordinate system into the flange pose of the robot base coordinate system.

[0019] The pose transformation between the flange and the tightening gun in the robot base coordinate system includes calculating the tool center point value of the tightening gun based on the theoretical and actual values ​​of the target, which is the transformation relationship from the robot flange to the tightening gun; the transformation relationship is the position and direction of the center point of the tightening gun relative to the center point of the robot flange.

[0020] The transformation of the tightening gun pose from the robot base coordinate system to the vehicle frame coordinate system includes calculating the transformation matrix from the vehicle frame coordinate system to the robot base coordinate system; the transformation relationship is the position and direction of the origin of the vehicle frame coordinate system relative to the origin of the robot base coordinate system.

[0021] Based on the above technical solution, the transformation relationship between the image coordinate system and the flange pose in the robot base coordinate system is as follows:

[0022] in, This represents a point P in the image coordinate system. This represents the position of point P in the robot's base coordinate system. Represents the camera intrinsic parameter matrix. This represents the depth value of point P in the camera coordinate system. and These represent the rotation and translation matrices in the robot flange's hand-eye matrix, respectively.

[0023] Based on the above technical solution, the transformation relationship from the flange to the tightening gun in the robot's base coordinate system is as follows:

[0024] in, This represents the theoretical value of the tightening gun's distance from the target center in the robot's base coordinate system. This represents the actual value from the flange to the target center in the robot's base coordinate system. This represents the transformation matrix from the robot flange to the tool.

[0025] Based on the above technical solution, the transformation relationship between the robot base coordinate system and the tightening gun pose in the vehicle frame coordinate system is as follows:

[0026] in, This indicates the pose of the tightening gun in the robot's frame coordinate system. This represents the transformation matrix from the vehicle frame coordinate system to the robot base coordinate system. This indicates the pose of the tightening gun in the robot's base coordinate system.

[0027] The present invention also provides a system for implementing the above-described 3D vision-guided robot bolt tightening method, comprising: Optimization Unit: Constructs a model for minimum number of photos based on camera and chassis data to solve for the optimal photo path; Visual unit: used to acquire target images and point cloud information; Image processing and computing unit: Calculates the position of the bolt through the intelligent bolt detection model, and then calculates its posture and size data by combining the point cloud information of the bolt; Calibration unit: Performs camera calibration, hand-eye calibration, and coordinate system transformation; Robot execution unit: Guides the robot to perform bolt tightening operations based on the identified bolt information.

[0028] The beneficial effects of this invention are as follows: This invention addresses the limitations of traditional robotic bolt tightening operations due to manufacturing errors and assembly deviations in the vehicle frame longitudinal beams. It introduces 3D vision-guided technology, employing high-precision visual inspection algorithms to identify the bolt's position, orientation, and dimensions. Combined with multi-coordinate system spatial alignment and dynamic path planning, it overcomes the impact of vehicle frame longitudinal beam bending and pose uncertainties on positioning accuracy. This allows the robot to adaptively adjust its bolt tightening actions, achieving precise assembly under complex conditions. This technology boasts high flexibility and intelligence, adapting to various bolt specifications and diverse production line requirements, providing an efficient and reliable automated assembly solution for aerospace, automotive manufacturing, and other fields. Attached Figure Description

[0029] Figure 1 This is a flowchart of a robot bolt tightening method based on 3D vision guidance in an embodiment of the present invention; Figure 2 This is a diagram of a robot bolt tightening system based on 3D vision guidance, as described in an embodiment of the present invention.

[0030] Figure label: 1-Industrial robot, 2-3D vision sensor, 3-Tightening gun, 4-Frame, 5-Hole, 6-Bolt, 7-Fixing fixture. Detailed Implementation

[0031] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout.

[0032] In the description of this invention, it should be noted that the directional terms such as "center", "lateral (X)", "longitudinal (Y)", "vertical (Z)", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", and "counterclockwise" indicate the orientation and positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. They should not be construed as limiting the specific protection scope of this invention.

[0033] The following description, in conjunction with the accompanying drawings, further illustrates specific embodiments of the present invention, making the technical solution and its beneficial effects clearer and more explicit. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the invention.

[0034] See Figure 1 As shown, this embodiment of the invention provides a 3D vision-guided robot bolt tightening method, including the following steps: S1. Construct a model for the minimum number of shots based on the known camera and chassis data, and solve the mathematical model to calculate the optimal shooting path; Problem Description: Given a set of bolts on a vehicle frame The position of each bolt The camera's field of view is Solve to minimize the number of photos taken, ensuring that each bolt is photographed at least once.

[0035] Decision variables: : Indicates whether to select bolts As the center of the camera's field of view

[0036] : Indicates the camera's field of view Did you take a picture of the bolts? ,

[0037] Objective function:

[0038] Constraints: Each bolt was tapped at least once:

[0039] Field of view When not selected, bolt Not counting being photographed:

[0040] Field of view It can only cover bolts within its range:

[0041]

[0042] in, and Representing the field of view respectively The coordinates of the center point, and These represent the width and height of the field of view, respectively.

[0043] S2. Acquire images and point cloud information of the vehicle frame; In this invention, a 3D vision sensor is fixed to the end effector of a robot to acquire frame images and point cloud data. The frame images include bolt images for both training and testing sets, as well as images of the bolts to be tightened. The bolt images in the training set are used to train the intelligent bolt detection model, while the bolt images in the testing set are used to evaluate the model. During bolt tightening, the accurate position of the bolts can be obtained by inputting the images of the bolts to be tightened into the intelligent bolt detection model; therefore, this invention also acquires frame images before applying the model.

[0044] S3. Input the image into the intelligent bolt detection model, which outputs the bolt size (number of pixels in the image) and position data;

[0045] in, This indicates the center coordinates of the bolt in the image. This represents the radius of the bolt in the image. Represents a convolutional neural network. This indicates the input bolt image.

[0046] S4. Cut out the point cloud information of the bolt surface based on the bolt data, and fit it to calculate the bolt's posture and specifications (the specifications are the bolt's specifications, such as M12, M14, M16, which are classified according to the physical dimensions). S4-1. Based on the bolt image position, crop the point cloud information of the bolt head, and use the plane fitting method to calculate the bolt's pose in the camera coordinate system.

[0047] Due to the camera's Z-axis and the vehicle frame coordinate system Z Since the axes coincide, the plane fitting equation can be expressed as:

[0048] in, a and bIn camera coordinate system X shaft and Y The coordinate components of the normal vector of the axis. c For the lower plane of the camera coordinate system Z The position of the axis.

[0049] For the coordinates of each point cloud on the bolt surface The equation is:

[0050] The matrix form of the above equation is , where the matrix and for:

[0051]

[0052] Using the least squares method to fit the plane coefficients :

[0053] Bolt attitude rotation matrix in camera coordinate system It can be represented as:

[0054] S4-2. Based on the point cloud information of the bolt head, calculate the size and specifications of the bolt head using the circle fitting method.

[0055] The equation of a standard circle is expressed as:

[0056] After unfolding and organizing, we get:

[0057] in, .

[0058] For the coordinates of each point cloud on the bolt surface The equation is:

[0059] The matrix form of the above equation is , where the matrix A and B for:

[0060]

[0061] Solve using the least squares method p :

[0062] The calculated center of the bolt is The physical radius of the bolt is Bolt specifications are classified according to the physical dimensions of the bolts and based on preset classification standards.

[0063] S5. Calculate the transformation relationship between the image coordinate system and the chassis coordinate system of the tightening gun pose; S5-1. The transformation of the flange pose between the image coordinate system and the robot base coordinate system includes acquiring the calibration plate image and recording the robot flange pose, calculating the pose of the calibration plate relative to the camera through camera calibration, and solving the transformation relationship between the image coordinate system and the flange pose under the robot base coordinate system using hand-eye calibration; the transformation relationship is to transform a point in the image coordinate system into the flange pose under the robot base coordinate system.

[0064] Camera calibration uses a 150×150mm checkerboard alumina calibration plate, with each square having a side length of 10mm and an image array of 12×9. First, the calibration plate is placed at different positions and angles within the camera's field of view, and 18 images are acquired. Next, the acquired images undergo preprocessing, including grayscale conversion and noise reduction. Feature points on the calibration plate are detected in the preprocessed images, and their pixel coordinates are calculated. Then, using the pixel coordinates of the feature points and their known world coordinates, the camera's intrinsic parameter matrix is ​​determined through the camera calibration algorithm. Finally, the accuracy of the calibration results is evaluated using indicators such as reprojection error. The smaller the reprojection error, the more accurate the calibration results, and the entire camera calibration process is then complete.

[0065] A 3D vision sensor was fixed to the robot's end effector, while the calibration board remained stationary. The robot's movement was controlled to move the camera to different poses. Eighteen sets of image data and corresponding robot pose information were acquired. The images were then preprocessed to detect feature points on the calibration board and calculate their corresponding pixel coordinates. Based on the acquired data, feature point coordinates, and robot pose information, the Tsai algorithm was used to solve for the hand-eye matrix, which represents the coordinate transformation relationship between the camera and the robot's base coordinate system. Finally, the hand-eye matrix was substituted to calculate the calibration board pose, and the calibration board's error was statistically analyzed to evaluate the accuracy of the calibration results.

[0066] The transformation relationship between the image coordinate system and the robot base coordinate system for the flange pose is as follows:

[0067] in, This represents a point P in the image coordinate system. This represents the position of point P in the robot's base coordinate system. Represents the camera intrinsic parameter matrix. This represents the depth value of point P in the camera coordinate system. and These represent the rotation and translation matrices in the robot flange's hand-eye matrix, respectively.

[0068] S5-2. The pose transformation between the flange and the tightening gun in the robot base coordinate system includes calculating the tool center point value of the tightening gun based on the theoretical and actual values ​​of the target, which is the transformation relationship from the robot flange to the tightening gun; the transformation relationship is the position and direction of the center point of the tightening gun relative to the center point of the robot flange.

[0069] Move the robot until the target is within the camera's field of view. Acquire images of the target and calculate the image coordinates of the target's center through image processing. Calculate the target's pose using the hand-eye matrix of the robot's flange; this value represents the target's theoretical pose. .

[0070] Move the robot again until it precisely touches the center of the target with the tightening gun. Record the robot's pose in the base coordinate system at this moment; this value is the actual value of the target. .

[0071] The transformation relationship from the flange to the tightening gun in the robot's base coordinate system is as follows:

[0072] in, This represents the theoretical value of the tightening gun's distance from the target center in the robot's base coordinate system. This represents the actual value from the flange to the target center in the robot's base coordinate system. This represents the transformation matrix from the robot flange to the tool.

[0073] S5-3. The transformation of the tightening gun's pose from the robot base coordinate system to the vehicle frame coordinate system includes calculating the transformation matrix from the vehicle frame coordinate system to the robot base coordinate system. The transformation relationship refers to the position and orientation of the origin of the vehicle frame coordinate system relative to the origin of the robot base coordinate system.

[0074] To accurately calculate this relationship, a vision-based method is employed. First, a positioning hole is selected on the chassis, and an image of that hole is acquired. Image processing is then used to obtain the image coordinates of the hole. Next, the hand-eye matrix is ​​used to calculate the coordinates of the hole in the robot's base coordinate system. Then, planar point cloud information of the chassis is acquired, and a plane fitting method is used to calculate the chassis's attitude, thus obtaining the transformation relationship from the chassis coordinate system to the robot's base coordinate system.

[0075] The transformation relationship between the robot's base coordinate system and the vehicle frame coordinate system for the tightening gun pose is as follows:

[0076] in, This indicates the pose of the tightening gun in the robot's frame coordinate system. This represents the transformation matrix from the vehicle frame coordinate system to the robot base coordinate system. This indicates the pose of the tightening gun in the robot's base coordinate system.

[0077] S6. Guide the robot to perform tightening operations based on the transformation relationship and the bolt's position and specifications; In bolt tightening operations, firstly, an image of the bolt to be tightened is acquired and input into the intelligent bolt detection model to obtain the bolt's position in the image coordinate system. Secondly, based on the transformation relationship between the image coordinate system and the tightening gun pose in the vehicle frame coordinate system, the bolt's image coordinates are mapped to the vehicle frame coordinate system. Simultaneously, combined with the bolt's attitude information, the insertion pose of the tightening gun sleeve is calculated. Finally, the industrial robot adjusts the pose of its end-effector tightening gun based on the calculation results to ensure the sleeve accurately aligns with the bolt and completes the tightening operation.

[0078] The present invention also provides a system for implementing the above-described 3D vision-guided robot bolt tightening method, comprising: Optimization Unit: Builds a model based on camera and chassis data to solve for the optimal shooting path; Visual unit: used to acquire target images and point cloud information; Image processing and computing unit: Calculates the position of the bolt through the intelligent bolt detection model, and then calculates its posture and size data by combining the point cloud information of the bolt; Calibration unit: Performs camera calibration, hand-eye calibration, and coordinate system transformation; Robot execution unit: Guides the robot to perform bolt tightening operations based on the identified bolt information.

[0079] See Figure 2As shown, in the application scenario of a 3D vision-guided robotic bolt tightening system, an image acquisition device is fixed to the end effector of an industrial robot 1, preferably a 3D vision sensor. Simultaneously, a tightening gun 3 is installed at the end effector of the industrial robot 1 for subsequent bolt tightening operations. A frame 4 is fixed to a fixture 7, which has multiple holes 5 pre-installed with bolts 6. Upon system startup, the industrial robot 1 first moves to the location of the positioning holes 5 on the frame, ensuring that the holes 5 are within the field of view of the 3D vision sensor 2. At this time, the 3D vision sensor 2 acquires images and point cloud information to calibrate the coordinate system of the frame 4. After calibration, the system solves the minimum number of image captures model based on the known parameters of the 3D vision sensor 2 and relevant data of the frame 4, thereby calculating the optimal image capture path. The industrial robot 1 moves to the designated image capture position and acquires image information of the bolts 6. The acquired images are processed using a bolt intelligent detection model, outputting the size and position data of the bolts 6. Furthermore, combined with point cloud fitting technology, the posture and dimensions of the bolts 6 can be accurately calculated. Based on the information about bolt 6, the sleeve of the tightening gun 3 is precisely inserted into the position of bolt 6. Once the sleeve is in place, the tightening gun 3 is activated to tighten bolt 6. After tightening is complete, the sleeve is withdrawn in an orderly manner, and the industrial robot 1 moves to the next camera position according to the preset program to prepare for the next round of bolt 6 tightening work, until all bolts 6 are accurately tightened.

[0080] This invention breaks through the reliance on manual positioning and fixing fixtures in traditional bolt assembly. Through high-precision visual inspection technology, it significantly improves the level of automation, accuracy and efficiency of assembly. It is especially suitable for the assembly needs of multi-specification bolts under complex working conditions and has broad application prospects in the fields of automobile manufacturing, aerospace and other fields.

[0081] In the description of this specification, references to terms such as "an embodiment," "preferred," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. Illustrative expressions of the above terms in this specification do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0082] This invention is not limited to the embodiments described above. Those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention. Contents not described in detail in this specification are prior art known to those skilled in the art.

Claims

1. A robot bolt tightening method based on 3D vision guidance, characterized in that, Includes the following steps: Step S1. Construct a model for the minimum number of photos based on the known camera and chassis data, and solve for the optimal photo path; Step S2. Acquire images and point cloud information of the vehicle frame; Step S3. Input the image into the intelligent bolt detection model to obtain the bolt size and position data; Step S4. Based on the bolt data, cut out the bolt surface point cloud information, and fit it to calculate the bolt's posture and specifications. Step S5. Calculate the transformation relationship between the image coordinate system and the vehicle frame coordinate system for the position of the tightening gun; Step S6. Guide the robot to perform the tightening operation according to the transformation relationship and the bolt's posture and specifications; In step S1, the minimum number of photos required model: Problem Description: Given a set of bolts on a vehicle frame The position of each bolt The camera's field of view is Solve to minimize the number of photos taken, ensuring that each bolt is photographed at least once; Decision variables: : Indicates whether to select bolts As the center of the camera's field of view ; : Indicates the camera's field of view Did you take a picture of the bolts? , ; Objective function: ; Constraints: Each bolt was tapped at least once: ; Field of view When not selected, bolt Not counting being photographed: ; Field of view It can only cover bolts within its range: ； ； in, and Representing the field of view respectively The coordinates of the center point, and represent the width and height of the field of view, respectively; In step S3, the intelligent bolt detection model is a convolutional neural network, which is established by training bolt images: ； in, This indicates the center coordinates of the bolt in the image. This represents the radius of the bolt in the image. Represents a convolutional neural network. This indicates the input bolt image; In step S4, the bolt's orientation is calculated using a plane fitting method based on the point cloud information of the bolt surface. Due to the camera Z Axis and frame coordinate system Z Since the axes coincide, the plane fitting equation can be expressed as: ； in, and In camera coordinate system shaft and The coordinate components of the normal vector of the axis. For the lower plane of the camera coordinate system The position of the axis; In step S4, the physical dimensions of the bolt head are calculated using a circle fitting method based on the point cloud information of the bolt surface, and the bolt specifications are classified according to the preset classification standards. The equation of a standard circle is: ； in, Indicates the position of the bolt's center. Represents point cloud coordinates, This indicates the physical dimensions of the bolt head.

2. The robot bolt tightening method based on 3D vision guidance as described in claim 1, characterized in that: In step S5, the transformation relationship of the tightening gun pose from the image coordinate system to the chassis coordinate system includes the transformation of the flange pose between the image coordinate system and the robot base coordinate system, the transformation of the flange and tightening gun poses between the robot base coordinate system and the chassis coordinate system, and the transformation of the tightening gun pose from the robot base coordinate system to the chassis coordinate system. The transformation between the image coordinate system and the robot base coordinate system of the flange pose includes acquiring the calibration plate image and recording the robot flange pose, calculating the pose of the calibration plate relative to the camera through camera calibration, and solving the transformation relationship between the image coordinate system and the flange pose of the robot base coordinate system using hand-eye calibration; the transformation relationship is to convert a point in the image coordinate system into the flange pose of the robot base coordinate system. The pose transformation between the flange and the tightening gun in the robot base coordinate system includes calculating the tool center point value of the tightening gun based on the theoretical and actual values ​​of the target, which is the transformation relationship from the robot flange to the tightening gun; the transformation relationship is the position and direction of the center point of the tightening gun relative to the center point of the robot flange. The transformation of the tightening gun pose from the robot base coordinate system to the vehicle frame coordinate system includes calculating the transformation matrix from the vehicle frame coordinate system to the robot base coordinate system; the transformation relationship is the position and direction of the origin of the vehicle frame coordinate system relative to the origin of the robot base coordinate system.

3. The robot bolt tightening method based on 3D vision guidance as described in claim 2, characterized in that: The transformation relationship between the image coordinate system and the flange pose in the robot base coordinate system is as follows: ； in, This represents a point P in the image coordinate system. This represents the position of point P in the robot's base coordinate system. This represents the camera intrinsic parameter matrix. This represents the depth value of point P in the camera coordinate system. and These represent the rotation and translation matrices in the robot flange's hand-eye matrix, respectively.

4. The robot bolt tightening method based on 3D vision guidance as described in claim 2, characterized in that: The transformation relationship from the flange to the tightening gun in the robot's base coordinate system is as follows: ； in, This represents the theoretical value of the tightening gun's distance from the target center in the robot's base coordinate system. This represents the actual value from the flange to the target center in the robot's base coordinate system. This represents the transformation matrix from the robot flange to the tool.

5. The robot bolt tightening method based on 3D vision guidance as described in claim 2, characterized in that: The transformation relationship between the robot base coordinate system and the vehicle frame coordinate system for the tightening gun pose is as follows: ； in, This indicates the pose of the tightening gun in the robot's frame coordinate system. This represents the transformation matrix from the vehicle frame coordinate system to the robot base coordinate system. This indicates the pose of the tightening gun in the robot's base coordinate system.

6. A system for implementing the 3D vision-guided robot bolt tightening method according to any one of claims 1-5, characterized in that, include: Optimization Unit: Constructs a model for minimum number of photos based on camera and chassis data to solve for the optimal photo path; Visual unit: used to acquire target images and point cloud information; Image processing and computing unit: Calculates the position of the bolt through the intelligent bolt detection model, and then calculates its posture and size data by combining the point cloud information of the bolt; Calibration unit: Performs camera calibration, hand-eye calibration, and coordinate system transformation; Robot execution unit: Guides the robot to perform bolt tightening operations based on the identified bolt information.

Images