A bolt high-precision identification, positioning and tightening method and intelligent tightening equipment

Abstract

The application relates to a bolt high-precision identification, positioning and tightening method and an intelligent tightening device, which comprises system environment establishment and calibration; determination of a camera shooting position, an optimal shooting height, a default end height and shooting parameters; determination of process parameters of a tightening gun; robot motion path planning and offline programming; shooting of a bolt to obtain a preliminary image, determination of a rotating shaft and a rotating angle, and tightening end direction alignment; tightening end height adjustment; determination of the position of a bolt center to be tightened, acquisition of the coordinates of the bolt center point under a base coordinate system, and then determination of the pose of a robot end flange under the base coordinate system required to be reached during tightening operation through pose transformation; and execution of the tightening operation. The application has the advantages of high accuracy and strong applicability, can realize high-precision identification of a bolt center in a narrow space such as a tank seat pad, and provides support for realizing full-automatic bolt tightening.

Description

Technical Field

[0001] This invention relates to the field of automated bolt tightening technology, and in particular to a method and intelligent tightening device for high-precision bolt identification, positioning and tightening. Background Technology

[0002] Bolt identification and positioning is a very important part of automated tightening. Bolt positioning in narrow spaces is difficult, the positioning accuracy is not high, and the tightening operation is difficult, which has always been a major problem that has plagued the industry. In order to solve this problem, many bolt positioning methods have emerged in recent years.

[0003] Traditional methods utilize three displacement sensors mounted in an isosceles right triangle on the same plane at the end of a robotic arm to simultaneously measure the distance to the bolt mounting plane. The end effector's posture is adjusted based on these distance differences to achieve end-effector alignment. A crosshair laser target mounted on the same plane as the displacement sensors is then used to locate and measure the threaded hole. This method effectively solves the problem of the sleeve end face not being parallel to the mounting plane. However, hole centering relies heavily on manual fitting of the target's aperture to the edge of the threaded hole, resulting in low positioning accuracy. Furthermore, the positioning device combining the three displacement sensors and the laser target is bulky, making it unsuitable for positioning in confined spaces.

[0004] On the other hand, a method combining a lever and a distance sensor is used. A mechanical device moves the lever across the mounting surface; when the lever contacts the bolt, it displaces, changing the distance between the distance sensor and the lever, thus detecting the bolt's presence. However, while this method is simple in structure and easy to use, it only detects the presence of bolts and has low positioning accuracy.

[0005] It is evident that the existing bolt positioning methods mostly fail to meet the positioning and guidance requirements of automatic tightening equipment, and are difficult to implement in confined spaces. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method for high-precision identification, positioning, and tightening of bolts, as well as an intelligent tightening device. This solves the problems that traditional methods cannot meet the positioning guidance requirements of automatic tightening devices in terms of positioning accuracy, and are difficult to implement in confined spaces.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a method for high-precision identification, positioning and tightening of bolts, comprising the following steps:

[0008] S1. System environment setup and calibration;

[0009] The calibration includes obtaining the camera's focal length f through camera calibration. x f y Optical center coordinates (cx ,c y The homogeneous transformation matrix f between the camera coordinate system and the robot end effector flange coordinate system is determined through hand-eye calibration, along with the depth value z0. c T; Determine the homogeneous transformation matrix f between the tightening tool coordinate system and the robot end flange coordinate system through tool coordinate system calibration. t T;

[0010] S2. Determine the camera's shooting position, optimal shooting height, default end height, and shooting parameters;

[0011] The optimal shooting height is determined by the shooting height h0 that can capture a clear image of the bolt. The tightening end height H at this height is the default end height. When determining the shooting position, try to ensure that the center of the bolt is located at the center of the image and record the pose information of the robot end flange in the base coordinate system when taking the picture.

[0012] S3. Determine the process parameters of the tightening gun to meet the requirements of the bolt tightening task;

[0013] S4. Perform robot motion path planning and offline programming to ensure that the robot can run and reach every point in an offline state;

[0014] S5. Take a picture of the bolt to obtain a preliminary image, filter the fitted ellipse of the preliminary image, determine the axis of rotation and the rotation angle, then determine and decompose the rotation matrix of the tightening end, and align the tightening end direction according to the step-by-step rotation angle obtained from the decomposition.

[0015] S6. Adjust the height of the tightening end; take another picture of the bolt and obtain the fitted circle of the inscribed circle of the bolt head after screening. Calculate the current shooting height. Based on the optimal shooting height determined in step S2, obtain the adjustment amount of the tightening end height, and then obtain the adjusted tightening end height.

[0016] S7. Take a clear image of the bolt for the last time, delineate the region of interest, and within this region, determine the coordinates of the center point of the bolt to be tightened in the image coordinate system based on the line segments that coincide with the regular hexagonal image of the bolt head, and perform coordinate system transformation to obtain the coordinates of the center point of the bolt to be tightened in the base coordinate system. Then, derive the homogeneous transformation matrix between each coordinate system in turn to determine the pose that the robot end flange needs to achieve in the base coordinate system during the tightening operation.

[0017] S8. Based on the pose obtained in step S7, first control the robot end flange to move horizontally so that the central axis of the sleeve and the central axis of the bolt to be tightened are on the same axis. Then move axially and adjust the axial movement according to the current photo height until the bottom surface of the sleeve is in contact with the mounting surface. First control the tightening gun to complete the cap finding task at a lower angular velocity, and then rotate the tightening gun sleeve at a higher angular velocity to complete the tightening operation.

[0018] Furthermore, in step S5, the specific process includes the following steps:

[0019] S51. Move the tightening equipment to the photo-taking position determined in step S2, call the camera to take a picture of the bolt, obtain a preliminary image, and find all the fitted ellipses in the preliminary image.

[0020] S52. Determine the axis of rotation; Filter the fitted ellipse obtained in step S51 to obtain the fitted ellipse of the bolt to be tightened at the current shooting position. Based on its rotation angle in the image coordinate system, derive the representation of the axis of rotation in the camera coordinate system based on the geometric invariance of rotation.

[0021] S53. Determine the rotation angle; determine the absolute value of the rotation angle based on the eccentricity of the ellipse fitted to the bolt to be tightened, and set positive and negative rules to derive the positive and negative signs of the rotation angle by simulating the camera's attitude after rotation.

[0022] S54. Based on the determined axis and angle, obtain the rotation matrix of the tightening end and perform zxz decomposition. Derive the expressions for the three step-by-step rotation angles of the tightening end, and perform three rotations on the tightening end in sequence to complete the orientation correction.

[0023] Furthermore, in step S52, the specific process includes the following steps:

[0024] S521. Set the length range of the major and minor axes of the fitted ellipse, and perform preliminary screening of all fitted ellipses based on the length range.

[0025] S522. The center distance l between each pair of fitted ellipses is obtained by using the distance formula between two points. The maximum value of the center distance is set to l0. Fitted ellipses with a center distance l less than l0 are screened out as duplicate ellipses. The remaining fitted ellipses are all fitted ellipses of the inscribed circle of the bolt cap.

[0026] S523. Calculate the distance from the center point of the fitted ellipse of the inscribed circle of each bolt cap to the center of the image, and select the fitted ellipse of the inscribed circle of the bolt cap with the smallest distance as the fitted ellipse of the bolt to be tightened under the current shooting position.

[0027] S524. Obtain the center coordinates (x0, y0), length a of the major axis, length b of the minor axis, and rotation angle α0 of the fitted ellipse of the bolt to be tightened in the image coordinate system; the rotation angle α0 is the angle between the major axis of the ellipse and the u-axis in the image coordinate system, 0°≤α0<180°;

[0028] S525. Based on the geometric invariance of rotation, a straight line passing through the origin of the camera coordinate system and parallel to the major axis of the ellipse fitted to the bolt to be tightened is selected as the axis of rotation. The axis of rotation can be expressed in the camera coordinate system as k = cosα0i + sinα0j, where i represents the unit vector in the positive direction of the x-axis of the camera coordinate system and j represents the unit vector in the positive direction of the y-axis of the camera coordinate system.

[0029] Furthermore, in step S53, the specific process includes the following steps:

[0030] S531. Calculate the eccentricity of the ellipse fitted to the bolt to be tightened, and then derive the absolute value of the tilt angle |θ0| of the ellipse fitted to the bolt to be tightened based on the projection principle. This tilt angle is the required rotation angle.

[0031] S532. Set the positive and negative rules for the turning angle:

[0032]

[0033] S533. Based on the positive and negative rules, simulate the camera's attitude after rotating around the rotation axis k in clockwise and counterclockwise directions respectively, and then calculate the rotation matrices of rotation angles θ0 and -θ0 respectively.

[0034] S534. Based on the rotation matrix, derive the z-axis direction vector p = (0,0,1) of the camera coordinate system after rotation in the directions θ0 and -θ0. T The coordinate values ​​p′ and p″ in the base coordinate system;

[0035] S535. Determine the sign of the rotation angle θ0 based on p′ and p″, using the following formula:

[0036]

[0037] In the formula p′ z and p″ z Let p′ and p″ represent the z-axis components of p′ and p″ in the base coordinate system, respectively.

[0038] Furthermore, in step S54, the specific process includes the following steps:

[0039] S541, Based on the rotating shaft and the rotation angle, determine the tightening end rotation matrix R(k,θ0);

[0040] S542. Decompose the rotation matrix R(k,θ0) at the tightening end into a zxz sequence, i.e., R(k,θ0)=R(z,α)R(x,β)R(z,γ), where:

[0041]

[0042] In the above formula, cα = cosα, sα = sinα, and cβ, sβ, cγ, and sγ are similar; R(z,α) represents the first rotation of the tightening end by α degrees along the z-axis of the camera coordinate system, R(x,β) represents the second rotation of the tightening end by β degrees along the x-axis of the coordinate system obtained after the first rotation, and R(z,γ) represents the third rotation of the tightening end by γ degrees along the z-axis of the coordinate system obtained after the second rotation.

[0043] S543. Based on the decomposition results of the tightening end rotation matrix R(k,θ0), derive the expressions for the step-by-step rotation angles around each axis:

[0044]

[0045] S544. Based on the expression derived in step S543, the tightening end is rotated three times in sequence as described in step S542 to complete the alignment of the tightening end.

[0046] Furthermore, in step S6, the specific process includes the following steps:

[0047] S61. After aligning the direction, take another image of the bolt. Set the length range of the fitted circle radius. Perform preliminary screening on all fitted circles according to the length range. Then, perform the same screening as in steps S522-S523 to obtain the fitted circle of the bolt head's inscribed circle. Obtain the pixel area S of this fitted circle and the center coordinates (x, y) of the circle in the image coordinate system. c0 ,y c0 The current shooting height h can be calculated using the following formula:

[0048]

[0049] In the formula, D represents the actual diameter of the inscribed circle of the bolt cap to be tightened, f is the focal length of the camera, and p x The size of a single pixel on a camera sensor;

[0050] S62. Compare the current shooting height h with the optimal shooting height h0 to obtain the adjustment amount h of the tightening end height. ml :

[0051] h m1 =h-h0

[0052] This allows us to obtain the adjusted tightening end height H1:

[0053]

[0054] In the formula, H represents the default end height.

[0055] Furthermore, in step S7, the specific process includes the following steps:

[0056] S71. The center coordinates (x, y) of the fitted circle of the bolt cap inscribed in step S61 in the image coordinate system. c0 ,y c0 ), delineate the region of interest, and within this region, filter line segments according to the properties of a regular hexagon to obtain a set of line segments that coincide with the regular hexagon of the bolt cap;

[0057] S72. Based on the line segment obtained in step S71, obtain the center point of the hexagon of the bolt cap, and then solve the coordinates of the center point of the bolt to be tightened in the image coordinate system.

[0058] S73. Perform coordinate system transformation. First, based on the depth value z0 obtained in step S1, transform the coordinates of the center point of the bolt to be tightened obtained in step S72 from the image coordinate system to the camera coordinate system, and then further transform it to the base coordinate system.

[0059] S74. Sequentially obtain the homogeneous transformation matrix of the tool coordinate system relative to the base coordinate system during the tightening operation. Homogeneous transformation matrix between the tool coordinate system and the robot end flange coordinate system This leads to the homogeneous transformation matrix of the robot's end flange coordinate system relative to the base coordinate system during the tightening operation. according to The required position of the robot's end flange in the base coordinate system during the tightening operation can be obtained. Based on this position and the position information of the robot's end flange in the base coordinate system when taking the picture recorded in step S2, the robot can carry the tightening end actuator to the coordinate position of the center point of the bolt to be tightened in the base coordinate system.

[0060] Furthermore, in step S71, the specific process includes the following steps:

[0061] S711, The center coordinates (x, y) of the fitted circle of the bolt cap in the image coordinate system. c0 ,y c0 Centered on the hexagonal region containing the projection of the bolt cap, the entire circular area is defined as the region of interest. All line segments are then selected from the captured image.

[0062]

[0063] Among them, R ROI The radius represents the region of interest, and (x1, y1) and (x2, y2) represent the coordinates of the two endpoints of the line segment, respectively.

[0064] S712. Calculate the cosine value between two line segments. Set a threshold P to group parallel line segments together:

[0065]

[0066] The threshold P is the minimum cosine value used to determine whether two line segments are parallel. S1 and S2 represent the vectors corresponding to the two line segments, and |S1| and |S2| are the magnitudes of these two vectors.

[0067] S713. Calculate the distance between the midpoints of two line segments and set a threshold L. c0 If the distance between the midpoints exceeds the threshold L, c0 The line segments are divided into two groups; thus, multiple groups of line segments are obtained, and each line segment in each group is determined to represent the same side of the regular hexagon of the bolt head.

[0068] S714. Calculate the distance from the midpoint of the line segment to the center of the fitted circle of the inscribed circle of the bolt head, and select the most suitable line segment from each group. The selection method is as follows:

[0069]

[0070] Among them, l 1c l 2c These represent the distances from the midpoints of line segments 1 and 2 within a group to the center of the fitted circle of the inscribed circle of the bolt head, respectively.

[0071] S715. Create a corresponding retention weight s for each line segment, and set the included angle thresholds θ1 and θ2. When the cosine values ​​of two line segments S1' and S'2 satisfy... When the included angle θ between two line segments satisfies θ2 < θ < θ1, the weight s is increased by 1, where 60° < θ1 < 70° and 50° < θ2 < 60°; an evaluation criterion s0 is established, and the number of line segments entering this screening step is n0, then:

[0072]

[0073] When the retention weight of a line segment satisfies s≥s0, the line segment is retained; finally, a set of line segments coinciding with the regular hexagon of the bolt cap is obtained.

[0074] Furthermore, in step S72, the specific process includes the following steps:

[0075] S721. Given the theoretical radius R of the inscribed circle of the bolt cap, calculate the theoretical radius r of the fitted circle of the bolt cap in the image plane at an image height of h0, based on the imaging principle:

[0076]

[0077] In the formula p x The size of a single pixel on a camera sensor;

[0078] S722. For the line segments obtained in step S71, take any two non-parallel line segments l1 and l2, and find the intersection points of four parallel lines at a distance r from each other. Take the coordinates (x, y) of the center of the fitted circle of the bolt cap in the image coordinate system. c0 ,y c0 The nearest intersection point is the average of all intersection points obtained by repeating this step for every two non-parallel line segments. This average value is the center point of the hexagonal bolt cap.

[0079] S723. Taking a step size of Δl, calculate the coordinates of the points around the center point of the bolt cap hexagon, including the top, bottom, left, and right points (x, y, y). cm ,y cm +Δl), (x cm ,y cm -Δl), (x cm -Δl,y cm ), (x cm +Δl,y cm Further calculate the standard deviation σ of the distances from the above four points and the starting point to each line segment obtained in step S71. u σ d σ l σ r σ m Compare the five standard deviations, take the point corresponding to the minimum value as the new starting point, and repeat this process until the minimum standard deviation appears at the current starting point. Then, denote the current starting point as (x cp ,y cp ), which is the coordinate of the center point of the bolt to be tightened in the image coordinate system.

[0080] Furthermore, the present invention also provides an intelligent tightening device for applying the aforementioned high-precision bolt identification, positioning and tightening method, comprising: an AGV mobile platform, a collaborative robot, a tightening end effector, an industrial camera and its accessories;

[0081] The collaborative robot is fixed to the AGV mobile platform and can move with the AGV mobile platform. All other components are installed at the end of the robot via connectors.

[0082] The tightening end effector consists of a tightening gun, a compressible sleeve, a cylinder, and a reaction arm. During installation, the bottom surface of the compressible sleeve should be kept horizontal with the industrial camera.

[0083] An industrial camera acquires images of the bolts to be tightened, and a series of algorithms adjust the pose of the robot's end flange to identify the position of the center point of the bolts and guide the tightening end effector to perform the tightening operation.

[0084] By employing the above technical solutions, the present invention provides a method for high-precision identification, positioning, and tightening of bolts, as well as an intelligent tightening device, which has at least the following beneficial effects:

[0085] (1) This invention can complete the tasks of aligning the tightening end, adjusting the height, and positioning the bolt center with only one monocular camera. While ensuring the recognition accuracy, it effectively reduces the complexity of the end device, reduces the volume of the tightening end, and is more conducive to bolt positioning in narrow spaces, thereby enhancing the accessibility and applicability of the tightening device.

[0086] (2) This invention utilizes the imaging principle and combines the changes in the geometric characteristics of the bolt cap image under different shooting conditions. By inferring the angle and height of the camera when taking the picture from the image situation, it realizes the alignment and height adjustment of the tightening end, providing a guarantee for high-precision identification of the bolt center.

[0087] (3) The present invention uses a visual recognition method, combined with a multi-layer screening method to achieve high-precision recognition of the center of the bolt cap to be tightened under different pre-tightening states, thereby guiding the end of the collaborative robot to accurately position to the tightening position. Attached Figure Description

[0088] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0089] Figure 1 This is a flowchart of a high-precision bolt identification, positioning, and tightening method proposed in this invention;

[0090] Figure 2 This is a flowchart of the tightening end direction alignment process of the present invention;

[0091] Figure 3 This is a schematic diagram showing the inscribed circle of the bolt cap and the major axis of the ellipse when the tightening end of the present invention rotates around different parallel axes.

[0092] Figure 4 This is a flowchart of the process for identifying the center point and solving the pose of the bolt to be tightened, as proposed in this invention.

[0093] Figure 5 A schematic diagram illustrating the principle of determining the center point of a regular hexagon based on two known non-parallel line segments and the radius of the inscribed circle in this invention.

[0094] Figure 6 This is a schematic diagram illustrating the process of solving for the optimal coordinates of the center point of the bolt to be tightened in this invention.

[0095] Figure 7 This is a schematic diagram of the intelligent tightening device of the present invention;

[0096] In the diagram: 1-AGV mobile platform, 2-collaborative robot, 3-tightening end effector, 4-industrial camera. Detailed Implementation

[0097] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects, and to facilitate its implementation.

[0098] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] Please refer to Figures 1-7 This illustration shows a specific implementation of the present embodiment. This embodiment utilizes imaging principles and geometric characteristics to align and adjust the height of the tightening end, and designs a multi-layer screening method to achieve high-precision identification of the center of the bolt cap to be tightened, guiding the end of the collaborative robot to complete the tightening operation. This achieves high-precision identification of the bolt center in a confined space, providing support for the realization of fully automated bolt tightening.

[0100] Please refer to Figure 1 This embodiment proposes a method for high-precision identification, positioning, and tightening of bolts, which includes the following steps:

[0101] S1. System environment setup and calibration;

[0102] The calibration includes obtaining the camera's focal length f through camera calibration. x f y Optical center coordinates (c x ,c y The camera coordinate system and depth value z0 are determined by hand-eye calibration, which establishes the homogeneous transformation matrix between the camera coordinate system and the robot end effector flange coordinate system. The homogeneous transformation matrix between the tightening tool coordinate system and the robot end flange coordinate system is determined through tool coordinate system calibration. f t T;

[0103] S2. Determine the camera's shooting position, optimal shooting height, default end height, and shooting parameters;

[0104] The optimal shooting height is determined by the shooting height h0 that can capture a suitable and clear image of the bolt. At this height, the tightening end height H is the default end height.

[0105] When determining the shooting position, try to ensure that the center of the bolt is located at the center of the image, and record the pose information of the robot end flange in the base coordinate system when taking the picture; the main shooting parameters include exposure time and gain.

[0106] S3. Determine the process parameters of the tightening gun. Set the process parameters of the tightening gun, such as tightening speed and tightening torque, according to the actual working conditions and process requirements, so as to meet the requirements of the bolt tightening task.

[0107] S4. Perform robot motion path planning and offline programming;

[0108] Based on the digital model information of the equipment to be tightened, the relative positional relationship of the tightening system, and the process parameters obtained in step S3, the robot's motion path is planned to ensure maximum tightening efficiency. Offline programming is used to ensure that the robot can run offline to reach each point and complete the entire workflow. Simultaneously, environmental interference issues must be considered during path planning to avoid collisions and ensure equipment safety. This completes the preparation process for the tightening operation. In this embodiment, the digital model information of the equipment to be tightened is used to avoid collisions between the robot and the equipment when planning the robot's motion path.

[0109] S5. Take photos of the bolt to obtain preliminary images. Filter the fitted ellipse from the preliminary images to determine the axis of rotation and rotation angle. Then determine and decompose the rotation matrix at the tightening end. Based on the decomposed step-by-step rotation angles, align the tightening end direction. The tightening end direction alignment process can be referred to... Figure 2 ;

[0110] As a preferred embodiment of step S5, the specific process includes the following steps:

[0111] S51. Begin the formal tightening operation. Move the tightening equipment to the photo-taking position determined in step S2, call the camera to take a picture of the bolt, and obtain a preliminary image. In order to ensure that the tightening sleeve can smoothly put the bolt into it, it is necessary to ensure that the axis of the sleeve and the axis of the bolt to be tightened are on the same straight line. Due to the influence of processing error and installation error, there is a certain angle between the two in actual operation. Therefore, it is necessary to perform end-of-tightening alignment.

[0112] Since the bolt to be tightened is a hexagonal head bolt, its bolt head top view consists of a regular hexagon and the inscribed circle of the hexagon. When the bolt head is photographed at an angle θ0, the image of the inscribed circle will be an ellipse. According to the imaging principle, the process of finding the alignment at the tightening end is the process of finding the fitted ellipse of the inscribed circle in the image to become a circle. Therefore, the alignment process requires finding all the fitted ellipses in the preliminary image first.

[0113] S52. The axis of rotation for end-point alignment should be parallel to the major axis of the fitted ellipse. The axis of rotation should be determined based on this principle.

[0114] To obtain information about the major axis of the fitted ellipse, the fitted ellipse obtained in step S51 is first filtered to obtain the fitted ellipse of the bolt to be tightened at the current shooting position. Based on its rotation angle in the image coordinate system, the representation of the rotation axis in the camera coordinate system is derived based on the geometric invariance of rotation.

[0115] As a preferred embodiment of step S52, the specific process includes the following steps:

[0116] S521. Define the length range of the major and minor axes of the fitted ellipse, and perform preliminary screening of all fitted ellipses based on the length range; the screening formula is:

[0117]

[0118] Where a and b are the major and minor axes of the currently selected fitted ellipse, respectively. min a max b min b max These represent the minimum and maximum values ​​of the major axis and the minor axis of the fitted ellipse, respectively, that satisfy the set conditions.

[0119] S522. The center distance *l* between any two points of the fitted ellipse is obtained using the formula for the distance between two points. A maximum center distance of *l*0 is set. Fitted ellipses with a center distance *l* less than *l*0 are discarded as duplicate ellipses. The remaining fitted ellipses are all fitted ellipses of the inscribed circle of the bolt cap. The selection formula is:

[0120]

[0121] S523. Calculate the distance from the center point of the fitted ellipse of the inscribed circle of each bolt cap to the center of the image, and select the fitted ellipse of the inscribed circle of the bolt cap with the smallest distance as the fitted ellipse of the bolt to be tightened under the current shooting position.

[0122] S524. Obtain the center coordinates (x0, y0), length a of the major axis, length b of the minor axis, and rotation angle α0 of the fitted ellipse of the bolt to be tightened in the image coordinate system; the rotation angle α0 is the angle between the major axis of the ellipse and the u-axis in the image coordinate system, 0°≤α0<180°;

[0123] S525, Figure 3This image shows the initial state of the tightened end (tilt angle θ0) and the imaging of the inscribed circle of the bolt head (camera top view) and the major axis of the ellipse when rotated around different parallel axes. Due to the geometric invariance of rotation, the end maintains the same orientation after rotation around any axis parallel to the rotation axis. To keep the camera's spatial position as constant as possible, a straight line passing through the origin of the camera coordinate system and parallel to the major axis of the fitted ellipse of the bolt to be tightened is chosen as the rotation axis. The unit vector parallel to the rotation axis can be represented in the image coordinate system as follows: img k = cosα0i + sinα0j; Since the xoy plane of the camera coordinate system is parallel to the uov plane of the image coordinate system, and the x-axis is parallel to the u-axis and in the same direction, and the y-axis is parallel to the v-axis and in the same direction, the rotation axis in the camera coordinate system can be expressed as k = cosα0i + sinα0j, where i represents the unit vector in the positive direction of the x-axis of the camera coordinate system, and j represents the unit vector in the positive direction of the y-axis of the camera coordinate system.

[0124] S53. Determine the rotation angle; determine the absolute value of the rotation angle based on the eccentricity of the ellipse fitted to the bolt to be tightened, and set positive and negative rules to derive the positive and negative signs of the rotation angle by simulating the camera's attitude after rotation.

[0125] As a preferred embodiment of step S53, the specific process includes the following steps:

[0126] S531. Calculate the eccentricity of the ellipse fitted to the bolt to be tightened using the eccentricity formula for an ellipse:

[0127]

[0128] According to the principle of projection, the eccentricity e of the ellipse and the tilt angle θ0 have the following relationship:

[0129] e = |sinθ0|

[0130] According to the above formula, the absolute value of the tilt angle |θ0| can be obtained, and this tilt angle is the required rotation angle.

[0131] S532. Set the positive and negative rules for the turning angle:

[0132]

[0133] S533. Based on the positive and negative rules, simulate the camera's attitude after rotating around the rotation axis k in clockwise and counterclockwise directions respectively, and then calculate the rotation matrix of the rotation angle θ0. Represented as:

[0134]

[0135] In the formula, Versθ0 = (1 - cosθ0), coordinate system {A} represents the camera coordinate system after rotation about axis k, and coordinate system {B} represents the original camera coordinate system. Similarly, the rotation matrix for rotating about axis k by -θ0 can be obtained.

[0136] S534. Based on the properties of rotation matrices:

[0137]

[0138] The rotation matrices of the rotated camera coordinate system relative to the original camera coordinate system are as follows: and Then, the coordinates of the z-axis direction vector of the rotated camera coordinate system in the base coordinate system are as follows:

[0139]

[0140] In the formula The homogeneous transformation matrix of the robot's end flange relative to the base coordinate system in the current state is derived from the pose information of the robot's end flange in the base coordinate system recorded in step S2 during the photo capture; p′ and p″ correspond to the z-axis direction vector p=(0,0,1) of the camera coordinate system after rotation in the directions θ0 and -θ0, respectively. T Coordinate values ​​in the base coordinate system;

[0141] S535. When rotating from top to bottom, the camera coordinate system's z-axis should be vertically downward after correct orientation. Therefore, the z-axis component of this direction vector in the base coordinate system should be less than that obtained by rotating in the wrong direction. The opposite is true when rotating from bottom to top. Therefore, the value of θ0 can be determined using the following formula:

[0142]

[0143] In the formula p′ z and p″ z Let p′ and p″ represent the z-axis components of p′ and p″ in the base coordinate system, respectively. At this point, the rotation axis k and the rotation angle θ0 are determined.

[0144] S54. Based on the determined axis and angle, obtain the rotation matrix of the tightening end and perform zxz decomposition. Derive the expressions for the three step rotation angles of the tightening end. Rotate the tightening end three times in sequence to make the sleeve axis parallel to the axis of the bolt to be tightened, thus completing the orientation alignment.

[0145] As a preferred embodiment of step S54, the specific process includes the following steps:

[0146] S541, Based on the rotating shaft and the rotation angle, determine the tightening end rotation matrix R(k,θ0);

[0147] S542. Since the shaft is located in the xoy plane, the rotation matrix R(k,θ0) at the tightening end is decomposed according to the zxz sequence, i.e., R(k,θ0)=R(z,α)R(x,β)R(z,γ), where:

[0148]

[0149] In the above formula, cα = cosα, sα = sinα, and cβ, sβ, cγ, and sγ are similar; R(z,α) represents the first rotation of the tightening end by α degrees along the z-axis of the camera coordinate system, R(x,β) represents the second rotation of the tightening end by β degrees along the x-axis of the coordinate system obtained after the first rotation, and R(z,γ) represents the third rotation of the tightening end by γ degrees along the z-axis of the coordinate system obtained after the second rotation.

[0150] S543. Based on the decomposition results of the tightening end rotation matrix R(k,θ0), derive the degree expression for the step-by-step rotation angles around each axis:

[0151]

[0152] S544. Based on the expression derived in step S543, the tightening end is rotated three times in sequence as described in step S542 to complete the alignment of the tightening end.

[0153] S6. Adjust the height of the tightening end; take another picture of the bolt and obtain the fitted circle of the inscribed circle of the bolt head after screening. Calculate the current shooting height. Based on the optimal shooting height determined in step S2, obtain the adjustment amount of the tightening end height, and then obtain the adjusted tightening end height.

[0154] As a preferred embodiment of step S6, the specific process includes the following steps:

[0155] S61. After orientation alignment, to obtain a clearer image, the shooting height needs to be adjusted. According to the imaging principle, the pixel area of ​​the fitted circle of the bolt cap's inscribed circle in the image is inversely proportional to the square of the current shooting height. Based on this relationship, the shooting height can be adjusted. Specifically: take another image of the bolt, set the length range of the fitted circle's radius, perform preliminary screening of all fitted circles based on the length range, and then perform the same screening as in steps S522-S523 to obtain the fitted circle of the bolt cap's inscribed circle. Obtain the pixel area S of this fitted circle and the center coordinates (x, y) of the circle in the image coordinate system. c0 ,y c0 The current shooting height h can be calculated using the following formula:

[0156]

[0157] In the formula, D represents the actual diameter of the inscribed circle of the bolt cap to be tightened, f is the focal length of the camera, and p x The size of a single pixel on a camera sensor;

[0158] S62. Compare the current shooting height h with the optimal shooting height h0 to obtain the adjustment amount h of the tightening end height. ml :

[0159] h m1 =h-h0

[0160] This allows us to obtain the adjusted tightening end height H1:

[0161]

[0162] In the formula, H represents the default end height.

[0163] In this embodiment, by utilizing the imaging principle and combining the changes in the geometric characteristics of the bolt cap image under different shooting conditions, the angle and height of the camera when taking the picture are deduced from the image situation, thereby realizing the orientation and height adjustment of the tightening end, and providing a guarantee for high-precision identification of the bolt center.

[0164] S7. After adjusting the end-effector height, take a final photo of the bolt to be tightened to obtain a clear image. Define the region of interest (ROI) within this image. Within this ROI, determine the coordinates of the bolt's center point in the image coordinate system based on the line segments coinciding with the bolt head's hexagonal image. Perform a coordinate system transformation to obtain the coordinates of the bolt's center point in the base coordinate system. Derive the homogeneous transformation matrices between the various coordinate systems sequentially to determine the required pose of the robot's end flange in the base coordinate system during the tightening operation. The process flow can be found in [reference needed]. Figure 4 ;

[0165] As a preferred embodiment of step S7, the specific process includes the following steps:

[0166] S71. Since adjusting the shooting height has little effect on the center coordinates of the fitted circle, the center coordinates (x, y) of the fitted circle of the bolt cap's inscribed circle obtained in step S61 in the image coordinate system are... c0 ,y c0 The region of interest is defined, which excludes other irrelevant parts of the image and leaves only the area of ​​the bolt to be tightened. Within this area, line segments are filtered according to the properties of a regular hexagon to obtain a set of line segments that coincide with the regular hexagon of the bolt head.

[0167] As a preferred embodiment of step S71, the specific process includes the following steps:

[0168] S711, The center coordinates (x, y) of the fitted circle of the bolt cap in the image coordinate system.c0 ,y c0 Centered on the hexagonal region containing the projection of the bolt cap, the entire circular area is defined as the region of interest. All line segments are then selected from the captured image.

[0169]

[0170] Among them, R ROI The radius represents the region of interest, and (x1, y1) and (x2, y2) represent the coordinates of the two endpoints of the line segment, respectively.

[0171] S712. Calculate the cosine value between two line segments. Set a threshold P to group parallel line segments together:

[0172]

[0173] The threshold P is the minimum cosine value for determining whether two line segments are parallel. Line segments that satisfy the above formula are considered parallel. S1 and S2 represent the vectors corresponding to the two line segments, and |S1| and |S2| are the magnitudes of these two vectors.

[0174] S713. Since parallel line segments can represent either the same side or opposite sides of a regular hexagonal bolt head, it is necessary to further distinguish parallel opposite sides within the same group using the midpoint distance. Calculate the midpoint distance between the two line segments and set a threshold L. c0 If the distance between the midpoints exceeds the threshold L, c0 The line segments are divided into two groups; the formula is expressed as:

[0175]

[0176] Among them, (x 1a ,y 1a ), (x 1b ,y 1b ), (x 1c ,y 1c (x) represents the coordinates of the two endpoints and the midpoint of line segment 1, respectively. 2a ,y 2a ), (x 2b ,y 2b ), (x 2c ,y 2c ) represent the coordinates of the two endpoints and the midpoint of line segment 2, respectively; threshold L c0 This is used to determine the maximum allowable distance between the midpoints of two line segments representing the same side of a regular hexagonal bolt head. When the distance between the midpoints of the two line segments is greater than L... c0 If the two parallel line segments represent the parallel opposite sides of the regular hexagon of the bolt cap respectively, they are divided into two groups; thus, multiple groups of line segments are obtained, and each line segment in each group is judged to represent the same side of the regular hexagon of the bolt cap.

[0177] S714. Calculate the distance from the midpoint of the line segment to the center of the fitted circle of the inscribed circle of the bolt head, and select the most suitable line segment from each group. The selection method is as follows:

[0178]

[0179] Among them, l 1c l 2c These represent the distances from the midpoints of line segments 1 and 2 within a group to the center of the fitted circle of the inscribed circle of the bolt head, respectively. In this embodiment, to avoid the influence of protrusions and scratches on the mounting plane on the identification, the line segment closest to the center is retained in each comparison. After traversing all line segments in the group, one line segment is retained at the end of each group.

[0180] S715. Due to issues such as unclean installation surfaces or scratches, the line segments selected in the previous step may not be sides of the bolt's regular hexagon. Therefore, further selection is required. A corresponding retention weight s is created for each line segment as the final criterion for whether to retain it. Since |cos120°|=|cos60°|, the absolute value of the cosine of the angle θ between each line segment and all other line segments is used. Whether the conditions are met is used as the weighting standard.

[0181] Set the included angle thresholds θ1 and θ2. When the cosine values ​​of the two line segments S1' and S'2 satisfy... When the included angle θ between two line segments satisfies θ2 < θ < θ1, the weight s is increased by 1, where 60° < θ1 < 70° and 50° < θ2 < 60°. θ1 and θ2 are adjusted within this range based on actual conditions. An evaluation criterion s0 is established. Let n0 be the number of line segments entering this screening step. Then:

[0182]

[0183] When the retention weight of a line segment satisfies s≥s0, the line segment is retained; finally, a set of line segments coinciding with the regular hexagon of the bolt cap is obtained.

[0184] S72. Due to environmental influences, the line segments obtained through the above steps may not represent every side of the bolt cap hexagon, resulting in missing sides and preventing the fitting of a complete bolt cap hexagon image. Therefore, the property that "the distance from the center of a regular hexagon to each side is equal" is used to further solve for the coordinates of the center point of the bolt to be tightened. This method only requires two non-parallel line segments to obtain the center point of the bolt cap hexagon. Specifically, the center point of the bolt cap hexagon is first obtained based on the line segments obtained in step S71, and then the coordinates of the center point of the bolt to be tightened in the image coordinate system are solved. The principle of obtaining the center point of the bolt cap hexagon can be found in [reference needed]. Figure 5 ;

[0185] As a preferred embodiment of step S72, the specific process includes the following steps:

[0186] S721. Since the bolt to be tightened is a standard part, the theoretical radius R of the inscribed circle of the bolt head is known. Based on the imaging principle, calculate the theoretical radius r of the fitted circle of the inscribed circle of the bolt head in the image plane when the shooting height is h0:

[0187]

[0188] In the formula p x The size of a single pixel on a camera sensor;

[0189] S722. For the line segments obtained in step S71, take any two non-parallel line segments l1 and l2, and find the coordinates of the center of the circle that is simultaneously tangent to both of them and has a radius of r. That is, find the intersection point of the parallel lines that are a distance r from both of them. In the image plane, each line segment has two parallel lines, so a total of four intersection points can be found. Among them, the center coordinates (x, y) of the fitted circle that is a distance r from the inscribed circle of the bolt cap in the image coordinate system are... c0 ,y c0 The nearest intersection point is the center point of the bolt cap hexagon obtained through these two line segments; the specific method for determining this point is as follows:

[0190] Given that (x1, y1) and (x2, y2) are two points on line segment l1, and two points on two parallel lines l1′ and l1″ at a distance r from l1, can be represented as follows:

[0191]

[0192]

[0193] Where (x′1,y′1) and (x′2,y′2) are two points on line segment l′1, and (x″1,y″1) and (x″2,y″2) are two points on line segment l″1. Similarly, we can find two points (x′3,y′3) and (x′4,y′4) on lines l′2 and l″2 parallel to line segment l2, and (x″3,y″3) and (x″4,y″4).

[0194] Next, find the intersection point (x) of l′1 and l′2. 1′2′ ,y 1′2′ ),in:

[0195]

[0196] Similarly, the intersection point (x) of l′1 and l″2 can be obtained. 1′2″ ,y 1′2″ The intersection point (x) of l″1 and l′2 1″2′ ,y 1″2′ The intersection point (x) of l″1 and l″21″2″ ,y 1″2″ Of the four intersection points, only one is needed. Calculate the coordinates (x, y) of each intersection point relative to the center of the circle using the distance formula between two points. c0 ,y c0 The distance between ) is denoted as the nearest intersection point.

[0197] Repeat the above steps for every two non-parallel line segments to obtain a set of intersection points. There are n1 intersection points. The average of these intersection points is taken as the approximate center point, i.e., the center point (x) of the bolt cap hexagon. cm ,y cm ):

[0198]

[0199] S723. Taking a step size of Δl, calculate the coordinates of the points around the center point of the bolt cap hexagon, including the top, bottom, left, and right points (x, y, y). cm ,y cm +Δl), (x cm ,y cm -Δl), (x cm -Δl,y cm ), (x cm +Δl,y cm Further calculate the standard deviation σ of the distances from the above four points and the starting point to each line segment obtained in step S71. u σ d σ l σ r σ m Compare the five standard deviations, take the point corresponding to the minimum value as the new starting point, and repeat this process until the minimum standard deviation appears at the current starting point. Then, denote the current starting point as (x cp ,y cp The coordinates of the center point of the bolt to be tightened in the image coordinate system are the desired coordinates. The process of finding the optimal center point coordinates of the bolt in this step can be found in [reference needed]. Figure 6 Starting from “center 0” in the diagram, obtain its four points “top 0”, “bottom 0”, “left 0”, and “right 0”. Then, based on the standard deviation of the distance from each point to the line segment, select “top 0” as the new starting point and record it as “center 1”. Then, obtain the four points “top 1”, “bottom 1”, “left 1”, and “right 1” of “center 1” (where “center 0” is updated to “bottom 1”). Repeat this process until the coordinates of the center point of the bolt to be tightened are determined.

[0200] In this embodiment, a visual recognition method is used in combination with a designed multi-layer screening method to achieve high-precision identification of the center of the bolt cap to be tightened under different pre-tightening states, thereby guiding the end effector of the collaborative robot to accurately position itself to the tightening position.

[0201] S73. After calculating the coordinates of the center point of the bolt to be tightened in the image coordinate system, it needs to be transformed to the base coordinate system in order to guide the robot's end effector movement. Specifically, firstly, based on the depth value z0 obtained in step S1, the coordinates of the center point of the bolt to be tightened obtained in step S72 are transformed from the image coordinate system to the camera coordinate system:

[0202]

[0203] Where (x) cc ,y cc ,z cc This represents the coordinates of the center point of the bolt to be tightened in the camera coordinate system. A matrix transformation is then performed on these coordinates to convert them to the base coordinate system.

[0204]

[0205] In the formula, This represents the homogeneous transformation matrix from the robot's end effector flange coordinate system to the base coordinate system under the current pose, obtained based on the pose information of the robot's end effector flange in the base coordinate system recorded during step S2; (x cb ,y cb ,z cb ) represents the coordinates of the center point of the bolt to be tightened in the base coordinate system.

[0206] S74. By following the steps above, the position that the center of the socket needs to reach in the base coordinate system during the tightening operation can be determined. The posture of the socket during the tightening operation should be the same as that during the photographing. Based on this, the homogeneous transformation matrix of the tool coordinate system (i.e., the socket coordinate system) relative to the base coordinate system during the tightening operation can be obtained. Then, by using the relative positional relationship between the sleeve and the robot end flange, the homogeneous transformation matrix between the tool coordinate system and the robot end flange coordinate system is obtained. This leads to the homogeneous transformation matrix of the robot's end flange coordinate system relative to the base coordinate system during the tightening operation. according to This allows us to obtain the required pose of the robot's end flange in the base coordinate system during the tightening operation:

[0207]

[0208] Based on this pose and the pose information of the robot's end effector flange in the base coordinate system recorded in step S2 during the photo capture, the robot can carry the tightening end effector to the center point coordinates of the bolt to be tightened in the base coordinate system. At this point, the identification of the center of the bolt to be tightened and the determination of the target point pose of the robot's end effector are complete.

[0209] S8. Based on the pose obtained in step S7, first control the robot's end flange to move horizontally, so that the central axis of the sleeve and the central axis of the bolt to be tightened are on the same axis. Then, move it axially until the bottom surface of the sleeve is in contact with the mounting surface. Since the height of the tightening end was adjusted before identifying the center point of the bolt to be tightened, the axial movement amount h m2 The adjustment needs to be made according to the current shooting height. When the tightening end height is the default H, it means that the bolt has been pre-tightened completely. That is, at this time, the distance between the camera and the bolt surface is h0, the relative height between the bottom surface of the sleeve and the camera remains constant at Δh, and the bolt head thickness is h. t In the default state, the distance between the sleeve and the bolt surface is h0+h t +Δh, then the axial movement adjusted after considering the shooting height should be:

[0210] h m2 =h0+h t +Δh-h m1

[0211] After being moved into place, the lower surface of the sleeve contacts the bolt surface, compresses under force, and the cylinder pushes the reaction arm to extend and abut against the edge of the workpiece. Then, the tightening gun is controlled to operate at a low angular velocity w. l The sleeve is slowly fed for one revolution. When the inner cavity of the sleeve completely aligns with the shape of the bolt head, the bolt head is inserted into the sleeve under the elastic force of the spring and can rotate with the sleeve, completing the nut-finding task. After the nut-finding task is completed, the sleeve rotates at a higher angular velocity. h Tighten the gun sleeve to tighten the bolts; when the tightening torque reaches the required level, the tightening operation is complete.

[0212] In this embodiment, a single monocular camera can complete tasks such as alignment, height adjustment, and bolt center positioning at the tightening end. While ensuring recognition accuracy, it effectively reduces the complexity of the end device, decreases the size of the tightening end, facilitates bolt positioning in confined spaces, and enhances the accessibility and applicability of the tightening device.

[0213] This application also provides an intelligent tightening device for applying the described high-precision bolt identification, positioning, and tightening method. Its structure can be found in [reference needed]. Figure 7 This includes: AGV mobile platforms, collaborative robots, tightening end effectors, industrial cameras and their accessories;

[0214] The collaborative robot is fixed to the AGV mobile platform and can move with the AGV mobile platform. All other components are installed at the end of the robot via connectors.

[0215] The tightening end effector consists of a tightening gun, a compressible sleeve, a cylinder, and a reaction arm. During installation, the bottom surface of the compressible sleeve should be kept horizontal with the industrial camera.

[0216] An industrial camera acquires images of the bolts to be tightened, and a series of algorithms adjust the pose of the robot's end flange to identify the position of the center point of the bolts and guide the tightening end effector to perform the tightening operation.

[0217] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0218] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0219] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for high-precision identification, positioning, and tightening of bolts, characterized in that, Includes the following steps: S1. System environment setup and calibration; The calibration includes obtaining the camera's focal length f through camera calibration. x f y Optical center coordinates (c x ,c y The homogeneous transformation matrix f between the camera coordinate system and the robot end effector flange coordinate system is determined through hand-eye calibration, along with the depth value z0. c T; Determine the homogeneous transformation matrix f between the tightening tool coordinate system and the robot end flange coordinate system through tool coordinate system calibration. t T; S2. Determine the camera's shooting position, optimal shooting height, default end height, and shooting parameters; The optimal shooting height is determined by the shooting height h0 that can capture a clear image of the bolt. The tightening end height H at this height is the default end height. When determining the shooting position, try to ensure that the center of the bolt is located at the center of the image and record the pose information of the robot end flange in the base coordinate system when taking the picture. S3. Determine the process parameters of the tightening gun to meet the requirements of the bolt tightening task; S4. Perform robot motion path planning and offline programming to ensure that the robot can run and reach every point in an offline state; S5. Take a picture of the bolt to obtain a preliminary image, filter the fitted ellipse of the preliminary image, determine the axis of rotation and the rotation angle, then determine and decompose the rotation matrix of the tightening end, and align the tightening end direction according to the step-by-step rotation angle obtained from the decomposition. S6. Adjust the height of the tightening end; take another picture of the bolt and obtain the fitted circle of the inscribed circle of the bolt head after screening. Calculate the current shooting height. Based on the optimal shooting height determined in step S2, obtain the adjustment amount of the tightening end height, and then obtain the adjusted tightening end height. S7. Take a clear image of the bolt for the last time, delineate the region of interest, and within this region, determine the coordinates of the center point of the bolt to be tightened in the image coordinate system based on the line segments that coincide with the regular hexagonal image of the bolt head, and perform coordinate system transformation to obtain the coordinates of the center point of the bolt to be tightened in the base coordinate system. Then, derive the homogeneous transformation matrix between each coordinate system in turn to determine the pose that the robot end flange needs to achieve in the base coordinate system during the tightening operation. S8. Based on the pose obtained in step S7, first control the robot end flange to move horizontally so that the central axis of the sleeve and the central axis of the bolt to be tightened are on the same axis. Then move axially and adjust the axial movement according to the current photo height until the bottom surface of the sleeve is in contact with the mounting surface. First control the tightening gun to complete the cap finding task at a lower angular velocity, and then rotate the tightening gun sleeve at a higher angular velocity to complete the tightening operation.

2. The method for high-precision identification, positioning, and tightening of bolts according to claim 1, characterized in that: Step S5 specifically includes the following steps: S51. Move the tightening equipment to the photo-taking position determined in step S2, call the camera to take a picture of the bolt, obtain a preliminary image, and find all the fitted ellipses in the preliminary image. S52. Determine the axis of rotation; Filter the fitted ellipse obtained in step S51 to obtain the fitted ellipse of the bolt to be tightened at the current shooting position. Based on its rotation angle in the image coordinate system, derive the representation of the axis of rotation in the camera coordinate system based on the geometric invariance of rotation. S53. Determine the rotation angle; determine the absolute value of the rotation angle based on the eccentricity of the ellipse fitted to the bolt to be tightened, and set positive and negative rules to derive the positive and negative signs of the rotation angle by simulating the camera's attitude after rotation. S54. Based on the determined axis and angle, obtain the rotation matrix of the tightening end and perform zxz decomposition. Derive the expressions for the three step-by-step rotation angles of the tightening end, and perform three rotations on the tightening end in sequence to complete the orientation correction.

3. The method for high-precision identification, positioning, and tightening of bolts according to claim 2, characterized in that: The specific process in step S52 includes the following steps: S521. Set the length range of the major and minor axes of the fitted ellipse, and perform preliminary screening of all fitted ellipses based on the length range. S522. The center distance l between each pair of fitted ellipses is obtained by using the distance formula between two points. The maximum value of the center distance is set to l0. Fitted ellipses with a center distance l less than l0 are screened out as duplicate ellipses. The remaining fitted ellipses are all fitted ellipses of the inscribed circle of the bolt cap. S523. Calculate the distance from the center point of the fitted ellipse of the inscribed circle of each bolt cap to the center of the image, and select the fitted ellipse of the inscribed circle of the bolt cap with the smallest distance as the fitted ellipse of the bolt to be tightened under the current shooting position. S524. Obtain the center coordinates (x0, y0), length a of the major axis, length b of the minor axis, and rotation angle α0 of the fitted ellipse of the bolt to be tightened in the image coordinate system; the rotation angle α0 is the angle between the major axis of the ellipse and the u-axis in the image coordinate system, 0°≤α0<180°; S525. Based on the geometric invariance of rotation, a straight line passing through the origin of the camera coordinate system and parallel to the major axis of the ellipse fitted to the bolt to be tightened is selected as the axis of rotation. The axis of rotation can be expressed in the camera coordinate system as k = cosα0i + sinα0j, where i represents the unit vector in the positive direction of the x-axis of the camera coordinate system and j represents the unit vector in the positive direction of the y-axis of the camera coordinate system.

4. The method for high-precision identification, positioning, and tightening of bolts according to claim 2, characterized in that: The specific process in step S53 includes the following steps: S531. Calculate the eccentricity of the ellipse fitted to the bolt to be tightened, and then derive the absolute value of the tilt angle |θ0| of the ellipse fitted to the bolt to be tightened based on the projection principle. This tilt angle is the required rotation angle. S532. Set the positive and negative rules for the turning angle: S533. Based on the positive and negative rules, simulate the camera's attitude after rotating around the rotation axis k in clockwise and counterclockwise directions respectively, and then calculate the rotation matrices of rotation angles θ0 and -θ0 respectively. S534. Based on the rotation matrix, derive the z-axis direction vector p = (0,0,1) of the camera coordinate system after rotation in the directions θ0 and -θ0. T The coordinate values ​​p′ and p″ in the base coordinate system; S535. Determine the sign of the rotation angle θ0 based on p′ and p″, using the following formula: In the formula p z ′ and p z "" represents the z-axis components of p′ and p″ in the base coordinate system, respectively.

5. The method for high-precision identification, positioning, and tightening of bolts according to claim 2, characterized in that: The specific process in step S54 includes the following steps: S541. Based on the rotating shaft k and the rotation angle θ0, determine the tightening end rotation matrix R(k,θ0); S542. Decompose the rotation matrix R(k,θ0) at the tightening end into a zxz sequence, i.e., R(k,θ0)=R(z,α)R(x,β)R(z,γ), where: In the above formula, cα = cosα, sα = sinα, and cβ, sβ, cγ, and sγ are similar; R(z,α) represents the first rotation of the tightening end by α degrees along the z-axis of the camera coordinate system, R(x,β) represents the second rotation of the tightening end by β degrees along the x-axis of the coordinate system obtained after the first rotation, and R(z,γ) represents the third rotation of the tightening end by γ degrees along the z-axis of the coordinate system obtained after the second rotation. S543. Based on the decomposition results of the tightening end rotation matrix R(k,θ0), derive the expressions for the step-by-step rotation angles around each axis: S544. Based on the expression derived in step S543, the tightening end is rotated three times in sequence as described in step S542 to complete the alignment of the tightening end.

6. The method for high-precision identification, positioning, and tightening of bolts according to claim 1, characterized in that: Step S6 specifically includes the following steps: S61. After aligning the direction, take another image of the bolt. Set the length range of the fitted circle radius. Perform preliminary screening on all fitted circles according to the length range. Then, perform the same screening as in steps S522-S523 to obtain the fitted circle of the bolt head's inscribed circle. Obtain the pixel area S of this fitted circle and the center coordinates (x, y) of the circle in the image coordinate system. c0 ,y c0 The current shooting height h can be calculated using the following formula: In the formula, D represents the actual diameter of the inscribed circle of the bolt cap to be tightened, f is the focal length of the camera, and p x The size of a single pixel on a camera sensor; S62. Compare the current shooting height h with the optimal shooting height h0 to obtain the adjustment amount h of the tightening end height. ml : h m1 =h-h0 This allows us to obtain the adjusted tightening end height H1: In the formula, H represents the default end height.

7. The method for high-precision identification, positioning, and tightening of bolts according to claim 6, characterized in that: Step S7 specifically includes the following steps: S71. The center coordinates (x, y) of the fitted circle of the bolt cap inscribed in step S61 in the image coordinate system. c0 ,y c0 ), delineate the region of interest, and within this region, filter line segments according to the properties of a regular hexagon to obtain a set of line segments that coincide with the regular hexagon of the bolt cap; S72. Based on the line segment obtained in step S71, obtain the center point of the hexagon of the bolt cap, and then solve the coordinates of the center point of the bolt to be tightened in the image coordinate system. S73. Perform coordinate system transformation. First, based on the depth value z0 obtained in step S1, transform the coordinates of the center point of the bolt to be tightened obtained in step S72 from the image coordinate system to the camera coordinate system, and then further transform it to the base coordinate system. S74. Sequentially obtain the homogeneous transformation matrix of the tool coordinate system relative to the base coordinate system during the tightening operation. Homogeneous transformation matrix between the tool coordinate system and the robot end flange coordinate system This leads to the homogeneous transformation matrix of the robot's end flange coordinate system relative to the base coordinate system during the tightening operation. according to The required position of the robot's end flange in the base coordinate system during the tightening operation can be obtained. Based on this position and the position information of the robot's end flange in the base coordinate system when taking the picture recorded in step S2, the robot can carry the tightening end actuator to the coordinate position of the center point of the bolt to be tightened in the base coordinate system.

8. The method for high-precision identification, positioning, and tightening of bolts according to claim 7, characterized in that: The specific process in step S71 includes the following steps: S711, The center coordinates (x, y) of the fitted circle of the bolt cap in the image coordinate system. c0 ,y c0 Centered on the hexagonal region containing the projection of the bolt cap, the entire circular area is defined as the region of interest. All line segments are then selected from the captured image. Among them, R ROI The radius represents the region of interest, and (x1, y1) and (x2, y2) represent the coordinates of the two endpoints of the line segment, respectively. S712. Calculate the cosine value between two line segments. Set a threshold P to group parallel line segments together: The threshold P is the minimum cosine value used to determine whether two line segments are parallel. S1 and S2 represent the vectors corresponding to the two line segments, and |S1| and |S2| are the magnitudes of these two vectors. S713. Calculate the distance between the midpoints of two line segments and set a threshold L. c0 If the distance between the midpoints exceeds the threshold L, c0 The line segments are divided into two groups; thus, multiple groups of line segments are obtained, and each line segment in each group is determined to represent the same side of the regular hexagon of the bolt head. S714. Calculate the distance from the midpoint of the line segment to the center of the fitted circle of the inscribed circle of the bolt head, and select the most suitable line segment from each group. The selection method is as follows: Among them, l 1c l 2c These represent the distances from the midpoints of line segments 1 and 2 within a group to the center of the fitted circle of the inscribed circle of the bolt head, respectively. S715. Create a corresponding retention weight s for each line segment, and set the included angle thresholds θ1 and θ2. When the cosine values ​​of two line segments S1' and S'2 satisfy... When the included angle θ between two line segments satisfies θ2 < θ < θ1, the weight s is increased by 1, where 60° < θ1 < 70° and 50° < θ2 < 60°; an evaluation criterion s0 is established, and the number of line segments entering this screening step is n0, then: When the retention weight of a line segment satisfies s≥s0, the line segment is retained; finally, a set of line segments coinciding with the regular hexagon of the bolt cap is obtained.

9. The method for high-precision identification, positioning, and tightening of bolts according to claim 7, characterized in that: The specific process in step S72 includes the following steps: S721. Given the theoretical radius R of the inscribed circle of the bolt cap, calculate the theoretical radius r of the fitted circle of the bolt cap in the image plane at an image height of h0, based on the imaging principle: In the formula p x The size of a single pixel on a camera sensor; S722. For the line segments obtained in step S71, take any two non-parallel line segments l1 and l2, and find the intersection points of four parallel lines at a distance r from each other. Take the coordinates (x, y) of the center of the fitted circle of the bolt cap in the image coordinate system. c0 ,y c0 The nearest intersection point is the average of all intersection points obtained by repeating this step for every two non-parallel line segments. This average value is the center point of the hexagonal bolt cap. S723. Taking a step size of Δl, calculate the coordinates of the points around the center point of the bolt cap hexagon, including the top, bottom, left, and right points (x, y, y). cm ,y cm +Δl), (x cm ,y cm -Δl), (x cm -Δl,y cm ), (x cm +Δl,y cm Further calculate the standard deviation σ of the distances from the above four points and the starting point to each line segment obtained in step S71. u σ d σ l σ r σ m Compare the five standard deviations, take the point corresponding to the minimum value as the new starting point, and repeat this process until the minimum standard deviation appears at the current starting point. Then, denote the current starting point as (x cp ,y cp ), which is the coordinate of the center point of the bolt to be tightened in the image coordinate system.

10. An intelligent tightening device for applying the high-precision bolt identification, positioning, and tightening method as described in any one of claims 1-9, characterized in that, include: AGV mobile platforms, collaborative robots, tightening end effectors, industrial cameras and accessories; The collaborative robot is fixed to the AGV mobile platform and can move with the AGV mobile platform. All other components are installed at the end of the robot via connectors. The tightening end effector consists of a tightening gun, a compressible sleeve, a cylinder, and a reaction arm. During installation, the bottom surface of the compressible sleeve should be kept horizontal with the industrial camera. An industrial camera acquires images of the bolts to be tightened, and a series of algorithms adjust the pose of the robot's end flange to identify the position of the center point of the bolts and guide the tightening end effector to perform the tightening operation.

Images