A three-dimensional screw tightening sequence foolproof control method and system based on visual positioning

Abstract

The present invention provides a three-dimensional screw tightening sequence fool-proof control method and system based on visual positioning, which relates to the field of screw tightening technology. The method includes constructing two world coordinate systems with a workpiece and a tool; determining a feature point set, the three-dimensional coordinates of all feature points in the feature point set, the three-dimensional coordinates of all screws in the workpiece coordinate system, and the three-dimensional coordinates of the tool execution end in the tool coordinate system; detecting the two-dimensional image coordinates of the workpiece feature point set and the tool feature point set; obtaining a homogeneous transformation matrix from the tool coordinate system to the workpiece coordinate system; calculating the three-dimensional coordinates of the tool execution end vertex in the workpiece coordinate system; obtaining the screw number corresponding to the current tool execution end vertex, and then controlling the screw tightening order. The present invention solves the problem that in the complex workpiece assembly process, the existing screw tightening process is difficult to implement spatial screw tightening sequence control, which easily causes potential quality risks.

Description

Technical Field

[0001] The present invention relates to the technical field of screw tightening, and in particular to a three-dimensional screw tightening sequence fool-proof control method and system based on visual positioning. Background Art

[0002] During the assembly of complex workpieces, scenarios requiring precise screw tightening sequences require real-time control. However, since the tightening points on complex workpieces may be located in different planes and at different angles, some of which are blind spots for camera vision, existing vision systems cannot directly determine these points. Consequently, spatial tightening sequence control is difficult to implement with existing tightening processes during the assembly of complex workpieces, leading to potential quality risks. Summary of the Invention

[0003] To this end, an embodiment of the present invention provides a three-dimensional screw tightening sequence fool-proof control method and system based on visual positioning, which is used to solve the problem in the prior art that in the process of assembling complex workpieces, the existing screw tightening process is difficult to implement spatial screw tightening sequence control, which easily causes potential quality risks.

[0004] In order to solve the above problems, an embodiment of the present invention provides a three-dimensional screwing sequence foolproof control method based on visual positioning, the method comprising:

[0005] S1: Construct two world coordinate systems based on the workpiece and tool, namely the workpiece coordinate system and the tool coordinate system;

[0006] S2: Based on the workpiece coordinate system, determine a feature point set A on the workpiece, the three-dimensional coordinates of all feature points in the feature point set A, and the three-dimensional coordinates of all screws in the workpiece coordinate system, automatically generate screw numbers according to the three-dimensional coordinates of the screws, and arrange the screw installation sequence; based on the tool coordinate system, determine a feature point set B on the tool, the three-dimensional coordinates of all feature points in the feature point set B, and the three-dimensional coordinates of the tool execution end in the tool coordinate system;

[0007] S3: Use the camera to capture the product image and detect the two-dimensional image coordinates of the workpiece feature point set A and the tool feature point set B;

[0008] S4: Obtain a homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system based on the two-dimensional image coordinates of feature point set A, the three-dimensional coordinates of feature point set A in the workpiece coordinate system, the two-dimensional image coordinates of feature point set B, and the three-dimensional coordinates of feature point set B in the tool coordinate system;

[0009] S5: Calculate the three-dimensional coordinates of the vertex of the tool execution end in the workpiece coordinate system according to the three-dimensional coordinates of the vertex of the tool execution end in the tool coordinate system and the homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system;

[0010] S6: Calculate the distance between the tool execution end vertex and all screws in the workpiece coordinate system, and obtain the screw number corresponding to the current tool execution end vertex. If the screw number matches the screw number specified in the prescribed installation sequence, enable the tool and set the corresponding tightening force of the screw. Otherwise, do not enable the tool and repeat steps S3-S6 until all screws are tightened.

[0011] Preferably, the method for determining the feature point set A on the workpiece or the feature point set B on the tool is:

[0012] If the feature points are in the same plane, the number of feature points shall be no less than 4; if the feature points are not in the same plane, the number of feature points shall be no less than 6.

[0013] The method for using a camera to capture product images and detect the two-dimensional image coordinates of the workpiece feature point set A and the tool feature point set B is as follows:

[0014] Use a camera to capture images of the product, and use deep learning or traditional vision to detect the two-dimensional image coordinates of the workpiece feature set A and the tool feature set B on the image.

[0015] Preferably, the camera intrinsic parameter matrix and camera distortion coefficient are obtained through camera calibration:

[0016]

[0017] distCoeffs = [k1,k2,p1,p2,k3]

[0018] Among them, cameraMatrix represents the camera internal parameter matrix, distCoeffs represents the camera distortion coefficient, k1, k2, k3 represent the radial distortion coefficient, p1, p2 represent the tangential distortion coefficient, focal length (f x ,f y ) and optical center (c x ,c y ).

[0019] Preferably, a method for obtaining a homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system is as follows based on the two-dimensional image coordinates of the feature point set A, the three-dimensional coordinates of the feature point set A in the workpiece coordinate system, the two-dimensional image coordinates of the feature point set B, and the three-dimensional coordinates of the feature point set B in the tool coordinate system:

[0020] According to the two-dimensional image coordinates of feature point set A and the three-dimensional coordinates of feature point set A in the workpiece coordinate system, the homogeneous transformation matrix T1 of the camera coordinate system relative to the workpiece coordinate system is obtained; according to the two-dimensional image coordinates of feature point set B and the three-dimensional coordinates of feature point set B in the tool coordinate system, the homogeneous transformation matrix T2 of the camera coordinate system relative to the tool coordinate system is obtained; through the above two homogeneous transformation matrices T1 and T2, the homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system is obtained.

[0021] Preferably, according to the two-dimensional image coordinates of the feature point set A, the three-dimensional coordinates of the feature point set A in the workpiece coordinate system, the camera intrinsic parameters and the camera distortion parameters, a method for obtaining the homogeneous transformation matrix T1 of the camera coordinate system relative to the workpiece coordinate system is:

[0022] According to the correspondence between the two-dimensional image coordinates of the feature point set A and the three-dimensional coordinates of the feature point set A in the workpiece coordinate system, the solvePnP function in OpenCV is used to obtain the initial pose estimation of the camera pose, the rotation vector and the translation vector of the pose, thereby obtaining the homogeneous transformation matrix T1 of the camera coordinate system relative to the workpiece.

[0023] Preferably, the initialized pose estimate is used as the initial value, and an iterative optimization is performed using a nonlinear optimization algorithm, including Levenberg-Marquardt and Gauss-Newton, to iteratively adjust the pose parameters by minimizing the reprojection error to minimize the reprojection error.

[0024] Preferably, the method for calculating the three-dimensional coordinates of the vertex of the tool execution end in the workpiece coordinate system is as follows:

[0025] p1=T*p2

[0026] Where p2 is in the form of homogeneous coordinates [x, y, z, 1], where x, y, and z are the three-dimensional coordinates of the tool end point in the tool coordinate system, and p1 represents the homogeneous coordinates of the vertex of the tool end point in the workpiece coordinate system [x', y', z', w]. Dividing the last element of the homogeneous coordinates by [x' / w, y' / w, z' / w] is the three-dimensional coordinates of the vertex of the tool end point in the tool coordinate system.

[0027] Preferably, the distance between the tool execution end vertex and all screws is calculated in the workpiece coordinate system, and the method for obtaining the screw number corresponding to the current tool execution end vertex is:

[0028] In the workpiece coordinate system, the distance between the tool execution end vertex and all screws is calculated, and the minimum distance is compared with the set threshold for judgment. If the minimum distance is less than the set threshold, the screw number corresponding to the current tool execution end vertex is determined to be the screw number corresponding to the minimum distance; otherwise, the current tool execution end vertex is determined not to be at any screw position.

[0029] The embodiment of the present invention further provides a three-dimensional screwing sequence foolproof control system based on visual positioning, the system comprising:

[0030] A coordinate system construction module is used to construct two world coordinate systems based on the workpiece and the tool, namely the workpiece coordinate system and the tool coordinate system;

[0031] A three-dimensional coordinate acquisition module is used to determine, based on the workpiece coordinate system, a feature point set A on the workpiece, the three-dimensional coordinates of all feature points in the feature point set A, and the three-dimensional coordinates of all screws in the workpiece coordinate system, automatically generate screw numbers based on the three-dimensional coordinates of the screws, and arrange the screw installation sequence; based on the tool coordinate system, determine a feature point set B on the tool, the three-dimensional coordinates of all feature points in the feature point set B, and the three-dimensional coordinates of the tool execution end in the tool coordinate system;

[0032] The detection module is used to use a camera to capture images of the product and detect the two-dimensional image coordinates of the workpiece feature point set A and the tool feature point set B;

[0033] a transformation matrix acquisition module, for obtaining a homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system based on the two-dimensional image coordinates of the feature point set A, the three-dimensional coordinates of the feature point set A in the workpiece coordinate system, the two-dimensional image coordinates of the feature point set B, and the three-dimensional coordinates of the feature point set B in the tool coordinate system;

[0034] a transformation module, configured to calculate the three-dimensional coordinates of the vertex of the tool execution end in the workpiece coordinate system according to the three-dimensional coordinates of the vertex of the tool execution end in the tool coordinate system and the homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system;

[0035] The control module is used to calculate the distance between the tool execution end vertex and all screws in the workpiece coordinate system, and obtain the screw number corresponding to the current tool execution end vertex. If the screw number matches the screw number specified in the prescribed installation sequence, the tool is enabled and the corresponding tightening force of the screw is set. Otherwise, the tool is not enabled, and the detection module, transformation matrix acquisition module, transformation module, and control module are repeatedly executed until all screws are tightened.

[0036] It can be seen from the above technical solutions that the present invention has the following advantages:

[0037] The present invention provides a three-dimensional screw tightening sequence foolproof control method and system based on visual positioning. The present invention automatically generates screw numbers according to the three-dimensional coordinates of the screws and arranges the screw installation order; uses deep learning to improve the detection accuracy of feature point sets in two-dimensional images; obtains the homogeneous transformation matrix of the tool coordinate system and the workpiece coordinate system through the homogeneous transformation matrix of the workpiece coordinate system and the camera coordinate system and the tool coordinate system and the camera coordinate system, and then obtains the three-dimensional coordinates of the tool execution end vertex in the workpiece coordinate system; finally, in the workpiece coordinate system, by calculating the distance between the tool execution end vertex and all screws, the screw number corresponding to the current tool execution end vertex is obtained, and then the order of screw tightening is controlled; the present invention can realize real-time MPI guidance and installation sequence management and control. And it can assist the force sensor to control the torque of each screw, and tighten each screw more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. Those skilled in the art can derive other drawings based on these drawings without inventive effort. Among them:

[0039] Figure 1 Flowchart of a three-dimensional screwing sequence foolproof control method based on visual positioning according to an embodiment;

[0040] Figure 2 A schematic diagram of establishing a workpiece coordinate system using a workpiece in an embodiment;

[0041] Figure 3 A schematic diagram of establishing a tool coordinate system using a tool in an embodiment;

[0042] Figure 4 Schematic diagram of characteristic points and screw coordinates in the workpiece coordinate system in the embodiment;

[0043] Figure 5 Schematic diagram of the coordinates of the feature points and tool execution end vertices in the tool coordinate system in the embodiment;

[0044] Figure 6 The block diagram is a three-dimensional screwing sequence fool-proof control system based on visual positioning provided in an embodiment. DETAILED DESCRIPTION

[0045] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0046] Example 1

[0047] like Figure 1 As shown, an embodiment of the present invention proposes a three-dimensional screwing sequence foolproof control method based on visual positioning, the method comprising:

[0048] S1: Construct two world coordinate systems based on the workpiece and tool, namely the workpiece coordinate system and the tool coordinate system;

[0049] S2: Based on the workpiece coordinate system, determine a feature point set A on the workpiece, the three-dimensional coordinates of all feature points in the feature point set A, and the three-dimensional coordinates of all screws in the workpiece coordinate system, automatically generate screw numbers according to the three-dimensional coordinates of the screws, and arrange the screw installation sequence; based on the tool coordinate system, determine a feature point set B on the tool, the three-dimensional coordinates of all feature points in the feature point set B, and the three-dimensional coordinates of the tool execution end in the tool coordinate system;

[0050] S3: Use the camera to capture the product image and detect the two-dimensional image coordinates of the workpiece feature point set A and the tool feature point set B;

[0051] S4: Obtain a homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system based on the two-dimensional image coordinates of feature point set A, the three-dimensional coordinates of feature point set A in the workpiece coordinate system, the two-dimensional image coordinates of feature point set B, and the three-dimensional coordinates of feature point set B in the tool coordinate system;

[0052] S5: Calculate the three-dimensional coordinates of the vertex of the tool execution end in the workpiece coordinate system according to the three-dimensional coordinates of the vertex of the tool execution end in the tool coordinate system and the homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system;

[0053] S6: Calculate the distance between the tool execution end vertex and all screws in the workpiece coordinate system, and obtain the screw number corresponding to the current tool execution end vertex. If the screw number matches the screw number specified in the prescribed installation sequence, enable the tool and set the corresponding tightening force of the screw. Otherwise, do not enable the tool and repeat steps S3-S6 until all screws are tightened.

[0054] The present invention provides a three-dimensional screw tightening sequence foolproof control method based on visual positioning, which automatically generates screw numbers according to the three-dimensional coordinates of the screws and arranges the screw installation sequence; uses deep learning to improve the detection accuracy of feature point sets in two-dimensional images; obtains the homogeneous transformation matrix of the tool coordinate system and the workpiece coordinate system through the homogeneous transformation matrix of the workpiece coordinate system and the camera coordinate system and the tool coordinate system and the camera coordinate system, and then obtains the three-dimensional coordinates of the tool execution end vertex in the workpiece coordinate system; finally, in the workpiece coordinate system, by calculating the distance between the tool execution end vertex and all screws, the screw number corresponding to the current tool execution end vertex is obtained, and then the order of screw tightening is controlled; the present invention can realize real-time MPI guidance and installation sequence management and control. It can also assist the force sensor in torque control of each screw, and tighten each screw more accurately.

[0055] Furthermore, in step S1, the origin of the world coordinate system is selected on the workpiece, and the x, y, and z directions of the world coordinate system are determined. This coordinate system is called the workpiece coordinate system. Figure 2 As shown, the coordinate system does not have to be taken at the edge of the workpiece, and it can be selected according to the actual situation. Select the world coordinate system on the tool and determine the x, y, and z directions of the world coordinate system. This coordinate system is called the tool coordinate system, as shown in Figure 3 As shown in the figure, the coordinate system does not have to be taken at the edge of the tool. Both the workpiece coordinate system and the tool coordinate system use the right-hand coordinate system.

[0056] Furthermore, in step S2, a feature point set A on the workpiece is determined. The number of feature points is not less than 4. If there are 4 points, these 4 points are in the same plane. If the feature points are not in the same plane, at least 6 feature points are required. Feature points can be corner points or any identifiable special points. Figure 4 The four corner points in the feature point set A are used to determine the three-dimensional coordinates of all feature points and all screws in the feature point set A in the workpiece coordinate system. The screw numbers are automatically generated based on the three-dimensional coordinates of the screws, for example Figure 4 The six screws, represented by the medium gray dots, are numbered 1, 2, 3, 4, 5, and 6. The MPI programmer arranges the screw installation sequence. Automatic screw numbering begins by grouping the screws into larger groups based on their z coordinates, then into smaller groups based on their y coordinates. The specific screw number is calculated based on the x coordinate and the first two group numbers. Figure 4 The three-dimensional coordinates of each screw in the workpiece coordinate system are: [[x01,y01,z01],[x02,y02,z02],...].

[0057] Figure 4The three-dimensional coordinates of the four feature points A1, A2, A3, and A4 of the feature set A in the workpiece coordinate system are: [[x11, y11, z11], [x12, y12, z12], [x13, y13, z13], [x14, y14, z14]].

[0058] Determine the feature point set B on the tool. The number of feature points should be no less than 4. If there are 4 points, these 4 points should be in the same plane. If the feature points are not in the same plane, at least 6 feature points are required. Feature points can be corner points or any identifiable special points. Figure 5 The four corner points in the red circle constitute feature point set B. Determine the three-dimensional coordinates of all feature points and the tool end point in feature point set B in the tool coordinate system. Figure 5 In the figure, the three-dimensional coordinates of the endpoint of the tool execution end in the workpiece coordinate system are: [x0, y0, z0].

[0059] Figure 5 The three-dimensional coordinates of the four feature points B1, B2, B3, and B4 of the feature set B in the workpiece coordinate system are: [[x21, y21, z21], [x22, y22, z22], [x23, y23, z23], [x24, y24, z24]].

[0060] Furthermore, in step S3, a camera is used to capture a picture of the product, and the camera intrinsic parameter matrix and the camera distortion coefficient are obtained through camera calibration:

[0061]

[0062] distCoeffs = [k1,k2,p1,p2,k3]

[0063] Among them, cameraMatrix represents the camera internal parameter matrix, distCoeffs represents the camera distortion coefficient, k1, k2, k3 represent the radial distortion coefficient, p1, p2 represent the tangential distortion coefficient, focal length (f x ,f y ) and optical center (c x ,c y ).

[0064] The two-dimensional image coordinates of the workpiece feature set A and the tool feature set B on the image are detected by deep learning or traditional vision. Among them, the corresponding coordinates of the four feature points A1, A2, A3, and A4 of feature set A on the two-dimensional image are: [[x11', y11'], [x12', y12'], [x13', y13'], [x14', y14']]

[0065] The corresponding coordinates of the four feature points B1, B2, B3, and B4 of feature set B on the two-dimensional image are: [[x21', y21'], [x22', y22'], [x23', y23'], [x24', y24']].

[0066] Furthermore, in step S4, the homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system is obtained based on the camera intrinsic parameters, the camera distortion coefficient, the two-dimensional image coordinates of the feature point set A, the three-dimensional coordinates of the feature point set A in the workpiece coordinate system, the two-dimensional image coordinates of the feature point set B, and the three-dimensional coordinates of the feature point set B in the tool coordinate system.

[0067] Specifically, according to the correspondence between the two-dimensional image coordinates of the feature point set A and the three-dimensional coordinates of the feature point set A in the workpiece coordinate system, the solvePnP function in OpenCV is used to obtain the initial pose estimation of the camera pose, and the rotation vector and translation vector of the pose are used to obtain the homogeneous transformation matrix T1 of the camera coordinate system relative to the workpiece coordinate system.

[0068] Using the initialized pose estimate as the initial value, iterative optimization is performed using nonlinear optimization algorithms such as Levenberg-Marquardt and Gauss-Newton. The pose parameters are iteratively adjusted by minimizing the reprojection error. Reprojection involves applying the estimated pose to 3D points and projecting them onto an image. The reprojection error between the projected points and the corresponding image is calculated and used as the optimization objective. By minimizing the reprojection error, the pose parameters are adjusted to ensure a better match between the projected points and the image points.

[0069] During this process, 3D feature points and corresponding image points form point pairs. The accuracy of feature point detection on the image significantly impacts the results. Depending on the actual workpiece, deep learning or traditional workpiece vision detection methods should be used. Furthermore, the selection of feature point pairs is crucial. The RANSAC (Random Sample Consensus) algorithm can be used to eliminate outliers and noise. Appropriate point pairs are iteratively selected from all point pairs, filtering out incorrectly matched pairs and improving the accuracy of pose calculations.

[0070] Depending on the actual workpiece, if conditions permit, the number of point pairs can be increased to provide more constraints and improve the accuracy of the pose calculation. Furthermore, the spatial distribution of point pairs on the workpiece also has a certain impact on the results. Try to distribute them throughout the entire space, rather than confining them to a small area within the workpiece.

[0071] Specifically, this step can obtain the camera's pose in the workpiece coordinate system, and the rotation vector r1 and translation vector t1 from the workpiece coordinate system to the camera coordinate system can be obtained. The rotation vector can be used to obtain the rotation matrix R1, and the homogeneous transformation matrix T1 from the workpiece coordinate system to the camera coordinate system can be constructed based on R1 and t1. This step can also be used to obtain the camera's pose in the tool coordinate system, and the rotation vector r2 and translation vector t2 from the tool coordinate system to the camera coordinate system can be obtained. The rotation vector can be used to obtain the rotation matrix R2, and the homogeneous transformation matrix T2 from the tool coordinate system to the camera coordinate system can be constructed based on R2 and t2. The homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system can be obtained through T1 and T2.

[0072] Furthermore, in step S5, the three-dimensional coordinates of the vertex of the tool execution end in the workpiece coordinate system are calculated based on the three-dimensional coordinates of the vertex of the tool execution end in the tool coordinate system and the homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system:

[0073] p1=T*p2

[0074] Where p2 is in the form of homogeneous coordinates [x, y, z, 1], where x, y, and z are the three-dimensional coordinates of the tool end point in the tool coordinate system, and p1 represents the homogeneous coordinates of the vertex of the tool end point in the workpiece coordinate system [x', y', z', w]. Dividing the last element of the homogeneous coordinates by [x' / w, y' / w, z' / w] is the three-dimensional coordinates of the vertex of the tool end point in the tool coordinate system.

[0075] Furthermore, in step S6, the distance between the tool's end vertex and all screws is calculated in the workpiece coordinate system. The minimum distance is compared with a set threshold for comparison. If the minimum distance is less than the set threshold, the screw number corresponding to the current tool's end vertex is determined to be the screw number corresponding to the minimum distance. Otherwise, the current tool's end vertex is determined to be not at any screw location. If the screw number matches the screw number specified in the specified installation sequence, the tool is enabled and the corresponding torque is set; otherwise, the tool is disabled. Steps S3-S6 are repeated until all screws are tightened in order.

[0076] Example 2

[0077] like Figure 6 As shown, the present invention provides a three-dimensional screwing sequence foolproof control system based on visual positioning, the system comprising:

[0078] A coordinate system construction module 10 is used to construct two world coordinate systems based on the workpiece and the tool, namely the workpiece coordinate system and the tool coordinate system;

[0079] The three-dimensional coordinate acquisition module 20 is used to determine, based on the workpiece coordinate system, a feature point set A on the workpiece, the three-dimensional coordinates of all feature points in the feature point set A, and the three-dimensional coordinates of all screws in the workpiece coordinate system, automatically generate screw numbers based on the three-dimensional coordinates of the screws, and arrange the screw installation sequence; based on the tool coordinate system, determine a feature point set B on the tool, the three-dimensional coordinates of all feature points in the feature point set B, and the three-dimensional coordinates of the tool execution end in the tool coordinate system;

[0080] A detection module 30 is used to capture a picture of the product using a camera and detect the two-dimensional image coordinates of the workpiece feature point set A and the tool feature point set B;

[0081] a transformation matrix acquisition module 40 for obtaining a homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system based on the two-dimensional image coordinates of the feature point set A, the three-dimensional coordinates of the feature point set A in the workpiece coordinate system, the two-dimensional image coordinates of the feature point set B, and the three-dimensional coordinates of the feature point set B in the tool coordinate system;

[0082] a transformation module 50 for calculating the three-dimensional coordinates of the vertex of the tool execution end in the workpiece coordinate system based on the three-dimensional coordinates of the vertex of the tool execution end in the tool coordinate system and a homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system;

[0083] The control module 60 is used to calculate the distance between the tool execution end vertex and all the screws in the workpiece coordinate system, and obtain the screw number corresponding to the current tool execution end vertex. If the screw number matches the screw number specified in the prescribed installation sequence, the tool is enabled and the corresponding tightening force of the screw is set. Otherwise, the tool is not enabled, and the detection module, transformation matrix acquisition module, transformation module, and control module are repeatedly executed until all screws are tightened.

[0084] The system is used to implement the above-mentioned three-dimensional screwing sequence fool-proof control method based on visual positioning. In order to avoid redundancy, it is not described here.

[0085] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

[0086] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0087] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0088] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0089] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A three-dimensional screw tightening sequence foolproof control method based on visual positioning, characterized in that: include: S1: Construct two world coordinate systems based on the workpiece and tool, namely the workpiece coordinate system and the tool coordinate system; S2: Based on the workpiece coordinate system, determine a feature point set A on the workpiece, the three-dimensional coordinates of all feature points in the feature point set A, and the three-dimensional coordinates of all screws in the workpiece coordinate system, automatically generate screw numbers according to the three-dimensional coordinates of the screws, and arrange the screw installation sequence; based on the tool coordinate system, determine a feature point set B on the tool, the three-dimensional coordinates of all feature points in the feature point set B, and the three-dimensional coordinates of the tool execution end in the tool coordinate system; S3: Use the camera to capture the product image and detect the two-dimensional image coordinates of the workpiece feature point set A and the tool feature point set B; S4: Based on the two-dimensional image coordinates of feature point set A, the three-dimensional coordinates of feature point set A in the workpiece coordinate system, the two-dimensional image coordinates of feature point set B, and the three-dimensional coordinates of feature point set B in the tool coordinate system, a homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system is obtained, including: According to the two-dimensional image coordinates of the feature point set A, the three-dimensional coordinates of the feature point set A in the workpiece coordinate system, the camera intrinsic parameters and the camera distortion parameters, the homogeneous transformation matrix T1 of the camera coordinate system relative to the workpiece coordinate system is obtained; According to the two-dimensional image coordinates of the feature point set B, the three-dimensional coordinates of the feature point set B in the tool coordinate system, the camera intrinsic parameters and the camera distortion parameters, the homogeneous transformation matrix T2 of the camera coordinate system relative to the tool coordinate system is obtained; through the above two homogeneous transformation matrices T1 and T2, the homogeneous transformation matrix T between the tool coordinate system and the workpiece coordinate system is obtained; The method for obtaining the homogeneous transformation matrix T1 of the camera coordinate system relative to the workpiece coordinate system is: According to the correspondence between the two-dimensional image coordinates of the feature point set A and the three-dimensional coordinates of the feature point set A in the workpiece coordinate system, the solvePnP function in OpenCV is used to obtain the initial pose estimation of the camera pose, the rotation vector and translation vector of the pose, and the homogeneous transformation matrix T1 of the camera coordinate system relative to the workpiece coordinate system; S5: Calculate the three-dimensional coordinates of the vertex of the tool execution end in the workpiece coordinate system according to the three-dimensional coordinates of the vertex of the tool execution end in the tool coordinate system and the homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system; S6: Calculate the distance between the tool execution end vertex and all screws in the workpiece coordinate system, and obtain the screw number corresponding to the current tool execution end vertex. If the screw number matches the screw number specified in the prescribed installation sequence, enable the tool and set the corresponding tightening force of the screw. Otherwise, do not enable the tool and repeat steps S3-S6 until all screws are tightened.

2. The three-dimensional screwing sequence foolproof control method based on visual positioning according to claim 1 is characterized in that: The method for determining the feature point set A on the workpiece or the feature point set B on the tool is: If the feature points are in the same plane, the number of feature points shall be no less than 4; if the feature points are not in the same plane, the number of feature points shall be no less than 6.

3. The three-dimensional screwing sequence foolproof control method based on visual positioning according to claim 1 is characterized in that: The method for using a camera to capture product images and detect the two-dimensional image coordinates of the workpiece feature point set A and the tool feature point set B is as follows: Use a camera to capture images of the product, and use deep learning or traditional vision to detect the two-dimensional image coordinates of the workpiece feature set A and the tool feature set B on the image.

4. The three-dimensional screwing sequence foolproof control method based on visual positioning according to claim 3 is characterized in that: The camera intrinsic parameter matrix and camera distortion coefficient are obtained through camera calibration: distCoeffs = [k1,k2,p1,p2,k3] Among them, cameraMatrix represents the camera internal parameter matrix, distCoeffs represents the camera distortion coefficient, k1, k2, k3 represent the radial distortion coefficient, p1, p2 represent the tangential distortion coefficient, focal length (f x ,f y ) and optical center (c x ,c y ).

5. The three-dimensional screwing sequence foolproof control method based on visual positioning according to claim 1 is characterized in that: The initialized pose estimate is used as the initial value, and nonlinear optimization algorithms are used for iterative optimization. The optimization algorithms include Levenberg-Marquardt and Gauss-Newton. The pose parameters are iteratively adjusted by minimizing the reprojection error to minimize the reprojection error.

6. The three-dimensional screwing sequence foolproof control method based on visual positioning according to claim 1 is characterized in that: According to the three-dimensional coordinates of the vertex of the tool execution end in the tool coordinate system and the homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system, the method for calculating the three-dimensional coordinates of the vertex of the tool execution end in the workpiece coordinate system is: p1=T*p2 Where p2 is in the form of homogeneous coordinates [x, y, z, 1], where x, y, and z are the three-dimensional coordinates of the tool end point in the tool coordinate system, and p1 represents the homogeneous coordinates of the vertex of the tool end point in the workpiece coordinate system [x', y', z', w]. Dividing the last element of the homogeneous coordinates by [x' / w, y' / w, z' / w] is the three-dimensional coordinates of the vertex of the tool end point in the tool coordinate system.

7. The three-dimensional screwing sequence foolproof control method based on visual positioning according to claim 1 is characterized in that: The method for calculating the distance between the tool end vertex and all screws in the workpiece coordinate system and obtaining the screw number corresponding to the current tool end vertex is as follows: In the workpiece coordinate system, the distance between the tool execution end vertex and all screws is calculated, and the minimum distance is compared with the set threshold for judgment. If the minimum distance is less than the set threshold, the screw number corresponding to the current tool execution end vertex is determined to be the screw number corresponding to the minimum distance; otherwise, the current tool execution end vertex is determined not to be at any screw position.

8. A three-dimensional screwing sequence foolproof control system based on visual positioning, characterized in that: The system is used to implement the three-dimensional screwing sequence foolproof control method based on visual positioning according to any one of claims 1 to 7, comprising: A coordinate system construction module is used to construct two world coordinate systems based on the workpiece and the tool, namely the workpiece coordinate system and the tool coordinate system; A three-dimensional coordinate acquisition module is used to determine, based on the workpiece coordinate system, a feature point set A on the workpiece, the three-dimensional coordinates of all feature points in the feature point set A, and the three-dimensional coordinates of all screws in the workpiece coordinate system, automatically generate screw numbers based on the three-dimensional coordinates of the screws, and arrange the screw installation sequence; based on the tool coordinate system, determine a feature point set B on the tool, the three-dimensional coordinates of all feature points in the feature point set B, and the three-dimensional coordinates of the tool execution end in the tool coordinate system; The detection module is used to use a camera to capture images of the product and detect the two-dimensional image coordinates of the workpiece feature point set A and the tool feature point set B; a transformation matrix acquisition module, for obtaining a homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system based on the two-dimensional image coordinates of the feature point set A, the three-dimensional coordinates of the feature point set A in the workpiece coordinate system, the two-dimensional image coordinates of the feature point set B, and the three-dimensional coordinates of the feature point set B in the tool coordinate system; a transformation module, configured to calculate the three-dimensional coordinates of the vertex of the tool execution end in the workpiece coordinate system according to the three-dimensional coordinates of the vertex of the tool execution end in the tool coordinate system and the homogeneous transformation matrix T from the tool coordinate system to the workpiece coordinate system; The control module is used to calculate the distance between the tool execution end vertex and all screws in the workpiece coordinate system, and obtain the screw number corresponding to the current tool execution end vertex. If the screw number matches the screw number specified in the prescribed installation sequence, the tool is enabled and the corresponding tightening force of the screw is set. Otherwise, the tool is not enabled, and the detection module, transformation matrix acquisition module, transformation module, and control module are repeatedly executed until all screws are tightened.

Images