Flange positioning method and system

Abstract

The invention discloses a flange positioning method and system. The system comprises a training module; the acquisition module is used for acquiring a flange image and point cloud data of the flange during actual operation; the detection module is used for predicting the flange image and outputting a flange label frame, a bolt hole label frame and a bolt hole overall label frame of the flange image; the judgment module is used for screening combinations of the flange label frames and the bolt hole label frames and judging whether the number of the screened combinations of the flange label frames and the bolt hole label frames is consistent with the number of flanges in actual operation or not; the fitting module is used for carrying out circle fitting on the bolt holes; the calculation module is used for calculating the pose of the flange in actual operation. According to the method, the camera image and the laser radar point cloud data are fused to position the flange, the matching precision of the camera image and the laser radar point cloud data is improved through geometric constraint, the flange positioning efficiency and stability in a complex industrial environment can be improved, and the method is suitable for an automatic assembly scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a flange positioning method and system. Background Art

[0002] In the fields of industrial manufacturing and automated assembly, as an important component for pipe connection, the accurate identification and positioning of flanges are crucial for robot operation, assembly accuracy, and production efficiency. Traditional flange positioning methods mainly rely on a single sensor, such as positioning based on monocular or binocular vision, or point cloud registration methods based on lidar. However, these methods cannot achieve the expected results when applied in actual industrial environments.

[0003] Single-sensor methods are difficult to simultaneously meet the requirements of high precision, high robustness, and real-time performance in industrial scenarios. Especially in complex environments, such as uneven lighting and background interference, the positioning accuracy and stability of flanges and their bolt holes are significantly reduced. Summary of the Invention

[0004] Object of the Invention: The first object of the present invention is to provide a flange positioning method that can be applied to complex environments with high precision and strong stability.

[0005] The second object of the present invention is to provide a flange positioning method system.

[0006] Technical Solution: The flange positioning method disclosed by the present invention includes the following steps

[0007] S1: Obtain a training set composed of multiple calibrated flange images, and use the training set to train a target detection algorithm model to obtain a detection model for detecting flange images;

[0008] S2: Use an industrial camera to collect flange images of the flange during actual operation, and simultaneously use a lidar to collect point cloud data of the flange during actual operation. Denote the point set composed of the point cloud data as P a ;

[0009] S3: Input the flange image in step S2 into the detection model, and the detection model outputs the flange label box, bolt hole label box, and overall bolt hole label box of the flange image;

[0010] S4: Set geometric screening conditions to screen combinations of qualified flange label boxes and bolt hole label boxes, and determine whether the number of combinations of qualified flange label boxes and bolt hole label boxes screened is consistent with the number of flanges during actual operation. If it is consistent, proceed to the next step; if it is inconsistent, return to step S2 to re-collect flange images and point cloud data;

[0011] S5: Use the Hough algorithm to perform circular fitting on the bolt holes in the combination of flange label frames and bolt hole label frames that meet the conditions, obtain the image coordinates of the center points of the bolt holes, and record the set of point cloud projection points of the center points of the bolt holes with the obtained image coordinates as P;

[0012] S6: Map the point set P a to the coordinate system where the industrial camera is located to obtain the point set P b , and based on the internal parameter matrix K of the industrial camera and the point set P b calculate to obtain the point set P c of the point cloud data mapped to the flange image coordinate system, and record the point cloud data mapped to the flange image coordinate system as the point cloud projection points;

[0013] S7: Set a threshold based on the flange label frame to filter the point cloud projection points in the point set P c that are located within the flange label frame;

[0014] S8: Map the point cloud projection points in the point set P c that are located within the flange label frame back to the coordinate system of the lidar, and perform statistical filtering on the point cloud projection points mapped back to the lidar coordinates to obtain the target point cloud; Take the intermediate point cloud corresponding to the target point cloud among the point cloud projection points, find the point cloud projection point closest to the point set P in the intermediate point cloud and form the point set P c ', map P to the coordinate system where the industrial camera is located to obtain P d , and based on P c ' and P calculate the three-dimensional coordinates of P d ;

[0015] S9: Calculate the pose of the flange in the coordinate system where the industrial camera is located during actual operation based on the three-dimensional coordinates of the point set P d .

[0016] Further, the geometric screening conditions include inclusive constraint, quantity constraint, relative scale constraint, and circular distribution constraint for screening;

[0017] The inclusive constraint is used to screen the combination of flange label frames and bolt hole label frames where both the bolt hole label frame and the overall bolt hole label frame are completely located within the boundary of the flange label frame;

[0018] The quantity constraint is used to screen the combination of flange label frames and bolt hole label frames where the number of bolt hole label frames located within the same flange label frame is the same as the number of bolt holes within the overall bolt hole label frame;

[0019] The relative scale constraint is used to screen the combination of flange label frames and bolt hole label frames where the size ratio of the flange to the bolt holes meets a preset ratio;

[0020] The annular distribution constraint is used to screen the center points of the bolt hole label frames for the combined flange label frames and bolt hole label frames that are annularly arrayed around the center point of the flange frame.

[0021] Furthermore, when performing geometric screening conditions, inclusive constraint screening, quantity constraint screening, relative scale constraint screening, and annular distribution constraint screening are carried out in sequence.

[0022] Furthermore, the screening method of the inclusive constraint is as follows:

[0023] Calculate the inclusion degree φ of the flange-bolt hole label frame contain and the overall flange-bolt hole label frame φ contain1 , φ contain and φ contain1 The calculation formulas are as follows:

[0024]

[0025] where S boxf∩boxh refers to the area of the intersection of the bolt hole label frame and the flange label frame, S boxh refers to the area of the bolt hole label frame, where S boxf∩box refers to the area of the intersection of the overall bolt hole label frame and the flange label frame, S box refers to the area of the overall bolt hole label frame;

[0026] When φ contain = 1 and φ contain1 = 1, it is determined that the combination of the flange label frame and the bolt hole label frame meets the conditions; otherwise, it is determined that the combination of the flange label frame and the bolt hole label frame does not meet the conditions.

[0027] Furthermore, the screening method of the relative scale constraint is as follows:

[0028] Calculate the scale constraint degree φ scale , φ scale The calculation formula is as follows:

[0029]

[0030] where w h refers to the width of the detected bolt hole label frame, w f refers to the width of the detected flange label frame, h h refers to the detected, h f refers to the height of the detected bolt hole label frame, α is the scale ratio between the bolt hole and the flange; σ w refers to the variance of the widths of all bolt holes in the training set, σ h refers to the variance of the heights of all bolt holes in the training set;

[0031] Based on σ wand σ h Set the threshold γ. When φ scale ≥γ, it is determined that the combination of the flange label box and the bolt hole label box meets the conditions; otherwise, it is determined that the combination of the flange label box and the bolt hole label box does not meet the conditions;

[0032] Furthermore, the screening method of the annular distribution constraint is as follows:

[0033] Calculate the angle θ of the center of each bolt hole relative to the center of the flange respectively i , θ i The calculation formula is as follows:

[0034] θ i = arctan2(u i - u f , v i - v f )

[0035] where (u i , v i ) is the coordinate of the center point of the bolt hole label box, and (u f , v f ) is the coordinate of the center point of the flange label box;

[0036] where the definition and sorting method of the coordinate of the center point of the bolt hole label box (u i , v i ) are as follows:

[0037] First, based on the geometric distance between the center point of the bolt hole label box and the minimum corner point of the flange label box, obtain the center point (u1, v1) of the first-order bolt hole label box;

[0038] Based on the center point of the remaining i-th order bolt hole label box, obtain the center point of the (i + 1)-th bolt hole label box until i + 1 = n, where n refers to the total number of bolt holes in the overall bolt hole label box, and i ∈ [1, n - 1];

[0039] Calculate the angle difference Δθ between two adjacent-order bolt holes i , Δθ i The calculation formula is as follows:

[0040] Δθ i = θ i+1 - θ i , Δθ n = 360° - θ n-1 ;

[0041]

[0042] Set judgment condition 1:

[0043] Set judgment condition 2:

[0044] When both judgment condition 1 and judgment condition 2 are satisfied, it is determined that the combination of the flange label frame and the bolt hole label frame meets the conditions; otherwise, it is determined that the combination of the flange label frame and the bolt hole label frame does not meet the conditions.

[0045] Furthermore, in step S6, the calculation of point set P b is as follows:

[0046] Use a calibrated camera and a lidar to synchronously collect images and point cloud data of multiple checkerboard calibration plates;

[0047] Calculate the internal parameter matrix K of the calibrated camera by the Zhang calibration method, and use the RANSAC algorithm to extract three-dimensional feature points from the point cloud data;

[0048] Use the PnP algorithm combined with nonlinear optimization to calculate the external parameter matrix of the calibrated camera and the lidar, and obtain the rotation matrix R and the translation vector t;

[0049] Point set P a is represented as Point set P b The expression is where m is the number of point cloud data;

[0050] P a and P b The conversion relationship is as follows:

[0051] In step S6, the calculation of point set P c is as follows:

[0052] When there is b in P it means that the point is behind the industrial camera and not in the field of view of the industrial camera, and it needs to be removed. Let the number of removed cloud point data be s;

[0053] Point set P c The expression is

[0054] P b and P c The conversion relationship is as follows:

[0055] Furthermore, in step S7, the method of screening the point cloud projection points in the flange label frame in point set P c is: Screen the point cloud projection points that satisfy the following relationship

[0056]

[0057] where u min , u max , v min , v max are the maximum and minimum values of the boundary coordinates of the flange label frame.

[0058] Furthermore, the method for obtaining the point set P c ' and the depth values of the 3D constraint points in step S8 is as follows:

[0059] The expression of the point set P is P = {p i = (u i , v i ) | p i ∈ R 2 , i ∈ (1, 2,..., n)}, where n refers to the number of bolt holes within the overall label frame of the bolt holes;

[0060] The expression of the point set P c ' is

[0061] P c 's representation in the coordinate system where the industrial camera is located is P c ' and P b 's conversion relationship is as follows:

[0062]

[0063] Converting the point set P to the coordinate system where the industrial camera is located gives P d , and the expression of P d is

[0064]

[0065] The conversion relationships among P, P b ', and P d are as follows:

[0066]

[0067] Furthermore, the method for calculating the pose of the flange in the coordinate system where the industrial camera is located in step S9 is as follows:

[0068] Calculate the planar centroid O f ,

[0069] Decentralize P d with respect to the planar centroid O f to obtain P f ,

[0070]

[0071] Construct the covariance matrix M.

[0072] Perform eigenvalue decomposition on M to obtain M = VΛV T , where Λ is a diagonal matrix, Λ = diag(λ1, λ2, λ3), λ1, λ2, λ3 are eigenvalues and λ1 > λ2 > λ3; the columns of V are eigenvectors v1, v2, v3;

[0073] Since the normal vector N of the plane is the column vector in V corresponding to the smallest eigenvalue, so N = v3, use Calculate to obtain the unit normal vector N in the Z-axis direction Z ;

[0074] Select the center point of the bolt hole label frame with the smallest geometric distance between the center point of the bolt hole label frame and the minimum corner point of the flange label frame, and select the point set P d The point in corresponding to the center point of this bolt hole label frame Define the connection direction between the point f and the plane centroid O as the X-axis direction, and the vector X is expressed as

[0075] Remove the component of the X vector in the Z-axis direction by projection: X' = X - (X · N Z )N Z , and unitize X' to obtain the unit normal vector N in the X-axis direction X ,

[0076] The unit vector N in the Y-axis direction Y , N Y = N X ×N Z ;

[0077] Construct the rotation matrix R f , R f = [N X , N Y , N Z ;

[0078] During actual operation, the pose matrix of the flange in the coordinate system where the industrial camera is located is

[0079] Based on the same inventive concept, the present invention also discloses a flange positioning system, including,

[0080] A training module, configured to obtain a training set composed of multiple flange images with calibrated information, and use the training set to train a target detection algorithm model to obtain a detection model for detecting flange images;

[0081] An acquisition module, which uses an industrial camera to acquire flange images of the flange during actual operation, and synchronously uses a lidar to acquire point cloud data of the flange during actual operation, and denots the point set formed by the point cloud data as P a ;

[0082] A detection module, which can use the detection model of the training module to predict the flange images in the acquisition module, and output the flange label box, bolt hole label box and overall bolt hole label box of the flange image;

[0083] A judgment module, which sets geometric screening conditions to screen the combinations of flange label boxes and bolt hole label boxes that meet the conditions, and judges whether the number of combinations of flange label boxes and bolt hole label boxes that meet the conditions is consistent with the number of flanges during actual operation. If they are consistent, it inputs the combinations of flange label boxes and bolt hole label boxes that meet the conditions into the fitting module; if they are inconsistent, it drives the acquisition module to re-acquire flange images and point cloud data, and drives the detection module and this module again until the number of combinations of flange label boxes and bolt hole label boxes that meet the conditions is consistent with the number of flanges during actual operation;

[0084] A fitting module, which uses the Hough algorithm to perform circle fitting on the bolt holes in the combinations of flange label boxes and bolt hole label boxes that meet the conditions to obtain the image coordinates of the bolt hole center points, and denots the point set of the bolt hole center points with the obtained image coordinates as P;

[0085] A mapping module, which maps the point set P a to the coordinate system where the industrial camera is located to obtain the point set P b , and calculates the point set P b of the point cloud data mapped onto the flange image coordinate system based on the internal parameter matrix K of the industrial camera and the point set P c , and denotes the point cloud data mapped onto the flange image coordinate system as point cloud projection points;

[0086] A screening module, which sets a threshold based on the flange label box to screen the point cloud projection points in the point set P c that are located within the flange label box;

[0087] A conversion module, which maps the point cloud projection points in the point set P c that are located within the flange label box back to the coordinate system of the lidar, and performs statistical filtering on the point cloud projection points mapped back to the lidar coordinates to obtain target point clouds; takes the intermediate point clouds corresponding to the target point clouds among the point cloud projection points, and finds the point cloud projection points in the intermediate point clouds that are closest to the point set P and forms the point set P c', map P to the coordinate system where the industrial camera is located to obtain P d , and based on P c ' and P, calculate the three-dimensional coordinates of P d ;

[0088] A calculation module calculates the pose of the flange in the coordinate system where the industrial camera is located during actual operation based on the three-dimensional coordinates of the point set P d .

[0089] Advantageous effects: Compared with the prior art, the present invention has the following remarkable advantages: The present invention fuses camera images and lidar point cloud data to locate the flange, and can achieve high-precision flange positioning in complex industrial environments such as uneven illumination and background interference, with better stability and accuracy than traditional single sensors or complex registration methods; While combining the images collected by the camera and the point cloud data collected by the lidar, the present invention uses geometric constraints to improve the matching accuracy of the two, and can improve the efficiency and stability of flange positioning in complex industrial environments, and is suitable for automated assembly; The present invention can locate flanges of different specifications simultaneously without knowing the flange size in advance. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 is a flowchart of the method of the present invention;

[0091] Figure 2 is a schematic structural diagram of the method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0092] The technical solutions of the present invention will be further described below with reference to the drawings.

[0093] Example 1

[0094] A flange positioning method disclosed by the present invention, as Figure 1 shown, includes the following steps:

[0095] S1: Obtain a training set composed of multiple calibrated flange images, and use the training set to train a target detection algorithm model to obtain a detection model for detecting flange images.

[0096] The information calibrated in the flange images in the training set includes flange label frames, bolt hole label frames, and bolt hole overall label frames, and the bolt hole overall label frame contains the quantity information of the bolt holes, that is, the total number of bolt holes is included in the bolt hole overall label frame; the total number of bolt holes on different flanges is different, and is represented by the bolt hole overall label frame.

[0097] Preferably, the target detection algorithm model is a YOLOv5 algorithm detection model.

[0098] S2: Use an industrial camera to collect the flange image during actual operation, and simultaneously use a lidar to collect the point cloud data of the flange during actual operation. Denote the point set formed by the point cloud data as P a .

[0099] S3: Input the flange image in step S3 into the detection model, and the detection model outputs the flange label box, bolt hole label box, and overall bolt hole label box of the flange image

[0100] S4: Set geometric screening conditions to screen the combinations of flange label boxes and bolt hole label boxes that meet the conditions, and determine whether the number of combinations of flange label boxes and bolt hole label boxes that meet the conditions screened is the same as the number of flanges during actual operation. If they are the same, proceed to the next step; if not, return to step S2 to re-collect the flange image and point cloud data

[0101] The geometric screening conditions include inclusive constraint, relative scale constraint, circular distribution constraint, and quantity constraint. The combination of flange label box and bolt hole label box that simultaneously satisfies the inclusive constraint, relative scale constraint, circular distribution constraint, and quantity constraint is the combination of flange label box and bolt hole label box that meets the conditions

[0102] The screening method of the inclusive constraint is as follows

[0103] Calculate the flange-bolt hole label box inclusion degree φ contain and the flange-bolt hole overall label box φ contain1 , φ contain and φ contain1 The calculation formulas are as follows

[0104]

[0105] where S boxf∩boxh refers to the area of the intersection of the bolt hole label box and the flange label box, S boxh refers to the area of the bolt hole label box, where S boxf∩box refers to the area of the intersection of the overall bolt hole label box and the flange label box, S box refers to the area of the overall bolt hole label box

[0106] When φ contain = 1 and φ contain1 = 1, it is determined that the combination of the flange label box and the bolt hole label box meets the conditions; otherwise, it is determined that the combination of the flange label box and the bolt hole label box does not meet the conditions

[0107] The inclusive constraint is used to screen the combination of flange label box and bolt hole label box in which both the bolt hole label box and the overall bolt hole label box are completely located within the boundary of the flange label box

[0108] The screening method for quantity constraint is as follows: Determine whether the number of bolt hole label frames within the same flange label frame is consistent with the number of bolt holes within the overall bolt hole label frame. If they are consistent, then determine that the combination of the flange label frame and the bolt hole label frame meets the conditions; if not, then determine that the combination of the flange label frame and the bolt hole label frame does not meet the conditions.

[0109] The quantity constraint is used to screen the combinations of flange label frames and bolt hole label frames where the number of bolt hole label frames within the same flange label frame is consistent with the number of bolt holes within the overall bolt hole label frame.

[0110] The screening method for relative scale constraint is as follows:

[0111] Calculate the scale constraint degree φ scale ,φ scale The calculation formula of φ is as follows:

[0112]

[0113] where w h refers to the width of the detected bolt hole label frame, w f refers to the width of the detected flange label frame, h h refers to the height of the detected bolt hole label frame, h f refers to the height of the detected flange label frame, and α is the scale ratio between the bolt hole and the flange; σ w refers to the variance of the widths of all bolt holes in the training machine, σ h refers to the variance of the heights of all bolt holes in the training set.

[0114] Based on σ w and σ h set the threshold γ. When φ scale ≥γ, then determine that the combination of the flange label frame and the bolt hole label frame meets the conditions; otherwise, determine that the combination of the flange label frame and the bolt hole label frame does not meet the conditions.

[0115] Preferably,

[0116] The relative scale constraint is used to screen the combinations of flange label frames and bolt hole label frames where the size ratio of the flange to the bolt hole meets a preset ratio, that is, the relative scale constraint estimates the reasonable size of the bolt hole based on the size of the flange to avoid misdetection of being too large or too small.

[0117] The screening method for circular distribution constraint is as follows:

[0118] Calculate the angle θ of the center of each bolt hole relative to the center of the flange respectively i ,θ i The calculation formula is as follows:

[0119] θi = arctan2(u i - u f , v i - v f )

[0120] where (u i , v i ) is the coordinate of the center point of the bolt hole label frame, and (u f , v f ) is the coordinate of the center point of the flange label frame.

[0121] The definition and sorting method of the coordinate of the center point of the bolt hole label frame (u i , v i ) are as follows:

[0122] First, based on the geometric distance between the center point of the bolt hole label frame and the minimum corner point of the flange label frame, obtain the center point (u1, v1) of the bolt hole label frame in the first order; that is, calculate the geometric distance between the center point of each bolt hole label frame and the minimum corner point of the flange label frame, and take the center point of the bolt hole label frame corresponding to the minimum geometric distance as the starting point (u1, v1);

[0123] Based on the geometric distance between the center points of the remaining bolt hole label frames and (u1, v1), obtain the center point (u2, v2) of the bolt hole label frame in the second order; that is, calculate the geometric distance between the center points of the remaining bolt hole label frames and (u1, v1) respectively, and take the center point of the bolt label frame corresponding to the minimum geometric distance as (u2, v2);

[0124] Repeat the above step. Based on the center point of the bolt hole label frame in the i-th order, obtain the center point of the bolt hole label frame in the (i + 1)-th order until i + 1 = n, where n refers to the total number of bolt holes in the overall bolt hole label frame, and i ∈ [1, n - 1].

[0125] Calculate the angular difference Δθ i , Δθ i The calculation formula is as follows:

[0126] Δθ i = θ i+1 - θ i , Δθ n = 360° - θ n-1 ;

[0127] Calculate the deviation γ i , γ i The calculation formula is as follows:

[0128]

[0129] Set judgment condition 1:

[0130] Set judgment condition 2:

[0131] When both judgment condition 1 and judgment condition 2 are satisfied, it is determined that the combination of the flange label frame and the bolt hole label frame meets the conditions; otherwise, it is determined that the combination of the flange label frame and the bolt hole label frame does not meet the conditions.

[0132] The annular distribution constraint is used to screen the combination of the flange label frame and the bolt hole label frame in which the center points of the bolt hole label frames are annularly arrayed around the center point of the flange frame.

[0133] Preferably, first perform inclusive constraint screening, then perform quantity constraint screening, then perform relative scale constraint screening, and finally perform annular distribution constraint screening.

[0134] S5: Use the Hough algorithm to perform circle fitting on the bolt holes in the combination of the flange label frame and the bolt hole label frame that meet the conditions, obtain the image coordinates of the center points of the bolt holes, and record the set of points of the center points of the bolt holes for which the image coordinates are obtained as P.

[0135] The set of image coordinates of the obtained center points of the bolt holes is P = {p i =(u i ,v i )|p i ∈R 2 , i ∈ (1, 2,..., n)}, and define P as the 2D constraint points of the flange during actual operation.

[0136] S6: Map the point set P a to the coordinate system where the industrial camera is located to obtain the point set P b , and calculate the point set P b obtained by mapping the point cloud data to the flange image coordinate system based on the internal parameter matrix K of the industrial camera c , and record the point cloud data mapped to the flange image coordinate system as the point cloud projection points.

[0137] First, map the point cloud data to the coordinate system where the industrial camera is located, calculate the intermediate coordinates of the point cloud data in the coordinate system where the industrial camera is located, and then calculate the conversion coordinates of the point cloud data mapped to the coordinate system of the flange image based on the internal parameter matrix of the industrial camera and the intermediate coordinates, and record the point cloud of the point cloud data mapped to the flange image as the point cloud projection points.

[0138] Calculate the external parameter matrix between the calibration camera and the lidar as follows:

[0139] Use the calibration camera and the lidar to synchronously collect multiple groups of images and point cloud data of the checkerboard calibration board;

[0140] Calculate the calibration camera internal parameter matrix by Zhang's calibration method, and use the RANSAC algorithm to extract the three-dimensional feature points in the point cloud data;

[0141] Use the PnP algorithm combined with nonlinear optimization to calculate the external parameter matrix of the calibration camera and the lidar, and obtain the rotation matrix R and the translation vector t.

[0142] The representation of the point cloud data is P a The representation of the intermediate coordinates converted to the coordinate system where the industrial camera is located is where m is the number of point cloud data;

[0143] P a and P b The conversion relationship is as follows: If then it means that the point is located behind the industrial camera, that is, the point is not visible in the field of view of the industrial camera and needs to be removed. Let the number of removed cloud point data be s.

[0144] P b The representation of the converted coordinates mapped to the coordinate system of the flange image is P b and P c The conversion relationship is as follows: where K refers to the internal parameter matrix of the industrial camera; Preferably, when the calculated value is a floating point number, and the integer needs to be retained according to the pixel characteristics.

[0145] S7: Set a threshold based on the flange label box to filter the point cloud projection points located within the flange label box.

[0146]

[0147] where u min ,u max ,v min ,v max are the maximum and minimum values of the boundary coordinates of the flange label box.

[0148] S8: Map the point cloud projection points located within the flange label box in the point set P c back to the coordinate system of the lidar, and perform statistical filtering on the point cloud projection points mapped back to the lidar coordinates to obtain the target point cloud; Take the intermediate point cloud corresponding to the target point cloud among the point cloud projection points, find the point cloud projection point closest to the point set P in the intermediate point cloud and form the point set P c ', map P to the coordinate system where the industrial camera is located to get P d ,and calculate P based on P c ' and P dThe three-dimensional coordinates. Let P d be defined as a 3D constraint point. Based on P c ', calculate the depth value of the 3D constraint point, where the depth value is the z-axis coordinate of the 3D point.

[0149] The point set P c ' is represented as P c ' is obtained after screening, that is, P c ' represents the set of points formed by the intermediate point cloud corresponding to each 2D constraint point.

[0150] P c 's representation in the coordinate system of the industrial camera after transformation is P c ' and P b 's transformation relationship is as follows:

[0151]

[0152] P, P b ', P d have a one-to-one correspondence. The value corresponding to the z-axis coordinate system of P b ' is the depth value corresponding to the 3D constraint point. Let the representation of the 3D constraint point be '

[0153] P, P b , P d 's transformation formula is as follows:

[0154]

[0155] That is, calculate the x-axis coordinate and y-axis coordinate of P d through P, and let the value corresponding to the z-axis coordinate system of P b ' replace the corresponding value on the z-axis of P d to obtain the z-axis coordinate of P d and finally obtain the three-dimensional coordinates of the 3D constraint point.

[0156] S9: Calculate the pose of the flange in the coordinate system of the industrial camera during actual operation based on the three-dimensional coordinates of the point set P d .

[0157] Since there will be errors in calculating 2D and 3D constraint points in the target detection module and the point cloud processing module, the obtained P d is not strictly coplanar in the camera coordinate system. Calculate the flange pose through the centroid:

[0158] Since there will be errors in calculating 2D and 3D constraint points in the target detection module and the point cloud processing module, the obtained P dNot strictly coplanar in the camera coordinate system, calculate the flange pose through the centroid:

[0159] Calculate the planar centroid O of the flange during actual operation f :

[0160]

[0161] Let P d Decentralize all points relative to the planar centroid to obtain P f :

[0162]

[0163] Construct the covariance matrix M:

[0164]

[0165] Since M is a symmetric matrix, perform eigenvalue decomposition on M to obtain:

[0166] M = VΛV T

[0167] where Λ refers to the diagonal matrix, Λ = diag(λ1, λ2, λ3), and λ1, λ2, λ3 are eigenvalues and λ1 > λ2 > λ3;

[0168] The columns of V are the eigenvectors v1, v2, v3;

[0169] The normal vector N of the plane is the column vector in V corresponding to the smallest eigenvalue; then N = v3, and normalizing N gives the unit vector in the Z-axis direction That is, the unit normal vector N is obtained Z .

[0170] Select the center point of the bolt hole label frame with the smallest geometric distance between the center point of the bolt hole label frame and the minimum corner point of the flange label frame, and select the point set P d The point in P corresponding to the center point of this bolt hole label frame That is, take the point in P d Corresponding to the center point (u1, v1) of the first-order bolt hole label frame in P Let The connection direction with the planar centroid O f Be defined as the X-axis direction, denoted as the vector X;

[0171]

[0172] To ensure that the projection of X is orthogonal to the Z-axis, remove its component in the Z-axis direction through projection:

[0173] X = X - (X · N Z )N Z

[0174] Normalize X′ to get the unit vector N in the X-axis direction X ,

[0175] Unit vector N in the Y-axis direction Y By calculating the outer product of the Z axis and the X axis, ensure that the right-hand coordinate system is formed:

[0176] N Y =N X ×N Z

[0177] Construct the rotation matrix R f :

[0178] R f =[N X ,N Y ,N Z ]

[0179] Since the coordinates of the center of mass O are calculated from the coordinates of the flange bolt holes, the center of mass O is taken as the origin of the flange coordinate system. The pose matrix of the flange in the coordinate system where the industrial camera is located is:

[0180] The present invention combines YOLO target detection to project point cloud data onto the flange image plane for screening. The extracted flange edge is clear and has very little noise. Compared with direct image edge detection or point cloud processing, it not only avoids waste of resources, but also improves positioning accuracy and robustness. In addition, the present invention makes full use of the advantages of multiple sensors (lidar), is suitable for various scenarios such as automated assembly and docking, and has higher practical value and environmental adaptability. The present invention can simultaneously locate flanges of different specifications without knowing the flange size in advance.

[0181] Example 2

[0182] The present invention discloses a flange positioning system, such as Figure 2 As shown, it includes a training module, an acquisition module, a detection module, a judgment module, a fitting module, a mapping module, a screening module, a conversion module and a calculation module.

[0183] The training module is used to obtain a training set consisting of multiple flange images with calibrated information, and use the training set to train the target detection algorithm model to obtain a detection model for detecting flange images.

[0184] The information of flange image calibration in the training set includes flange label frame, bolt hole label frame and bolt hole overall label frame, and the bolt hole overall label frame contains the number information of bolt holes, that is, the total number of bolt holes contained in the bolt hole overall label frame; the total number of bolt holes on different flanges is different, which is represented by the bolt hole overall label frame.

[0185] Preferably, the object detection algorithm model is the YOLOv5 algorithm detection model.

[0186] The acquisition module uses an industrial camera to acquire the flange image of the flange during actual operation, and simultaneously uses a lidar to acquire the point cloud data of the flange during actual operation. The point set composed of the point cloud data is denoted as P a 。

[0187] The detection module can use the detection model of the training module to predict the flange image in the acquisition module, and output the flange label box, bolt hole label box and overall bolt hole label box of the flange image.

[0188] The judgment module sets geometric screening conditions to screen the combinations of flange label boxes and bolt hole label boxes that meet the conditions, and judges whether the number of combinations of flange label boxes and bolt hole label boxes that meet the conditions is consistent with the number of flanges during actual operation. If they are consistent, the combinations of flange label boxes and bolt hole label boxes that meet the conditions are input into the fitting module; if they are inconsistent, the acquisition module is driven to re-acquire the flange image and point cloud data, and the detection module and this module are driven again until the number of combinations of flange label boxes and bolt hole label boxes that meet the conditions is consistent with the number of flanges during actual operation.

[0189] The geometric screening conditions include inclusive constraint, relative scale constraint, circular distribution constraint and quantity constraint. The combination of flange label box and bolt hole label box that simultaneously meets the inclusive constraint, relative scale constraint, circular distribution constraint and quantity constraint is the combination of flange label box and bolt hole label box that meets the conditions.

[0190] The screening method of the inclusive constraint is as follows:

[0191] Calculate the flange-bolt hole label box inclusion degree φ contain and the flange-bolt hole overall label box φ contain1 , φ contain and φ contain1 The calculation formulas of are as follows:

[0192]

[0193] where S boxf∩boxh refers to the area of the intersection of the bolt hole label box and the flange label box, S boxh refers to the area of the bolt hole label box, where S boxf∩box refers to the area of the intersection of the overall bolt hole label box and the flange label box, S box refers to the area of the overall bolt hole label box;

[0194] When φ contain = 1 and φ contain1When = 1, it is determined that the combination of the flange label frame and the bolt hole label frame meets the conditions; otherwise, it is determined that the combination of the flange label frame and the bolt hole label frame does not meet the conditions.

[0195] The inclusive constraint is used to screen the combination of the flange label frame and the bolt hole label frame where both the bolt hole label frame and the overall bolt hole label frame are completely within the boundary of the flange label frame.

[0196] The screening method of the quantity constraint is as follows: Determine whether the number of bolt hole label frames within the same flange label frame is consistent with the number of bolt holes within the overall bolt hole label frame. If they are consistent, it is determined that the combination of the flange label frame and the bolt hole label frame meets the conditions; if not, it is determined that the combination of the flange label frame and the bolt hole label frame does not meet the conditions.

[0197] The quantity constraint is used to screen the combination of the flange label frame and the bolt hole label frame where the number of bolt hole label frames within the same flange label frame is consistent with the number of bolt holes within the overall bolt hole label frame.

[0198] The screening method of the relative scale constraint is as follows:

[0199] Calculate the scale constraint degree φ scale , φ scale The calculation formula of is as follows:

[0200]

[0201] where w h refers to the width of the detected bolt hole label frame, w f refers to the width of the detected flange label frame, h h refers to the height of the detected bolt hole label frame, h f refers to the height of the detected flange label frame, α is the scale ratio between the bolt hole and the flange; σ w refers to the variance of the widths of all bolt holes in the training set, σ h refers to the variance of the heights of all bolt holes in the training set.

[0202] Based on σ w and σ h Set the threshold γ. When φ scale ≥γ, it is determined that the combination of the flange label frame and the bolt hole label frame meets the conditions; otherwise, it is determined that the combination of the flange label frame and the bolt hole label frame does not meet the conditions.

[0203] Preferably,

[0204] The relative scale constraint is used to screen the combination of flange label frames and bolt hole label frames where the size ratio of the flange to the bolt holes meets a preset ratio. That is, the relative scale constraint estimates the reasonable size of the bolt holes based on the size of the flange, avoiding misdetection of overly large or small sizes.

[0205] The screening method of the annular distribution constraint is as follows:

[0206] Calculate the angle θ of the center of each bolt hole relative to the center of the flange respectively i , θ i The calculation formula is as follows:

[0207] θ i = arctan2(u i -u f , v i -v f )

[0208] where, (u i , v i ) are the coordinates of the center point of the bolt hole label frame, and (u f , v f ) are the coordinates of the center point of the flange label frame.

[0209] where (u i , v i ) the definition and sorting method of the coordinates of the center point of the bolt hole label frame are as follows:

[0210] First, based on the geometric distance between the center point of the bolt hole label frame and the minimum corner point of the flange label frame, obtain the center point (u1, v1) of the first-order bolt hole label frame; that is, calculate the geometric distance between the center point of each bolt hole label frame and the minimum corner point of the flange label frame, and take the center point of the bolt hole label frame corresponding to the minimum geometric distance as the starting point (u1, v1);

[0211] Based on the geometric distance between the center points of the remaining bolt hole label frames and (u1, v1), obtain the center point (u2, v2) of the second-order bolt hole label frame; that is, calculate the geometric distance between the center points of the remaining bolt hole label frames and (u1, v1) respectively, and take the center point of the bolt label frame corresponding to the minimum geometric distance as (u2, v2);

[0212] Repeat the above step, and based on the i-th order bolt hole label frame, obtain the center point of the (i + 1)-th bolt hole label frame until i + 1 = n, where n refers to the total number of bolt holes in the overall bolt hole label frame, and i ∈ [1, n - 1].

[0213] Calculate the angle difference Δθ between two adjacent-order bolt holes i , Δθ i The calculation formula is as follows:

[0214] Δθi = θ i+1 -θ i , Δθ n = 360° - θ n-1 ;

[0215] Calculate the deviation γ i , γ i The calculation formula of is as follows:

[0216]

[0217] Set judgment condition 1:

[0218] Set judgment condition 2:

[0219] When both judgment condition 1 and judgment condition 2 are satisfied, it is determined that the combination of the flange label frame and the bolt hole label frame meets the conditions; otherwise, it is determined that the combination of the flange label frame and the bolt hole label frame does not meet the conditions.

[0220] The annular distribution constraint is used to screen the combination of the flange label frame and the bolt hole label frame in which the center points of the bolt hole label frames are annularly arrayed with the center point of the flange frame as the center.

[0221] Preferably, first perform inclusive constraint screening, then perform quantity constraint screening, then perform relative scale constraint screening, and finally perform annular distribution constraint screening.

[0222] The fitting module uses the Hough algorithm to perform circular fitting on the bolt holes in the combination of the flange label frame and the bolt hole label frame that meet the conditions, obtains the image coordinates of the bolt hole center points, and records the set of bolt hole center points with the obtained image coordinates as P.

[0223] The set of image coordinates of the obtained bolt hole center points is P = {p i = (u i , v i ) | p i ∈R 2 , i ∈ (1, 2,..., n)}, and defines P as the 2D constraint points of the flange during actual operation.

[0224] The mapping module maps the point set P a to the coordinate system where the industrial camera is located to obtain the point set P b , and calculates the point set P b of the point cloud data mapped to the flange image coordinate system based on the internal parameter matrix of the industrial camera and the point set P c , and records the point cloud data mapped to the flange image coordinate system as the point cloud projection points.

[0225] First, map the point cloud data to the coordinate system where the industrial camera is located, calculate the intermediate coordinates of the point cloud data in the coordinate system where the industrial camera is located, and then calculate the conversion coordinates of the point cloud data mapped to the coordinate system of the flange image based on the internal parameter matrix of the industrial camera and the intermediate coordinates. The point cloud mapped to the flange image is denoted as the point cloud projection point.

[0226] Calculate the external parameter matrix between the calibration camera and the lidar as follows:

[0227] Use the calibration camera and the lidar to synchronously collect images and point cloud data of multiple checkerboard calibration plates;

[0228] Calculate the internal parameter matrix of the calibration camera through the Zhang's calibration method, and use the RANSAC algorithm to extract the three-dimensional feature points in the point cloud data;

[0229] Use the PnP algorithm combined with nonlinear optimization to calculate the external parameter matrix between the calibration camera and the lidar, and obtain the rotation matrix R and the translation vector t.

[0230] The representation of the point cloud data is P a The representation of the intermediate coordinates converted to the coordinate system where the industrial camera is located is where m is the number of point cloud data;

[0231] P a and P b The conversion relationship is as follows: If then it means that the point is behind the industrial camera, that is, the point is not visible in the field of view of the industrial camera and needs to be removed. Let the number of removed cloud point data be s.

[0232] P b The representation of the conversion coordinates mapped to the coordinate system of the flange image is P b and P c The conversion relationship is as follows: where K refers to the internal parameter matrix of the industrial camera; Preferably, when The calculated value is a floating point number, and the integer needs to be retained according to the pixel characteristics.

[0233] The screening module sets a threshold based on the flange label box to screen the point cloud projection points in the point set P c that are located within the flange label box.

[0234]

[0235] where u min ,u max ,v min ,vmax are the maximum and minimum values of the boundary coordinates of the flange label frame.

[0236] A conversion module that projects the point cloud projection points of the point set P c inside the flange label frame back onto the coordinate system of the lidar, and performs statistical filtering on the point cloud projection points mapped back to the lidar coordinates to obtain the target point cloud; select the intermediate point cloud corresponding to the target point cloud among the point cloud projection points, find the point cloud projection point closest to the point set P in the intermediate point cloud and form the point set P c ', map P to the coordinate system where the industrial camera is located to obtain P d , and calculate the three-dimensional coordinates of P c ′ and P. Define P d as the 3D constraint point, and calculate the depth value of the 3D constraint point based on P d '. The depth value is the z-axis coordinate of the 3D point. c '

[0237] The point set P c ' is represented as P c ' is obtained after screening, that is, P c ' represents the point set composed of the intermediate point cloud corresponding to each 2D constraint point.

[0238] P c ' The representation after being converted to the coordinate system where the industrial camera is located is P c ' The conversion relationship with P b ' is as follows:

[0239]

[0240] P, P b ', P d have a one-to-one correspondence. The value corresponding to the z-axis coordinate system of P b ' is the depth value corresponding to the 3D constraint point. Let the representation of the 3D constraint point be '

[0241] P, P b , P d ' The conversion formula is as follows:

[0242]

[0243] That is, calculate the x-axis coordinate and y-axis coordinate of P d through P, and let the value corresponding to the z-axis coordinate system of P b ' replace the corresponding value on the z-axis of P d to obtain P dThe z-axis coordinate, and finally obtain the three-dimensional coordinates of the 3D constraint points.

[0244] A calculation module, based on the point set P d The three-dimensional coordinates are calculated to obtain the pose of the flange in the coordinate system where the industrial camera is located during actual operation.

[0245] Since there will be errors in calculating the 2D and 3D constraint points in the target detection module and the point cloud processing module, the obtained P d Is not strictly coplanar in the camera coordinate system. Calculate the flange pose through the centroid:

[0246] Since there will be errors in calculating the 2D and 3D constraint points in the target detection module and the point cloud processing module, the obtained P d Is not strictly coplanar in the camera coordinate system. Calculate the flange pose through the centroid:

[0247] Calculate the planar centroid O of the flange during actual operation f :

[0248]

[0249] Decentralize all points in P d Relative to the planar centroid to obtain P f :

[0250]

[0251] Construct the covariance matrix M:

[0252]

[0253] Since M is a symmetric matrix, perform eigenvalue decomposition on M to obtain:

[0254] M = VΛV T

[0255] Where Λ refers to the diagonal matrix, Λ = diag(λ1, λ2, λ3), and λ1, λ2, λ3 are eigenvalues and λ1 > λ2 > λ3;

[0256] The columns of V are the eigenvectors v1, v2, v3;

[0257] The normal vector N of the plane is the column vector in V corresponding to the smallest eigenvalue; then N = v3, and normalizing N gives the unit vector in the Z-axis direction That is, obtain the unit normal vector N Z .

[0258] Select the center point of the bolt hole label box with the smallest geometric distance between the center point of the bolt hole label box and the minimum corner point of the flange label box, and select the point set P d The point in P corresponding to the center point of this bolt hole label box That is, take P d The point corresponding to the center point (u1, v1) of the first-order bolt hole label frame in P Let The connection direction with the plane centroid O f be defined as the X-axis direction, denoted as vector X;

[0259]

[0260] To ensure that the projection of X is orthogonal to the Z-axis, remove its component in the Z-axis direction by projection:

[0261] X = X - (X · N Z )N Z

[0262] Normalize X' to obtain the unit vector N in the X-axis direction X ,

[0263] The unit vector N in the Y-axis direction Y Calculate through the cross product of the Z-axis and the X-axis to ensure a right-handed coordinate system:

[0264] N Y = N X × N Z

[0265] Construct the rotation matrix R f : R f = [N X , N Y , N Z

[0266] Since the coordinates of the centroid O are calculated from the flange bolt hole coordinates, the centroid O is used as the origin of the flange coordinate system. Then, the pose matrix of the flange in the coordinate system where the industrial camera is located is: ​

Claims

1. A flange positioning method, characterized in that: It includes the following steps, S1: Obtain a training set composed of multiple calibrated flange images, and use the training set to train an object detection algorithm model to obtain a detection model for detecting flange images; S2: Use an industrial camera to collect the flange image of the flange during actual operation, and simultaneously use a lidar to collect the point cloud data of the flange during actual operation. Denote the point set formed by the point cloud data as P a ; S3: Input the flange image in step S2 into the detection model, and the detection model outputs the flange label box, bolt hole label box, and overall bolt hole label box of the flange image; S4: Set geometric screening conditions, screen the combinations of flange label boxes and bolt hole label boxes that meet the conditions, and judge whether the number of combinations of flange label boxes and bolt hole label boxes that meet the conditions is the same as the number of flanges during actual operation. If they are the same, proceed to the next step; if they are different, return to step S2 to re-collect flange images and point cloud data; S5: Use the Hough algorithm to perform circle fitting on the bolt holes in the combinations of flange label boxes and bolt hole label boxes that meet the conditions to obtain the image coordinates of the center points of the bolt holes, and record the set of center points of the bolt holes with the obtained image coordinates as P; S6: Map the point set P a onto the coordinate system where the industrial camera is located to obtain the point set P b , and based on the internal parameter matrix K of the industrial camera and the point set P b , calculate the point set P of the point cloud data mapped onto the flange image coordinate system c , and record the point cloud data mapped onto the flange image coordinate system as the point cloud projection points; S7: Set a threshold based on the flange label box and filter the point set P c The point cloud projection points within the flange label box in S8: Map the point cloud projection points of the point set P c located within the flange label frame back to the coordinate system of the lidar, and perform statistical filtering on the point cloud projection points mapped back to the lidar coordinates to obtain the target point cloud; select the intermediate point cloud corresponding to the target point cloud from the point cloud projection points, find the point cloud projection point closest to the point set P in the intermediate point cloud and form the point set P c ', map P to the coordinate system where the industrial camera is located to obtain P d , and based on P c ' and P, calculate the three-dimensional coordinates of P d ; S9: Calculate the pose of the flange in the coordinate system where the industrial camera is located during actual operation based on the three-dimensional coordinates of the point set P d ​ 2. The flange positioning method according to claim 1, wherein: The geometric screening conditions include inclusive constraint, quantity constraint, relative scale constraint, and annular distribution constraint; The inclusive constraint is used to screen the combinations of flange label boxes and bolt hole label boxes where both the bolt hole label box and the overall bolt hole label box are completely within the boundary of the flange label box; The quantity constraint is used to screen the combinations of flange label boxes and bolt hole label boxes where the number of bolt hole label boxes within the same flange label box is the same as the number of bolt holes within the overall bolt hole label box; The relative scale constraint is used to screen the combinations of flange label boxes and bolt hole label boxes where the size ratio of the flange to the bolt hole meets a preset ratio; The annular distribution constraint is used to screen the combinations of flange label boxes and bolt hole label boxes where the center points of the bolt hole label boxes are distributed in an annular array around the center point of the flange box.

3. The flange positioning method according to claim 2, wherein: The screening method of the inclusive constraint is as follows: Calculate the inclusion degree φ of the flange-bolt hole label box contain and the overall label box φ of the flange-bolt hole contain1 , φ contain and φ contain1 The calculation formulas are as follows: where S boxf∩boxh refers to the area of the intersection of the bolt hole label frame and the flange label frame, S boxh refers to the area of the bolt hole label frame, where S boxf∩box refers to the area of the intersection of the overall bolt hole label frame and the flange label frame, S box refers to the area of the overall bolt hole label frame; When φ contain = 1 and φ contain1 = 1, it is determined that the combination of the flange label frame and the bolt hole label frame meets the conditions; otherwise, it is determined that the combination of the flange label frame and the bolt hole label frame does not meet the conditions.

4. The flange positioning method according to claim 2, wherein: The screening method of the relative scale constraint is as follows: Calculate the scale constraint degree φ scale , φ scale The calculation formula is as follows: where w h represents the width of the detected bolt hole label box, w f represents the width of the detected flange label box, h h represents the height of the detected bolt hole label box, h f represents the height of the detected flange label box, and α is the scale ratio between the bolt hole and the flange; σ w represents the variance of all bolt hole widths in the training set, σ h represents the variance of all bolt hole heights in the training set; Based on σ w and σ h Set a threshold value γ. When φ scale ≥γ, it is determined that the combination of the flange label box and the bolt hole label box meets the conditions; otherwise, it is determined that the combination of the flange label box and the bolt hole label box does not meet the conditions.

5. The flange positioning method according to claim 2, characterized in that: The screening method of the annular distribution constraint is as follows: Calculate the angle θ of the center of each bolt hole relative to the center of the flange respectively i , θ i The calculation formula is as follows: θ i = arctan2(u i - u f , v i - v f ) Among them, (u i , v i ) is the center point coordinate of the bolt hole label frame, and (u f , v f ) is the center point coordinate of the flange label frame; Among them (u i , v i ), the definition and sorting method of the center point coordinates of the bolt hole label frame are as follows: First, based on the geometric distance between the center point of the bolt hole label box and the minimum corner point of the flange label box, obtain the center point (u1, v1) of the first-order bolt hole label box; Based on the center point of the remaining i-th order bolt hole label box of the bolt hole label box, obtain the center point of the (i + 1)-th bolt hole label box until i + 1 = n, where n refers to the total number of bolt holes within the overall bolt hole label box and i ∈ [1, n - 1]; Calculate the angular difference Δθ between two bolt holes at adjacent ordinal positions i , Δθ i The calculation formula is as follows: Δθ i = θ i+1 - θ i , Δθ n = 360° - θ n-1 ; Set judgment condition 1: Set judgment condition 2: When both judgment condition 1 and judgment condition 2 are satisfied, it is judged that the combination of the flange label box and the bolt hole label box meets the conditions; otherwise, it is judged that the combination of the flange label box and the bolt hole label box does not meet the conditions.

6. The flange positioning method according to claim 5, characterized in that: The steps of calculating the point set P in step S6 b are as follows: Use a calibrated camera and a lidar to synchronously collect images and point cloud data of multiple checkerboard calibration plates; Calculate the internal parameter matrix K of the calibrated camera through the Zhang calibration method, and use the RANSAC algorithm to extract three-dimensional feature points from the point cloud data; Use the PnP algorithm combined with nonlinear optimization to calculate the external parameter matrix of the calibrated camera and the lidar to obtain the rotation matrix R and the translation vector t; Point set P a is represented as Point set P b has the expression where m is the number of point cloud data; P a and P b The conversion relationship is as follows: The step of calculating the point set P in step S6 c is as follows: When P b exists in it means that the point is located behind the industrial camera and is not within the field of view of the industrial camera, so it needs to be removed. Let the number of the removed cloud point data be s; Point set P c The expression of P b and P c The conversion relationship is as follows:

7. The flange positioning method according to claim 6, wherein: Filter the point set P in step S7 c The method for the point cloud projection points located within the flange label frame in where u min and u max , v min and v max are the maximum and minimum values of the boundary coordinates of the flange label frame.

8. The flange positioning method according to claim 7, characterized in that: Obtain the point set P in step S8 d The coordinate method is as follows: The expression of the point set P is P = {p i = (u i , v i ) | p i ∈ R 2 , i ∈ (1, 2, …, n)}, where n refers to the number of bolt holes within the overall label frame of the bolt holes; Point set P c The expression of P c 'The representation in the coordinate system where the industrial camera is located is P c 'and P b 'The conversion relationship is as follows: Convert the point set P to the coordinate system where the industrial camera is located to obtain P d , let P d 's expression be P, P b ', P d The conversion relationship is as follows:

9. The flange positioning method according to claim 8, wherein: The method for calculating the pose of the flange in the coordinate system where the industrial camera is located during actual operation in step S9 is as follows: Calculate the planar centroid O of the flange during actual operation f , Decentralize P d with respect to the planar centroid O f to obtain P f , Construct the covariance matrix M, Performing eigen - decomposition on M gives M = VΛV T , where Λ is a diagonal matrix, Λ = diag(λ1, λ2, λ3), λ1, λ2, λ3 are eigenvalues and λ1 > λ2 > λ3; the columns of V are eigenvectors v1, v2, v3; Since the normal vector N of the plane is the column vector corresponding to the minimum eigenvalue in V, so N = v3. Using calculate to obtain the unit normal vector N in the Z-axis direction Z ; Select the center point of the bolt hole label frame with the smallest geometric distance between the center point of the bolt hole label frame and the minimum corner point of the flange label frame, and select the point set P d The point corresponding to the center point of the bolt hole label frame in Take the point Connect it with the plane centroid O f The connecting direction is defined as the X-axis direction, and the vector X is expressed as Remove the component of the X vector in the Z-axis direction by projection: X' = X - (X · N Z )N Z , and unitize X' to obtain the unit normal vector N in the X-axis direction X , Unit vector N in the Y-axis direction Y , N Y = N X × N Z ; Construct the rotation matrix R f , R f = [N X , N Y , N Z ; The pose matrix of the flange in the coordinate system where the industrial camera is located during actual operation is 10. A flange positioning system according to any one of claims 1 to 9, characterized in that: It includes, A training module, configured to obtain a training set composed of multiple flange images with calibrated information, and use the training set to train an object detection algorithm model to obtain a detection model for detecting flange images; The acquisition module uses an industrial camera to acquire the flange image of the flange during actual operation, and simultaneously uses a lidar to acquire the point cloud data of the flange during actual operation. The point set formed by the point cloud data is denoted as P a ; A detection module, capable of predicting the flange images in the acquisition module by using the detection model of the training module, and outputting a flange label box, a bolt hole label box, and a total bolt hole label box of the flange image; A judgment module, configured to set geometric screening conditions to screen combinations of qualified flange label boxes and bolt hole label boxes, and judge whether the number of combinations of qualified flange label boxes and bolt hole label boxes screened is consistent with the number of flanges during actual operation. If they are consistent, input the combinations of qualified flange label boxes and bolt hole label boxes into the fitting module; if they are inconsistent, drive the acquisition module to re-acquire flange images and point cloud data, and drive the detection module and this module again until the number of combinations of qualified flange label boxes and bolt hole label boxes screened is consistent with the number of flanges during actual operation; A fitting module, which uses the Hough algorithm to perform circle fitting on the bolt holes in the combinations of qualified flange label boxes and bolt hole label boxes to obtain the image coordinates of the bolt hole center points, and records the set of bolt hole center points with the obtained image coordinates as P; Mapping module, mapping the point set P a to the coordinate system where the industrial camera is located to obtain the point set P b , and based on the internal parameter matrix K of the industrial camera and the point set P b calculate the point set P obtained by mapping the point cloud data to the flange image coordinate system c , and record the point cloud data mapped to the flange image coordinate system as the point cloud projection points; Filtering module, based on the flange label box to set a threshold, filters the point set P c the point cloud projection points within the flange label box; The conversion module projects the point cloud projection points within the flange label frame in the point set P c back onto the coordinate system of the lidar, and performs statistical filtering on the point cloud projection points mapped back to the lidar coordinates to obtain the target point cloud; take the intermediate point cloud corresponding to the target point cloud among the point cloud projection points, find the point cloud projection point closest to the point set P in the intermediate point cloud and form the point set P c ', map P to the coordinate system where the industrial camera is located to obtain P d and calculate the three-dimensional coordinates of P c ' and P d ; A calculation module that calculates the pose of the flange in the coordinate system where the industrial camera is located during actual operation based on the three-dimensional coordinates of the point set P d ​

Images