A method and system for real-time estimation of flange pose

Abstract

This invention discloses a method and system for real-time estimation of flange pose. The system includes: a camera capturing module for real-time image capture; a detection module for identifying the flange and flange bolt holes in the captured images and obtaining flange images and flange bolt hole images; a first-frame flange pose recognition and localization module for calculating the initial flange pose T0 of the first frame based on the bolt hole image of the first frame and the PNP algorithm; and a non-first-frame flange pose recognition and localization module for calculating the true flange pose T of the b-th frame based on the flange image and the initial flange pose T0. b This invention enables real-time detection of the true position and orientation of a flange. It can extract the flange edge undergoing low-speed translational motion in space and achieve flange positioning in real time. Except for the initial frame positioning which requires the use of flange bolt holes for auxiliary positioning, subsequent frame flange edge extraction and positioning only require image processing of the flange image, ensuring accuracy while avoiding resource waste.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method and system for real-time estimation of flange pose. Background Technology

[0002] Edge detection is one of the most fundamental problems in computer vision, with wide applications such as image segmentation, object detection, and video object segmentation. Given an input image, edge detection aims to extract accurate object boundaries and visually salient edges. However, most edge detection algorithms rely on low-level local cues (such as color and texture) to obtain edges, resulting in poor edge detection performance against complex image backgrounds. In the assembly of flanges or the handling of flange workpieces, the working environment is complex, and the flanges are in operation during the process. Existing edge detection algorithms perform poorly on flanges, failing to meet the demands of practical operations. Summary of the Invention

[0003] Purpose of the invention: The first objective of this invention is to provide a method for real-time estimation of flange pose that is suitable for complex working environments, can extract moving flange edges, and has high detection accuracy.

[0004] A second objective of this invention is to provide a system for real-time estimation of flange pose.

[0005] Technical solution: This invention discloses a method for real-time estimation of flange pose, comprising the following steps:

[0006] S1: The camera captures images in real time during the operation;

[0007] S2: Use the YOLO neural network algorithm to pre-train a detection model for identifying flanges, and use the trained detection model to identify flanges and flange bolt holes in the captured images to obtain flange images and flange bolt hole images;

[0008] S3: Combine the bolt hole image and the PNP algorithm to calculate the initial flange pose T0 of the first frame flange;

[0009] S4: Crop the flange image from the image captured in frame b;

[0010] S5: Use the Canny edge detection algorithm to detect the flange edges in the cropped flange image to obtain the original flange edge image;

[0011] S6: Transform the coordinate point set of the original flange edge image to the flange pose T of the (b-1)th frame. b-1 The planar point cloud located in the XY plane;

[0012] S7: Use a consistency check algorithm to perform circle fitting on the planar point cloud to obtain the coordinates of the flange center;

[0013] S8: Select target point cloud based on flange radius to form flange edge point set;

[0014] S9: Back-project the flange edge point set onto the captured image in step S1 to obtain the point set of the target flange edge image;

[0015] S10: Use the least squares method to fit the point set of the target flange edge image to an ellipse to obtain the ambiguous flange pose.

[0016] S11: Eliminate the ambiguity of the fuzzy flange pose and obtain the true flange pose T of the flange in the current b-th frame. b ;

[0017] S12: Repeat steps S6 to S11 to detect the actual flange position in real time until the operation is completed.

[0018] Furthermore, the steps for calculating the initial flange pose T0 of the first frame flange in step S3 are as follows:

[0019] Cropping the flange bolt hole image from the first frame of the captured image;

[0020] The Canny edge detection algorithm was used to detect the edges of the flange bolt holes in each flange bolt hole image.

[0021] Hough circle fitting is performed on the flange bolt hole image after edge detection to calculate the image coordinates of the center point of the flange bolt hole.

[0022] Establish a world coordinate system at the center point of the flange, and obtain the actual coordinates of the center point of the flange bolt holes in the world coordinate system based on the data of the flange bolt holes and the center point of the flange obtained by prior actual measurement.

[0023] Calculate the distance between all flange bolt holes and the upper left corner of the flange YOLO frame based on the image coordinates of the center point. Record the center of the flange bolt hole corresponding to the minimum distance as Center. Calculate the polar angle between the center point of the remaining flange bolt holes and the upper left corner of the flange YOLO frame, and sort the polar angles from largest to smallest. Determine the quadrant of all flange bolt holes in the world coordinate system based on the sorting.

[0024] The initial flange pose T0 of the first frame flange is calculated using the PNP algorithm, which combines the actual coordinates, the quadrant it belongs to, and the image coordinates.

[0025] Furthermore, the steps in step S6 to convert the planar point cloud are as follows:

[0026] Obtain the coordinate point set Q of the original flange edge image;

[0027] Mapping the coordinate point set Q onto the camera coordinate system yields the 3D point set P mapped onto the camera coordinate system.c ;

[0028] The three-dimensional point set P c Transformed into flange pose T b-1 The planar point cloud located in the XY plane.

[0029] Furthermore, the coordinate point set Q is transformed into a three-dimensional point set P. c The formula is:

[0030] ,

[0031] ,

[0032] Q and P c Between , ,

[0033] Where R refers to the set of real numbers. Used to describe the coordinates of a two-dimensional point. Used to describe the coordinates of a 3D point, K refers to the camera intrinsic parameter matrix, ( () refers to the principal point of the plane of the image captured in step S1, ( (This refers to the camera's focal length.)

[0034] Furthermore, the three-dimensional point set P c Transformed into flange pose T b-1 The transformation method for the planar point cloud in the XY plane is as follows:

[0035] Let P be a three-dimensional point set. c Transition to flange pose T b-1 The three-dimensional point set in coordinates is ,

[0036] 3D point set P c With the three-dimensional point set P f The transformation formula is:

[0037] ,

[0038] The three-dimensional point set P f Projected onto flange pose T b-1 On the XY plane, after eliminating the influence of the Z-axis, the projected planar point cloud is obtained as follows: .

[0039] Furthermore, the steps for obtaining the flange edge point set in step S8 are as follows: traverse all point clouds in the planar point cloud, calculate the distance from each point cloud to the flange center, set a distance threshold based on the flange radius, select point clouds that meet the threshold conditions to form an intermediate point set, and reassign the z-component of the point clouds on the Z-axis back to the intermediate point set to obtain the flange edge point set. .

[0040] Furthermore, the formula for obtaining the point set of the target flange edge image in step S9 is as follows:

[0041] The point set of the target flange edge image is ,

[0042] And P w and satisfy .

[0043] Furthermore, the steps in step S10 to obtain the ambiguous fuzzy flange pose are as follows:

[0044] The points in the flange edge image are determined by the coordinates of the captured image ( The coordinates projected onto the camera coordinate system are (X, Y, Z), and the two satisfy the following relationship:

[0045] ,

[0046] The point set of the target flange edge image is obtained using the least squares method. By performing ellipse fitting, the general form of the ellipse is obtained.

[0047]

[0048] ,

[0049] Combining the above formula, we can obtain

[0050] =0,

[0051] This formula is the expression for the elliptical cone formed by the optical center of the camera and the edge of the target flange. The ellipse in the captured image, obtained by projecting the flange edge, is a cross-section of the elliptical cone.

[0052] Solving for the spatial circular pose using the general formula for an ellipse reveals the existence of a real symmetric matrix P that satisfies...

[0053] ,

[0054] ,

[0055] There exist orthogonal matrices Real symmetric matrices can be Diagonalization, i.e.: );

[0056] A point (X,Y,Z) in the camera coordinate system is transformed into a point in the world coordinate system after being rotated using the P matrix. , , The transformation formula is as follows:

[0057] ,

[0058] From the above equation, we can obtain the general form of a standard elliptic cone: =0;

[0059] The formulas for calculating the center of the spatial circle and the normal vector of the cross-section are as follows:

[0060] ,

[0061] ,

[0062] ,

[0063] ,

[0064] Where r is the flange radius, and t1 and t2 are the world coordinates of the flange face. The flange face normal vector, Let be the eigenvalues ​​of matrix P.

[0065] Furthermore, the step of eliminating ambiguity in step S11 is as follows: Calculate With N b-1 The included angle The flange pose corresponding to the minimum included angle is taken as the true flange pose T. b , where N b-1 T b-1 The normal vector corresponding to the XY plane

[0066] ,

[0067] .

[0068] Based on the same inventive concept, this invention also discloses a system for real-time estimation of flange pose, comprising,

[0069] Camera capturing module, used for real-time image capture;

[0070] The detection module is used to identify flanges and flange bolt holes in captured images and to obtain flange images and flange bolt hole images;

[0071] The first-frame flange pose recognition and positioning module calculates the initial flange pose T0 of the first-frame flange based on the bolt hole image of the first frame and the PNP algorithm.

[0072] The non-first frame flange pose recognition and localization module calculates the true flange pose T of the b-th frame based on the flange image and the initial flange pose T0. bReal-time detection of the flange's true pose; the non-first frame flange pose recognition and positioning module includes the following sub-modules:

[0073] The acquisition submodule is used to crop the flange image from the captured image in frame b;

[0074] The edge detection submodule uses the Canny edge detection algorithm to detect the flange edges in the cropped flange image and obtain the original flange edge image.

[0075] The planar point cloud submodule is used to obtain the flange pose T of the flange in the (b-1)th frame by transforming the coordinate point set of the original flange edge image. b-1 The planar point cloud located in the XY plane;

[0076] The circular fitting submodule uses a consistency check algorithm to perform circular fitting on the planar point cloud to obtain the coordinates of the flange center.

[0077] The filtering submodule filters the target point cloud based on the flange radius to form a flange edge point set;

[0078] The back projection submodule is used to back project the flange edge point set onto the image captured by the camera module to obtain the point set of the target flange edge image;

[0079] The ellipse fitting submodule uses the least squares method to fit the point set of the target flange edge image to an ellipse, resulting in an ambiguous flange pose.

[0080] The ambiguity elimination module is used to eliminate the ambiguity of the fuzzy flange pose and obtain the true flange pose T of the flange in the current b-th frame. b .

[0081] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: The present invention can extract the flange edge that is moving at low speed in space in real time and realize flange positioning; the present invention eliminates the need for flange bolt hole-assisted positioning in the first frame positioning, and the flange edge extraction and positioning in subsequent consecutive frames only requires image processing of the flange image, which avoids resource waste while ensuring accuracy; in addition, the present invention can extract the flange edge with relatively clear and low noise through the mapping-projection method, which improves the accuracy compared with ellipse detection directly on the image. Attached Figure Description

[0082] Figure 1 This is a flowchart of the method of the present invention;

[0083] Figure 2 This is a flowchart illustrating the calculation of the initial flange pose T0 using the method of the present invention;

[0084] Figure 3 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0085] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0086] Example 1

[0087] The present invention provides a method for real-time estimation of flange pose, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0088] S1: The camera captures images in real time during the operation.

[0089] S2: A detection model for flange recognition is pre-trained using the YOLO neural network algorithm. The trained detection model is then used to identify the flange and flange bolt holes in the captured images, resulting in flange images and flange bolt hole images. During the pre-training of the detection model, an image of the target flange is selected, its dimensions are obtained, and the target flange image is input into the YOLO neural network algorithm model for training.

[0090] Preferably, when the detection model identifies the flange and flange bolt holes in the captured image, it first determines whether the coordinates of the flange bolt hole YOLO frame are within the range of the flange YOLO frame. Then, it combines the flange and flange bolt holes according to the number of flange bolt holes on the flange. That is, it judges the number of flange bolt holes in the flange YOLO frame based on prior knowledge and filters the flange image and flange bolt hole image that meet the conditions, which helps to reduce the probability of misidentification.

[0091] S3: Combining the bolt hole image and the PNP algorithm, the initial flange pose T0 of the first frame flange is calculated.

[0092] Cropping the flange bolt hole image from the first frame of the captured image;

[0093] The flange bolt hole images are preprocessed, and then the Canny edge detection algorithm is used to detect the flange bolt hole edges in each flange bolt hole image; preferably, the flange bolt hole images are subjected to median filtering before edge detection.

[0094] Hough circle fitting is performed on the flange bolt hole image after edge detection to calculate the image coordinates of the center point of the flange bolt hole.

[0095] Establish a world coordinate system at the center point of the flange, and obtain the actual coordinates of the center point of the flange bolt holes in the world coordinate system based on the data of the flange bolt holes and the center point of the flange obtained by prior actual measurement.

[0096] Calculate the distance between all flange bolt holes and the upper left corner of the flange YOLO frame based on the image coordinates of the center point. Record the center of the flange bolt hole corresponding to the minimum distance as Center. Calculate the polar angle between the center point of the remaining flange bolt holes and the upper left corner of the flange YOLO frame, and sort the polar angles from largest to smallest. Determine the quadrant of all flange bolt holes in the world coordinate system based on the sorting.

[0097] The initial flange pose T0 of the first frame flange is calculated using the PNP algorithm, which combines the actual coordinates, the quadrant it belongs to, and the image coordinates.

[0098] S4: Crop the flange image from the captured image in frame b. Preferably, remove redundant information from the captured image in step S1.

[0099] S5: Use the Canny edge detection algorithm to detect the flange edges in the cropped flange image to obtain the original flange edge image.

[0100] S6: Transform the coordinate point set of the original flange edge image to the flange pose T of the (b-1)th frame. b-1 The planar point cloud in the XY plane. When solving for the flange pose of the first frame, the coordinate point set of the flange edge image is transformed into the planar point cloud in the XY plane where T0 is located.

[0101] The original flange edge image is mapped to the camera coordinate system as follows:

[0102] Obtain the set of coordinate points of the original flange edge image Where R refers to the set of real numbers, Used to describe the coordinates of a two-dimensional point;

[0103] Let the coordinate point set Q of the original flange edge image be mapped to the three-dimensional point set in the camera coordinate system as follows: ,in Used to describe the coordinates of a three-dimensional point;

[0104] Q and P c The conversion relationship between them is:

[0105] ,

[0106] In the formula, K refers to the camera intrinsic parameter matrix. ,

[0107] in The principal point of the plane of the captured image in step S1 ("principal point" refers to the optical center of the camera). This refers to the camera's focal length.

[0108] The 3D point set P in the camera coordinate system c Transformed into flange pose T b-1The planar point cloud in the XY plane is transformed as follows:

[0109] Let P be a three-dimensional point set. c Transition to flange pose T b-1 The three-dimensional point set in coordinates is ,

[0110] 3D point set P c With the three-dimensional point set P f The transformation formula is:

[0111] ,

[0112] The three-dimensional point set P f Projected onto flange pose T b-1 On the XY plane, after eliminating the influence of the Z-axis, the projected planar point cloud is obtained as follows: .

[0113] S7: Use a consistency check algorithm to perform circle fitting on the planar point cloud to obtain the coordinates of the flange center.

[0114] S8: Select target point cloud based on flange radius to form flange edge point set.

[0115] Traverse all point clouds, calculate the distance from each point cloud to the flange center, set a distance threshold based on the flange radius, select point clouds that meet the threshold to form an intermediate point set, and reassign the Z-axis components back to the intermediate point set to obtain the flange edge point set. .

[0116] S9: Back-project the flange edge point set onto the image to obtain the point set of the target flange edge image.

[0117] The flange edge point set P w The target flange edge image is obtained by projecting it onto the captured image in step S1, and the projection method is as follows:

[0118] Let the set of points in the coordinates of the image captured in step S1 after the target flange edge image is imaged be: ,

[0119] P w and The transformation formula is:

[0120] .

[0121] S10: Use the least squares method to fit the point set of the target flange edge image to an ellipse, and obtain the ambiguous flange pose.

[0122] The points in the flange edge image are determined by the coordinates of the captured image ( The coordinates projected onto the camera coordinate system are (X, Y, Z), and the two satisfy the following relationship:

[0123] ,

[0124] The point set of the target flange edge image is obtained using the least squares method. By performing ellipse fitting, the general form of the ellipse is obtained.

[0125]

[0126]

[0127] Combining the above formula, we can obtain

[0128] =0

[0129] This formula is the expression for the elliptical cone formed by the optical center of the camera and the edge of the target flange, while the ellipse in the captured image obtained by projecting the flange edge is a cross-section of the elliptical cone.

[0130] Solving for the spatial circular pose using the general formula for an ellipse, and based on the general formula for an ellipse and the projection formula from world coordinates to the coordinates of the captured image in step S1, it is known that there exists a real symmetric matrix P satisfying...

[0131] ,

[0132] ,

[0133] According to the properties of matrices, there exist orthogonal matrices. Real symmetric matrices can be Diagonalization, i.e.: );

[0134] A point (X,Y,Z) in the camera coordinate system is transformed into a point in the world coordinate system after being rotated using the P matrix. , , ),

[0135] ,

[0136] From the above equation, we can obtain the general form of a standard elliptic cone: =0,

[0137] The spatial circle pose derived from the general formula of an ellipse is ambiguous, and the formulas for calculating the center and normal vector of the spatial circle with this cross-section are as follows:

[0138] ,

[0139] ,

[0140] ,

[0141] ,

[0142] Where r is the flange radius, and t1 and t2 are the world coordinates of the flange face. The flange face normal vector, Let b be the eigenvalue of matrix P, and b refers to the b-th frame image.

[0143] S11: Eliminate the ambiguity of the fuzzy flange pose and obtain the true flange pose T of the flange in the current b-th frame. b .

[0144] Eliminate the ambiguity of the spatial circular pose and calculate With N b-1 The included angle The flange pose corresponding to the minimum included angle is taken as the true flange pose T. b N b-1 T b-1 The normal vector corresponding to the XY plane;

[0145] , ,

[0146] When solving for the flange pose in the first frame, the ambiguity is eliminated by using the normal vector N0 of the XY plane where T0 is located.

[0147] S12: Repeat steps S6 to S11 to detect the actual flange position in real time until the operation is completed.

[0148] This embodiment can extract flange edge positioning in real time, that is, it can detect flange edge and estimate pose in real time. The first frame positioning is used for mapping-projection and eliminating ambiguity. The flange pose is the flange pose obtained from bolt hole positioning. Subsequent flange edge detection and localization are used for mapping-projection and deambiguity resolution. The flange pose is obtained from the localization of the previous frame. In this embodiment, aside from the need for flange bolt hole-assisted positioning in the first frame, flange edge extraction and positioning in subsequent consecutive frames only require image processing of the flange image, ensuring accuracy while avoiding resource waste. Furthermore, the mapping-projection method can extract relatively clear flange edges with minimal noise, improving accuracy compared to directly performing ellipse detection on the image.

[0149] Example 2

[0150] The present invention provides a system for real-time estimation of flange pose, such as... Figure 3As shown, it includes a camera shooting module, a detection module, a first-frame flange pose recognition and positioning module, and a non-first-frame flange pose recognition and positioning module.

[0151] Camera capturing module, used for real-time image capture.

[0152] The detection module is used to identify flanges and flange bolt holes in captured images, and to obtain flange images and flange bolt hole images. An image of the target flange is selected, its dimensions are obtained, and the target flange image is input into a YOLO neural network algorithm model for training. This model is then used to identify the flanges and flange bolt holes in the captured images.

[0153] Preferably, when the detection model identifies the flange and flange bolt holes in the captured image, it first determines whether the coordinates of the flange bolt hole YOLO frame are within the range of the flange YOLO frame. Then, it combines the flange and flange bolt holes according to the number of flange bolt holes on the flange. That is, it judges the number of flange bolt holes in the flange YOLO frame based on prior knowledge and filters the flange image and flange bolt hole image that meet the conditions, which helps to reduce the probability of misidentification.

[0154] The first-frame flange pose recognition and positioning module calculates the initial flange pose T0 of the first-frame flange based on the bolt hole image of the first frame and the PNP algorithm.

[0155] Cropping the flange bolt hole image from the first frame of the captured image;

[0156] The flange bolt hole images are preprocessed, and then the Canny edge detection algorithm is used to detect the flange bolt hole edges in each flange bolt hole image; preferably, the flange bolt hole images are subjected to median filtering before edge detection.

[0157] Hough circle fitting is performed on the flange bolt hole image after edge detection to calculate the image coordinates of the center point of the flange bolt hole.

[0158] Establish a world coordinate system at the center point of the flange, and obtain the actual coordinates of the center point of the flange bolt holes in the world coordinate system based on the data of the flange bolt holes and the center point of the flange obtained by prior actual measurement.

[0159] Calculate the distance between all flange bolt holes and the upper left corner of the flange YOLO frame based on the image coordinates of the center point. Record the center of the flange bolt hole corresponding to the minimum distance as Center. Calculate the polar angle between the center point of the remaining flange bolt holes and the upper left corner of the flange YOLO frame, and sort the polar angles from largest to smallest. Determine the quadrant of all flange bolt holes in the world coordinate system based on the sorting.

[0160] The initial flange pose T0 of the first frame flange is calculated using the PNP algorithm, which combines the actual coordinates, the quadrant it belongs to, and the image coordinates.

[0161] The non-first frame flange pose recognition and localization module calculates the true flange pose T of the b-th frame based on the flange image and the initial flange pose T0. b The real-time detection of the flange's true pose; the non-first frame flange pose recognition and positioning module includes the following sub-modules: acquisition sub-module, edge detection sub-module, planar point cloud sub-module, circle fitting sub-module, filtering sub-module, back projection sub-module, ellipse fitting sub-module, and ambiguity elimination module.

[0162] The acquisition submodule is used to crop the flange image in the b-th frame of the captured image. Preferably, redundant information in the captured image from the camera acquisition module is removed.

[0163] The edge detection submodule uses the Canny edge detection algorithm to detect the flange edges in the cropped flange image, obtaining the original flange edge image.

[0164] The planar point cloud submodule is used to obtain the flange pose T of the flange in the (b-1)th frame by transforming the coordinate point set of the original flange edge image. b-1 The planar point cloud in the XY plane. When solving for the flange pose of the first frame, the coordinate point set of the flange edge image is transformed into the planar point cloud in the XY plane where T0 is located.

[0165] The original flange edge image is mapped to the camera coordinate system as follows:

[0166] Obtain the set of coordinate points of the original flange edge image Where R refers to the set of real numbers, Used to describe the coordinates of a two-dimensional point;

[0167] Let the coordinate point set Q of the original flange edge image be mapped to the three-dimensional point set in the camera coordinate system as follows: ,in Used to describe the coordinates of a three-dimensional point;

[0168] Q and P c The conversion relationship between them is:

[0169] ,

[0170] In the formula, K refers to the camera intrinsic parameter matrix. ,

[0171] in( () refers to the principal point of the plane on which the image is taken. (This refers to the camera's focal length.)

[0172] The 3D point set P in the camera coordinate systemc Transformed into flange pose T b-1 The planar point cloud in the XY plane is transformed as follows:

[0173] Let P be a three-dimensional point set. c Transition to flange pose T b-1 The three-dimensional point set in coordinates is ,

[0174] 3D point set P c With the three-dimensional point set P f The transformation formula is:

[0175] ,

[0176] The three-dimensional point set P f Projected onto flange pose T b-1 On the XY plane, after eliminating the influence of the Z-axis, the projected planar point cloud is obtained as follows: .

[0177] The circular fitting submodule uses a consistency check algorithm to perform circular fitting on the planar point cloud to obtain the coordinates of the flange center.

[0178] The filtering submodule filters the target point cloud based on the flange radius to form a flange edge point set.

[0179] Traverse all point clouds, calculate the distance from each point cloud to the flange center, set a distance threshold based on the flange radius, select point clouds that meet the threshold to form an intermediate point set, and reassign the Z-axis components back to the intermediate point set to obtain the flange edge point set. .

[0180] The back-projection submodule is used to back-project the flange edge point set onto the captured image to obtain the point set of the target flange edge image.

[0181] The flange edge point set P w The target flange edge image is obtained by projecting it onto the captured image, and the projection method is as follows:

[0182] Let the set of points in the captured image coordinates after the target flange edge image is imaged be... ,

[0183] P w and The transformation formula is:

[0184] ,

[0185] The ellipse fitting submodule uses the least squares method to fit an ellipse to the point set of the target flange edge image, resulting in an ambiguous flange pose.

[0186] The points in the flange edge image are determined by the coordinates of the captured image ( The coordinates projected onto the camera coordinate system are (X, Y, Z), and the two satisfy the following relationship:

[0187] ,

[0188] The point set of the target flange edge image is obtained using the least squares method. By performing ellipse fitting, the general form of the ellipse is obtained.

[0189]

[0190]

[0191] Combining the above formula, we can obtain

[0192] =0,

[0193] This formula is the expression for the elliptical cone formed by the optical center of the camera and the edge of the target flange, while the ellipse in the captured image obtained by projecting the flange edge is a cross-section of the elliptical cone.

[0194] Solving for the spatial circular pose using the general formula for an ellipse, and based on the general formula for an ellipse and the projection formula from world coordinates to the coordinates of the captured image in step S1, it is known that there exists a real symmetric matrix P satisfying...

[0195] ,

[0196] ,

[0197] According to the properties of matrices, there exist orthogonal matrices. Real symmetric matrices can be Diagonalization, i.e.: );

[0198] A point (X,Y,Z) in the camera coordinate system is transformed into a point in the world coordinate system after being rotated using the P matrix. , , ),

[0199] ,

[0200] From the above equation, we can obtain the general form of a standard elliptic cone: =0,

[0201] The spatial circle pose derived from the general formula of an ellipse is ambiguous, and the formulas for calculating the center and normal vector of the spatial circle with this cross-section are as follows:

[0202] ,

[0203] ,

[0204] ,

[0205] ,

[0206] Where r is the flange radius, t1 and t2 are the world coordinates of the flange face, and is the flange face normal vector. Let be the eigenvalues ​​of matrix P.

[0207] The ambiguity elimination module is used to eliminate the ambiguity of the fuzzy flange pose and obtain the true flange pose T of the flange in the current b-th frame. b .

[0208] Eliminate the ambiguity of the spatial circular pose and calculate With N b-1 The included angle The flange pose corresponding to the minimum included angle is taken as the true flange pose T. b N b-1 T b-1 The normal vector on the XY plane;

[0209] ,

[0210] ,

[0211] When solving for the flange pose in the first frame, the ambiguity is eliminated by using the normal vector N0 of the XY plane where T0 is located.

Claims

1. A method for real-time estimation of flange pose, characterized in that: Includes the following steps: S1: The camera captures images in real time during the operation; S2: Use the YOLO neural network algorithm to pre-train a detection model for identifying flanges, and use the trained detection model to identify flanges and flange bolt holes in the captured images to obtain flange images and flange bolt hole images; S3: Combine the bolt hole image and the PNP algorithm to calculate the initial flange pose T0 of the first frame flange; S4: Crop the flange image from the image captured in frame b; S5: Use the Canny edge detection algorithm to detect the flange edges in the cropped flange image to obtain the original flange edge image; S6: Transform the coordinate point set of the original flange edge image to the flange pose T of the (b-1)th frame. b-1 The planar point cloud located in the XY plane; S7: Use a consistency check algorithm to perform circle fitting on the planar point cloud to obtain the coordinates of the flange center; S8: Select target point cloud based on flange radius to form flange edge point set; S9: Back-project the flange edge point set onto the captured image in step S1 to obtain the point set of the target flange edge image; S10: Use the least squares method to fit the point set of the target flange edge image to an ellipse to obtain the ambiguous flange pose. S11: Eliminate the ambiguity of the fuzzy flange pose and obtain the true flange pose T of the flange in the current b-th frame. b ; S12: Repeat steps S6 to S11 to detect the actual flange position in real time until the operation is completed.

2. The method for real-time estimation of flange pose according to claim 1, characterized in that: The steps for calculating the initial flange pose T0 of the first frame flange in step S3 are as follows: Cropping the flange bolt hole image from the first frame of the captured image; The Canny edge detection algorithm was used to detect the edges of the flange bolt holes in each flange bolt hole image. Hough circle fitting is performed on the flange bolt hole image after edge detection to calculate the image coordinates of the center point of the flange bolt hole. Establish a world coordinate system at the center point of the flange, and obtain the actual coordinates of the center point of the flange bolt holes in the world coordinate system based on the data of the flange bolt holes and the center point of the flange obtained by prior actual measurement. Calculate the distance between all flange bolt holes and the upper left corner of the flange YOLO frame based on the image coordinates of the center point. Record the center of the flange bolt hole corresponding to the minimum distance as Center. Calculate the polar angle between the center point of the remaining flange bolt holes and the upper left corner of the flange YOLO frame, and sort the polar angles from largest to smallest. Determine the quadrant of all flange bolt holes in the world coordinate system based on the sorting. The initial flange pose T0 of the first frame flange is calculated using the PNP algorithm, which combines the actual coordinates, the quadrant it belongs to, and the image coordinates.

3. The method for real-time estimation of flange pose according to claim 1, characterized in that: The steps in step S6 to obtain the planar point cloud are as follows: Obtain the coordinate point set Q of the original flange edge image; Mapping the coordinate point set Q onto the camera coordinate system yields the 3D point set P mapped onto the camera coordinate system. c ; The three-dimensional point set P c Transformed into flange pose T b-1 The planar point cloud located in the XY plane.

4. The method for real-time estimation of flange pose according to claim 3, characterized in that: Transform a coordinate point set Q into a three-dimensional point set P c The formula is: ， ， Q and P c Between , , Where R refers to the set of real numbers. Used to describe the coordinates of a two-dimensional point. Used to describe the coordinates of a 3D point, K refers to the camera intrinsic parameter matrix, ( () refers to the principal point of the plane of the image captured in step S1, ( (This refers to the camera's focal length.) 5. The method for real-time estimation of flange pose according to claim 4, characterized in that: The three-dimensional point set P c Transformed into flange pose T b-1 The transformation method for the planar point cloud in the XY plane is as follows: Let P be a three-dimensional point set. c Transition to flange pose T b-1 The three-dimensional point set in coordinates is , 3D point set P c With the three-dimensional point set P f The transformation formula is: ， The three-dimensional point set P f Projected onto flange pose T b-1 On the XY plane, after eliminating the influence of the Z-axis, the projected planar point cloud is obtained as follows: .

6. The method for real-time estimation of flange pose according to claim 5, characterized in that: The steps for obtaining the flange edge point set in step S8 are as follows: traverse all point clouds in the planar point cloud, calculate the distance from each point cloud to the flange center, set a distance threshold based on the flange radius, select point clouds that meet the threshold conditions to form an intermediate point set, and reassign the z-component of the point clouds on the Z-axis back to the intermediate point set to obtain the flange edge point set. .

7. The method for real-time estimation of flange pose according to claim 6, characterized in that: The formula for obtaining the point set of the target flange edge image in step S9 is as follows: The point set of the target flange edge image is , And P w and satisfy .

8. The method for real-time estimation of flange pose according to claim 7, characterized in that: The steps in step S10 to obtain the ambiguous fuzzy flange pose are as follows: The points in the flange edge image are determined by the coordinates of the captured image ( The coordinates projected onto the camera coordinate system are (X, Y, Z), and the two satisfy the following relationship: ， The point set of the target flange edge image is processed using the least squares method. By performing ellipse fitting, the general form of the ellipse is obtained. ， ， Combining the above formula, we can obtain =0， This formula is the expression for the elliptical cone formed by the optical center of the camera and the edge of the target flange. The ellipse in the captured image, obtained by projecting the flange edge, is a cross-section of the elliptical cone. Solving for the spatial circular pose using the general formula for an ellipse reveals the existence of a real symmetric matrix P that satisfies... ， ， There exist orthogonal matrices Real symmetric matrices can be Diagonalization, i.e.: ); A point (X,Y,Z) in the camera coordinate system is transformed into a point in the world coordinate system after being rotated using the P matrix. , , The transformation formula is as follows: ， From the above equation, we can obtain the general form of a standard elliptic cone: =0; The formulas for calculating the center of the spatial circle and the normal vector of the cross-section are as follows: ， ， ， ， Where r is the flange radius, and t1 and t2 are the world coordinates of the flange face. The flange face normal vector, Let b be the eigenvalue of matrix P, and b refers to the b-th frame image.

9. The method for real-time estimation of flange pose according to claim 8, characterized in that: The steps to eliminate ambiguity in step S11 are as follows: Calculate With N b-1 The included angle The flange pose corresponding to the minimum included angle is taken as the true flange pose T. b , where N b-1 T b-1 The normal vector corresponding to the XY plane ， 。 10. A system for real-time estimation of flange pose, characterized in that: include, Camera capturing module, used for real-time image capture; The detection module is used to identify flanges and flange bolt holes in captured images, and to obtain flange images and flange bolt hole images; The first-frame flange pose recognition and positioning module calculates the initial flange pose T0 of the first-frame flange based on the bolt hole image of the first frame and the PNP algorithm. The non-first frame flange pose recognition and localization module calculates the true flange pose T of the b-th frame based on the flange image and the initial flange pose T0. b Real-time detection of the flange's true pose; the non-first frame flange pose recognition and positioning module includes the following sub-modules: The acquisition submodule is used to crop the flange image from the captured image in frame b; The edge detection submodule uses the Canny edge detection algorithm to detect the flange edges in the cropped flange image and obtain the original flange edge image. The planar point cloud submodule is used to obtain the flange pose T of the flange in the (b-1)th frame by transforming the coordinate point set of the original flange edge image. b-1 The planar point cloud located in the XY plane; The circular fitting submodule uses a consistency check algorithm to perform circular fitting on the planar point cloud to obtain the coordinates of the flange center. The filtering submodule filters the target point cloud based on the flange radius to form a flange edge point set; The back projection submodule is used to back project the flange edge point set onto the image captured by the camera module to obtain the point set of the target flange edge image; The ellipse fitting submodule uses the least squares method to fit the point set of the target flange edge image to an ellipse, resulting in an ambiguous flange pose. The ambiguity elimination module is used to eliminate the ambiguity of the fuzzy flange pose and obtain the true flange pose T of the flange in the current b-th frame. b .

Images