Segment bolt detection positioning method and system based on ellipse detection, medium and equipment

Abstract

The invention discloses a segment bolt detecting and positioning method and system based on ellipse detection, a medium and equipment. Firstly, a region of interest of a bolt is determined by using an SSD algorithm; then, aiming at the problem that the detection precision is reduced due to incomplete elliptic arc segments during ellipse detection, an ellipse detection algorithm based on arc clustering is provided, and the algorithm mainly comprises the steps of image edge detection, edge clustering, least square fitting and the like; and finally, a monocular positioning scheme is given, and the duct pieces are automatically grabbed. According to the invention, the error of positioning the center pixel coordinate of the ellipse during ellipse detection is smaller, and accurate detection and positioning of the bolt can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic splicing of shield segments, and in particular relates to a segment bolt detection and positioning method, system, medium and equipment based on ellipse detection. Background Art

[0002] Tunnel boring machines are increasingly used in tunnel excavation. During the drilling process of tunnel boring machines, automated segment assembly technology has significantly improved the construction speed and quality. Automatic segment grabbing is the primary link in the automatic segment assembly. The key to automatic segment grabbing is to accurately identify and position the segments. When grabbing a segment, the target is the segment bolt, so accurate detection and positioning of the bolt is crucial. The existing segment automatic grabbing method and algorithm are time-consuming. For example, the segment bolt position extraction algorithm based on SIFT feature point matching is used for segment positioning, but this algorithm is time-consuming. By attaching a mark on the surface of the bolt head, the bolt is positioned by detecting the mark, but this method may introduce human errors. Based on the circular shape of the upper surface of the lifting bolt, the segment positioning can be achieved by detecting the bolt surface. Since the circle is an ellipse at all angles except when observed vertically downward when imaging, the ellipse detection algorithm can be used to detect the bolt.

[0003] Hough Transform (HT) algorithms are one of the commonly used methods for ellipse detection. The Standard Hough Transform (SHT) is known for its robustness to image defects and noise, but this algorithm has a high demand for computer memory. Although the Randomized Hough Transform (RHT) has a fast calculation speed, it may have problems with missed detection or false detection when dealing with overlapping ellipses. Optimization methods such as least squares and genetic algorithms have excellent accuracy in ellipse detection, but they often require high computing resources, resulting in low efficiency. In previous work on ellipse detection, when the ellipse has more arc segments, the overall efficiency of the algorithm is low, and when the ellipse is defective or occluded, the detection accuracy decreases, making it difficult to achieve accurate and effective detection. Summary of the invention

[0004] The present invention focuses on the problems existing in the above-mentioned prior art and provides a segment bolt detection and positioning method, system, medium and equipment based on ellipse detection.

[0005] A first aspect of the present invention is a segment bolt detection and positioning method based on ellipse detection, the method comprising the following steps:

[0006] Step S100: collecting an original shield segment image and identifying an area of ​​interest in the original image, wherein the area of ​​interest is an area containing bolts;

[0007] Step S200: performing edge detection on the region of interest based on the Canny operator to obtain a bolt edge image;

[0008] Step S300: performing boundary pixel connection and line segment fitting extraction on the bolt edge image to obtain a valid line segment sequence;

[0009] Step S400: extracting arc segments from the ordered line segments in the valid line segment sequence, performing integrity judgment and same-region division on the arc segments, and then clustering the arc segments belonging to the same ellipse to obtain candidate ellipse arcs and candidate ellipse arc pairs;

[0010] Step S500: extracting corresponding directed boundary sequences from the candidate elliptical arcs and candidate elliptical arc pairs, and fitting the ellipse parameters using a direct least squares method to obtain multiple ellipses;

[0011] Step S600: acquiring the ordinates of the multiple ellipses, and performing comparative analysis on the ordinates of the multiple ellipses, and the ellipse with the largest ordinate is the target ellipse;

[0012] Step S700: Acquire the center pixel coordinates of the target ellipse, transform the center pixel coordinates of the target bolt based on the monocular measurement model and the point laser, and acquire the real-time position of the bolt.

[0013] Preferably, the step S100 specifically includes: using a camera to capture an original shield segment image, identifying a region of interest in the original shield segment image by training a deep neural network, and specifically using an SSD network to detect the region of interest.

[0014] Preferably, step S200 specifically includes:

[0015] Step S210: performing Gaussian filtering on the image of the region of interest, using a 3×3 Gaussian filter to perform a convolution operation on the image of the region of interest;

[0016] Step S220: Obtain the partial derivatives f of each pixel in the x and y directions by performing finite difference calculation of the first-order partial derivatives on the filtered image. x and f y Then, the gradient amplitude value G and gradient direction θ of the point are calculated according to formulas (1) and (2);

[0017]

[0018] θ=arctan(f y / fx ) (2)

[0019] Step S230: traverse the entire image and perform non-maximum suppression of the gradient amplitude;

[0020] Step S240: Use double thresholds to detect the edge distribution image and perform edge connection. Set two high and low thresholds. Consider points with edge strength greater than the high threshold as edge points, and eliminate points with edge strength less than the low threshold. Consider points between the two as weak edge points. Use 8-connected neighborhood pixels to find surrounding edges that can be connected to the contour until the edge is closed.

[0021] Preferably, the step 300 specifically includes:

[0022] Step S310: Based on the edge pixel linking technology, the bolt edge image is scanned from top to bottom and from left to right through run-length encoding and the "8-adjacent" criterion, and the edge pixel points are connected into an equivalent edge sequence; redundant edge sequences are eliminated according to the number of pixels in the equivalent edge sequence and the minimum diagonal length of the circumscribed rectangle;

[0023] Step S320: extracting a line segment sequence from the edge of the bolt according to a line segment fitting algorithm, and using multiple segments of the line segment sequence to approximate the arc contour; at the same time, using the end of the edge pixel and the bifurcation point at the intersection of the curve to determine the boundaries of different connected domains, divide them into several subsets, set thresholds, and exclude line segment sequences with shorter lengths;

[0024] Step 330: Perform a cross product calculation based on the convex point pixel on the arc contour and the vector formed by the two adjacent pixel points, and introduce the Z direction coordinate to determine the rotation direction of the line segment; if the Z value of the cross product is positive, the rotation direction is determined to be counterclockwise; if the Z value of the cross product is negative, a reverse operation will be performed on the line segment sequence and the directed edge sequence to determine the consistency of the direction.

[0025] Step 340: For all continuous points in the line segment sequence except the first and last two points, calculate the spatial vector product between three points, judge the concave points and corner points according to the spatial vector product, and filter the valid line segment sequence based on the judgment result.

[0026] Preferably, the step S400 specifically includes:

[0027] Step S401: Determine the integrity of the arc segment extracted by the valid line segment by measuring the angle formed by the arc segment, connect the starting point, the end point and the midpoint of the arc segment, construct two vectors, and determine the integrity of the arc segment according to the angle of the two vectors;

[0028] Step S402: perform the same area division, take the endpoint of the arc to be clustered as the starting point, emit two rays, the direction of which is consistent with the direction of the tangent of the arc, and the two rays and the arc constitute an effective search range.

[0029] Step S403: further determining whether the arc segments within the effective search range are in the same ellipse as the circular arcs to be clustered based on two constraint conditions;

[0030] The two constraints specifically include: the distance between the center of the arc and the center of the chord; and the distance between the centers of the fitted ellipse.

[0031] Preferably, the step S500 specifically includes: calculating the shortest distance d between the edge pixel point and the fitted ellipse boundary by setting a threshold K i , then, for all d i <K, and calculate the ratio ρ of the edge pixels in the total number of edge points. If ρ>60%, the ellipse really exists and the shortest distance d i The calculation is as follows:

[0032]

[0033] In the formula, (x i ,y i ) represents the coordinates of an edge pixel point, (p,q) is the center coordinate of the fitted ellipse, θ represents the rotation angle of the fitted ellipse, r1 and r2 are the major axis and minor axis radii respectively.

[0034] Preferably, the step S700 specifically includes: according to the linear model of the monocular camera, obtaining the vertical height h of the gripping mechanism from the pipe segment in real time through the point laser, and calculating the actual position coordinates (x W ,y W ), x W ,y W The calculation method is as follows:

[0035]

[0036] Among them, h is the vertical distance from the optical center of the camera to the object plane, f is the focal length, α is the fixed angle of the camera, and α x With y are the angles of the camera's vertical field of view to the x and y axes, v o -v Q’ It represents the image distance difference between point o and point Q', and u(p')-u0 represents the object distance difference between point o and point p'.

[0037] A second aspect of the present invention provides a segment bolt detection and positioning system based on ellipse detection, the system comprising:

[0038] Image acquisition unit: collects the original shield segment image and identifies the region of interest in the original image, wherein the region of interest is the region containing the bolts;

[0039] Edge detection unit: performing edge detection on the region of interest based on the Canny operator to obtain a bolt edge image;

[0040] The arc clustering ellipse detection unit: performs boundary pixel connection and line segment fitting on the bolt edge image to obtain a valid line segment sequence; extracts arc segments from the ordered line segments in the valid line segment sequence, performs integrity judgment and same area division on the arc segments, and then clusters the arc segments belonging to the same ellipse to obtain candidate ellipse arcs and candidate ellipse arc pairs; extracts corresponding directed boundary sequences from the candidate ellipse arcs and candidate ellipse arc pairs, and uses the direct least squares method to fit the ellipse parameters to obtain multiple ellipses; obtains the ordinates of the multiple ellipses and performs comparative analysis, and the one with the largest ordinate is the target ellipse;

[0041] Bolt monocular positioning unit: obtains the central pixel coordinates of the target ellipse, transforms the central pixel coordinates of the target bolt based on the monocular measurement model and point laser, and obtains the real-time position of the bolt.

[0042] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0043] According to a fourth aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above method when executed by the processor.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1) The deep neural network is used to determine the region of interest of the bolt, which improves the accuracy of ellipse detection.

[0046] 2) Through multiple processes such as image preprocessing, boundary pixel connection, line segment column extraction, arc clustering, ellipse fitting and screening, the morphological information of the ellipse is effectively used to optimize the threshold, thereby improving the accuracy of ellipse detection when the arc is missing.

[0047] 3) Based on the characteristic that the circular imaging of the bolt head is an ellipse, the bolts are detected using the ellipse detection algorithm. A monocular positioning solution is given based on the fixed circular size of the bolt head to solve the problem that the height information cannot be provided when the point laser is not above the segment, and finally the automatic grasping of the segment is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the segment bolt detection and positioning method based on ellipse detection;

[0049] Figure 2 Schematic diagram of the camera observation angle principle in the embodiment;

[0050] Figure 3 Obtaining a result map for the bolt region of interest in the embodiment;

[0051] Figure 4 is a schematic diagram of an edge segment sequence in an embodiment;

[0052] Figure 5 A schematic diagram of selecting edge segment vectors in an embodiment;

[0053] Figure 6 Schematic diagram of the angles of different edge line segments in the embodiment;

[0054] Figure 7 Schematic diagram of the same-area arc division principle in the embodiment;

[0055] Figure 8 Schematic diagram of the same arc judgment principle in the embodiment;

[0056] Fig. 9 Schematic diagram of the principle of judging the ellipse polar line in the embodiment;

[0057] Fig.10 This is a comparison chart of the results before and after the target ellipse detection screening in the embodiment;

[0058] Fig.11 It is the monocular camera positioning model in the embodiment;

[0059] Fig.12 is a schematic diagram of camera observation imaging in another embodiment;

[0060] Fig.13 A monocular positioning model for ellipse detection in another embodiment;

[0061] Fig.14 It is an experimental platform for automatic assembly of pipe segments;

[0062] Fig.15 This is the automatic segment grabbing process.

[0063] Figure numerals: red and blue cylinder 1, camera 2, grabbing mechanism 3, point laser 4. DETAILED DESCRIPTION

[0064] The technology described below can be transformed in many ways and can have many embodiments, which are described in detail with reference to the accompanying drawings in specific embodiments. However, this does not mean that the technology described below is limited to specific embodiments. It should be understood that the present invention includes all similar modifications, equivalents and substitutions without departing from the spirit and technical scope of the technology described below.

[0065] Example 1

[0066] like Figure 1 As shown, a segment bolt detection and positioning method based on ellipse detection includes the following steps:

[0067] Step S100: collecting an original shield segment image and identifying an area of ​​interest in the original image, wherein the area of ​​interest is an area containing bolts;

[0068] Preferably, the step S100 further comprises: the ROI extraction in the present invention is regarded as an object detection problem, by training a deep neural network, and then using it to accurately identify the ROI in the original image. In the present invention, the SSD network is used for object detection, firstly the original image is collected, the object boundary box is drawn in the image using an annotation tool, the original image and the annotation file are placed in a specified folder, a data set (including a training set and a test set) is constructed, the SSD network is configured and trained for object detection, the trained deep neural network is verified, and the training results are updated, and the target detection model is integrated into the software system for ROI extraction. Figure 3 Shown is the ROI acquisition result of the segment bolts.

[0069] When the camera observes the bolt, if it is not photographed vertically above the bolt, the upper surface of the bolt will be imaged as an ellipse. In the present invention, the camera forms a certain angle when observing the bolt, such as Figure 2 The observation angle is 2. Therefore, when locating the center of the bolt, the ellipse recognition algorithm is used for identification.

[0070] Step S200: performing edge detection on the region of interest based on the Canny operator to obtain a bolt edge image;

[0071] Preferably, the step S200 further comprises: step S210: performing Gaussian filtering on the bolt edge image, using a 3×3 Gaussian filter to perform image convolution operation;

[0072] Step S220: By performing finite difference calculation of the first-order partial derivatives on the entire filtered image, the partial derivatives f of each pixel in the x and y directions are obtained. x and f y Next, the gradient magnitude G and gradient direction θ of the point are calculated according to formulas (6) and (7);

[0073]

[0074] θ=arctan(f y / f x ) (7)

[0075] Step S230: traverse the entire image and perform non-maximum suppression of the gradient amplitude;

[0076] Step S240: Use dual thresholds to detect the edge distribution image and perform edge connection, set two high and low thresholds. Points with edge strength greater than the high threshold are considered edge points, points with edge strength less than the low threshold are removed, and points between the two are considered weak edge points. Use 8-connected neighborhood pixels to find surrounding edges that can be connected to the contour until the edge is closed.

[0077] Step S300: performing boundary pixel connection and line segment fitting extraction on the bolt edge image to obtain a valid line segment sequence;

[0078] Preferably, the step S300 further comprises:

[0079] Step S310: First, based on the edge pixel linking technology, the binary image is scanned from top to bottom and from left to right through run-length coding and the “8-adjacent” criterion, and the edge pixels are connected into an equivalent edge sequence. Then, according to the number of pixels in the equivalent edge sequence [N min ,N max ] and the minimum circumscribed rectangle diagonal length [L min ,L max ] two thresholds to eliminate redundant edge sequences; edge sequences with pixel counts that do not reach the preset threshold are considered noise and are removed. The threshold range of pixel count is set to the maximum and minimum values ​​of the number of pixels in the major axis of the ellipse multiplied by π; the threshold range of diagonal length is the maximum and minimum values ​​of 1.5 times the major axis of the ellipse; the maximum and minimum values ​​of the major axis of the ellipse are calculated based on the number of pixels in the width direction of the image.

[0080] Step S320: Next, a line segment sequence is identified and extracted from the edge through a line segment fitting algorithm, and the arc contour is approximated with multiple line segments. The connected areas in the binary image are analyzed to achieve pixel clustering and improve the accuracy of arc contour aggregation. At the same time, the ends of the edge pixels and the bifurcation points at the intersection of the curves are used to determine the boundaries of different connected domains and divide them into several subsets. In this process, the threshold interval of the number of line segments is set to M1 and the threshold interval of the directed edge sequence pixels is set to M2. These thresholds are adjusted specifically according to the size of the ellipse. In the present invention, the range of M1 is set to [3,5] and the range of M2 is set to [10,20]. Using these thresholds, line segment sequences with shorter lengths are excluded.

[0081] Step S330: In order to solve the problem of inconsistent direction of line segment sequence, a strategy is adopted, that is, to adjust all line segments and their directed edge sequences in the counterclockwise direction. The cross product calculation is performed based on the vector formed by the convex point pixel on the arc and the two adjacent pixels to determine the rotation direction of the line segment. This method ensures the consistent processing of geometric characteristics in the image and effectively improves the stability and accuracy of the algorithm. Figure 4 As shown, in the image coordinate system, a specific point on the edge is determined by random scanning, denoted as P i (x i ,y i ), and the two pixels adjacent to the point are recorded as P i-1 (x i-1 ,y i-1 ) and P i+1 (x i+1 ,y i+1 ). Next, we need to determine the direction of rotation of the line segment sequence. Introduce the z-direction coordinate, represent the vector formed by the three points in space, and set the z-coordinate value z i 、z i-1 、z i+1 is zero, and the vector product of these three points is calculated as shown in formula (8).

[0082]

[0083] When performing the above vector product calculation on edge pixels, this operation is performed on all edge pixels except the first and last two points. By analyzing the z-direction values ​​of the i-2 vectors in these vector products, the overall rotation direction can be determined. If the z-values ​​of most vector products are positive, the rotation direction is determined to be counterclockwise. On the contrary, if the z-values ​​of most vector products are negative, the reverse operation will be performed on the line segment sequence and the directed edge sequence to ensure the consistency of the direction.

[0084] Step S340: Finally, detect the concave points and corner points. Figure 5 As shown, an angle threshold η1 is set, and in the present invention, η1=10°. For all the consecutive points in the line segment sequence except the first and last two points, the spatial vector product between the three points is calculated. i >0 and p i-1 p i With point p i p i+1 If the angle between them exceeds η1, then the point p i is a concave point. Set the angle threshold η2 = 45°. i >0 and p i-1 p i With p i p i+1When the angle between exceeds η2, it is determined as a corner point. At these corner points, the line segments will be segmented, and the line segment sequences that do not meet the requirements of the number of pixels will be eliminated according to the thresholds M1 and M2. The screened line segment sequences will be used for further arc clustering. Figure 5 As shown, point p2 may be determined as a concave point, while point p5 may be determined as a corner point.

[0085] Step S400: extracting arc segments from the ordered line segments in the valid line segment sequence, performing integrity judgment and same-region division on the arc segments, and then clustering the arc segments belonging to the same ellipse to obtain candidate ellipse arcs and candidate ellipse arc pairs;

[0086] Preferably, the step S400 specifically includes:

[0087] Step S410: First, the integrity of the arc segment is determined by measuring the angle formed by the arc segment extracted from the line segment. The starting point, end point and midpoint of the arc segment are connected to construct two vectors. Figure 6 As shown, P1 and P2 are the two endpoints of the arc segment, P3 is the midpoint, and the vector and The angle formed is the angle to be calculated. The measured value of this angle is used to evaluate the completeness of the arc: the smaller the angle, the closer the arc is to being complete.

[0088] In order to adapt to the diversity of arc clustering, the present invention defines an angle coverage threshold M ac If the measured angle is lower than the threshold M ac , the arc is considered to have high integrity and can be directly used for cluster analysis, and the least squares method is used to fit the ellipse; on the contrary, if the angle exceeds the threshold M ac , then the arc is not suitable for accurate ellipse fitting and should be excluded. Figure 6 As shown, if M is set ac =90°, then Figure 6 The arc in (a) is complete and can be fitted directly; Figure 6 The arc in (b) needs further cluster analysis. ac =160°.

[0089] Step S420: Then, perform the same area division. Take the endpoint of the arc to be clustered as the starting point and send out two rays, the direction of which is consistent with the tangent direction of the arc. The two rays and the arc constitute the effective search range S. Only the arc segments within this range may belong to the same ellipse as the arc. Figure 7As shown in the figure, l1 and l2 are rays emitted from arc a1. Based on the chord l3 of arc a1, two parallel lines parallel to l3 are drawn through the two endpoints of the arc to be judged. If the pair of parallel lines intersects with the tangent l1l2, and the end of the arc segment is between the intersection points, then the task is that this arc segment is within the search range of the arc to be clustered. Based on this method, it can be determined Figure 7 The arcs a2 and a3 are within the search range, but a4 is not. This strategy limits the search range of clustered arcs, ensuring that only arcs that are most likely to belong to the same ellipse are taken into consideration, thereby improving detection efficiency.

[0090] Step S430: Finally, based on two constraints, it is further determined whether the arc segments within the search range are in the same ellipse as the arcs to be clustered.

[0091] Preferably, the two constraints are:

[0092] 1) The distance between the center of the arc and the center of the chord.

[0093] Judge based on the distance between the arc center and the chord center, such as Figure 8 As shown, select a center point that is equidistant from the two end points of the arc segment and is denoted by P. n / 2,1 (p u / 2,2 ,p v / 2,3 ), let the center point of the arc segment chord be C1.

[0094] If the directions of adjacent arcs are opposite to each other, the judgment condition is:

[0095] p n / 2,1 p n / 2,2 >p n / 2,1 C2 and p n / 2,1 p n / 2,2 >C1p n / 2,2 C2;

[0096] If the directions of adjacent arcs are opposite to each other, the judgment condition is:

[0097] P n / 2，1 P v / 2，3 >P n / 2，1 C3 and P n / 2，1 P v / 2，3 >C1P v / 2，3 ;

[0098] like Figure 8 As shown in (a), the arc where C2 is located meets the conditions and is thus retained, while the arc where C3 is located in (b) does not meet the conditions and is therefore removed.

[0099] 2) The distance from the center of the fitted ellipse.

[0100] The distance between the center points of the ellipse corresponding to two arc segments is calculated based on the characteristics of the ellipse polar line. Fig. 9 As shown, using two points P on the directed edge columns e1 and e2 4,1 With P u-3,2 , determine the tangent of the arc segment. Similarly, using the midpoint p of the arc segment u / 2 ,2 and P n / 2,1 , the other two tangents τ3 and τ4 can be determined. Through these tangents, the center points O1 and O2 of the ellipse and the distance between them can be determined. A threshold d is set, and the judgment is based on the size of the ellipse. If the distance between the center points O1 and O2 is less than the threshold d, the two arcs are considered to be part of the same ellipse. In the present invention, the threshold d is set to 5.

[0101] Step S500: extracting corresponding directed boundary sequences from the candidate elliptical arcs and candidate elliptical arc pairs, and fitting the ellipse parameters using a direct least squares method to obtain multiple ellipses;

[0102] Preferably, step S500 specifically includes: extracting corresponding directed edge sequences from candidate elliptical arcs and candidate elliptical arc pairs, and fitting ellipse parameters (p, q, r1, r2, θ) using a robust direct least squares method. By setting a threshold K, the shortest distance d between the edge pixel point and the fitted ellipse boundary is calculated according to formula (9): i Then, for all d i The edge pixels with a value less than K are counted, and their proportion ρ in the total number of edge points is calculated. If ρ>60%, the ellipse really exists.

[0103]

[0104] In the formula, (x i ,y i ) represents the coordinates of an edge pixel point, (p,q) is the center coordinate of the fitted ellipse, θ represents the rotation angle of the fitted ellipse, r1 and r2 are the major axis and minor axis radii respectively.

[0105] Step S600: acquiring the ordinates of the multiple ellipses, and performing comparative analysis on the ordinates of the multiple ellipses, and the ellipse with the largest ordinate is the target ellipse;

[0106] Preferably, step S600 specifically includes: due to the shape characteristics of the bolt, the bolt head and the bolt bottom are both cylinders. When detecting the circle on the surface of the bolt head, the circle at the bottom of the bolt may also be detected. Therefore, it is necessary to distinguish the detected ellipses and filter out the target ellipse that needs to be located in the end.

[0107] The circle of the bolt head is above the entire ROI area. Therefore, when multiple ellipses are detected, the Y coordinate of the ellipse of the bolt head is the largest. Then, the Y coordinates of multiple ellipses can be compared, and the one with the largest Y coordinate is the target ellipse. The steps are as follows:

[0108] Assume that the detected ellipse set is E = {E1, E2, ... E n}, where n is the number of detected ellipses. Each ellipse E i The center coordinates are (x center,i ,y center,i ), and then select the ellipse with the maximum Y coordinate as the target ellipse, the expression is as follows:

[0109]

[0110] In the formula, E t is the target ellipse, and max represents the ellipse with the maximum value.

[0111] like Fig.10 The figure shows the comparison of the results before and after the target ellipse detection and screening. After the ellipse screening, the non-target ellipse under the bolt is eliminated, and only the target ellipse on the upper surface of the bolt is retained.

[0112] Step S700: Acquire the center pixel coordinates of the target ellipse, transform the center pixel coordinates of the target bolt based on the monocular measurement model and the point laser, and acquire the real-time position of the bolt.

[0113] Preferably, step S700 specifically includes: after obtaining the pixel coordinates of the bolt center, the pixel coordinates need to be further converted to obtain the position of the bolt center. Since the monocular camera can only measure the distance of one height plane, the point laser ranging sensor is used to compensate for the height information, and a positioning solution is given based on the monocular measurement model and the point laser.

[0114] According to the linear model of the monocular camera, under the premise of knowing the internal and external parameters of the camera, the monocular vision system cannot uniquely determine the world coordinates of the three-dimensional space point corresponding to the image pixel point. Therefore, a fixed height plane constraint is introduced to realize monocular ranging, so for each height plane, monocular ranging can be realized. The height of each height plane is obtained by real-time measurement of the point laser, that is, the height distance from the camera to the pipe segment is measured in real time by the point laser.

[0115] like Fig.11 The figure shows the monocular camera positioning model, and the coordinate system of the camera is defined as O C X C Y C Z C , where O CRepresents the optical center of the camera. The distance from the optical center of the camera to the object plane X W O W Y W The vertical distance of the camera to the object plane is h, and the inclination angle α with the plane is fixed. W As the world coordinate system O W X W Y W Z W The intersection points of the optical axis, the imaging plane and the object plane are O and O' respectively. W , so that O to O C The distance between the two objects is defined as the focal length f. Let the coordinates of a point P on the object plane in the world coordinate system be (x w ,y w ,z w ); P' is the projection point of P on the imaging plane. Q and Q' are the projection points of P and P' on the vertical axis respectively. The image pixel coordinate system is expressed as fuv , where u and v correspond to the columns and rows of the image matrix respectively.

[0116] In plane O C O W Y W Inside, the vertical coordinate y of point P W , that is, line segment O W The length of Q is calculated as follows:

[0117]

[0118] |OQ'| can be expressed as (v o -v Q ')α y , then formula (11) can be rewritten as:

[0119]

[0120] In plane O C Determine the horizontal coordinate x of point P within QP W When , we can use the proportional relationship to calculate:

[0121] x w =|PQ|=|O c Q|tan∠PO c Q. (13)

[0122]

[0123] |P'Q'| can be expressed as [u(P′)-u o ]a x , express |OQ'|, |P'Q'| in image pixel coordinates:

[0124]

[0125] Where |OQ'|, |P'Q'| represent the image distances from the camera to P and PQ respectively, h is the vertical distance from the camera optical center to the object plane, f is the focal length, α is the fixed angle of the camera, α x With y are the angles of the camera's vertical field of view to the x and y axes, v o -v Q’ It represents the image distance difference between point o and point Q', and u(p')-u0 represents the object distance difference between point o and point p'.

[0126] Through the above calculation, the coordinates of any point on a certain height plane can be obtained. Then, the height h of the grabbing mechanism from the pipe segment can be obtained in real time through point laser, and the real-time position of the bolt can be obtained.

[0127] On the other hand, preferably, obtaining the position of the bolt based on the monocular positioning model may also include: after obtaining the pixel coordinates of the center of the bolt, these coordinates need to be converted to determine the actual position of the bolt. Since the visual measurement of the monocular camera is limited to a specific height plane, when using a point laser to assist in height acquisition, it is necessary to ensure that the laser can irradiate the pipe segment, but during the grasping process, there is a situation where the point laser cannot irradiate the pipe segment. To this end, the present invention uses the fixed circular size of the bolt head to estimate the depth. When the camera angle is fixed, the projection of the bolt head in the image will change according to the relative distance between the camera and the bolt. This change can be analyzed by quantifying the pixel size of the ellipse. When the camera moves along the Y and Z axes, the projection of the bolt head appears as an ellipse in the image, and the pixel length of its axis changes with the change of the camera position. As Fig.12 As shown, Figure (b) moves relative to Figure (a) on the Y and Z axes, and the pixels of the major axis of the ellipse change with the movement, thereby providing assistance for the positioning of the monocular camera when the point laser fails.

[0128] According to the linear model of the monocular camera, a monocular positioning model based on ellipse detection is established under the premise of knowing the internal and external parameters of the camera. Fig.13 As shown in the figure, the pixel coordinates of the center of the bolt ellipse are (x0, y0). Taking point O as the origin, the coordinates of the center of the bolt are (x d ,y d , z d ), positioning is performed based on the principle of triangulation and the parameters of the ellipse. Fig.13 (a) shows the horizontal viewing angle model of the camera. The horizontal field of view angle of the camera is β, X L is the major axis pixel of the ellipse, width is the pixel width of the camera image, according to the triangle theorem, we can get:

[0129]

[0130] like Fig.13 (b) shows the vertical viewing angle model, α is the fixed angle of the camera, θ is the vertical field of view of the camera, R is the radius of the bolt head circle, Yw is the distance from the closest position observable by the camera on the Y axis to the target, and Y0 is the closest distance observable by the camera. From equations (17) and (18), we can get

[0131]

[0132] exist Fig.13 In (b), we can obtain from the triangle theorem:

[0133] Y0+Yw=Z d *tan(α+d θ ) (twenty one)

[0134]

[0135] Y0=Z d *tanα (23)

[0136] From equations (20), (21), (22) and (23), we can get the height of the bolt relative to the camera:

[0137]

[0138] We can further get the Y-axis distance of the bolt relative to the camera:

[0139]

[0140] Through the above calculations, the position of the bolt can be obtained based on monocular vision and ellipse parameters.

[0141] Preferably, the automatic segment assembly process is as follows: Fig.14 The figure shows an automatic segment assembly experimental platform, including a red and blue oil cylinder 1, a camera 2, a gripping mechanism 3, and a point laser 4. After the camera 2 captures the picture, the bolt is identified based on the SSD network, and then the target position is obtained based on the segment bolt detection and positioning method of the ellipse detection of the present invention. The point laser 4 obtains the height of the target bolt, and the gripping logic is set in the gripping mechanism 3 to control the assembly machine gripping mechanism to reach the target position to grab the segment. Fig.15The figure shows the process of segment grabbing. During 0-10 seconds, the assembler moves toward the bolt and descends. During 10-15 seconds, the assembler moves downward and makes fine adjustments to align the grabbing mechanism with the bolt. During 15-20 seconds, the assembler moves toward the bolt. During 20-25 seconds, the assembler moves downward to fit the segment and moves toward the bolt to the grabbing position. Then the assembler can perform the grabbing action.

[0142] Example 2

[0143] The present invention also provides a segment bolt detection and positioning system based on ellipse detection, the system comprising:

[0144] A segment bolt detection and positioning system based on ellipse detection, the system comprising:

[0145] Image acquisition unit: collects the original shield segment image and identifies the region of interest in the original image, wherein the region of interest is the region containing the bolts;

[0146] Edge detection unit: performing edge detection on the region of interest based on the Canny operator to obtain a bolt edge image;

[0147] The arc clustering ellipse detection unit: performs boundary pixel connection and line segment fitting on the bolt edge image to obtain a valid line segment sequence; extracts arc segments from the ordered line segments in the valid line segment sequence, performs integrity judgment and same area division on the arc segments, and then clusters the arc segments belonging to the same ellipse to obtain candidate ellipse arcs and candidate ellipse arc pairs; extracts corresponding directed boundary sequences from the candidate ellipse arcs and candidate ellipse arc pairs, and uses the direct least squares method to fit the ellipse parameters to obtain multiple ellipses; obtains the ordinates of the multiple ellipses and performs comparative analysis, and the one with the largest ordinate is the target ellipse;

[0148] Bolt monocular positioning unit: obtains the central pixel coordinates of the target ellipse, transforms the central pixel coordinates of the target bolt based on the monocular measurement model and point laser, and obtains the real-time position of the bolt.

[0149] Example 3

[0150] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0151] Example 4

[0152] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above method when executed by the processor.

[0153] Although the present invention has been described in detail above with general description and specific embodiments, some modifications or improvements can be made on the basis of the present invention. The above description is only a preferred embodiment of the present invention and is not limited to the scope of the present invention. Other changes and modifications made by those skilled in the art without departing from the spirit and scope of protection of the present invention are still included in the scope of protection of the present invention.

Claims

1. A segment bolt detection and positioning method based on ellipse detection, characterized in that: The method comprises the following steps: Step S100: collecting an original shield segment image and identifying an area of ​​interest in the original image, wherein the area of ​​interest is an area containing bolts; Step S200: performing edge detection on the region of interest based on the Canny operator to obtain a bolt edge image; Step S300: performing boundary pixel connection and line segment fitting extraction on the bolt edge image to obtain a valid line segment sequence; Step S400: extracting arc segments from the ordered line segments in the valid line segment sequence, performing integrity judgment and same-region division on the arc segments, and then clustering the arc segments belonging to the same ellipse to obtain candidate ellipse arcs and candidate ellipse arc pairs; Step S500: extracting corresponding directed boundary sequences from the candidate elliptical arcs and candidate elliptical arc pairs, and fitting the ellipse parameters using a direct least squares method to obtain multiple ellipses; Step S600: acquiring the ordinates of the plurality of ellipses, and performing comparative analysis on the ordinates of the plurality of ellipses to determine that the ellipse with the largest ordinate is the target ellipse; Step S700: Acquire the center pixel coordinates of the target ellipse, transform the center pixel coordinates of the target bolt based on the monocular measurement model and the point laser, and acquire the real-time position of the bolt.

2. The method according to claim 1, characterized in that The step S100 specifically includes: using a camera to collect an original shield segment image, identifying a region of interest in the original shield segment image by training a deep neural network, and specifically using an SSD network to detect the region of interest.

3. The method according to claim 1, characterized in that The step S200 specifically includes: Step S210: performing Gaussian filtering on the image of the region of interest, using a 3×3 Gaussian filter to perform a convolution operation on the image of the region of interest; Step S220: Obtain the partial derivatives f of each pixel in the x and y directions by performing finite difference calculation of the first-order partial derivatives on the filtered image. x and f y Then, the gradient amplitude value G and gradient direction θ of the point are calculated according to formulas (1) and (2); θ=arctan(f y / f x ) (2) Step S230: traverse the entire image and perform non-maximum suppression of the gradient amplitude; Step S240: Use double thresholds to detect the edge distribution image and perform edge connection. Set two high and low thresholds. Consider points with edge strength greater than the high threshold as edge points, and eliminate points with edge strength less than the low threshold. Consider points between the two as weak edge points. Use 8-connected neighborhood pixels to find surrounding edges that can be connected to the contour until the edge is closed.

4. The method according to claim 1, characterized in that The step 300 specifically includes: Step S310: Based on the edge pixel linking technology, the bolt edge image is scanned from top to bottom and from left to right through run-length encoding and the "8-adjacent" criterion, and the edge pixel points are connected into an equivalent edge sequence; redundant edge sequences are eliminated according to the number of pixels in the equivalent edge sequence and the minimum diagonal length of the circumscribed rectangle; Step S320: extracting a line segment sequence from the edge sequence according to a line segment fitting algorithm, and using multiple segments of the line segment sequence to approximate the arc contour; at the same time, using the end of the edge pixel and the bifurcation point at the intersection of the curve to determine the boundaries of different connected domains, divide them into several subsets, and set a threshold to exclude line segment sequences with shorter lengths; Step 330: Perform a cross product calculation based on the convex point pixel on the arc contour and the vector formed by the two adjacent pixel points, and introduce the Z direction coordinate to determine the rotation direction of the line segment; if the Z value of the cross product is positive, the rotation direction is determined to be counterclockwise; if the Z value of the cross product is negative, a reverse operation will be performed on the line segment sequence and the directed edge sequence to determine the consistency of the direction. Step 340: For all continuous points in the line segment sequence except the first and last two points, calculate the spatial vector product between three points, judge the concave points and corner points according to the spatial vector product, and filter the valid line segment sequence based on the judgment result.

5. The method according to claim 1, characterized in that The step S400 specifically includes: Step S401: Determine the integrity of the arc segment by measuring the angle formed by the arc segment extracted from the valid line segment, connect the starting point, the end point and the midpoint of the arc segment, construct two vectors, and determine the integrity of the arc according to the angle of the two vectors; Step S402: perform the same area division, specifically, take the endpoint of the arc to be clustered as the starting point, emit two rays, the direction of which is consistent with the direction of the tangent of the arc, and the two rays and the arc constitute an effective search range. Step S403: further determining whether the arc segments within the effective search range are in the same ellipse as the circular arcs to be clustered based on two constraint conditions; The two constraints specifically include: the distance between the center of the arc and the center of the chord; and the distance between the centers of the fitted ellipse.

6. The method according to claim 1, characterized in that The step S500 specifically includes: calculating the shortest distance d between the edge pixel point and the fitted ellipse boundary by setting a threshold K i , then, for all d i <K, and calculate the ratio ρ of the edge pixels in the total number of edge points. If ρ>60%, the ellipse really exists and the shortest distance d i The calculation is as follows: In the formula, (x i ,y i ) represents the coordinates of an edge pixel point, (p,q) is the center coordinate of the fitted ellipse, θ represents the rotation angle of the fitted ellipse, r1 and r2 are the major axis and minor axis radii respectively.

7. The method according to claim 1, characterized in that The step S700 specifically includes: according to the linear model of the monocular camera, the vertical height h of the gripping mechanism from the pipe segment is obtained in real time by the point laser, and the actual position coordinates (x w ,y w ), x w ,y w The calculation method is as follows: Among them, h is the vertical distance from the optical center of the camera to the object plane, f is the focal length, α is the fixed angle of the camera, and α x With y are the angles of the camera's vertical field of view to the x and y axes, v o -v Q, represents the image distance difference between point o and point Q, u(p , )-u0 represents the point o and p , The object distance difference between the points.

8. A segment bolt detection and positioning system based on ellipse detection, characterized in that: The system comprises: Image acquisition unit: collects the original shield segment image and extracts the region of interest in the original image, wherein the region of interest is the region containing the bolts; Edge detection unit: performing edge detection on the region of interest based on the Canny operator to obtain a bolt edge image; The arc clustering ellipse detection unit: performs boundary pixel connection and line segment fitting on the bolt edge image to obtain a valid line segment sequence; extracts arc segments from the ordered line segments in the valid line segment sequence, performs integrity judgment and same area division on the arc segments, and then clusters the arc segments belonging to the same ellipse to obtain candidate ellipse arcs and candidate ellipse arc pairs; extracts corresponding directed boundary sequences from the candidate ellipse arcs and candidate ellipse arc pairs, and uses the direct least squares method to fit the ellipse parameters to obtain multiple ellipses; obtains the ordinates of the multiple ellipses detected and performs comparative analysis, and the one with the largest ordinate is the target ellipse; Bolt monocular positioning unit: obtains the central pixel coordinates of the target ellipse, transforms the central pixel coordinates of the target bolt based on the monocular measurement model and point laser, and obtains the real-time position of the bolt.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images