A color segmentation-based bolt positioning and loosening full-angle detection method

Abstract

The present application relates to a bolt positioning and loosening full-angle detection method based on color segmentation, and belongs to the technical field of industrial fastener structure safety monitoring, and comprises the following steps: (1) installing a color difference gasket between a bolt and a base, (2) marking anti-loosening lines for measuring the bolt loosening angle on the bolt surface and the base respectively, (3) obtaining a picture to be detected with the bolt, (4) performing color segmentation and edge contour extraction on the color difference gasket, (5) performing perspective transformation, (6) performing an open operation on the anti-loosening line connected domain to optimize the connected domain, (7) calculating the centroid and direction by using the geometric moment features of the rotating anti-loosening line connected domain and the fixed anti-loosening line connected domain, (8) measuring the loosening angle, and (9) calculating the loosening full angle of the bolt. The vector processing mode adopted eliminates the range limitation of the bolt loosening angle detection, greatly improves the bolt loosening angle detection range, and can realize the full-angle measurement of the bolt loosening.

Description

Technical Field

[0001] This invention relates to a method for bolt positioning and loosening detection at all angles based on color segmentation. It can be applied to industrial scenarios to locate bolts and measure their loosening angles using machine vision, and belongs to the field of industrial fastener structural safety monitoring technology. Background Technology

[0002] Bolts, as a typical basic fastener, are widely used in the fastening of structural components in various fields. However, due to vibration or impact in service environments, bolts are prone to loosening and other failure behaviors, which can affect the safety of the overall structure, causing serious economic losses or even endangering the lives of on-site personnel. Therefore, effective detection of bolt loosening behavior is extremely important.

[0003] To ensure the safe and effective service of bolts, regular inspection and tightening are crucial. Compared to traditional manual or contact-based inspection and repair methods, machine vision-based automated bolt inspection equipment offers a more advanced solution. This method inspects the working condition of bolts through image processing and analysis, offering advantages such as non-contact operation, traceability, and high levels of intelligence. Currently, the main research directions for visual detection of bolt loosening include traditional image processing and deep learning methods. In the field of traditional image processing, the mainstream approach is to first locate the bolt using the Hough circle detection method; then, algorithms such as Hough transform and Rammer are used to detect the slope change of the edge line before and after bolt loosening, thereby calculating the loosening angle. In the field of deep learning, the main approach is to first detect the bolt target using a series of deep learning algorithms; then, methods such as optical flow-based target detection, target methods, and feature matching are used to detect the differences before and after bolt loosening, thereby quantifying the loosening angle.

[0004] Bolts play a crucial role in various applications, making it difficult for deep learning-based methods to accurately locate all types of bolts using a single recognition model. Furthermore, traditional image processing methods, such as the commonly used Hough transform, may produce false detections when handling complex scenes and environments with varying lighting conditions. Additionally, the aforementioned methods typically require comparing images before and after bolt loosening angle detection—at least two frames—making it impossible to perform "on-demand" detection and reducing efficiency. Moreover, existing methods often have a limited detection range for bolt loosening angles, failing to achieve full-angle (0-360 degree) coverage. Summary of the Invention

[0005] This invention addresses the shortcomings of existing machine vision-based methods for bolt positioning and full-angle loosening detection by proposing a color segmentation-based method. This method locates the bolt by identifying color-difference shims between the bolt and its base, and measures the full angle of bolt loosening using only a single frame image by processing anti-loosening lines marked on the bolt and base. This invention can be applied to the positioning and full-angle loosening measurement of bolts using drones, tightening robots, or inspection robots.

[0006] One of the objectives of this invention is achieved through the following technical solution:

[0007] A method for bolt positioning and loosening detection at all angles based on color segmentation includes the following steps:

[0008] Step 1: Install a color difference shim between the bolt and the base. The purpose is to use its color characteristics to locate the area of ​​the bolt to be inspected. The color difference shim is color C, which is different from the color of the bolt.

[0009] Step 2: Mark the rotational anti-loosening line and the fixed anti-loosening line on the bolt surface and the base respectively for measuring the bolt loosening angle. The color of the rotational anti-loosening line and the fixed anti-loosening line is C.

[0010] Step 3: Use a camera to acquire the image of the bolt to be detected, and use a color segmentation method based on CIELAB color space, which performs nonlinear stretching, normalization and optimal threshold segmentation on the a component in sequence to segment the region where the color difference gasket is located in the image, obtain the connected component set, and select the connected component with the largest number of pixels in the connected component set, which is the color difference gasket connected component in step 1, and then locate the bolt region.

[0011] Step 4: Extract the edge contour of the connected region of the color difference pad in Step 3, and use a corner detection method based on calculating the Euclidean distance between points and between points and lines to detect four or more vertices of the connected region of the color difference pad.

[0012] Step 5: Use the vertices obtained in Step 4 to perform perspective transformation on the image taken in Step 3, and then crop the bolt and anti-loosening line areas to be detected in the color difference gasket area of ​​the transformed image.

[0013] Step 6: Use the color segmentation method used in Step 2 to segment the rotating anti-loosening line image and the fixed anti-loosening line image representing the bolt loosening angle, obtain the connected components of the rotating anti-loosening line and the fixed anti-loosening line, and perform an opening operation on the connected components of the rotating anti-loosening line and the fixed anti-loosening line to optimize the connected components.

[0014] Step 7: Calculate the zeroth and first order primitives of the connected regions of the rotating and fixed anti-loosening lines in Step 6 to determine the centroids of the connected regions of the rotating and fixed anti-loosening lines. Use the second central moments of the connected regions of the rotating and fixed anti-loosening lines to calculate the direction of the connected regions relative to the horizontal axis in the image.

[0015] Step 8: Combining with Step 7, use the characteristics of the anti-loosening line to distinguish between the rotational anti-loosening line on the bolt and the fixed anti-loosening line on the base, and determine the direction vectors of both for subsequent loosening angle measurement.

[0016] Step 9: Calculate the full angle of bolt loosening using the four-quadrant arctangent function and an angle adjustment.

[0017] Furthermore, the color C is one of the following: red, orange, yellow, green, cyan, blue, and purple.

[0018] Furthermore, the bolts and color difference washers are provided with circular rulers;

[0019] Furthermore, the color difference pad is a regular polygon, a rectangle, or a trapezoid;

[0020] Furthermore, the specific steps of the color segmentation method proposed in step 3 are as follows:

[0021] Step 3-1: First, convert the image acquired by the camera from the RGB color space to the XYZ transition color space. The conversion process is shown in equation (1).

[0022]

[0023] Then, the color space is converted from XYZ to CIELab color space. The conversion process is shown in (2).

[0024]

[0025] Where X n Y n and Z n These are the stimulation values ​​of the three primary colors X, Y, and Z under white light conditions, and their values ​​are shown in equation (3).

[0026]

[0027] Step 3-2: Stretch the a component using a nonlinear function. The goal of stretching is to highlight the color C pixel while minimizing the impact on the colors of other pixels. The stretched a component is given by equation (4).

[0028]

[0029] Where a(i,j) represents the value of the a component of pixel (i,j) in the image before stretching; a new(i,j) represents the value of the a component of pixel (i,j) in the stretched image; k1 is the parameter in this nonlinear transformation.

[0030] Step 3-3: Use normalization to preprocess the data, transforming it to a standard scale. Normalization reduces the impact of outliers caused by nonlinear stretching, improves the stability of threshold segmentation, and reduces the sensitivity of the threshold segmentation algorithm to image brightness or contrast. The normalization operation is shown in Equation (5).

[0031]

[0032] Where a normalized (i,j) represents the value of the a component of pixel (i,j) in the normalized image; a min For a new The minimum value in; a max For a new The minimum value in the a component is determined. Finally, the normalized a component is segmented using the optimal threshold to obtain the binarized image I. m×n .

[0033] Furthermore, the specific steps of the vertex calculation method for the connected region of the color difference pad proposed in step 4 are as follows:

[0034] Step 4-1, define the definition in step 3 in I m×n The connected domain of the obtained color difference pad is C, and the set of its edge pixel coordinates is E obtained by using the edge contour extraction algorithm.

[0035] Step 4-2: Choose any point p0 = (x0, y0) in E;

[0036] Step 4-3: Calculate the Euclidean distance between any point p = (x, y) and p0 = (x0, y0) in E, and record the point with the largest Euclidean distance to p0 = (x0, y0) as p1 = (x1, y1), which is the first vertex. The calculation process is shown in equation (6).

[0037]

[0038] Step 4-4: Calculate the Euclidean distance between any point p = (x, y) and p1 = (x1, y1) in E, and record the point with the largest Euclidean distance to p1 = (x1, y1) as p2 = (x2, y2), which is the second vertex. The calculation process is shown in equation (7).

[0039]

[0040] Steps 4-5: Record the points p1 = (x1, y1) and p2 = (x2, y2) as L = w1x + w2y + w3 = 0. Calculate the Euclidean distance between any point p = (x, y) in E and L, and record the point with the largest Euclidean distance from L as p3 = (x3, y3), which is the third vertex. The calculation process is shown in equation (8).

[0041]

[0042] Steps 4-6: Calculate the Euclidean distance between any point p = (x, y) and p3 = (x3, y3) in E, and record the point with the largest Euclidean distance to p3 = (x3, y3) as p4 = (x4, y4), which is the fourth vertex. The calculation process is shown in equation (9).

[0043]

[0044] Steps 4-7: Denote the set of the above four vertices as V = {p1, p2, p3, p4}, and calculate the Euclidean distance between any point p = (x, y) in V and point (0, 0). Rename the point with the smallest Euclidean distance to (0, 0) as p1 = (x1, y1), and the point with the largest Euclidean distance to (0, 0) as p3 = (x3, y3). The calculation process is shown in Equation (10).

[0045]

[0046] Steps 4-8: Denote the set of the above four vertices as V = {p1, p2, p3, p4}, and calculate the Euclidean distance between any point p = (x, y) in V and point (0, m). Rename the point with the smallest Euclidean distance to (0, m) as p2 = (x2, y2), and rename the point with the largest Euclidean distance to (0, 0) as p4 = (x4, y4). The calculation process is shown in Equation (11).

[0047]

[0048] The above algorithm can be used to obtain the coordinates of the four vertices in E arranged clockwise.

[0049] As a further limiting embodiment of the present invention, step 8 utilizes the characteristics of the anti-loosening line to distinguish between the rotational anti-loosening line on the bolt and the fixed anti-loosening line on the base, and determines the direction vectors of both. The specific steps are as follows:

[0050] Step 8-1, using the linear equation L of the two connected regions obtained in Step 7. c1 =-tan(θ) c1 )x+yy c1 +tan(θ c1 )x c1 =A1x+B1y+C1=0 and Lc2 =-tan(θ) c2 )x+yy c2 +tan(θ c2 )x c2 =A²x + B²y + C² = 0, thus obtaining the intersection point p. c =(x c ,y c The calculation process is shown in equation (12);

[0051]

[0052] Step 8-2: To distinguish between the rotating and fixed loosening wire connected regions in the connected regions obtained in Step 6, calculate p. c =(x c ,y c (and the centroids p of the two connected domains in step 7) c1 =(x c1 ,y c1 ) and p c2 =(x c2 ,y c2 The Euclidean distance is calculated, and the point with the smaller calculated Euclidean distance is designated as the centroid p of the rotational anti-loosening line. r =(x r ,y r The other centroid is p. f =(x f ,y f The calculation process is shown in equation (13);

[0053]

[0054] Step 8-3: Determine the unitized direction vectors of the rotating anti-loosening line and the fixed anti-loosening line according to equation (14), which are e respectively. r and e f ;

[0055]

[0056] Furthermore, in step 9, the specific steps for calculating the loosening angle of the bolt using the four-quadrant arctangent function and an angle adjustment are as follows:

[0057] Step 9-1: After determining the direction vectors of the two anti-loosening lines, calculate the dot product and cross product of the two vectors.

[0058]

[0059] Where α is the vector e f to e r The included angle.

[0060] Step 9-2, see equation (16), use the atan2 (four-quadrant arctangent) function to calculate the arctangent of the two values, and return the angle calculation value with the range of (-π, π].

[0061] α=atan2(e f ×e r e f ·e r (16)

[0062] Step 9-3: Since the bolt loosening direction is unidirectional, counterclockwise can be set as the bolt loosening direction. The same process applies if clockwise is the loosening direction. Since the full angle of bolt loosening in counterclockwise direction is θ∈(0, 2π], the following adjustment is made, see equation (17). Add 360 degrees to the angle obtained in equation (16) and perform a modeling operation with 360 degrees to ensure that all loosening angles meet the detection requirements.

[0063] α=mod(α+360,360) (17).

[0064] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0065] Compared to deep learning methods, this approach leverages the significant color difference between the color-difference pads used in this solution and the surrounding environment to locate bolt areas, thus generalizing the bolt target detection function to locate bolts in multiple scenes. Simultaneously, since cameras are often not directly facing the bolt detection area, perspective distortion occurs in the image, raising concerns about the accuracy of subsequent bolt loosening angle measurements. Using red square pads can also perform perspective transformation on the image, ensuring the accuracy of bolt loosening angle measurements. By marking anti-loosening lines on the bolt and base, the bolt loosening angle is quantified using the angle formed by the anti-loosening lines, avoiding false detections caused by edge detection methods in traditional approaches. Existing methods use two frames of images to measure the loosening angle; however, due to the special structure of the anti-loosening lines, this application can accurately obtain the bolt loosening angle using only a single frame, greatly improving detection efficiency. Existing detection methods have limited detection ranges for loosening angles; the vector processing approach eliminates these limitations, greatly expanding the bolt loosening angle detection range and enabling full-angle measurement of bolt loosening.

[0066] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0067] Figure 1This is an overall flowchart of the bolt positioning and loosening detection method based on color segmentation of the present invention.

[0068] Figure 2 This is a schematic diagram of the placement of color difference gaskets and the marking of anti-loosening lines in the bolt positioning and loosening detection method based on color segmentation of the present invention.

[0069] Figure 3 This is a schematic diagram of an experiment for the bolt positioning and loosening detection method based on color segmentation of the present invention.

[0070] Figure 4 These are experimental camera images of the bolt positioning and loosening detection method based on color segmentation according to the present invention.

[0071] Figure 5 This is a schematic diagram of the connected domain of the color difference gasket in the bolt positioning and loosening full-angle detection method based on color segmentation of the present invention.

[0072] Figure 6 This is a schematic diagram of the four vertices of the edge of the connected domain of the color difference gasket in the bolt positioning and loosening full-angle detection method based on color segmentation of the present invention.

[0073] Figure 7 This is a schematic diagram of the perspective transformation process of the bolt positioning and loosening detection method based on color segmentation of the present invention.

[0074] Figure 8 This is a perspective transformation diagram of the bolt positioning and loosening detection method based on color segmentation of the present invention.

[0075] Figure 9 This is an image of the cut bolt and anti-loosening line from the bolt positioning and loosening full-angle detection method based on color segmentation of the present invention.

[0076] Figure 10 This is a binarized image of the anti-loosening line of the bolt positioning and loosening full-angle detection method based on color segmentation of the present invention.

[0077] Figure 11 This is a schematic diagram of the two anti-loosening line direction vectors of the bolt positioning and loosening full-angle detection method based on color segmentation of the present invention.

[0078] Figure 12 This is a schematic diagram of the circular ruler used in the bolt positioning and loosening full-angle detection method based on color segmentation of the present invention. Detailed Implementation

[0079] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0080] It should be noted that when a component is described as "fixed to" another component, it can be directly on the other component or may have a component in between. When a component is considered "connected to" another component, it can be directly connected to the other component or may have a component in between. When a component is considered "set on" another component, it can be directly set on the other component or may have a component in between. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0081] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0082] The present invention will now be further described with reference to the accompanying drawings. Figure 1-3 As shown, the main steps of the bolt positioning and loosening detection method based on color segmentation of the present invention are as follows:

[0083] Step 1, please refer to Figure 2 The test setup shown has a red square washer installed between the bolt and the base. The purpose of this was to use the color feature to locate the area of ​​the bolt to be tested.

[0084] Step 2, please refer to Figure 2 The test apparatus shown has red anti-loosening lines marked on the bolt surface and base with a paint pen, respectively, for measuring the bolt loosening angle. These lines represent the rotational anti-loosening line and the fixed anti-loosening line.

[0085] Step 3, please refer to Figure 3 As shown, a camera is used to acquire images of the objects to be inspected containing bolts. The resulting images are shown below. Figure 4 The red regions in the image were segmented using a color segmentation method based on the CIELAB color space to obtain a set of connected components. The connected component with the largest number of pixels was selected from these components; this is the connected component containing the red square pad from step 1. The results are shown in [see image]. Figure 5 .

[0086] Step 3-1: First, convert the image acquired by the camera from the RGB color space to the XYZ transition color space. The conversion process is shown in equation (1).

[0087]

[0088] Then, the color space is converted from XYZ to CIELab color space. The conversion process is shown in (2).

[0089]

[0090] Where X n Y n and Z n These are the stimulation values ​​of the three primary colors X, Y, and Z under white light conditions, and their values ​​are shown in equation (3).

[0091]

[0092] Step 3-2: Stretch the a component using a nonlinear function. The goal of stretching is to highlight the red pixels while minimizing the impact on the colors of other pixels. The stretched a component is given by equation (4).

[0093]

[0094] Where a(i,j) represents the value of the a component of pixel (i,j) in the image before stretching; a new (i,j) represents the value of the a component of pixel (i,j) in the stretched image; k1 is the parameter in this nonlinear transformation.

[0095] Step 3-3: Use normalization to preprocess the data, transforming it to a standard scale. Normalization reduces the impact of outliers caused by nonlinear stretching, improves the stability of threshold segmentation, and reduces the sensitivity of the threshold segmentation algorithm to image brightness or contrast. The normalization operation is shown in Equation (5).

[0096]

[0097] Where a normalized (i,j) represents the value of the a component of pixel (i,j) in the normalized image; a min For a new The minimum value in; a max For a new The minimum value in the a component is determined. Finally, the normalized a component is segmented using the optimal threshold to obtain the binarized image I. m×n .

[0098] Steps 3-4, in the binarized image I m×n The connected component with the largest number of pixels in the set of connected components is the red square pad connected component, as shown in the diagram. Figure 5 .

[0099] Step 4: Extract the edge contour of the connected domain of the red square pad in Step 3, and use a corner detection method based on calculating the Euclidean distance between points and between points and lines to detect the four vertices of the connected domain of the red square pad.

[0100] Step 4-1, define the definition in step 3 in I m×n The red square connected component obtained is C, and its edge pixel coordinate set is E obtained by using the edge contour extraction algorithm.

[0101] Step 4-2: Choose any point p0 = (x0, y0) in E;

[0102] Step 4-3: Calculate the Euclidean distance between any point p = (x, y) and p0 = (x0, y0) in E, and record the point with the largest Euclidean distance to p0 = (x0, y0) as p1 = (x1, y1), which is the first vertex. The calculation process is shown in equation (6).

[0103]

[0104] Step 4-4: Calculate the Euclidean distance between any point p = (x, y) and p1 = (x1, y1) in E, and record the point with the largest Euclidean distance to p1 = (x1, y1) as p2 = (x2, y2), which is the second vertex. The calculation process is shown in equation (7).

[0105]

[0106] Steps 4-5: Record the points p1 = (x1, y1) and p2 = (x2, y2) as L = w1x + w2y + w3 = 0. Calculate the Euclidean distance between any point p = (x, y) in E and L, and record the point with the largest Euclidean distance from L as p3 = (x3, y3), which is the third vertex. The calculation process is shown in equation (8).

[0107]

[0108] Steps 4-6: Calculate the Euclidean distance between any point p = (x, y) and p3 = (x3, y3) in E, and record the point with the largest Euclidean distance to p3 = (x3, y3) as p4 = (x4, y4), which is the fourth vertex. The calculation process is shown in equation (9).

[0109]

[0110] Steps 4-7: Denote the set of the above four vertices as V = {p1, p2, p3, p4}, and calculate the Euclidean distance between any point p = (x, y) in V and point (0, 0). Rename the point with the smallest Euclidean distance to (0, 0) as p1 = (x1, y1), and the point with the largest Euclidean distance to (0, 0) as p3 = (x3, y3). The calculation process is shown in Equation (10).

[0111]

[0112] Steps 4-8: Denote the set of the above four vertices as V = {p1, p2, p3, p4}, and calculate the Euclidean distance between any point p = (x, y) in V and point (0, m). Rename the point with the largest Euclidean distance to (0, m) as p2 = (x2, y2), and rename the point with the smallest Euclidean distance to (0, 0) as p4 = (x4, y4). The calculation process is shown in Equation (11).

[0113]

[0114] The above algorithm yields the coordinates of the four vertices in E arranged clockwise, as shown in the diagram. Figure 6 .

[0115] Step 5: Use the four vertices obtained in Step 4 to perform perspective transformation on the image taken in Step 3, and then crop the bolt and anti-loosening line areas to be detected in the red square area of ​​the transformed image.

[0116] Step 5-1: In order to eliminate perspective distortion of the image and correct the tilted image to a vertically projected image, perspective transformation of the image is required.

[0117] Step 5-2, refer to Figure 7 The midpoint of the tilted image obtained in step 4 is denoted as follows:

[0118] v i =(x i ,y i ,1) (12)

[0119] Let the points in the vertical projection diagram be denoted as

[0120] u i =(x i ,y i ,1) (13)

[0121] According to equation (14), the acquired image is subjected to perspective transformation.

[0122]

[0123] Where T is the perspective transformation matrix.

[0124] Step 5-3: Substitute the four vertices calculated in Step 4-8 into Equation (15). By referring to the actual physical dimensions of the length and width of the red square pad, and specifying the four corresponding points of the vertices obtained above, the perspective transformation matrix T can be obtained.

[0125]

[0126] See the corrected image Figure 8 .

[0127] Step 5-4, select a certain size ratio, in Figure 8 Cut the bolts and anti-loosening line areas to be inspected in the middle section. See the schematic diagram. Figure 9 .

[0128] Step 6: Use the color segmentation method used in Step 2 to segment the anti-loosening line image representing the bolt loosening angle, obtain the connected components of the anti-loosening line, and perform an opening operation on the connected components to optimize the connected components.

[0129] Step 6-1: Use the color segmentation method used in Step 2 to segment the anti-loosening line image representing the bolt loosening angle, see... Figure 10 .

[0130] Step 6-2, referring to equation (16), the opening operation first uses the structuring element B to erode the connected component set A in the image, and then B performs a dilation operation on the erosion result.

[0131]

[0132] Where ⊙ represents the erosion operation in morphological processing. This refers to the dilation operation in morphological processing.

[0133] Step 7: Calculate the zeroth and first primitive orders of the connected components from Step 6 to determine the centroid of the connected components, and use the second central moment of the connected components to calculate the direction of the connected components relative to the horizontal axis in the image.

[0134] Step 7-1, first, will Figure 10 The two connected components are stored in blank images of the same size as the original image, using their original coordinates. The centroids of the two connected components are calculated using the zeroth and first order primitive moments of the connected components. c1 =(x c1 ,y c1 ) and p c2 =(x c2 ,y c2 The calculation process is shown in equation (17).

[0135]

[0136] Where (x′, y′) is the centroid of the connected component, m and n are the number of rows and columns of the image, respectively; I(i, j) is the value of the image at position (i, j); M 00 M is the zeroth primitive moment of a connected domain; 10 and M 01 It is the first-order primitive moment of the connected domain.

[0137] Step 7-2: Calculate the direction θ of the connected component relative to the horizontal axis in the image using the second central moments of the connected components. Let the directions of the two connected components be θ0 and θ1 respectively. c1 and θ c2 The calculation process is shown in equation (18).

[0138]

[0139] Where μ 00 μ 11 These are the zeroth and first-order central moments of the connected domain, respectively; μ 20 and μ 02 It is the second central moment of the connected domain; M 02 and M 20 It is the second-order primitive moment of the connected domain.

[0140] Step 8: Combining with Step 7, use the characteristics of the anti-loosening line to distinguish between the rotational anti-loosening line on the bolt and the fixed anti-loosening line on the base, and determine the direction vectors of both for subsequent loosening angle measurement.

[0141] Step 8-1, using the linear equation L of the two connected regions obtained in Step 7. c1 =-tan(θ) c1 )x+yy c1 +tan(θ c1 )x c1 =A1x+B1y+C1=0 and L c2 =-tan(θ) c2 )x+yy c2 +tan(θ c2 )x c2 =A²x + B²y + C² = 0, thus obtaining the intersection point p. c =(x c ,y c The calculation process is shown in equation (19);

[0142]

[0143] Step 8-2: To distinguish between the rotating and fixed loosening wire connected regions in the connected regions obtained in Step 6, calculate p. c =(x c ,y c (and the centroids p of the two connected domains in step 7) c1 =(xc1 ,y c1 ) and p c2 =(x c2 ,y c2 The Euclidean distance is calculated, and the point with the smaller calculated Euclidean distance is designated as the centroid p of the rotational anti-loosening line. r =(x r ,y r The other centroid is p. f =(x f ,y f The calculation process is shown in equation (20);

[0144]

[0145] Step 8-3: Determine the unitized direction vectors of the rotating anti-loosening line and the fixed anti-loosening line according to equation (21), which are e respectively. r and e f ;

[0146]

[0147] See the test diagram. Figure 11 The yellow arrow indicates the direction vector of the rotating anti-loosening line, and the green arrow indicates the direction vector of the fixed anti-loosening line.

[0148] Step 9: Calculate the full angle of bolt loosening using the four-quadrant arctangent function and an angle adjustment.

[0149] Step 9-1: After determining the direction vectors of the two anti-loosening lines, calculate the dot product and cross product of the two vectors.

[0150]

[0151] Where α is the vector e f to e r The included angle.

[0152] Step 9-2, see equation (23), use the atan2 (four-quadrant arctangent) function to calculate the arctangent of the two values, and return the angle calculation value with the range of (-π, π].

[0153] α=atan2(e f ×e r e f ·e r ) (twenty three)

[0154] Step 9-3: Since the bolt loosening direction is unidirectional, counterclockwise can be set as the bolt loosening direction. The same process applies if clockwise is the loosening direction. Since the full angle of counterclockwise bolt loosening is θ∈(0, 2π], the following adjustment is made, see equation (24). Add 360 degrees to the angle obtained by equation (23) and perform a modeling operation with 360 degrees so that all loosening angles meet the detection requirements. Since there is no solution for equation (19) when the loosening angle is 180°, a case is added to the angle representation;

[0155]

Claims

1. A method for bolt positioning and loosening detection at all angles based on color segmentation, characterized in that: Includes the following steps: Step 1: Install a color difference shim between the bolt and the base. The color difference shim is color C, and the color C is different from the color of the bolt. Step 2: Mark the rotational anti-loosening line and the fixed anti-loosening line on the bolt surface and the base respectively for measuring the bolt loosening angle. The color of the rotational anti-loosening line and the fixed anti-loosening line is C. Step 3: Use a camera to acquire the image of the bolt to be detected, and use a color segmentation method based on CIELAB color space, which performs nonlinear stretching, normalization and optimal threshold segmentation on the a component in sequence, to segment the region where the color difference gasket is located in the image, obtain the connected component set, and select the connected component with the largest number of pixels in the connected component set, which is the color difference gasket connected component in step 1, and then locate the bolt region. Step 4: Extract the edge contour of the connected region of the color difference pad in Step 3, and use a corner detection method based on calculating the Euclidean distance between points and between points and lines to detect four or more vertices of the connected region of the color difference pad. Step 5: Use the vertices obtained in Step 4 to perform perspective transformation on the image taken in Step 3, and then crop the bolt and anti-loosening line areas to be detected in the color difference gasket area of ​​the transformed image. Step 6: Use the color segmentation method used in Step 2 to segment the rotating anti-loosening line image and the fixed anti-loosening line image representing the bolt loosening angle, obtain the connected components of the rotating anti-loosening line and the fixed anti-loosening line, and perform an opening operation on the connected components of the rotating anti-loosening line and the fixed anti-loosening line to optimize the connected components. Step 7: Calculate the zeroth and first order primitives of the connected regions of the rotating and fixed anti-loosening lines in Step 6 to determine the centroids of the connected regions of the rotating and fixed anti-loosening lines. Use the second central moments of the connected regions of the rotating and fixed anti-loosening lines to calculate the direction of the connected regions relative to the horizontal axis in the image. Step 8: Combining with Step 7, use the characteristics of the anti-loosening line to distinguish between the rotational anti-loosening line on the bolt and the fixed anti-loosening line on the base, and determine the direction vectors of both for subsequent loosening angle measurement. Step 9: Calculate the full angle of bolt loosening using the four-quadrant arctangent function and an angle adjustment.

2. The bolt positioning and loosening detection method based on color segmentation as described in claim 1, characterized in that: The color C is one of the following: red, orange, yellow, green, cyan, blue, and purple.

3. The bolt positioning and loosening detection method based on color segmentation as described in claim 1, characterized in that: The color difference pads are regular polygons, rectangles, or trapezoids.

4. The bolt positioning and loosening detection method based on color segmentation as described in claim 1, characterized in that: The specific steps of the color segmentation method proposed in step 3 are as follows: Step 3-1: First, convert the image acquired by the camera from the RGB color space to the XYZ transition color space. The conversion process is shown in equation (1). Then, the color space is converted from XYZ to CIELab color space. The conversion process is shown in (2). Where X n Y n and Z n These are the stimulation values ​​of the three primary colors XYZ under white light conditions, and their values ​​are shown in equation (3). Step 3-2: Use a nonlinear function to stretch the a component. The stretching goal is to highlight the color C pixel, but at the same time, to minimize the impact on the colors of other pixels. The stretched a component is given by equation (4). Where a(i,j) represents the value of the a component of pixel (i,j) in the image before stretching; a new (i,j) represents the value of the a component of pixel (i,j) in the stretched image; k1 is the parameter in this nonlinear transformation; Step 3-3: Use normalization to preprocess the data, transforming it to a standard scale. Normalization can reduce the influence of outliers caused by nonlinear stretching, improve the stability of threshold segmentation, and reduce the sensitivity of the threshold segmentation algorithm to the brightness or contrast of the image. The normalization operation is shown in equation (5). Where a normalized (i,j) represents the value of the a component of pixel (i,j) in the normalized image; a min For a new The minimum value in; a max For a new The minimum value in the a component is used to perform optimal threshold segmentation on the normalized a component, resulting in a binarized image I. m×n .

5. The bolt positioning and loosening detection method based on color segmentation as described in claim 1, characterized in that: The specific steps of the vertex calculation method for the connected region of the color difference pad proposed in step 4 are as follows: Step 4-1, define the definition in step 3 in I m×n The connected component of the obtained color difference pad is C, and the set of its edge pixel coordinates is E obtained by using the edge contour extraction algorithm; Step 4-2: Choose any point p0 = (x0, y0) in E; Step 4-3: Calculate the Euclidean distance between any point p = (x, y) and p0 = (x0, y0) in E, and record the point with the largest Euclidean distance to p0 = (x0, y0) as p1 = (x1, y1), which is the first vertex. The calculation process is shown in equation (6). Step 4-4: Calculate the Euclidean distance between any point p = (x, y) and p1 = (x1, y1) in E, and record the point with the largest Euclidean distance to p1 = (x1, y1) as p2 = (x2, y2), which is the second vertex. The calculation process is shown in equation (7). Steps 4-5: Record the points p1 = (x1, y1) and p2 = (x2, y2) as L = w1x + w2y + w3 = 0. Calculate the Euclidean distance between any point p = (x, y) in E and L, and record the point with the largest Euclidean distance from L as p3 = (x3, y3), which is the third vertex. The calculation process is shown in equation (8). Steps 4-6: Calculate the Euclidean distance between any point p = (x, y) and p3 = (x3, y3) in E, and record the point with the largest Euclidean distance to p3 = (x3, y3) as p4 = (x4, y4), which is the fourth vertex. The calculation process is shown in equation (9). Steps 4-7: Denote the set of the above four vertices as V = {p1, p2, p3, p4}, and calculate the Euclidean distance between any point p = (x, y) in V and point (0, 0). Rename the point with the smallest Euclidean distance to (0, 0) as p1 = (x1, y1), and the point with the largest Euclidean distance to (0, 0) as p3 = (x3, y3). The calculation process is shown in Equation (10). Steps 4-8: Denote the set of the above four vertices as V = {p1, p2, p3, p4}, and calculate the Euclidean distance between any point p = (x, y) in V and point (0, m). Rename the point with the smallest Euclidean distance to (0, m) as p2 = (x2, y2), and rename the point with the largest Euclidean distance to (0, 0) as p4 = (x4, y4). The calculation process is shown in Equation (11). The above algorithm can be used to obtain the coordinates of the vertices in E arranged clockwise.

6. The bolt positioning and loosening detection method based on color segmentation as described in claim 1, characterized in that: Step 8 utilizes the characteristics of the anti-loosening line to distinguish between the rotational anti-loosening line on the bolt and the fixed anti-loosening line on the base, and determines the direction vectors of both. The specific steps are as follows: Step 8-1, using the linear equation L of the two connected regions obtained in Step 7. c1 = -tan(θ) c1 )x+yy c1 +tan(θ c1 )x c1 =A1x+B1y+C1=0 and L c2 =-tan(θ) c2 )x+yy c2 +tan(θ c2 )x c2 =A²x + B²y + C² = 0, thus obtaining the intersection point p. c =(x c ,y c The calculation process is shown in equation (12); Step 8-2: To distinguish between the rotating and fixed loosening wire connected regions in the connected regions obtained in Step 6, calculate p. c =(x c ,y c (and the centroids p of the two connected domains in step 7) c1 =(x c1 ,y c1 ) and p c2 =(x c2 ,y c2 The Euclidean distance is calculated, and the point with the smaller calculated Euclidean distance is designated as the centroid p of the rotational anti-loosening line. r =(x r ,y r The other centroid is p. f =(x f ,y f The calculation process is shown in equation (13); Step 8-3: Determine the unitized direction vectors of the rotating anti-loosening line and the fixed anti-loosening line according to equation (14), which are e respectively. r and e f ; 7. The bolt positioning and loosening detection method based on color segmentation as described in claim 1, characterized in that: Step 9 involves using the four-quadrant arctangent function and an angle adjustment to calculate the bolt loosening angle. The specific steps are as follows: Step 9-1: After determining the direction vectors of the two anti-loosening lines, calculate the dot product and cross product of the two vectors. Where α is the vector e f to e r The included angle; Step 9-2, see equation (16), use the atan2 (four-quadrant arctangent) function to calculate the arctangent of the two values, and return the angle calculation value with the range of (-π, π]. αthen2(e f ×e r ,e f ·e r ) (16) Step 9-3: Since the loosening direction of the bolt is unidirectional, counterclockwise can be set as the loosening direction of the bolt. The research process is the same if clockwise is the loosening direction. Since the full angle of bolt loosening in counterclockwise direction is θ∈(0,2π], the following adjustment is made, see equation (17). Add 360 degrees to the angle obtained by equation (16) and perform a modeling operation with 360 degrees so that all loosening angles meet the detection requirements. Since there is no solution for equation (12) when the loosening angle is 180°, a case is added to the angle representation.

8. The bolt positioning and loosening detection method based on color segmentation as described in claim 1, characterized in that: The bolts and color difference washers are equipped with circular rulers.

Images