A highly robust visual positioning method for aero-engine bolts

Abstract

The present invention discloses a highly robust visual positioning method for aircraft engine bolts. Using the color and depth images acquired by a RealSense depth camera, the bolt color and depth images are segmented through deep learning. An ellipse detection method based on super-resolution preprocessing is used to detect multiple elliptical targets in the bolt color image. An ellipse screening method based on the centroid offset vector method is proposed to screen out single elliptical targets corresponding to the bolt bottom surface. The symmetric depth mean method is used to obtain the depth value of the center of the bolt bottom surface, and a three-dimensional coordinate calculation model for the bolt bottom surface center is established based on the camera pinhole model. This invention solves the problem of the current method's limitation on camera shooting angles, achieving precise and high-precision positioning of the bottom surface of the bolt in the aircraft engine cavity at any shooting angle.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision intelligent detection, and specifically relates to an aircraft engine bolt visual positioning method with strong shooting angle robustness, which is mainly applied to the connection structure between an aircraft engine rotor drum and a grate disc. Background Art

[0002] Bolted connections are widely used in the connection between aircraft engine rotor disks. In order to ensure the assembly quality of aircraft engine rotors, it is necessary to measure key connection parameters such as the bolt connection stiffness and bolt preload in the rotor cavity. Taking the bolt preload measurement as an example, in the closed and narrow rotor cavity, it is necessary to ensure that the ultrasonic probe and the bottom surface of the bolt have a high degree of parallelism. The existing method requires the disassembly of some parts and relies on manual operation to position the detection probe. This method is not only cumbersome to operate, but also has poor placement, which seriously affects the measurement results. Therefore, the measurement of the connection status is heavily dependent on the visual positioning of the bolts, and it is of great significance to carry out in-situ automated detection of the connection status.

[0003] Currently, most visual positioning of bolts is still limited to the pixel plane. Although simple template matching or deep learning methods can quickly detect bolts, the results are usually rectangular regression frames, which make it difficult to locate the center of the bottom of the bolt. In addition, existing bolt visual positioning methods have high requirements for the camera's shooting angle, requiring the camera's optical axis to be perpendicular to the bottom surface of the bolt being measured. However, the internal space of the aircraft engine rotor is small, making it difficult to meet the vertical shooting requirements. Therefore, the bolt can only be visually positioned by tilting at a large angle. Summary of the Invention

[0004] In order to solve the problems raised in the background technology, the present invention proposes a highly robust visual positioning method for aircraft engine bolts, which can achieve precise positioning of aircraft engine inner cavity bolts at any shooting angle.

[0005] The technical solution of the present invention comprises the following steps:

[0006] Step 1: Obtain color images of the aircraft engine cavity at different shooting angles, manually annotate the acquired color images by surrounding the bolts with rectangular frames, and divide the annotated results into training and validation sets. Send them into the deep learning-based target detection network model for training until convergence.

[0007] Step 2: Mount the RealSense depth camera on the connected automated inspection device. The camera's shooting angle is not critical, as long as it can capture the bottom surface of the bolt. Once the RealSense depth camera is mounted, it will capture a color image and a depth map corresponding to each pixel in the color image.

[0008] Step 3: Deploy the trained neural network weights on the deep learning-based target detection network model to automatically identify and select the bolt targets from the color image captured by the RealSense depth camera. The selected area is called the bolt color map. At the same time, the bolt depth map is extracted from the depth map based on the results of the bolt color map.

[0009] Step 4: The bolt color image undergoes super-resolution processing based on a generative adversarial network, magnifying the image by a factor of n to obtain more pixels. Because the RealSense depth camera's optical axis is not perpendicular to the bolt's base, the circular bolt base appears as an ellipse in the acquired bolt color image. Using a pixel gradient-based ellipse detection method, multiple elliptical targets are detected in the magnified bolt color image. The center coordinates and semi-axis sizes of the acquired elliptical targets are then reduced to 1 / nth the size. Super-resolution processing is a method of extracting additional image detail and information based on known image information.

[0010] Step 5: Based on the ellipse detection results in the bolt color image, the center of mass offset vector method is used, combined with the bolt depth map corresponding to the bolt color image, to screen out the unique ellipse corresponding to the bolt bottom surface from multiple ellipse targets. The center position of this ellipse is the center position of the bolt bottom surface.

[0011] Step 5 includes the following steps:

[0012] Step 5.1: The bolt color map obtained in step 3 is called CPRB, and the area ratio of the ellipse is defined as:

[0013]

[0014] Where a is the semi-major axis of the ellipse, b is the semi-minor axis of the ellipse, and W CPRB is the width of CPRB, H CPRB is the height of CPRB. The flattening of the ellipse is:

[0015]

[0016] Therefore, the constraint equations are:

[0017]

[0018] As the angle between the optical axis of the RealSense depth camera and the bottom surface of the bolt gradually decreases, will gradually increase, and γ will gradually decrease. According to the actual situation, set The upper and lower thresholds of γ are used to include the area ratio and flattening of all annotated ellipses. A large number of false ellipses can be removed through the constraint equation, thereby achieving the initial screening of ellipse area and shape.

[0019] Step 5.2: First, set the size of W CPRB ×H CPRB The bolt depth map is expanded, and the points with a depth value of 0 are invalid depth points. The invalid depth points are eliminated and sorted in ascending order according to the depth value to obtain a 1×N z data, where N z is the number of effective depth points in a bolt depth map. Then create a frequency distribution histogram based on the depth value, and divide it into N equal widths. H Since the depth values ​​of the foreground background and the target bolts differ greatly, discontinuous columns will appear in the frequency distribution histogram. The depth values ​​at the left end of these gaps are used as the truncation threshold T d For different shooting angles and shooting environments, there will be multiple truncation thresholds T d Therefore, it is necessary to correctly use the truncation threshold to segment the depth information of the target bolt: in a CPRB, the target bolt accounts for more than 50% of the entire bolt depth map, and find the depth point number N closest to the truncation threshold. t , if N t >N z / 2, this threshold is the background threshold, so the depth values ​​of all depth points in the bolt depth map that are greater than the cutoff threshold are set to 0. On the contrary, if N t <N z / 2, this threshold is the foreground threshold, and the depth values ​​of all depth points in the bolt depth map that are less than the truncation threshold are set to 0, ultimately achieving the removal of the foreground and background of the bolt depth map.

[0020] Step 5.3: Restore the 3D point cloud data after foreground and background removal to the form of depth map. Let D s is the slice depth:

[0021] D S =D min +τ·(D max -D min ) (4)

[0022] Among them D max With D min are the maximum and minimum depth values ​​in the depth map, respectively, and τ is the slicing factor. When the slicing factor is fixed, D s According to D max With D min The value of is constantly adjusted, so D s Is an adaptive threshold. All depth values ​​less than D s The depth of the effective point is projected onto a two-dimensional plane, which is called a slice of the depth map. Taking the depth value as the mass, the centroid of the slice is:

[0023]

[0024] where x c with y c are the horizontal and vertical coordinates of the slice centroid, Di is the depth value of the depth point (xi,yi), ns is the number of valid depth points in the slice. By gradually increasing the value of τ, multiple consecutive slices are obtained. Assuming that there are a total of S c slices, and correspondingly get S c A centroid.

[0025] Step 5.4: Project the centroid of each slice onto the same two-dimensional plane P c In the above example, the offset direction of the center of mass is obtained by linear fitting.

[0026] The model of the linear regression line is as follows:

[0027] f(x ci |θ)=θ0+θ1x ci (6)

[0028] Where θ0 and θ1 are the intercept and slope respectively, x ci is the horizontal coordinate of the centroid of the i-th slice.

[0029] The fitting method adopts the quadratic fitting method based on Huber Loss. The Huber Loss loss function is as follows:

[0030]

[0031] Where δ is a hyperparameter and yci is the vertical coordinate of the centroid of the actual i-th slice. The linear parameters θ0 and θ1 can be obtained through iterative optimization.

[0032] Calculate the square error of each mass point relative to the linear regression line:

[0033] ε i =(y ci -f(x ci )) 2 (8)

[0034] The mean of the squared error is:

[0035]

[0036] Traverse the square error of each particle, when v i The value is greater than When the center of mass (x ci ,yci) are outliers and are removed. Finally, the filtered particles are subjected to quadratic fitting to obtain the final θ0 and θ1.

[0037] Define the centroid offset vector as ρx,ρ y ∈{-1,+1} is the direction factor, by adjusting ρx and ρ y The value of the vector It is from the nut to the bottom of the bolt. Calculate the vectors of all adjacent mass centers and get the vector group as Count the number of vectors whose horizontal coordinates are greater than 0 N x ,if Then ρx takes the value of +1, otherwise it takes the value of -1. Similarly, count the number of vectors whose vertical coordinate is greater than 0, N y ,when Time ρ y The value is +1, otherwise the value is -1.

[0038] In order to screen out the final ellipse, the confidence score of the ellipse is defined as:

[0039]

[0040] in The confidence score of each ellipse is calculated as the vector from the center of the ellipse to the origin of the pixel coordinate. The ellipse with the highest score is the final filtered ellipse.

[0041] Step 6: Let the ellipse corresponding to the bottom of the bolt be E1, and E2 be a concentric ellipse that is proportionally reduced by α times the original size, where α∈(0,1). By selecting an appropriate value for α, find all symmetrical valid points within ellipse E2 and obtain the depth values ​​of these pixels based on the depth map. Average these depth values ​​to obtain the final depth value of the center of the bolt bottom.

[0042] Step 7: Using the bolt bottom center depth value obtained in step 6 and the intrinsic parameters of the RealSense depth camera, calculate the coordinates of the bolt bottom center in the camera coordinate system based on the camera pinhole model.

[0043] Compared with the prior art, the present invention has the following significant advantages: (1) The super-resolution algorithm is combined with the ellipse detection algorithm, which greatly improves the effect of the ellipse detection algorithm and reduces the missed detection rate of the ellipse on the bottom surface of the bolt. (2) A centroid offset vector method is proposed to greatly reduce the interference of other ellipse features and can accurately screen out the ellipse corresponding to the bottom surface of the bolt. (3) A symmetrical depth mean method is proposed, so that when a hole appears in the depth map of the bottom surface of the bolt, the depth value of the center of the bottom of the bolt can still be calculated. The present invention has been experimentally verified in a real aero-engine rotor cavity environment. The experimental results show that the present invention has high positioning accuracy and strong robustness for aero-engine bolts and can guide the automated detection probe. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of a highly robust visual positioning method for aircraft engine bolts;

[0045] Figure 2 This is a flow chart of the ellipse screening method based on the centroid offset vector method;

[0046] Figure 3 This is the principle diagram of the symmetric depth mean method;

[0047] Figure 4 The three-dimensional positioning results of the bolts at different shooting angles in the embodiment are shown, where (a) is the mean absolute error in the X direction, (b) is the mean absolute error in the Y direction, and (c) is the mean absolute error in the Z direction;

[0048] Figure 5 is a color image obtained in the embodiment;

[0049] Figure 6 is a depth image obtained in the embodiment;

[0050] Figure 7 is a color image of the bolt obtained in the embodiment;

[0051] Figure 8 is a bolt depth map obtained in the embodiment;

[0052] Figure 9 The result of ellipse detection on the bolt color image in the embodiment;

[0053] Figure 10 The result of slicing the bolt depth map in the embodiment;

[0054] Figure 11 This is the result after ellipse screening in the embodiment. DETAILED DESCRIPTION

[0055] The following is a further detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings. This description is not restrictive and the actual use is not limited to this. An embodiment of the present invention is as follows:

[0056] Step 1: Use mobile phones, RealSense depth cameras, and other devices to obtain 800 color images of aircraft engine cavities containing aircraft bolts. Manually annotate the obtained color images by surrounding the bolts with rectangular frames. The annotated results are divided into training and validation sets in an 8:2 ratio and sent to the deep learning-based object detection network model for 100 generations of training until convergence.

[0057] Step 2: Secure the RealSense depth camera to the camera bracket, ensuring it can capture the bottom surface of the bolt. There are no restrictions on the shooting angle. Once the RealSense depth camera is secured, obtain the color image and the depth map corresponding to its pixels.

[0058] Step 3: Deploy the trained neural network weights on the deep learning-based target detection network model, automatically identify and select the bolt targets from the color image captured by the RealSense depth camera, and extract the bolt color image and bolt depth image from the depth image and color image respectively.

[0059] Step 4: Using a super-resolution method based on a generative adversarial network, the bolt color image is magnified by 4 times to obtain more pixels. Super-resolution processing refers to the process of extracting more image detail and information based on known image information. Because the optical axis of the RealSense depth camera is not perpendicular to the bolt bottom surface, the circular bolt bottom surface appears as an ellipse in the acquired bolt color image. Using an ellipse detection method based on pixel gradients, multiple elliptical targets are detected in the magnified bolt color image. The center coordinates and semi-axis sizes of the acquired elliptical targets are then reduced to 1 / 4 of their original size.

[0060] Step 5: Based on the ellipse detection results in the bolt color image and the bolt depth map corresponding to the bolt color image, a unique ellipse corresponding to the bolt bottom surface is selected from multiple ellipse targets. The center position of the ellipse is the center position of the bolt bottom surface.

[0061] Step 5 includes the following steps:

[0062] Step 5.1: Based on the multiple bolt color images obtained from the actual measurement scene, manually mark the bolt bottom ellipse and calculate the area ratio of the marked ellipse and the flattening γ, the area ratio of the ellipse is defined as:

[0063]

[0064] Where a is the semi-major axis of the ellipse, b is the semi-minor axis of the ellipse, and W CPRB is the width of CPRB, H CPRB is the height of CPRB. The flattening of the ellipse is:

[0065]

[0066] Therefore, the constraint equations are:

[0067]

[0068] As the angle between the optical axis of the RealSense depth camera and the bottom surface of the bolt gradually decreases, will gradually increase, and γ will gradually decrease. Set according to the calculation results The upper and lower thresholds of γ are used to include the area ratio and flattening of all annotated ellipses. By using the constraint equation, a large number of false ellipses can be removed, thus achieving the initial screening of ellipse area and shape.

[0069] Step 5.2: In the actual shooting environment, there may be random interference targets that are too close or too far away, which makes the depth values ​​of different pixels in the depth map span too large, making it difficult to extract useful information. To solve this problem, first, CPRB ×H CPRB The bolt depth map is expanded, and the points with a depth value of 0 are invalid depth points. The invalid depth points are eliminated and sorted in ascending order according to the depth value to obtain a 1×N z data, where N z is the number of valid depth points in a bolt depth map. Then, a frequency distribution histogram is created based on the depth values, and divided into 20 groups of equal width. Due to the large difference in depth between the foreground background and the target bolt, there will be discontinuous columns in the frequency distribution histogram. The depth value at the left end of these gaps is used as the truncation threshold T d For different shooting angles and shooting environments, there will be multiple truncation thresholds T d Therefore, it is necessary to correctly use the cutoff threshold to segment the depth information of the target bolt. Obviously, in a CPRB, the target bolt accounts for much more than 50% of the entire bolt depth map. Find the depth point number N closest to the cutoff threshold. t , if N t >N z / 2, this threshold is the background threshold, so the depth values ​​of all depth points in the bolt depth map that are greater than the cutoff threshold are set to 0. On the contrary, if N t <N z / 2, this threshold is the foreground threshold, and the depth values ​​of all depth points in the depth map that are less than the truncation threshold are set to 0, ultimately achieving the removal of the foreground and background of the bolt depth map.

[0070] Step 5.3: Restore the 3D point cloud data after foreground and background removal into the form of a depth map, let D s is the slice depth:

[0071] D S =D min +τ·(D max -D min ) (4)

[0072] Among them D max With D min are the maximum and minimum depth values ​​in the depth map, respectively, and τ is the slicing factor. When the slicing factor is fixed, D s According to Dmax With D min The value of is constantly adjusted, so D s Is an adaptive threshold. All depth values ​​less than D s The depth of the effective point is projected onto a two-dimensional plane, which is called a slice of the depth map. Taking the depth value as the mass, the centroid of the slice is:

[0073]

[0074] where x c with y c are the horizontal and vertical coordinates of the slice centroid, Di is the depth value of the depth point (xi,yi), ns is the number of valid depth points in a slice. The slicing factor τ is set to a base value of 0.3 and is incremented 10 times in steps of 0.03 to obtain 10 slices. Using the depth value as the mass, the horizontal and vertical coordinates of the centroids of the 10 slices are calculated.

[0075] Step 5.4: By observation, it can be found that the displacement of the center of mass can reflect the direction of the bolt to a certain extent. Therefore, the coordinates of the 10 centers of mass are plotted in a two-dimensional plane and the displacement direction of the center of mass is obtained using a linear fit method. The linear model is established as follows:

[0076] f(x ci |θ)=θ0+θ1x ci (6)

[0077] Where θ0 and θ1 are the intercept and slope respectively, x ci is the horizontal coordinate of the centroid of the i-th slice.

[0078] The mean square error is sensitive to outliers, and the fitting results are easily disturbed and are not applicable. A quadratic fitting method based on Huber Loss is used to perform linear fitting on these centroids. The Huber Loss loss function is as follows:

[0079]

[0080] yci is the ordinate of the centroid of the actual i-th slice. The hyperparameter δ in the Huber Loss is set to 1.5. The linear parameters θ0 and θ1 can be obtained through iterative optimization. To further improve the regression accuracy of the line, the squared error of each mass point relative to the regression line is calculated using the fitting results:

[0081] ε i =(y ci -f(x ci )) 2 (8)

[0082] The mean of the squared error is:

[0083]

[0084] Traverse the square error of each particle, when ε i The value is greater than When the center of mass (x ci ,yci) are outliers and are removed. Finally, the filtered particles are subjected to quadratic fitting to obtain the final θ0 and θ1.

[0085] Define the centroid offset vector as ρx,ρ y ∈{-1,+1} is the direction factor, by adjusting ρx and ρ y The value of can ensure that the vector It is from the nut to the bottom of the bolt. Calculate the vectors of all adjacent mass centers and get the vector group as Count the number of vectors whose horizontal coordinates are greater than 0 N x ,if Then ρx takes the value of +1, otherwise it takes the value of -1. Similarly, count the number of vectors whose vertical coordinate is greater than 0, N y ,when Time ρ y The value is +1, otherwise it is -1. Based on the coordinates of the multiple ellipse centers obtained in the previous steps, the confidence score proje of each ellipse is calculated, and the confidence score of the ellipse is defined as:

[0086]

[0087] in is the vector from the center of the ellipse to the origin of the pixel coordinates. After calculating the confidence score of each ellipse, the ellipse with the highest score is the final filtered ellipse.

[0088] Step 6: Due to factors such as lighting noise and reflection, the actual depth image will have many holes, resulting in the inability to obtain the depth value of the center position of the bottom surface of some bolts. In order to solve this problem, a symmetrical depth averaging method is proposed. The valid points are the pixel points in the depth map that can obtain valid depth values. If two valid points are symmetrical about the center of the bottom surface of the bolt, the two points are called a set of symmetrical valid points. Figure 3 As shown in the figure, E1 is the detected ellipse at the bottom of the bolt, Q2 is the center point of the ellipse, Q1 and Q3 are two points on the edge of ellipse E1, and they are symmetrical about Q2. P1, P2, and P3 are planes containing Q1, Q2, and Q3 and parallel to the camera plane. The depth value measured by the RealSense depth camera is the distance from the object to the camera plane. Therefore, Z1, Z2, and Z3 are the depth values ​​of points Q1, Q2, and Q3 relative to the RealSense depth camera, respectively. The geometric properties of the trapezoid can be used to obtain:

[0089]

[0090] Therefore, even if the depth value of the ellipse center is missing, the depth value of the ellipse center can be solved by knowing the depth values ​​of any two points symmetrical about the center of the ellipse. To reduce the error of the estimated value, it is necessary to select a sufficient number of calculation points, and they should be as close to the ellipse center as possible. E2 is a concentric ellipse E1 scaled down to 0.5 times its original size. All symmetrical valid points within ellipse E2 are found and the depth values ​​of these pixels are obtained based on the depth map information. These depth values ​​are averaged to finally obtain the depth value of the ellipse center.

[0091] Step 7: Using the bolt bottom center depth value obtained in step 6 and the intrinsic parameters of the RealSense depth camera, calculate the coordinates of the bolt bottom center in the camera coordinate system based on the camera pinhole model.

Claims

1. A highly robust visual positioning method for aircraft engine bolts, characterized in that: Here are the steps: Step 1: Obtain color images of the aircraft engine cavity at different shooting angles. Manually annotate the acquired color images by surrounding the bolts with rectangular boxes. The annotated results are divided into training and validation sets and fed into a deep learning-based object detection network model for training until convergence. Step 2: Fix the RealSense depth camera to the connected automated inspection device. The shooting angle is not required, as long as it can capture the bottom surface of the bolt. After the RealSense depth camera is fixed, obtain the color image and the depth map corresponding to the color image pixels. Step 3: Deploy the trained neural network weights on the deep learning-based object detection network model to automatically identify and select bolt targets from the color image captured by the RealSense depth camera. The selected area is called the bolt color map. At the same time, the bolt depth map is extracted from the depth map based on the bolt color map results. Step 4: The bolt color image is subjected to super-resolution processing based on a generative adversarial network, which magnifies the bolt color image by a factor of n to obtain more pixels. An ellipse detection method based on pixel gradients is used to detect multiple elliptical targets in the magnified bolt color image, and the center coordinates and semi-axis sizes of the obtained elliptical targets are reduced to 1 / n. Super-resolution processing refers to a method of obtaining more image details and information based on known image information. Step 5: Based on the ellipse detection results in the bolt color image, the center of mass offset vector method is used, combined with the bolt depth map corresponding to the bolt color image, to screen out the unique ellipse corresponding to the bolt bottom surface from multiple ellipse targets. The center position of this ellipse is the center position of the bolt bottom surface; Step 6: Let the ellipse corresponding to the bottom of the bolt be E1, and E2 be a concentric ellipse that is proportionally reduced to α times the original E1, where α∈(0,1). By selecting an appropriate α value, find all symmetrical valid points inside the ellipse E2 and obtain the depth values ​​of these pixels based on the depth map. Finally, average these depth values ​​to obtain the depth value of the center of the bolt bottom. Step 7: Using the bolt bottom center depth value obtained in step 6 and the intrinsic parameters of the RealSense depth camera, calculate the coordinates of the bolt bottom center in the camera coordinate system based on the camera pinhole model.

2. A highly robust visual positioning method for aircraft engine bolts according to claim 1, characterized in that: Step 5 includes the following steps: Step 5.1: The bolt color map obtained in step 3 is called CPRB, and the area ratio of the ellipse is defined as: Where a is the semi-major axis of the ellipse, b is the semi-minor axis of the ellipse, and W CPRB is the width of CPRB, H CPRB is the height of CPRB; the flattening of the ellipse is: Therefore, the constraint equations are: As the angle between the optical axis of the RealSense depth camera and the bottom surface of the bolt gradually decreases, will gradually increase, and γ will gradually decrease; according to the actual situation, set The upper and lower thresholds of γ are used to include the area ratio and flattening of all annotated ellipses. A large number of false ellipses are removed through the constraint equation, thus achieving the initial screening of ellipse area and shape. Step 5.2: First, set the size of W CPRB ×H CPRB The bolt depth map is expanded, and the points with a depth value of 0 are invalid depth points. The invalid depth points are eliminated and sorted in ascending order according to the depth value to obtain a 1×N z data, where N z is the number of valid depth points in a bolt depth map; then a frequency distribution histogram is created based on the depth value, divided into N equal width H Since the depth values ​​of the foreground background and the target bolts differ greatly, discontinuous columns will appear in the frequency distribution histogram. The depth values ​​at the left end of these gaps are used as the cutoff threshold T. d ; For different shooting angles and shooting environments, there will be multiple truncation thresholds T d , use the cutoff threshold to segment the depth information of the target bolt: in a CPRB, the target bolt accounts for more than 50% of the entire bolt depth map, and find the depth point number N closest to the cutoff threshold t , if N t >N z / 2, this threshold is the background threshold, so the depth values ​​of all depth points in the bolt depth map that are greater than the cutoff threshold are set to 0. On the contrary, if N t <N z / 2, this threshold is the foreground threshold, and the depth values ​​of all depth points in the bolt depth map that are less than the truncation threshold are set to 0, ultimately achieving the removal of the foreground and background of the bolt depth map; Step 5.3: Restore the 3D point cloud data after foreground and background removal to the form of depth map; let D s is the slice depth: D S =D min +τ·(D max -D min ) (4) Among them D max With D min are the maximum and minimum depth values ​​in the depth map, respectively, and τ is the slicing factor. When the slicing factor is fixed, D s According to D max With D min The value of is constantly adjusted, so D s is an adaptive threshold; all depth values ​​less than D s The depth of the effective point is projected onto a two-dimensional plane, which is called a slice of the depth map. Taking the depth value as the mass, the centroid of the slice is: where x c with y c are the horizontal and vertical coordinates of the slice centroid, Di is the depth value of the depth point (xi,yi), ns is the number of effective depth points in the slice; by gradually increasing the value of τ, multiple consecutive slices are obtained. Suppose there are a total of S c slices, and correspondingly get S c centroid; Step 5.4: Project the centroid of each slice onto the same two-dimensional plane P c In the example, the linear fitting method is used to obtain the offset direction of the center of mass: The model of the linear regression line is as follows: f(x ci |θ)=θ0+θ1x ci (6) Where θ0 and θ1 are the intercept and slope respectively, x ci is the horizontal coordinate of the centroid of the i-th slice; The fitting method adopts the quadratic fitting method based on Huber Loss. The Huber Loss loss function is as follows: Where δ is a hyperparameter, yci is the vertical coordinate of the centroid of the actual i-th slice; the linear parameters θ0 and θ1 can be obtained by iteration through the optimization method; Calculate the square error of each mass point relative to the linear regression line: ε i =(y ci -f(x ci )) 2 (8) The mean of the squared error is: Traversing the square error of each particle, when ε i The value is greater than When the center of mass (x ci ,yci) are outliers and are removed. Finally, the filtered particles are subjected to quadratic fitting to obtain the final θ0 and θ1; Define the centroid offset vector as ρx,ρ y ∈{-1,+1} is the direction factor, by adjusting ρx and ρ y The value of the vector is from the nut to the bottom of the bolt; calculate the vectors of all adjacent mass centers and get the vector group as Count the number of vectors whose horizontal coordinates are greater than 0 N x ,if Then ρx takes the value of +1, otherwise it takes the value of -1; similarly, count the number of vectors with a vertical coordinate greater than 0 N y ,when Time ρ y The value is +1, otherwise the value is -1; In order to screen out the final ellipse, the confidence score of the ellipse is defined as: in The confidence score of each ellipse is calculated as the vector from the center of the ellipse to the origin of the pixel coordinate. The ellipse with the highest score is the final filtered ellipse.

Images