A bolt defect detection method based on image information and point cloud information

Abstract

The present application belongs to the field of computer vision, and particularly relates to a bolt defect detection method based on image information and point cloud information, comprising: collecting an inspection image and point cloud data of an inspection area containing bolts to be inspected, wherein the inspection image and the point cloud data are pixel-aligned; based on the inspection image, a pre-trained target recognition model is used to regress and predict a target frame of the bolt to be inspected; if the predicted result is that there is no bolt to be inspected, the bolt defect detection is completed; if the predicted result is that there is a bolt to be inspected, point cloud mapping of the bolt area is performed according to the target frame obtained by regression, and a RANSAC plane segmentation algorithm is used to segment and extract the upper surface of the bolt and the mounting reference surface in the point cloud of the bolt area; the distance between the two planes is calculated and compared with a set threshold value, so as to determine whether the bolt is loose, and the bolt defect detection is completed. The present application can efficiently realize bolt defect detection.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and more specifically, relates to a bolt defect detection method based on image information and point cloud information. Background Technology

[0002] Bolt defect identification is fundamental to ensuring the normal operation of inspected equipment. If loose bolts are not detected and repaired by maintenance personnel in a timely manner, they may fall off further. Loose bolts can cause a series of problems such as oil leaks and air leaks, and in the case of bolt detachment, they may even lead to more serious safety issues. Therefore, the task of identifying bolt defects in inspected equipment is particularly important.

[0003] Currently, the main methods for bolt defect identification are to use image information or point cloud information alone. However, when using image information for bolt defect identification, the recognition accuracy is not high because the image features are not obvious, especially for images taken from a viewpoint perpendicular to the bolt surface. On the other hand, when using point cloud information alone for bolt defect identification, the processing is performed on the global point cloud, which involves a large amount of computation and is time-consuming.

[0004] Therefore, existing bolt defect detection methods suffer from many problems, such as low detection accuracy, difficulty in adapting to various shooting angles, and low real-time performance. Summary of the Invention

[0005] To address the shortcomings and improvement needs of existing technologies, this invention provides a bolt defect detection method based on image information and point cloud information. Its purpose is to ensure timeliness while avoiding the problem of low accuracy when identifying bolt defects based on images taken from a viewpoint perpendicular or approximately perpendicular to the bolt surface.

[0006] To achieve the above objectives, according to one aspect of the present invention, a bolt defect detection method based on image information and point cloud information is provided, comprising:

[0007] Collect inspection images and point cloud data of the inspection area containing the bolts to be inspected, wherein the inspection images and point cloud data are pixel-aligned;

[0008] Based on the inspection images, a pre-trained target recognition model is used to regress and predict the target bounding box of the bolt to be inspected.

[0009] If the prediction result indicates that there is no bolt to be inspected, then the bolt defect detection is completed; if the prediction result indicates that there is a bolt to be inspected, the point cloud of the bolt region is mapped according to the target box obtained by regression, and the RANSAC plane segmentation algorithm is used to segment and extract the upper surface of the bolt and the mounting reference surface in the point cloud of the bolt region; the distance between the two planes is calculated and compared with the set threshold to determine whether the bolt has become loose, thus completing the bolt defect detection.

[0010] Furthermore, a depth camera is used to acquire the inspection images and point cloud data.

[0011] Furthermore, the target recognition model is an improved YOLOv8, wherein the improved YOLOv8 is obtained through the following improvement methods:

[0012] In the C2f structure of the backbone feature extraction network of YOLOv8, a coordinate attention mechanism (CA) is added. The CA is added to the output of the Bottleneck module in C2f to assign different weights to different channels and regions of the feature map output by the Bottleneck module.

[0013] In the neck structure of YOLOv8, a progressive feature pyramid fusion structure AFPN is used. First, the CBS module is used to convolve the P3 and P4 feature maps output by the backbone feature extraction network to extract features. Then, two ASFF2 modules are used to adaptively fuse the convolutional feature maps. The two fused feature maps are then processed by the C2f module to obtain intermediate layer feature maps C3 and C4. Three ASFF3 modules are used to adaptively fuse the intermediate layer feature maps C3 and C4 with the P5 feature map output by the backbone feature extraction network and then process them by C2f to extract features. Finally, progressively fused feature maps P'3, P'4, and P'5 are obtained. Subsequently, the fused feature layers are used for target bounding box regression and prediction.

[0014] Furthermore, the improved YOLOv8 uses the Focal-SIoU bounding box loss function during training.

[0015] Furthermore, the Focal-SIoU bounding box loss function is specifically as follows:

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023] In the formula, IoU represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box; Λ represents the angle loss term; Ω represents the shape loss term; ρ x ρ y Both γ and γ are intermediate variables; and These represent the x-coordinates of the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively. and c represents the ordinate of the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively; w and c h Let represent the height and width of the minimum bounding boxes of the predicted and ground truth bounding boxes, respectively; α represents the angle between the line connecting the center points of the predicted and ground truth bounding boxes and the horizontal direction of the image; β represents the angle between the line connecting the center points of the predicted and ground truth bounding boxes and the vertical direction of the image; b h This represents the height difference between the center point of the predicted bounding box and the center point of the actual bounding box; b w σ represents the width difference between the center points of the predicted and ground truth bounding boxes; σ represents the distance between the center points of the predicted and ground truth bounding boxes; w and h represent the width and height of the ground truth bounding box, respectively; w gt h gt These represent the width and height of the predicted bounding box, respectively; w w w h All are intermediate variables; θ represents the degree of attention the Focal-SIoU bounding box loss function pays to shape loss.

[0024] Furthermore, before employing the RANSAC plane segmentation algorithm, statistical filtering is performed on the point cloud of the bolt region obtained by mapping to remove outliers, and the filtered point cloud is uniformly downsampled by a factor of 4. The point cloud of the bolt region obtained in this way is used in the subsequent fitting algorithm.

[0025] Furthermore, the RANSAC plane segmentation algorithm is used to segment and extract the upper surface of the bolt and the mounting reference plane in the point cloud of the bolt region. The first segmentation extraction uses the RANSAC plane segmentation algorithm, and the second segmentation extraction uses the RANSAC plane segmentation algorithm with normal vector constraints.

[0026] The RANSAC plane segmentation algorithm with normal vector constraints is as follows: In the RANSAC plane segmentation algorithm, after determining the current iteration plane and before traversing the point cloud to calculate the distance from each point to the current iteration plane, the following judgment step is added:

[0027] Calculate the cosine of the normal vector of the current iteration plane and the normal vector of the plane extracted in the first segmentation. If the cosine value is less than the preset value, return to determine a new plane as the next iteration plane and repeat the judgment step. Otherwise, execute the subsequent steps of the RANSAC plane segmentation algorithm.

[0028] Furthermore, the preset value is 0.995.

[0029] Furthermore, the method for calculating the distance between the two planes is as follows:

[0030] Based on the bolt's upper surface obtained from the segmentation and the installation reference surface, the distance is calculated using the set of interior points of one plane and the four parameters of the other plane. The calculated distance is an array with the number of elements equal to the number of data points in the set of interior points. The distance between the two planes is obtained by averaging this array.

[0031] The present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed by a processor, it controls the device containing the storage medium to perform a bolt defect detection method based on image information and point cloud information as described above.

[0032] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0033] (1) This invention proposes a bolt defect detection method based on image information and point cloud information. First, RGB images and point cloud data are acquired, where the RGB images and point cloud data are automatically pixel-aligned. The bolt area is located using the RGB images, and then point cloud mapping is performed. Based on the point cloud data, planar segmentation and identification of looseness are conducted. This invention overcomes the limitations of existing methods that rely solely on image data to effectively identify bolt loosening defects in images taken within a certain angle range perpendicular to the bolt's upper surface. It also overcomes the problem of excessive computation when using global point cloud processing. This improves the accuracy and timeliness of bolt defect identification.

[0034] (2) This invention proposes a method based on inspection images, utilizing an improved YOLOv8 to locate bolt areas and identify bolt loss defects. This reduces interference from complex backgrounds and enhances the model's feature extraction capability for bolt targets in complex environments. While fusing low-level feature details, it retains as much semantic information as possible from high-level features, thereby improving detection accuracy. It addresses the issues of the overall small size of bolt and nut samples, their greater sensitivity to angular offsets between the target bounding box and the predicted target bounding box, and the varying degrees of difficulty in locating bolts of different shapes, thus improving the accuracy of coordinate frame localization.

[0035] (3) This invention proposes to preprocess the bolt region point cloud obtained by point cloud mapping based on the bolt coordinate frame by using statistical filtering and downsampling, which reduces the interference of noisy point clouds and improves the running efficiency of the algorithm.

[0036] (4) This invention proposes that when performing point cloud plane segmentation, the second plane segmentation uses the RANSAC algorithm with normal vector constraints to perform plane segmentation, which reduces invalid iterative operations and reduces computational complexity. Attached Figure Description

[0037] Figure 1 A flowchart of a bolt defect detection method based on image information and point cloud information provided in an embodiment of the present invention;

[0038] Figure 2 An improved YOLOv8 model structure diagram provided for embodiments of the present invention;

[0039] Figure 3 This is a schematic diagram of bolt defect identification based on image information and point cloud information provided in an embodiment of the present invention;

[0040] Figure 4 A schematic diagram illustrating the meaning of each parameter for calculating the Focal-SIoU loss, provided for embodiments of the present invention;

[0041] Figure 5 The flowcharts for two different RANSAC algorithm plane extraction methods provided in the embodiments of the present invention are shown. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0043] Example 1

[0044] A bolt defect detection method based on image information and point cloud information, such as Figure 1 As shown, it includes:

[0045] Collect inspection images and point cloud data of the inspection area containing the bolts to be inspected, wherein the inspection images and point cloud data are pixel-aligned;

[0046] Based on the inspection images, a pre-trained target recognition model is used to regress and predict the target bounding box of the bolt to be inspected.

[0047] If the prediction result indicates that there is no bolt to be inspected, then the bolt defect detection is completed; if the prediction result indicates that there is a bolt to be inspected, the point cloud of the bolt region is mapped according to the target box obtained by regression, and the RANSAC plane segmentation algorithm is used to segment and extract the upper surface of the bolt and the mounting reference surface in the point cloud of the bolt region; the distance between the two planes is calculated and compared with the set threshold to determine whether the bolt has become loose, thus completing the bolt defect detection.

[0048] This embodiment utilizes inspection photography to collect inspection images and point cloud data of various areas of the inspected object. A preferred implementation involves using an inspection robot to collect these images and point cloud data. Specifically, the inspection robot uses a depth camera at its end to take pictures of the inspected object. For example, if the inspected object is the bottom bogie of a high-speed train parked in a maintenance workshop, the inspection robot carries two robotic arms—a left and a right robotic arm—each equipped with a precision depth camera to achieve high-resolution image and point cloud data acquisition of the bottom bogie of the high-speed train.

[0049] This embodiment proposes a bolt defect detection method based on image and point cloud information. First, RGB images and point cloud data are acquired, with automatic pixel alignment between the RGB images and point cloud data. The bolt area is located using the RGB images, followed by point cloud mapping. Based on the point cloud data, planar segmentation and looseness detection are performed. This method overcomes the limitations of existing methods that rely solely on image data to effectively identify bolt loosening defects within a certain angle range perpendicular to the bolt's upper surface, and also overcomes the computational burden associated with global point cloud processing. This improves the accuracy and timeliness of bolt defect identification.

[0050] Considering that the bolts in the inspected objects are relatively small in size and occupy a small proportion in the image, and that the difficulty of locating bolts of different shapes varies, this embodiment proposes to use an improved YOLOv8 for bolt area location and bolt loss defect identification. The specific improvement method is as follows:

[0051] In the C2f structure of the backbone feature extraction network of YOLOv8, a coordinate attention mechanism (CA) is added. The CA is added to the output of the Bottleneck module in C2f to assign different weights to different channels and regions of the feature map output by the Bottleneck module.

[0052] In the neck structure of YOLOv8, the progressive feature pyramid fusion structure AFPN is used. First, the P3 and P4 feature maps output by the backbone feature extraction network are adaptively fused. Then, the fused C3 and C4 feature maps are adaptively fused with the P5 feature map output by the backbone feature extraction network. Finally, the progressively fused feature maps P'3, P'4, and P'5 are obtained. Finally, the fused feature layers are used for subsequent target box regression and prediction.

[0053] Specifically, such as Figure 2 As shown, a coordinate attention mechanism (CA), named C2f_Attentation, is added to the C2f structure in the Backbone feature extraction network. CA is added to the output of the Bottleneck module in C2f to perform weight processing on the output of the Bottleneck module, thereby improving the attention of the improved YOLOv8 model to the region of interest. By assigning different weights to different channels and regions of the input data, the model pays more attention to important information, thus improving the target detection performance.

[0054] like Figure 2 As shown, a progressive feature pyramid fusion structure (AFPN) is used in the neck structure. First, the CBS module is used to convolve the P3 and P4 feature maps output by the backbone network to extract features. Then, two ASFF2 modules are used to adaptively fuse the obtained feature maps. The fused feature maps are then processed by the C2f module to obtain intermediate layer feature maps C3 and C4. Next, the fused C3 and C4 feature layers are adaptively fused with the P5 feature map output by the backbone feature extraction network using three ASFF3 modules and then processed by C2f for feature extraction. Finally, the progressively fused feature layers P'3, P'4, and P'5 are obtained. These fused feature maps are used for target box regression and prediction.

[0055] In general, this preferred solution is as follows: Figure 3As shown, based on each inspection image, an improved YOLOv8 is used to locate bolt areas and identify bolt loss defects. The improvement strategies include: adding a coordinate attention mechanism (CA) to the backbone feature extraction network, which, by introducing horizontal and vertical coordinate information, more effectively captures spatial distribution features in the image and aggregates spatial information globally, helping the model capture long-distance dependencies in the image. This reduces interference from complex backgrounds and improves the model's feature extraction capability for bolts and nuts in complex environments. In the neck structure, an AFPN structure is used instead of the original PAN-FAN for multi-scale feature fusion, preserving semantic information of high-level features as much as possible while fusing low-level feature details, thus improving detection accuracy. An improved version of YOLOv8 can be used to identify bolt loss defects. If the bolt is present, the coordinate frame of the bolt area can be located. Based on the location of the bolt area, point cloud mapping of the bolt area is performed. Then, the RANSAC plane segmentation algorithm is used to segment and extract the upper surface of the bolt and the mounting reference plane in the bolt area point cloud. Finally, the distance between the two planes is calculated and compared with a set threshold to determine whether the bolt is loose. Finally, the bolt loss identification result is combined with the bolt loosening identification result to obtain the bolt defect identification result.

[0056] As a preferred option, the Focal-SIoU bounding box loss function is used instead of the original CIoU loss during training. This solves the problems that the overall target size of bolt samples is relatively small, they are more sensitive to the angular offset between the target box and the predicted target box, and the difficulty of positioning bolts and nuts of different shapes varies, thus improving the accuracy of coordinate box positioning.

[0057] In other words, the Focal-SIoU bounding box loss function is used instead of the original CIoU loss during training, which solves the problems of the overall target size of bolt samples being too small, being more sensitive to the angular offset between the real target box and the predicted target box, and the different difficulty of positioning between bolts and nuts of different shapes, thus improving the accuracy of coordinate box positioning. The improved YOLOv8 uses the Focal-SIoU bounding box loss function during training.

[0058] As a preferred implementation, the Focal-SIoU bounding box loss function is specifically as follows:

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066] In the formula, IoU represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box; Λ represents the angle loss term; Ω represents the shape loss term; ρ x ρ y Both γ and γ are intermediate variables; for example Figure 4 As shown, and These represent the x-coordinates of the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively. and c represents the ordinate of the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively; w and c h Let represent the height and width of the minimum bounding boxes of the predicted and ground truth bounding boxes, respectively; α represents the angle between the line connecting the center points of the predicted and ground truth bounding boxes and the horizontal direction of the image; β represents the angle between the line connecting the center points of the predicted and ground truth bounding boxes and the vertical direction of the image; b h This represents the height difference between the center point of the predicted bounding box and the center point of the actual bounding box; b w σ represents the width difference between the center points of the predicted and ground truth bounding boxes; σ represents the distance between the center points of the predicted and ground truth bounding boxes; w and h represent the width and height of the ground truth bounding box, respectively; w gt h gt These represent the width and height of the predicted bounding box, respectively; w w w h All are intermediate variables; θ represents a parameter that adjusts the degree of attention given to the shape loss of the total bounding box loss, and its value is 4.

[0067] This embodiment uses Focal-SIoU as the bounding box regression loss, incorporating the angle between the line connecting the center point of the predicted target box and the center point of the true target box and the horizontal or vertical direction into the loss function. It comprehensively considers four factors: shape, IoU, angle, and distance. It also takes into account the different contributions of predicted target boxes of different qualities to the loss, which can improve the accuracy of bolt bounding box localization. At the same time, it serves as the basis for subsequent bolt loosening judgment based on the bolt area point cloud, thus improving the accuracy of bolt loosening defect identification.

[0068] As a preferred implementation, before using the RANSAC plane segmentation algorithm, statistical filtering is performed on the point cloud of the bolt region obtained by mapping to remove outliers, and the filtered point cloud is uniformly downsampled by 4 times. The point cloud of the bolt region obtained in this way is used in the subsequent fitting algorithm. This method reduces the amount of computation without reducing the detection accuracy.

[0069] As a preferred implementation, the RANSAC plane segmentation algorithm is used. When segmenting and extracting the bolt's upper surface and mounting reference plane in the point cloud of the bolt region, the first segmentation extraction uses the RANSAC plane segmentation algorithm, and the second segmentation extraction uses the RANSAC plane segmentation algorithm with normal vector constraints. Figure 3 As shown;

[0070] The RANSAC plane segmentation algorithm with normal vector constraints described above is as follows: In the RANSAC plane segmentation algorithm, after determining the current iteration plane and before traversing the point cloud to calculate the distance from each point to the current iteration plane, the following judgment step is added:

[0071] Calculate the cosine of the normal vector of the current iteration plane and the normal vector of the plane extracted in the first segmentation. If the cosine value is less than the preset value, return to determine a new plane as the next iteration plane and repeat the judgment step. Otherwise, execute the subsequent steps of the RANSAC plane segmentation algorithm.

[0072] Specifically, when determining whether a bolt has loosened, the first plane extraction uses the RANSAC plane segmentation algorithm, and the second plane extraction uses the RANSAC plane segmentation algorithm with normal vector constraints. The processes for the two plane extraction methods are as follows: Figure 5 As shown.

[0073] During planar extraction, the formula for calculating the planar parameters (a, b, c, d) based on three coordinate points (x1, y1, z1), (x2, y2, z2), and (x3, y3, z3) is as follows:

[0074] a=(y2-y1)·(z3-z1)-(z2-z1)·(y3-y1)

[0075] b = (z2-z1)·(x3-x1)-(x2-x1)·(z3-z1)

[0076] c=(x2-x1)·(y3-y1)-(y2-y1)·(x3-x1)

[0077] Determine a point (x) in a point cloud k ,y k ,z kWhen determining whether a point is in a plane, the formula for calculating the distance from the point to the plane is as follows:

[0078] Based on the segmented bolt surface and the mounting reference plane, the distance can be calculated using the set of interior points of one plane and the four parameters (a, b, c, d) of the other plane. The calculated distance is an array with the number of elements equal to the number of data points in the interior point set. By averaging this array, the distance between the two planes can be obtained. Finally, by comparing the calculated plane distance with a set distance threshold, it can be determined whether the bolt has become loose.

[0079] The RANSAC plane segmentation algorithm with normal vector constraints is implemented as follows for plane segmentation:

[0080] S1. Randomly select three points (x1, y1, z1), (x2, y2, z2), and (x3, y3, z3) as the set of interior points in the unordered point cloud (i.e., the point cloud of the bolt region), and calculate the four parameters a, b, c, and d of a three-dimensional plane.

[0081] S2. Calculate the cosine value of the normal vector of the current iteration plane and the normal vector of the plane extracted in the first calculation. For example, if cos < 0.995, return to step S1 for the next iteration and increment the iteration number n by one; otherwise, proceed to the next step S3.

[0082] S3. Traverse the next data point (x, y, z) in the point cloud, calculate the distance h from the point to the current iteration plane. If h is less than the set threshold H, add the point to the inner point set; otherwise, skip it. Repeat this step until all data points (i.e., the point cloud of the bolt region) have been traversed.

[0083] S4. Determine if the number of points in the interior point set obtained in this iteration is greater than the optimal interior point set. If it is, update the interior point set. Continuously return to step S1 and loop until the set maximum number of iterations N is reached, at which point the iteration stops.

[0084] S5. Returns the set of interior points and the plane parameters (a, b, c, d).

[0085] Compared to the RANSAC plane segmentation algorithm with normal vector constraints, the RANSAC plane segmentation algorithm does not have an S2 step.

[0086] For example, the distance threshold H is set to 0.1 in both RANSAC plane segmentation algorithms, and the maximum number of iterations N is 50.

[0087] Example 2

[0088] A computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed by a processor, it controls the device where the storage medium is located to perform a bolt defect detection method based on image information and point cloud information as described above.

[0089] The relevant technical solutions are the same as in Embodiment 1, and will not be repeated here.

[0090] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A bolt defect detection method based on image information and point cloud information, characterized by, include: Collect inspection images and point cloud data of the inspection area containing the bolts to be inspected, wherein the inspection images and point cloud data are pixel-aligned; Based on the inspection images, a pre-trained target recognition model is used to regress and predict the target bounding box of the bolt to be inspected. If the prediction result indicates that there is no bolt to be inspected, the bolt defect detection is completed; if the prediction result indicates that there is a bolt to be inspected, the point cloud of the bolt region is mapped according to the target box obtained by regression, and the RANSAC plane segmentation algorithm is used to segment and extract the upper surface of the bolt and the mounting reference plane in the point cloud of the bolt region; the distance between the two planes is calculated and compared with the set threshold to determine whether the bolt has become loose, thus completing the bolt defect detection. The target recognition model is an improved YOLOv8, which is obtained through the following improvement methods: In the C2f structure of the backbone feature extraction network of YOLOv8, a coordinate attention mechanism (CA) is added. The CA is added to the output of the Bottleneck module in C2f to assign different weights to different channels and regions of the feature map output by the Bottleneck module. In the neck structure of YOLOv8, the progressive feature pyramid fusion structure AFPN is used. First, the CBS module is used to convolve the P3 and P4 feature maps output by the backbone feature extraction network to extract features. Then, two ASFF2 modules are used to adaptively fuse the convolutional feature maps. The two fused feature maps are then processed by the C2f module to obtain intermediate layer feature maps C3 and C4. Three ASFF3 modules are used to adaptively fuse the intermediate layer feature maps C3 and C4 with the P5 feature map output by the backbone feature extraction network and then process them by C2f to extract features. Finally, progressively fused feature maps P'3, P'4, and P'5 are obtained. The fused feature layers are then used for target bounding box regression and prediction. The improved YOLOv8 uses the Focal-SIoU bounding box loss function during training. The Focal-SIoU bounding box loss function is specifically as follows: In the formula, This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. Represents the angle loss term; Represents the shape loss term; , , All are intermediate variables; and These represent the x-coordinates of the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively. and These represent the ordinates of the predicted and actual target bounding box center points, respectively. and These represent the height and width of the minimum bounding boxes for the predicted and ground truth bounding boxes, respectively. This represents the angle between the line connecting the center point of the predicted bounding box and the center point of the ground truth bounding box and the horizontal direction of the image. This represents the angle between the line connecting the center point of the predicted bounding box and the center point of the ground truth bounding box and the vertical direction of the image. This represents the height difference between the center point of the predicted bounding box and the center point of the actual bounding box. This represents the width difference between the center point of the predicted bounding box and the center point of the actual bounding box. This represents the distance between the center point of the predicted bounding box and the center point of the ground truth bounding box. , These represent the width and height of the actual target bounding box, respectively. , These represent the width and height of the predicted bounding box, respectively. , All are intermediate variables; This indicates the degree of attention the Focal-SIoU bounding box loss function pays to shape loss.

2. The bolt defect detection method according to claim 1, characterized in that, The inspection images and point cloud data are acquired using a depth camera.

3. The bolt defect detection method according to claim 1, characterized in that, Before using the RANSAC plane segmentation algorithm, statistical filtering is performed on the point cloud of the bolt region obtained by mapping to remove outliers, and the filtered point cloud is uniformly downsampled by 4 times. The point cloud of the bolt region obtained in this way is used in the subsequent fitting algorithm.

4. The bolt defect detection method according to claim 1, characterized in that, When using the RANSAC plane segmentation algorithm to segment and extract the upper surface of the bolt and the mounting reference plane in the point cloud of the bolt region, the first segmentation extraction uses the RANSAC plane segmentation algorithm, and the second segmentation extraction uses the RANSAC plane segmentation algorithm with normal vector constraints. The RANSAC plane segmentation algorithm with normal vector constraints is as follows: In the RANSAC plane segmentation algorithm, after determining the current iteration plane and before traversing the point cloud to calculate the distance from each point to the current iteration plane, the following judgment step is added: Calculate the cosine of the normal vector of the current iteration plane and the normal vector of the plane extracted in the first segmentation. If the cosine value is less than the preset value, return to determine a new plane as the next iteration plane and repeat the judgment step. Otherwise, execute the subsequent steps of the RANSAC plane segmentation algorithm.

5. The bolt defect detection method according to claim 4, characterized in that, The preset value is 0.

995.

6. The bolt defect detection method according to claim 1, characterized in that, The method for calculating the distance between the two planes is as follows: Based on the bolt's upper surface obtained from the segmentation and the installation reference surface, the distance is calculated using the set of interior points of one plane and the four parameters of the other plane. The calculated distance is an array with the number of elements equal to the number of data points in the set of interior points. The distance between the two planes is obtained by averaging this array.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is run by a processor, it controls the device where the storage medium is located to perform a bolt defect detection method based on image information and point cloud information as described in any one of claims 1 to 6.

Images