A bolt defect intelligent detection system and method based on 2D-3D information fusion

Abstract

This application provides an intelligent bolt defect detection system and method based on 2D-3D information fusion, including hardware acquisition, data processing, data fusion, defect detection, and result output units. The method includes: simultaneously acquiring two-dimensional images and laser stripe images of the bolt; preprocessing the two-dimensional image and extracting features; extracting sub-pixel centers from the laser stripe image and reconstructing a three-dimensional point cloud to extract features; registering the two-dimensional and three-dimensional feature spaces; setting prior coefficients for two-dimensional and three-dimensional regions for different detection areas of the bolt; dynamically calculating fusion weights and generating fusion feature vectors based on the credibility of the two-dimensional image and the three-dimensional point cloud for each region; and completing the identification, classification, and parameter quantization of surface and geometric defects based on the fusion feature vectors. This application, through regionalized dynamic weighted fusion, can simultaneously achieve bolt surface defect identification and three-dimensional dimension quantization, and is applicable to bolt quality inspection in fields such as machinery manufacturing and aerospace.

Description

Technical Field

[0001] This application belongs to the field of bolt defect detection technology, specifically relating to an intelligent bolt defect detection system and method based on 2D-3D information fusion. Background Technology

[0002] Bolts, as core fasteners in industrial equipment, directly determine the operational safety of the equipment due to their structural integrity. Defects such as cracks and deformation can easily lead to equipment loosening, component failure, or even safety accidents. Currently, bolt defect detection technologies are mainly divided into two categories. One category is 2D vision-based detection methods. These methods acquire bolt surface images using industrial cameras and utilize image processing algorithms to identify surface defects such as cracks, scratches, and corrosion, offering advantages such as fast detection speed and relatively low cost. The other category is 3D point cloud-based detection methods. These methods acquire three-dimensional contour data of the bolt through laser scanning or structured light technology, enabling precise measurement of geometric parameters such as bolt diameter, length, and thread pitch, and exhibiting high sensitivity to dimensional deviations and deformation defects. Both of these technologies have achieved good application results in their respective preferred detection dimensions.

[0003] However, existing technologies still have the following core limitations: First, they rely on a single detection dimension, making it difficult to simultaneously address surface defect identification and three-dimensional size quantification. While 2D vision-based methods can identify surface cracks and corrosion, the lack of depth information prevents the quantification of key three-dimensional dimensional parameters such as bolt diameter and length, resulting in insufficient detection accuracy for defects requiring dimensional assessment, such as "deep cracks" and "minor deformations." 3D point cloud-based dimensional measurement methods, while capable of precise geometric parameter measurement, suffer from low sensitivity in identifying minute surface defects (especially cracks smaller than 0.5mm), with a false negative rate exceeding 15%. Both of these technologies suffer from the inability to simultaneously address both "defect identification" and "size quantification." Second, most technologies require offline inspection after removing bolts from equipment, posing safety risks to personnel under conditions such as high altitude and high temperature. Furthermore, reassembly can increase clearance and accelerate bolt wear. Third, there is an imbalance between efficiency and accuracy. Traditional manual inspection is inefficient, subjective, and has a high rate of missed detection. Existing automated technologies either have cumbersome data processing procedures that are time-consuming or lack effective fusion strategies that result in insufficient accuracy, thus failing to meet the needs of online real-time inspection. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this application provides an intelligent bolt defect detection system and method based on 2D-3D information fusion. The technical problem to be solved by this application is achieved through the following technical solution: A bolt defect intelligent detection system based on 2D-3D information fusion includes: The hardware acquisition unit is used to simultaneously acquire two-dimensional images and laser stripe images of the bolt to be inspected; The data processing unit is used to preprocess the two-dimensional image and extract its features, as well as to extract the center of the laser stripe image at the subpixel level and extract the three-dimensional point cloud features by combining the triangulation principle. The data fusion unit is used to spatially register the two-dimensional image features with the three-dimensional point cloud features to obtain spatially aligned two-dimensional image features and three-dimensional point cloud features; for different detection areas of the bolt to be detected, two-dimensional region prior coefficients and three-dimensional region prior coefficients are set for each, and the two-dimensional image confidence and three-dimensional point cloud confidence of each detection area are obtained; based on the two-dimensional region prior coefficients, the three-dimensional region prior coefficients, the two-dimensional image confidence and the three-dimensional point cloud confidence, combined with the spatially aligned two-dimensional image features and three-dimensional point cloud features, the fusion weight of each detection area is dynamically calculated, and the fusion feature vector of each detection area is generated; The defect detection unit is used to identify, classify, and quantize the parameters of surface defects and geometric defects based on the fused feature vector. The results output unit is used to generate structured inspection reports and visualize defect areas.

[0005] A method for intelligent bolt defect detection based on 2D-3D information fusion includes: S1: Simultaneously acquire two-dimensional images and laser stripe images of the bolt to be inspected; S2: Preprocess the two-dimensional image and extract its features, and extract the center of the laser stripe image at the subpixel level, and extract the three-dimensional point cloud features by combining the triangulation principle. S3: Spatial registration is performed between the two-dimensional image features and the three-dimensional point cloud features to obtain spatially aligned two-dimensional image features and three-dimensional point cloud features; S4: For different detection areas of the bolt to be detected, set two-dimensional region prior coefficients and three-dimensional region prior coefficients respectively, and obtain the two-dimensional image confidence and three-dimensional point cloud confidence of each detection area; based on the two-dimensional region prior coefficients, the three-dimensional region prior coefficients, the two-dimensional image confidence and the three-dimensional point cloud confidence, and combined with the spatially aligned two-dimensional image features and three-dimensional point cloud features, dynamically calculate the fusion weight of each detection area, and generate the fusion feature vector of each detection area; S5: Based on the fused feature vector, identify, classify and quantify the surface defects and geometric defects, and output the detection results.

[0006] Beneficial effects: 1. This application uses a hardware acquisition unit to simultaneously acquire two-dimensional images and laser stripe images of the bolt to be inspected. The data processing unit extracts two-dimensional surface texture and edge features, and extracts geometric size features after sub-pixel-level center extraction and triangulation to reconstruct the three-dimensional point cloud. The data fusion unit then performs spatial registration and feature fusion of the two dimensions, so that the final fused feature vector used for defect detection contains both rich surface texture information and accurate three-dimensional spatial information. This enables the simultaneous identification of bolt surface defects and quantification of three-dimensional dimensions, solving the limitation of single-dimensional detection technology that cannot simultaneously address defect identification and size quantification.

[0007] 2. This application configures differentiated two-dimensional and three-dimensional region prior coefficients for three different structural regions: bolt head, thread, and bolt shaft. Based on this, it further obtains the two-dimensional image reliability and three-dimensional point cloud reliability of each detection region, and calculates dynamic fusion weights based on the region prior coefficients and real-time data reliability. When drastic changes in illumination or metal reflection cause a decrease in two-dimensional image quality, the two-dimensional feature fusion weights automatically decrease, and the system relies more on three-dimensional geometric features. When laser stripe breaks or the point cloud is sparse, causing a decrease in three-dimensional data quality, the three-dimensional feature fusion weights automatically decrease, and the system relies more on two-dimensional surface features. This strategy abandons the traditional fixed weight mode and can adaptively adjust the contribution of 2D and 3D features under different working conditions, significantly improving the stability and reliability of detection in complex industrial scenarios.

[0008] 3. This application employs an improved YOLOv8 neural network in the defect detection unit. By embedding SE attention and CBAM attention mechanisms into the backbone network, the extraction capability of defect-related features is enhanced from the channel and spatial dimensions, respectively. Simultaneously, a multi-scale feature fusion structure combining the FPN feature pyramid network and the PANet path aggregation network enables the network to have good detection capabilities for bolt defects of different scales, especially enhancing the sensitivity to small target defects such as thread breakage and minor scratches. Furthermore, when the credibility of both the 2D image and the 3D point cloud of a certain detection area falls below the corresponding preset threshold, the system controller automatically adjusts the acquisition parameters and re-acquires data, forming a closed-loop detection process of acquisition, evaluation, feedback, and re-acquisition. This effectively avoids false detections and missed detections caused by data acquisition quality issues, ensuring the reliability of the detection results.

[0009] 4. This application utilizes an electric rotary platform to rotate the bolt to be inspected around its own axis, while a linear motion platform drives the sensor to move along the bolt's axial direction. The system controller generates a synchronous trigger signal based on the rotation angle and axial position signals, achieving full-angle, full-coverage acquisition of both two-dimensional and laser stripe images. This acquisition method can be completed while the bolt is clamped or in place, eliminating the need for offline inspection after disassembly. This reduces the operational risks for personnel in complex working conditions and is applicable to various inspection scenarios, including in-place, semi-in-place, and out-of-place inspections. Furthermore, the regionalized dynamic fusion strategy of this application does not rely on a specific defect classification network structure and can be extended to defect detection tasks of other mechanical parts by adjusting the inspection head category and partitioning inspection strategy, demonstrating good versatility and promotional value.

[0010] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0011] Figure 1 A schematic diagram of a bolt 2D vision and 3D point cloud synchronous acquisition device provided in this application; Figure 2 A schematic diagram of the overall architecture of a bolt defect detection system based on 2D-3D information fusion provided in this application; Figure 3 A schematic diagram of the data acquisition and preprocessing process for a detection method provided in this application; Figure 4 A flowchart of 3D point cloud reconstruction based on laser triangulation is provided for this application; Figure 5 A schematic diagram of a bolt defect detection network architecture based on improved YOLOv8 is provided for this application; Figure 6 This is a schematic diagram of a 2D-3D data fusion and regionalized defect detection process provided in this application.

[0012] in, Figure 1 The annotations in the attached figures are explained as follows: 1—Base; 2—Line laser generator; 3—First industrial camera; 4—Frame beam; 5—Second industrial camera; 6—Bolt to be tested; 7—Clamping assembly; 8—Electric rotating platform. Detailed Implementation

[0013] The present application will be described in further detail below with reference to specific embodiments, but the implementation of the present application is not limited thereto.

[0014] Example 1: Combination Figure 1 and Figure 2 , Figure 1This is a schematic diagram of the structure of the bolt two-dimensional vision and three-dimensional point cloud synchronous acquisition device in this application. Figure 2 This is a schematic diagram of the overall architecture of the bolt defect detection system based on 2D-3D information fusion, illustrating the composition of the hardware acquisition unit, data processing unit, data fusion unit, defect detection unit, and result output unit, as well as the data flow transmission paths between each unit. This embodiment provides a specific hardware configuration scheme for an intelligent bolt defect detection system based on 2D-3D information fusion. The synchronous acquisition device includes a base 1, a line laser generator 2, a first industrial camera 3, a frame beam 4, a second industrial camera 5, a bolt to be inspected 6, a clamping assembly 7, and an electric rotating platform 8. The clamping assembly 7 is mounted on the electric rotating platform 8 and is used to fix the bolt to be inspected 6 and drive it to rotate around its axis. The line laser generator 2 is used to project line structured light onto the surface of the bolt to be inspected 6. The first industrial camera 3 is used to acquire a two-dimensional image of the bolt to be inspected 6, and the second industrial camera 5 is used to acquire the laser stripe image formed by the line laser acting on the surface of the bolt to be inspected 6. The electric rotating platform 8 drives the bolt 6 to be inspected to rotate, and with the synchronous triggering of the first industrial camera 3, the second industrial camera 5 and the line laser generator 2, the two-dimensional visual image and three-dimensional point cloud data of the outer peripheral surface of the bolt 6 to be inspected are acquired.

[0015] The image acquisition module employs two 5-megapixel industrial cameras, one for 2D color imaging and the other for laser stripe acquisition. The laser projection module uses a line laser to project high-contrast laser stripes onto the bolt surface. The motion control module utilizes a high-precision linear motion platform with a stroke of at least 20cm and a repeatability of at least 0.01mm. The control module employs a high-performance processor and graphics processing unit to meet the real-time processing requirements of multimodal data. The system controller generates a synchronous trigger signal based on the rotation angle position signal of the electric rotary platform and the axial position signal of the linear motion platform, ensuring that the 2D image, laser stripe image, angle position, and axial position share the same timestamp index or the same acquisition code, thereby guaranteeing the spatial and temporal correspondence between 2D image features and 3D point cloud features.

[0016] Example 2: This application also provides a method for intelligent detection of bolt defects based on 2D-3D information fusion, including: S1: Simultaneously acquire two-dimensional images and laser stripe images of the bolt to be inspected; S2: Preprocess the two-dimensional image and extract its features, and extract the center of the laser stripe image at the subpixel level, and extract the three-dimensional point cloud features by combining the triangulation principle. S3: Spatial registration is performed between the two-dimensional image features and the three-dimensional point cloud features to obtain spatially aligned two-dimensional image features and three-dimensional point cloud features; S4: For different detection areas of the bolt to be detected, set two-dimensional region prior coefficients and three-dimensional region prior coefficients respectively, and obtain the two-dimensional image confidence and three-dimensional point cloud confidence of each detection area; based on the two-dimensional region prior coefficients, the three-dimensional region prior coefficients, the two-dimensional image confidence and the three-dimensional point cloud confidence, and combined with the spatially aligned two-dimensional image features and three-dimensional point cloud features, dynamically calculate the fusion weight of each detection area, and generate the fusion feature vector of each detection area; S5: Based on the fused feature vector, identify, classify and quantify the surface defects and geometric defects, and output the detection results.

[0017] Example 3: This embodiment details the system calibration process and the specific implementation method of S1 synchronous acquisition, such as... Figure 2 As shown, Figure 3 This is a schematic diagram of the data acquisition and preprocessing flow of the detection method in this application. The calibration process includes camera calibration, dual-target calibration, and laser plane calibration. A checkerboard calibration board with 49 marker points and a center-to-center distance of 12.5 mm is used. At least 15 images with different poses are acquired. The camera intrinsic parameter matrix is ​​calculated using the Zhang Zhengyou calibration method, and the reprojection error is controlled within 0.3 pixels. The rotation and translation parameters between the cameras are obtained through dual-target calibration. Laser plane calibration involves fitting a plane equation to multiple frames of images of the laser line at different positions, with a fitting residual of no more than 0.01 mm.

[0018] Synchronous data acquisition specifically includes: controlling the bolt to be inspected to rotate around its own axis with a preset angle step distance, and controlling the sensor to move along the axial direction of the bolt with a preset scanning step distance. At each rotation angle position and each axial position, a two-dimensional image and a laser stripe image are acquired, and the acquired two-dimensional image and laser stripe image are associated and stored with the corresponding rotation angle position and axial position. In one specific implementation, the linear slide moves the sensor along the bolt's axial direction at a speed of 5 mm / s, triggering a data acquisition every 0.3 mm of movement, simultaneously acquiring a 2D color image and a laser stripe image to ensure 360° full-angle coverage. The system controller reads the angle position signal of the rotation drive component and the axial position signal of the linear slide, triggering the industrial camera and line laser emission component to acquire data when a preset angle or preset axial step distance is reached. Each set of data is associated and stored in the form of a timestamp, angle position, axial position, two-dimensional image, and laser stripe image.

[0019] Example 4: Figure 4This is a flowchart illustrating the 3D point cloud reconstruction process based on laser triangulation in this application. This embodiment details the specific implementation process of extracting sub-pixel-level centers from laser stripe images and combining this with triangulation principles to extract 3D point cloud features. This process mainly includes hardware joint calibration, laser stripe sub-pixel center extraction, and multi-frame point cloud stitching reconstruction.

[0020] (1) Hardware joint calibration Before 3D reconstruction, the system needs to be pre-calibrated to establish a precise mapping relationship between 2D image pixels and 3D physical points. This includes camera calibration, light plane calibration, and motion platform calibration.

[0021] Camera calibration: The intrinsic parameter matrix and distortion coefficients of the industrial camera were obtained using Zhang Zhengyou's checkerboard calibration method. The intrinsic parameter matrix includes the focal length. , and optical center coordinates , The distortion coefficients include radial distortion. , , With tangential distortion , By using the camera's intrinsic and extrinsic parameter matrices, rigid body transformation relationships are constructed between the pixel coordinate system, image coordinate system, camera coordinate system, and world coordinate system.

[0022] Light plane calibration: The fan-shaped plane formed by the projection of the linear laser is the light plane. By changing the pose of the calibration plate in space, multiple images of the intersection of the laser stripes and the calibration plate are obtained. Using the calibrated camera extrinsics, the extracted laser line pixel coordinates are transformed to three-dimensional coordinates in the camera coordinate system. Then, the least squares method is used for plane fitting to obtain the mathematical equation of the light plane in the camera coordinate system:

[0023] Motion platform calibration: To obtain the complete 3D contour of the bolt across its entire circumference, a high-precision linear motion platform with motion-driven relative translation scanning is required. By extracting the translation matrix between adjacent frames during the scanning process, the average translation vector of each frame scan by the displacement stage is calculated. .

[0024] (2) Extraction of sub-pixel center lines of laser stripes To address the center extraction errors caused by ambient light interference in industrial settings, bolt surface reflections, and laser speckle, this application employs an improved Steger algorithm for high-precision centerline positioning. S2.1: First, the original laser stripe image is preprocessed with Gaussian filtering to eliminate random noise and make the light intensity distribution of the stripe cross section strictly approximate a Gaussian distribution.

[0025] S2.2: Calculate the second-order partial derivatives of each pixel in the image using the Hessian matrix. , , The eigenvalues ​​and eigenvectors of the Hessian matrix are calculated, and the eigenvector corresponding to the largest absolute eigenvalue is extracted as the normal direction of the light stripe.

[0026] S2.3: Under the determined normal direction, perform a Taylor series expansion on the gray-scale distribution function of the light stripe, and solve for the extreme points where the first derivative is zero, thereby obtaining the sub-pixel level coordinates of the laser stripe center. .

[0027] (3) 3D coordinate calculation and multi-frame point cloud stitching reconstruction Based on the extracted sub-pixel centerline and the aforementioned calibration parameters, the system performs the following calculations to generate a 3D point cloud of the bolt: S2.4: First, combine the camera intrinsic parameter matrix to determine the sub-pixel level center pixel coordinates. Convert to normalized image physical coordinates after distortion correction ,Right now:

[0028] Next, substitute the expression for the ray into the equation for the light plane. The physical depth value of the pixel in the camera coordinate system is calculated. :

[0029] This allows us to obtain the three-dimensional spatial coordinates of the point in the camera coordinate system:

[0030] S2.5: During the process of the linear slide scanning the bolt frame by frame at fixed intervals, the system uses the single-frame translation vector obtained from the motion platform calibration. , for the Global displacement compensation is performed on single-frame local point cloud data acquired through frame acquisition.

[0031] in It is the image index, with a value of By performing the aforementioned spatial alignment and stitching compensation on multiple frames of continuously scanned data, a high-precision 3D point cloud model containing the complete morphology and accurate three-dimensional dimensions of the bolt surface is finally reconstructed. Three-dimensional point cloud features are extracted from the complete three-dimensional point cloud, providing reliable high-dimensional data support for subsequent assessment of whether the bolt has geometric defects such as deformation, chipped teeth, or dimensional deviations.

[0032] Example 5: This embodiment details the specific implementation method of spatial registration of two-dimensional image features and three-dimensional point cloud features in S3.

[0033] S3.1: Obtain the camera intrinsic parameter matrix, distortion coefficients, and rotation matrix and translation vector between the camera and the world coordinate system. These parameters are obtained in advance through the system calibration process in Example 2.

[0034] S3.2: Based on the camera intrinsic parameter matrix, distortion coefficients, rotation matrix, and translation vector, establish a perspective projection model between points in the world coordinate system and points in the image pixel coordinate system. ,in As a scale factor, Image pixel coordinates, For the camera intrinsic parameter matrix, Let be a rotation matrix. It is a translation vector. Used as world coordinates.

[0035] S3.3: Select no fewer than four non-coplanar world coordinate system control points and their corresponding image pixel coordinate system projection points, and use the EPnP algorithm to solve for the optimal solutions of the rotation matrix and translation vector.

[0036] S3.4: Based on the rotation matrix and translation vector, project the 3D point cloud features onto the 2D image plane, or back-project the 2D image features onto the 3D space, so that the 2D image features and the 3D point cloud features are aligned in space, and obtain the spatially aligned 2D image features and 3D point cloud features.

[0037] The aforementioned data fusion unit consists of a spatial registration module and a feature fusion module. The spatial registration module establishes a coordinate transformation chain based on calibration parameters and uses the EPnP algorithm to achieve high-precision alignment of 2D-3D data; see Example 3 for a specific implementation. The feature fusion module, based on spatial registration, eliminates the dimensional differences between 2D and 3D features and achieves dynamic weighted fusion by combining data quality and regional characteristics; see Examples 4 and 5 for specific implementations.

[0038] Example 6: Figure 6This is a schematic diagram of the 2D-3D data fusion and regionalized defect detection process in this application. The system first calculates the diameter variation curve based on the contour changes of the 3D point cloud along the bolt's axial direction. When a sudden diameter change is detected, this location is used as a candidate boundary between the bolt head and the threaded area. When a periodic undulation in surface height along the axial direction is detected, this area is identified as the threaded area. For bolts of known specifications, the system can also call preset structural dimension parameters to correct the above boundaries, thereby obtaining the head area, threaded area, and threaded area. Therefore, the different detection areas of the bolt to be detected include the head area, threaded area, and threaded area, which are determined based on at least one of the bolt axial point cloud contour, diameter abrupt change location, periodic thread height change, or preset structural dimension parameters.

[0039] After completing the region segmentation, the system configures region prior coefficients for the two-dimensional and three-dimensional features of different regions. The region prior coefficients are configured differently based on the typical defect manifestations of each region. Head region: Common defects in this region include surface scratches, corrosion, and abnormal end-face textures, with anomalies primarily manifesting as abnormal two-dimensional surface texture features. Therefore, the two-dimensional region prior coefficients are... Set to a prior coefficient greater than that of the three-dimensional region The preferred value is , .

[0040] Threaded region: The main defects in this region are chipped teeth, wear, and pitch deviation, with the fault characteristics mainly manifested as abnormal three-dimensional geometric morphology. Therefore, the prior coefficients for this three-dimensional region are... Set to a prior coefficient greater than that of a two-dimensional region The preferred value is , .

[0041] Screw area: This area exhibits both surface scratches (two-dimensional texture abnormalities) and bending deformation (three-dimensional morphological abnormalities). A comprehensive assessment will be conducted... and Set the values ​​to be similar or exactly equal, with the preferred value being [value]. , .

[0042] As a preferred implementation method, the two-dimensional region prior coefficients and three-dimensional region prior coefficients satisfy .

[0043] Example 7: This embodiment details the complete implementation process of dynamically calculating the fusion weights of each detection region and generating fusion feature vectors in S4.

[0044] S4.1: For different detection areas of the bolt to be tested, set two-dimensional region prior coefficients and three-dimensional region prior coefficients respectively. For specific configuration methods, please refer to Example 5.

[0045] S4.2: Obtain two-dimensional image quality scores for different detection areas of the bolt to be inspected. 3D point cloud quality score The two-dimensional image quality score and the three-dimensional point cloud quality score were normalized into two-dimensional image confidence scores. and 3D point cloud credibility .in, This indicates the detection area of ​​the bolt to be inspected. .

[0046] Two-dimensional image quality rating The quality score is obtained based on at least one of image sharpness, contrast, exposure uniformity, and texture integrity. (3D point cloud quality score) It is obtained based on at least one of point cloud density, curvature continuity, laser stripe integrity, and outlier ratio. Specifically, It can be obtained by normalizing at least one of the following indicators: Laplacian sharpness, grayscale contrast, exposure uniformity, and texture integrity. It can be obtained by normalizing at least one of the following indicators: point cloud density per unit area, local curvature continuity, laser stripe continuity, and outlier ratio.

[0047] S4.3: Based on two-dimensional region prior coefficients 3D region prior coefficients 2D image credibility and 3D point cloud credibility Calculate the two-dimensional feature fusion weights for each detection region. 3D feature fusion weights The calculation formula is as follows:

[0048]

[0049] in, To prevent tiny positive numbers with a denominator of zero, it is recommended to take... .

[0050] S4.4: Weight fusion based on two-dimensional features 3D feature fusion weights For the two-dimensional image feature vectors of each detection region and 3D point cloud feature vectors Perform weighted summation to generate fused feature vectors for each detection region. :

[0051] In the above formula, This indicates the bolt detection area, with a value range of {head area, thread area, bolt area}. Indicates the first Two-dimensional image feature fusion weights for each detection region; Indicates the first Weighting of 3D point cloud features in each detection region; Indicates the first The credibility of the two-dimensional image of each detection area; Indicates the first The reliability of the 3D point cloud of each detection area; Indicates the first Two-dimensional image feature vectors of each detection region; Indicates the first The 3D point cloud feature vector of each detection region; Indicates the first The fused feature vector of each detection region; Indicates the first Two-dimensional region prior coefficients for each detection area; Indicates the first Three-dimensional region prior coefficients for each detection area; This represents a positive constant to prevent the denominator from being zero.

[0052] The aforementioned regionalized dynamic weight fusion strategy enables the surface defect detection in the head region, the geometric defect detection in the thread region, and the comprehensive defect detection in the screw region to obtain more suitable data sources, avoiding the over-reliance on low-quality modes under conditions of strong reflection, sparse point cloud, or partial occlusion, thereby improving detection stability and defect identification accuracy.

[0053] During dynamic adjustment, when the bolt surface reflects light, has abnormal exposure, or lacks texture, resulting in... When reduced, the weights of the two-dimensional feature fusion are... Automatic reduction means the system relies more on 3D geometric features. This is exacerbated by issues such as broken laser stripes, missing point clouds, or an increase in outliers. When reduced, the 3D feature fusion weights The system automatically lowers its load, relying more on two-dimensional surface features.

[0054] Explanation of adaptive fusion effect under working conditions: Taking thread area detection as an example, when the ambient lighting changes drastically and metal reflection is severe, the clarity of the two-dimensional image decreases significantly. Consequently, the two-dimensional feature weights decrease. The system automatically reduces noise and prioritizes extracting geometric features such as tooth height, tooth spacing deviation, and cross-sectional depth from the 3D point cloud for detection. However, it may encounter issues such as localized breaks in the laser stripes or point cloud gaps due to equipment malfunction. Reduce, 3D feature weights As the system descends synchronously, it shifts to analyzing the texture and edge features of two-dimensional images. This solution dynamically allocates weights by combining the inherent characteristics of the bolt area with the dual constraints of real-time data acquisition quality, abandoning the traditional fixed weight mode. This effectively solves the problems of false detection and missed detection caused by the deterioration of a single data modality, and significantly improves the overall stability and reliability of bolt defect detection under harsh conditions such as complex lighting, metal reflection, laser stripe fracture, and point cloud defects.

[0055] Example 8: This embodiment details the specific implementation method of the system performing a re-inspection when the credibility of the two-dimensional image and the credibility of the three-dimensional point cloud of a certain detection area are both lower than the corresponding preset threshold.

[0056] In S4.2, the confidence level of the two-dimensional image of each detection region is obtained. and 3D point cloud credibility The system then determines whether the credibility of the 2D image and the 3D point cloud for each detection region is lower than the corresponding preset threshold. A 2D credibility threshold of 0.6 and a 3D credibility threshold of 0.5 are recommended.

[0057] When the reliability of both the 2D image and the 3D point cloud of a certain detection area is lower than the corresponding preset threshold, the system enters a re-inspection mode, adjusts at least one of the following: industrial camera exposure parameters, line laser power, scanning step size, or rotation angle, and re-acquires 2D images and laser stripe images for the detection area. Specifically, the system can control the industrial camera to adjust the exposure time or gain, control the line laser generator to adjust the laser power, and adjust the scanning step size or rotation angle.

[0058] After completing the re-inspection data acquisition, the system re-executes S1 to S4 on the newly acquired data until the reliability of the detection area meets the threshold requirements. Through the above method, this application performs feedback re-acquisition under abnormal conditions such as strong reflection, partial occlusion, broken laser stripes, or missing point clouds, forming a closed-loop detection process of acquisition, evaluation, feedback, and re-acquisition, thereby improving the stability and repeatability of the detection results.

[0059] Example 9: Figure 6This is a schematic diagram of the improved bolt defect detection based on YOLOv8 in this application. This embodiment details the specific implementation method in S5 for identifying, classifying, and quantizing surface and geometric defects based on fused feature vectors. The improved YOLOv8 network architecture includes an input layer, an improved backbone network, an FPN+PANet neck network, and a multi-scale detection head. The improved backbone network embeds the SE attention mechanism module and the CBAM attention mechanism module sequentially on top of the CSPDarknet53 base layer, then outputs to the neck network via a multi-scale feature extraction layer, and finally outputs the defect identification result by the multi-scale detection head. This embodiment uses an improved YOLOv8 neural network, specifically optimizing the YOLOv8 network architecture to meet the multi-scale defect detection requirements of different regions of the bolt, achieving accurate identification of eight types of defects: bolt scratches, corrosion, deformation, chipped teeth, wear, bending, misalignment, and loosening.

[0060] I. Input Layer: Image Preprocessing and Input Standardization The input to this method is a preprocessed 2D bolt image, with a uniform input image size. The dimensions are defined as follows: The spatial dimension consists of an image width and height of 640 pixels each, adapting to the standard input scale of the YOLOv8 series models, achieving an optimal balance between detection accuracy and inference efficiency, while also meeting the downsampling calculation requirements for multi-scale feature extraction. The channel dimension is a 3-channel RGB color image, fully preserving the color, texture, contour, and other visual features of bolt surface defects, providing sufficient original information for subsequent feature extraction. The input source is the image preprocessing module, which performs image size normalization, noise reduction, contrast enhancement, distortion correction, and other operations to eliminate interference from the shooting environment and ensure that the input data meets the standardized requirements for model training and inference.

[0061] II. Improved YOLOv8 core architecture This embodiment optimizes the native YOLOv8 network into three modules to construct a dedicated network architecture adapted for bolt defect detection: (a) Improve the backbone network Based on the CSPDarknet53 architecture, an attention mechanism and a multi-scale feature extraction layer are embedded to enhance the defect feature extraction capability. The specific hierarchical structure is as follows: 1. CSPDarknet53 base layer: It retains the core structure of the cross-stage local network, realizes feature reuse through residual connections, efficiently extracts the basic visual features of bolt images, and controls the number of model parameters and computational cost to ensure inference efficiency.

[0062] 2. SE Attention Mechanism Module (Channel Attention Enhancement): The SE module is inserted during the key feature extraction stage of the backbone network to enhance channel features through three steps: compression, activation, and recalibration.

[0063] Compression: Compression of the input feature map (size: , For height, For width, Perform global average pooling for the number of channels, reducing the spatial dimension. Compress to The generated length is The channel description vectors are aggregated to gather global spatial information. This process is expressed by the following formula:

[0064] in, For the first Global average pooling results for each channel. For the input feature map, the first Each channel is in The pixel value of the location.

[0065] Excitation: The channel description vector is nonlinearly transformed through two fully connected layers, first reducing the number of channels to a smaller dimension. (Compression ratio) To reduce computational load, the number of channels is restored to its original value. A 0-1 channel weight vector is then generated via sigmoid activation to learn the importance of different channels for defect detection. This process is expressed by the following formula:

[0066] in, It is the ReLU activation function. It is the Sigmoid activation function. and These are the learning parameters.

[0067] Recalibration: The channel weight vector is multiplied channel by channel of the original feature map, assigning high weights to defect-related key channels and low weights to redundant background channels. This strengthens channel features valuable for defect detection and improves the model's feature representation capability. This process is expressed by the following formula:

[0068] in For the first The weighting coefficients of each channel, This is the recalibrated feature map.

[0069] 3. CBAM Attention Mechanism Module (Spatial + Channel Dual Attention Enhancement): This module, connected in series with the CBAM module, supplements spatial attention to the SE channel attention, accurately capturing spatial details of defects. CBAM comprises two sub-modules: Channel attention submodule: Simultaneously employs global average pooling and global max pooling to aggregate spatial information. The results of the two pooling are input into a multilayer perceptron with shared weights, and after fusion, channel weights are generated via Sigmoid to complete channel dimension feature enhancement.

[0070] Spatial Attention Submodule: Performs global average pooling and global max pooling along the channel dimension on the feature map output by the channel attention module, resulting in two... The spatial feature maps. The two feature maps are concatenated to form... The fusion characteristics, through The convolutional layer extracts spatial features, which are then processed by a sigmoid function to generate a spatial weight map of 0-1. This weight map is multiplied pixel-by-pixel with the input feature map to focus on the spatial region where the bolt defect is located. The calculation formula for the spatial attention submodule is as follows:

[0071] in, Indicates the kernel size as The convolution operation [;] represents the feature concatenation operation along the channel dimension. Through this operation, the system can significantly enhance the spatial response to minor geometric defects such as thread breakage.

[0072] 4. Multi-scale feature extraction layer: Design a multi-path downsampling structure to generate... , , Feature maps of different resolutions correspond to small, medium, and large-scale bolt defects, respectively, providing multi-dimensional feature inputs for neck network feature fusion.

[0073] The sub-modules of the backbone network are connected as follows: the CSPDarknet53 base layer, the SE attention mechanism module, the CBAM attention mechanism module, and the multi-scale feature extraction layer are connected in sequence.

[0074] (II) FPN+PANet Neck Network A bidirectional fusion of multi-scale features is achieved using an FPN (Feature Pyramid Network) and PANet (Path Aggregation Network) architecture. The FPN feature pyramid fuses high-level semantic features with low-level detail features via a top-down path, enhancing the semantic information of small-scale defects. The PANet path aggregation supplements feature transmission via a bottom-up path, strengthening the spatial localization capability of low-level features. For small defects such as bolt thread chipping and minor scratches, a dedicated small-target feature enhancement path is designed to improve the expressive power of small-target features. Efficient fusion of features at different resolutions is achieved through cross-scale connections, balancing the detection needs of both large-scale deformation defects and small-scale minute defects.

[0075] The sub-module connections of the neck network are as follows: multi-scale feature map output by the backbone network, top-down feature fusion of FPN, bottom-up feature aggregation of PANet, small target feature enhancement module, and cross-scale feature fusion output to the multi-scale detection head.

[0076] (III) Optimization of Multi-Scale Detection Head To address the dimensional differences in bolt defects, the inspection head structure was optimized: a large-scale inspection head was designed for... Feature maps are used to detect large-sized defects such as bolt bending and severe deformation. The medium-scale inspection head corresponds to... Feature maps are used to detect medium-sized defects such as bolt corrosion and head scratches. Small-scale inspection heads (dedicated branches) are also available. Feature maps are used to detect small-sized defects such as thread breakage and minor wear. In addition, an adaptive anchor frame generator is implemented, which adaptively generates anchor frames that fit the defect size based on the statistical distribution of the bolt defect dataset, replacing the original fixed anchor frames and improving defect location accuracy.

[0077] III. Definition of Loss Function and Optimization Objective To achieve high-precision location and classification of bolt defects, this application employs a multi-task loss function for network training:

[0078] Among them, regression loss The CIoU (Complete IoU) loss method is adopted, taking into account the overlapping area of ​​the bounding boxes, the distance between the center points, and the aspect ratio, to improve the regression accuracy of the bolt defect bounding boxes. (Classification Loss) Binary cross-entropy loss is used to determine the category attributes of eight types of defects. Distributed focal length loss is also employed. To address the issue of ambiguous defect boundaries, the robustness of bounding box localization in complex industrial contexts is improved by optimizing the probability distribution function.

[0079] IV. Targeted Inspection of Bolt Zones To address the differences in defect types across different structural regions of bolts, a zoned inspection strategy was designed to achieve accurate defect classification and identification. Head area inspection: For bolt heads, focus on inspecting three types of defects: scratches, corrosion, and deformation, and identify surface damage and shape distortion of the head.

[0080] Thread area inspection: For bolt thread sections, the focus is on detecting three types of defects: chipped threads, wear, and corrosion, and identifying thread profile defects, surface wear, and corrosion damage.

[0081] Screw area inspection: For bolt and screw sections, focus on detecting two types of defects: bending and scratches. At the same time, combine visual features to identify two types of installation defects: misalignment and loosening, ultimately covering 8 types of target defects.

[0082] V. Post-processing and Result Optimization The post-processing module optimizes the detection results to improve output accuracy. Non-maximum suppression, with a threshold of 0.5, eliminates duplicate detection boxes for the same defect, retaining the best detection result. Confidence filtering, with a confidence threshold of 0.7, filters out low-confidence false detections, improving detection reliability. Defect bounding box optimization fine-tunes the coordinates of the detection boxes, optimizing defect location accuracy and ensuring a closer fit to the actual defect contour.

[0083] VI. Output Layer: Output of Defect Identification Results The output layer generates standardized bolt defect identification results, containing three core types of information: eight defect categories, namely scratches, corrosion, deformation, chipped teeth, wear, bending, misalignment, and loosening; defect location and confidence level, with bounding box coordinates marking the defect's position in the image, and outputting a defect confidence level of 0 to 1 to characterize the reliability of the detection results; and a 2D image quality score, which scores the clarity and integrity of the input image, providing a basis for evaluating the credibility of the detection results.

[0084] This application selected commonly used bolt specifications in industrial scenarios and used its system for inspection. Results showed that: in terms of defect identification, it can reliably identify surface cracks and minor corrosion defects larger than 0.5mm with high accuracy. Regarding dimensional quantification, the measurement error of bolt diameter, length, and other parameters can be controlled within 0.02mm under specific test conditions, meeting industrial precision requirements. In terms of inspection efficiency, the inspection time for a single bolt is no more than 2 seconds, significantly higher than traditional manual inspection efficiency, making it suitable for online real-time inspection scenarios.

[0085] To enhance the detection capability of minute chipped and worn defects in bolt threads, this application makes targeted improvements to the YOLOv8 network. A CBAM module is embedded before the SPPF layer of the backbone network, using a channel attention mechanism to filter out feature frequencies sensitive to thread structure, and a spatial attention mechanism to lock the edge region of the thread. This improvement effectively solves the problem that subtle geometric distortions of the thread are easily masked by background noise under complex lighting conditions, reducing the missed detection rate of extremely small target defects by more than 15%.

[0086] The improved architecture of this application can be implemented using the PyTorch framework. The training dataset uses a self-made bolt defect dataset, which includes bolt images under different lighting, angles, and working conditions, covering full-scene samples of 8 types of defects. The embedding positions of the SE and CBAM modules can be adjusted according to actual needs, and they are preferentially embedded after the CSP module of the backbone network to achieve the best feature enhancement effect. This method can be extended to defect detection of other mechanical parts, and only the detection head category and partition detection strategy need to be adjusted for adaptation, demonstrating good versatility.

[0087] It is worth noting that the terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0088] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of this application and should not be construed as limiting the specific implementation of this application to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of this application, and all such modifications or substitutions should be considered within the scope of protection of this application.

Claims

1. A bolt defect intelligent detection system based on 2D-3D information fusion, characterized in that, include: The hardware acquisition unit is used to simultaneously acquire two-dimensional images and laser stripe images of the bolt to be inspected; The data processing unit is used to preprocess the two-dimensional image and extract its features, as well as to extract the center of the laser stripe image at the subpixel level and extract the three-dimensional point cloud features by combining the triangulation principle. The data fusion unit is used to spatially register the two-dimensional image features with the three-dimensional point cloud features to obtain spatially aligned two-dimensional image features and three-dimensional point cloud features. For different detection areas of the bolt to be inspected, two-dimensional region prior coefficients and three-dimensional region prior coefficients are set for them respectively, and the two-dimensional image confidence and three-dimensional point cloud confidence of each detection area are obtained. Based on the two-dimensional region prior coefficients, the three-dimensional region prior coefficients, the two-dimensional image confidence and the three-dimensional point cloud confidence, combined with the spatially aligned two-dimensional image features and three-dimensional point cloud features, the fusion weight of each detection area is dynamically calculated, and the fusion feature vector of each detection area is generated. The defect detection unit is used to identify, classify, and quantize the parameters of surface defects and geometric defects based on the fused feature vector. The results output unit is used to generate structured inspection reports and visualize defect areas.

2. The intelligent bolt defect detection system based on 2D-3D information fusion according to claim 1, characterized in that, The hardware acquisition unit includes: a first industrial camera, a second industrial camera, a line laser generator, a clamping assembly, an electric rotary platform, a linear motion platform, and a system controller; The clamping assembly is disposed on the electric rotating platform for clamping the bolt to be tested. The electric rotating platform is used to drive the bolt to be tested to rotate around its own axis. The linear motion platform is used to drive the sensor to move along the axial direction of the bolt to be tested. The line laser generator is used to project laser stripes onto the surface of the bolt to be tested. The first industrial camera is used to acquire a two-dimensional image of the bolt to be inspected, and the second industrial camera is used to acquire a laser stripe image on the surface of the bolt to be inspected. The system controller is connected to the first industrial camera, the second industrial camera, the line laser generator, the electric rotary platform, and the linear motion platform respectively, and is used to control the coordinated operation of each component.

3. A method for intelligent detection of bolt defects based on 2D-3D information fusion, characterized in that, include: S1: Simultaneously acquire two-dimensional images and laser stripe images of the bolt to be inspected; S2: Preprocess the two-dimensional image and extract its features, and extract the center of the laser stripe image at the subpixel level, and extract the three-dimensional point cloud features by combining the triangulation principle. S3: Spatial registration is performed between the two-dimensional image features and the three-dimensional point cloud features to obtain spatially aligned two-dimensional image features and three-dimensional point cloud features; S4: For different detection areas of the bolt to be detected, set two-dimensional region prior coefficients and three-dimensional region prior coefficients respectively, and obtain the two-dimensional image confidence and three-dimensional point cloud confidence of each detection area; based on the two-dimensional region prior coefficients, the three-dimensional region prior coefficients, the two-dimensional image confidence and the three-dimensional point cloud confidence, and combined with the spatially aligned two-dimensional image features and three-dimensional point cloud features, dynamically calculate the fusion weight of each detection area, and generate the fusion feature vector of each detection area; S5: Based on the fused feature vector, identify, classify and quantify the surface defects and geometric defects, and output the detection results.

4. The intelligent bolt defect detection method based on 2D-3D information fusion according to claim 3, characterized in that, S1 includes: The bolt to be tested is controlled to rotate around its own axis with a preset angle step distance, and the sensor is controlled to move along the axis of the bolt to be tested with a preset scanning step distance. At each rotation angle position and each axial position, a two-dimensional image and a laser stripe image are acquired, and the acquired two-dimensional image and the laser stripe image are associated and stored with the corresponding rotation angle position and axial position.

5. The intelligent bolt defect detection method based on 2D-3D information fusion according to claim 3, characterized in that, In S2, sub-pixel-level center extraction is performed on the laser stripe image, and three-dimensional point cloud features are extracted using the triangulation principle, including: S2.1, Perform Gaussian smoothing preprocessing on the laser stripe image to obtain the processed laser stripe image; S2.2 Calculate the Hessian matrix of each pixel in the processed laser stripe image, solve for the eigenvalues ​​and eigenvectors of the Hessian matrix, and then take the direction of the eigenvector corresponding to the largest absolute eigenvalue as the normal direction of the laser stripe. S2.3, perform a Taylor series expansion on the grayscale distribution function of the light stripe in the normal direction, solve for the extreme points where the first derivative is zero, and obtain the coordinates of the center of the laser stripe at the sub-pixel level. S2.4, Based on the preset camera intrinsic parameters, the coordinates of the center of the laser stripe are converted into normalized image physical coordinates, and the depth value of the point in the camera coordinate system is solved by combining the preset laser plane equation to obtain the three-dimensional spatial coordinates of the point. S2.5, S2.1 to S2.4 are repeated for multiple frames of laser stripe images continuously scanned along the axial direction to obtain the local three-dimensional point cloud corresponding to each frame. The local three-dimensional point cloud of each frame is displacement compensated and stitched according to the preset scanning translation vector to obtain the complete three-dimensional point cloud of the bolt to be detected. The three-dimensional point cloud features are extracted from the complete three-dimensional point cloud.

6. The intelligent bolt defect detection method based on 2D-3D information fusion according to claim 3, characterized in that, S3 include: S3.1: Obtain the camera intrinsic parameter matrix, distortion coefficients, and rotation matrix and translation vector between the camera and the world coordinate system; S3.2: Based on the camera intrinsic parameter matrix, the distortion coefficients, the rotation matrix, and the translation vector, establish a perspective projection model between world coordinate system points and image pixel coordinate system points; S3.3: Select no fewer than four non-coplanar world coordinate system control points and their corresponding image pixel coordinate system projection points, and use the EPnP algorithm to solve for the optimal solution of the rotation matrix and the translation vector; S3.4: Based on the rotation matrix and the translation vector, project the three-dimensional point cloud features onto the two-dimensional image plane, or back-project the two-dimensional image features onto the three-dimensional space, so that the two-dimensional image features and the three-dimensional point cloud features are aligned in space, thereby obtaining the spatially aligned two-dimensional image features and three-dimensional point cloud features.

7. The intelligent bolt defect detection method based on 2D-3D information fusion according to claim 3, characterized in that, The different detection areas of the bolt to be tested include the head area, the thread area, and the screw area. The head area, the thread area, and the screw area are determined based on at least one of the bolt axial point cloud profile, diameter abrupt change position, periodic height change of thread, or preset structural dimension parameters. The two-dimensional prior coefficient of the head region is greater than the three-dimensional prior coefficient, the three-dimensional prior coefficient of the thread region is greater than the two-dimensional prior coefficient, and the two-dimensional prior coefficient of the screw region is equal to or close to the three-dimensional prior coefficient.

8. The intelligent bolt defect detection method based on 2D-3D information fusion according to claim 3, characterized in that, S4 include: S4.1, for different detection areas of the bolt to be tested, set two-dimensional region prior coefficients and three-dimensional region prior coefficients respectively; S4.2, obtain the two-dimensional image quality score and three-dimensional point cloud quality score of different detection areas of the bolt to be detected, and normalize the two-dimensional image quality score and three-dimensional point cloud quality score into two-dimensional image confidence and three-dimensional point cloud confidence respectively; S4.3, based on the prior coefficients of the two-dimensional region, the prior coefficients of the three-dimensional region, the confidence level of the two-dimensional image, and the confidence level of the three-dimensional point cloud, calculate the two-dimensional feature fusion weights and three-dimensional feature fusion weights of each detection region; S4.4, Based on the two-dimensional feature fusion weights and the three-dimensional feature fusion weights, the two-dimensional image feature vectors and three-dimensional point cloud feature vectors of each detection region are weighted and summed to generate the fused feature vectors of each detection region.

9. The intelligent bolt defect detection method based on 2D-3D information fusion according to claim 8, characterized in that, Following S4.2, the intelligent bolt defect detection method based on 2D-3D information fusion further includes: when the credibility of the two-dimensional image and the credibility of the three-dimensional point cloud of a certain detection area are both lower than the corresponding preset threshold, adjusting at least one of the industrial camera exposure parameters, line laser power, scanning step distance or rotation angle, and re-acquiring the two-dimensional image and laser stripe image of the detection area.

10. The intelligent bolt defect detection method based on 2D-3D information fusion according to claim 3, characterized in that, The identification and classification of surface defects and geometric defects described in S5 adopts an improved YOLOv8 neural network. The improved YOLOv8 neural network integrates the SE attention mechanism module and the CBAM attention mechanism module in the backbone network part, and adopts a structure that combines the FPN feature pyramid network and the PANet path aggregation network in the neck network part.

Images