A precision fastener surface flaw detection method based on visual recognition and three-dimensional model comparison

Abstract

The application discloses a precision fastener surface flaw detection method and system based on visual identification and three-dimensional model comparison, which comprises the following steps: synchronously collecting surface point cloud data and two-dimensional gray images of the precision fastener, and registering the point cloud data with a standard CAD model; according to the included angle between the scanning direction of the surface element and the normal line and the adjacent tooth top shielding condition, the model surface is divided into a three-dimensional reliable area and a three-dimensional unreliable area. The deviation of the point cloud and the model surface in the three-dimensional reliable area is calculated, and the flaw is determined according to the tolerance limit value; according to the registration pose and the camera parameters, a synthetic reference image is generated for the three-dimensional unreliable area, and the synthetic reference image is compared with the corresponding two-dimensional gray image in pixels to identify the flaw. The two kinds of detection results are uniformly mapped to the standard CAD model surface to generate a flaw distribution map. The method can reduce the influence of thread shielding and large incident angle on three-dimensional detection, and improve the detection coverage rate and flaw positioning accuracy of the complex thread surface.

Description

Technical Field

[0001] This application relates to the field of precision fastener surface quality inspection technology, specifically to a method and system for detecting surface defects in precision fasteners based on visual recognition and 3D model comparison. Background Technology

[0002] Precision fasteners are widely used in aerospace, automotive, rail transportation, and energy equipment. Cracks, pits, burrs, indentations, or poor forming on their threaded surfaces can affect assembly accuracy, preload performance, and service reliability. Current inspection methods mainly include manual visual inspection, two-dimensional machine vision, and three-dimensional topography measurement. Manual inspection is affected by personnel experience, fatigue levels, and observation angles, making it difficult to meet the requirements for consistently detecting minute defects in mass production. Two-dimensional machine vision inspection is faster, but surface reflections, lighting fluctuations, and variations in brightness on the threaded surface can easily create false defects, and two-dimensional images cannot directly reflect the depth and geometric deviations of defects.

[0003] Three-dimensional inspection methods such as structured light scanning and laser profilometry can acquire surface point clouds and determine morphological deviations by comparing them with standard models. However, thread profiles have continuous helical grooves, and the tooth crest can obscure the lower part of the tooth lateral surface and the tooth root. When the scanning direction makes a large angle with the local surface normal, the accuracy of fringe imaging and depth calculation will also decrease. For highly reflective metal surfaces, point cloud defects, outliers, or local reconstruction distortions may also occur. Increasing the scanning angle or the number of inspection stations can improve coverage, but it will increase the cost of equipment, calibration, and inspection cycle time. Existing two-dimensional and three-dimensional inspection results are usually processed separately, lacking a mechanism to select the inspection method based on the reliability of local measurements and uniformly map it to the standard model surface, making it difficult to balance the inspection coverage, positioning consistency, and inspection efficiency of complex thread areas. Summary of the Invention

[0004] In view of this, this disclosure proposes a method and system for detecting surface defects in precision fasteners based on visual recognition and 3D model comparison.

[0005] According to one aspect of this disclosure, a method for detecting surface defects in precision fasteners based on visual recognition and 3D model comparison is provided, including:

[0006] Three-dimensional scanning is performed on the precision fastener to obtain surface point cloud data, and two-dimensional grayscale images are acquired simultaneously. The point cloud data is rigidly registered with the standard CAD model of the precision fastener to obtain a transformation matrix.

[0007] Based on the thread geometry parameters of the CAD model, calculate the angle between the scanning direction and the normal of each facet element, perform ray projection along the scanning direction to determine whether the facet elements in the thread region are occluded by adjacent tooth crests, and mark each facet element as a three-dimensional reliable region or a three-dimensional unreliable region based on the angle and occlusion determination results.

[0008] For the point cloud data within the three-dimensional reliable region, calculate the deviation from the surface of the CAD model, and determine the defect based on whether the deviation exceeds the tolerance limit;

[0009] Based on the transformation matrix and pre-calibrated camera parameters, perspective projection and grayscale rendering are performed on the facets of the three-dimensional unreliable region in the CAD model to generate a synthetic reference image;

[0010] The region corresponding to the three-dimensional unreliable area in the two-dimensional grayscale image is compared pixel by pixel with the synthetic reference image, and the defects are identified based on the difference distribution.

[0011] The defect detection results of the three-dimensional reliable region and the three-dimensional unreliable region are merged into the surface coordinate system of the CAD model to generate a defect distribution map.

[0012] In one possible implementation, marking each surface element as a three-dimensional reliable region or a three-dimensional unreliable region includes: marking unobstructed surface elements with an included angle less than an included angle threshold as three-dimensional reliable regions, and marking obstructed surface elements or surface elements with an included angle not less than the included angle threshold as three-dimensional unreliable regions; the included angle threshold is determined by calibration test: point clouds are collected at different incident angles on a standard planar reference part of the same material as the fastener being tested, the root mean square error of the point cloud distance at each incident angle is calculated, and the incident angle corresponding to the first time the root mean square error exceeds the detection accuracy requirement is taken as the included angle threshold.

[0013] In one possible implementation, the ray projection includes: emitting a ray from the center of a face element along the scanning direction toward one side of the scanner, and detecting whether the ray intersects with a face element in an adjacent tooth crest region of the CAD model; the search range for the intersection detection is limited to face elements within ±1.5 times the pitch from the center of the current face element along the fastener axis direction.

[0014] In one possible implementation, the grayscale rendering employs a Lambertian reflection model, calculating the expected grayscale value of each pixel based on the cosine relationship between the normal direction of each surface element and the pre-calibrated light source direction; the light source direction and reference illumination intensity are obtained by acquiring images of a reference object with known geometry and fitting the Lambertian reflection model parameters using the least squares method.

[0015] In one possible implementation, before the pixel-by-pixel comparison, local mean normalization processing is performed on the two-dimensional grayscale image and the synthetic reference image respectively: a local window is taken with each pixel as the center, the grayscale mean and grayscale standard deviation within the window are calculated, and the grayscale value is obtained by subtracting the grayscale mean from the original grayscale value and dividing by the grayscale standard deviation; the pixel-by-pixel comparison is performed based on the normalized grayscale value.

[0016] In one possible implementation, identifying defects based on the difference distribution includes: calculating the pixel-by-pixel absolute difference between the two-dimensional grayscale image and the synthetic reference image at corresponding pixels in the three-dimensional unreliable region; calculating the mean and standard deviation of the absolute difference; using the sum of the product of the mean, a threshold coefficient, and the standard deviation as a defect judgment threshold; and marking pixels whose absolute difference exceeds the defect judgment threshold as candidate defect pixels.

[0017] In one possible implementation, the half-angle vector of each pixel is calculated based on the normal direction, light source direction, and camera direction of the CAD model surface element. When the cosine value of the surface element normal and the half-angle vector exceeds the cosine value corresponding to the specular reflection angle threshold, the pixel is marked as a specular reflection risk area. The increased threshold coefficient is used to determine the candidate defective pixels in the specular reflection risk area. The specular reflection angle threshold is determined by the angle distribution between the normal and the half-angle vector corresponding to the actual occurrence of a bright spot on the sample fastener.

[0018] In one possible implementation, connected component labeling is performed on the candidate defective pixels, spatially adjacent candidate defective pixels are grouped into the same connected component, connected components with an area smaller than a minimum area threshold are filtered out, and the remaining connected components are used as the final detected defective regions; the minimum area threshold is determined based on the square of the ratio of the minimum detectable defect size to the actual size corresponding to a single pixel.

[0019] In one possible implementation, the rigid body registration includes coarse registration and fine registration: coarse registration determines the initial pose based on the macroscopic geometric features of the fastener, and fine registration uses an iterative nearest-point algorithm to minimize the distance error between the point cloud data and the surface of the CAD model; during the registration process, the starting end features of the thread helix are extracted from the point cloud data and aligned with the starting position of the thread in the CAD model to lock the axial rotational degree of freedom.

[0020] According to another aspect of this disclosure, a precision fastener surface defect detection system based on visual recognition and 3D model comparison is provided, comprising:

[0021] The data acquisition and registration module is used to perform three-dimensional scanning of the precision fastener to obtain surface point cloud data and simultaneously acquire two-dimensional grayscale images; the point cloud data is rigidly registered with the standard CAD model of the precision fastener to obtain a transformation matrix.

[0022] The reliability partitioning module is used to calculate the angle between the scanning direction and the normal of each facet element according to the thread geometry parameters of the CAD model, perform ray projection on the facet elements in the thread area along the scanning direction to determine whether they are occluded by adjacent tooth crests, and mark each facet element as a three-dimensional reliable area or a three-dimensional unreliable area according to the angle and occlusion determination results.

[0023] The three-dimensional defect detection module is used to calculate the deviation between the point cloud data in the three-dimensional reliable area and the surface of the CAD model, and to determine the defect based on whether the deviation exceeds the tolerance limit.

[0024] The reference image generation module is used to perform perspective projection and grayscale rendering on the surface elements of the three-dimensional unreliable area in the CAD model according to the transformation matrix and the pre-calibrated camera parameters to generate a synthetic reference image.

[0025] The two-dimensional defect detection module is used to compare the region corresponding to the three-dimensional unreliable area in the two-dimensional grayscale image with the synthetic reference image pixel by pixel, and identify defects based on the difference distribution.

[0026] The result fusion module is used to merge the defect detection results of the three-dimensional reliable region and the three-dimensional unreliable region into the CAD model surface coordinate system to generate a defect distribution map.

[0027] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing a computer program executable by the processor; wherein the processor is configured to execute the computer program in the memory to implement the method.

[0028] According to another aspect of this disclosure, a computer-readable storage medium is provided that enables the method to be implemented when an executable computer program in the storage medium is executed by a processor.

[0029] This application utilizes the thread geometry parameters contained in a standard CAD model, combined with the scanning direction, surface element normals, and the occlusion relationship between adjacent tooth crests, to regionally determine the reliability of 3D measurement data of fastener surfaces. Based on this, 3D deviation detection and 2D image comparison are configured in corresponding surface areas. This allows areas with high point cloud quality to accurately identify protrusions, depressions, and other morphological anomalies according to dimensional tolerances. Simultaneously, areas affected by occlusion or large incident angles, such as the thread root and lower part of the thread flank, are detected using a synthetic reference image matched to the actual pose. This avoids missed detections caused by directly using distorted point clouds and eliminates the need to compensate for blind spots by adding scanning stations and angles. Through local grayscale normalization and specular reflection risk correction, non-defect differences between the actual image and the synthetic reference image are suppressed. Furthermore, by mapping both types of detection results to the CAD model surface coordinate system, the location and extent of cross-regional defects can be continuously expressed. This improves the detection coverage, defect location accuracy, and result consistency of complex thread surfaces under controlled equipment input and detection cycle time.

[0030] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0031] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0032] Figure 1 The flowchart illustrates a method for detecting surface defects in precision fasteners based on visual recognition and 3D model comparison according to an embodiment of the present disclosure.

[0033] Figure 2 A flowchart illustrating the identification of a three-dimensional reliable region according to an embodiment of the present disclosure is shown.

[0034] Figure 3 A flowchart illustrating the process of generating a composite reference image according to an embodiment of this disclosure is shown.

[0035] Figure 4 A flowchart illustrating the process of identifying defects based on the difference distribution according to an embodiment of the present disclosure is shown.

[0036] Figure 5 This diagram illustrates the structure of a precision fastener surface defect detection system based on visual recognition and 3D model comparison according to an embodiment of the present disclosure. Detailed Implementation

[0037] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0038] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0039] According to embodiments of this disclosure, this method is applicable to the detection of surface defects in precision threaded fasteners in fields such as aerospace, automotive, and energy. The objects being tested include high-strength bolts, precision screws, and special studs, among other metal threaded fasteners, with thread profiles conforming to national or industry standards. The detection system consists of a structured light 3D scanner, an industrial camera, and a computing unit, and pre-stores a standard CAD model corresponding to the specifications of the fastener being tested. The measurement accuracy of the structured light scanner should be better than 1 / 10 of the critical dimensional tolerance of the fastener being tested; for example, when the tolerance band is 50 micrometers, the scanning accuracy should not be less than 5 micrometers. The relative pose between the industrial camera and the scanner is pre-calibrated and fixed using, for example, the Zhang Zhengyou calibration method. The lighting conditions remain stable during the detection process, and the direction and intensity of the light source are obtained through pre-calibration.

[0040] The implementation process of the method described in this application will be described in detail below with reference to specific embodiments. It should be noted that this embodiment is only used to explain this application and is not intended to limit the scope of protection of this application. Conventional adjustments or substitutions of each step by those skilled in the art without departing from the concept of this application should be included in the scope of protection of this application.

[0041] Figure 1 A flowchart illustrating a method for detecting surface defects in precision fasteners based on visual recognition and 3D model comparison, according to an embodiment of this disclosure, is shown. Figure 1 As shown, the method may include:

[0042] In step S11, the precision fastener is 3D scanned to obtain surface point cloud data, and a 2D grayscale image is acquired simultaneously; the point cloud data is rigidly registered with the standard CAD model of the precision fastener to obtain a transformation matrix.

[0043] According to an embodiment of this disclosure, after the fastener to be tested is fixed at the scanning station, a structured light scanner projects a coded stripe pattern onto the surface of the fastener and acquires deformed stripe images. Point cloud data of the fastener surface is generated through phase calculation and triangulation. Within the same scanning cycle, an industrial camera synchronously acquires two-dimensional grayscale images of the fastener under the same lighting conditions. The two types of data share a unified time and spatial reference, providing corresponding three-dimensional geometric information and two-dimensional appearance information for subsequent comparison.

[0044] After point cloud data acquisition, rigid body registration is performed between the data and a standard CAD model corresponding to the measured fastener to determine the rotation and translation relationships between their coordinate systems. Specifically, the registration process is divided into two steps: coarse registration and fine registration. In the coarse registration stage, the initial pose is determined through feature matching based on macroscopic geometric features such as the fastener head outer diameter, shank diameter, and total length. In the fine registration stage, the Iterative Closest Point (ICP) algorithm is used to optimize the initial pose. The basic principle of the ICP algorithm is to search for the nearest corresponding point on the CAD model surface for each point in the point cloud in each iteration, construct a point-to-surface distance error function, update the transformation parameters by minimizing this error function, iterate repeatedly until convergence, and finally output the transformation matrix. . It is a 4×4 homogeneous transformation matrix, containing a 3×3 rotation matrix and a 3×1 translation vector.

[0045] For example, the point cloud data after coarse registration ,in For the number of points in the point cloud, each point Three-dimensional coordinate vectors; and triangular mesh surfaces of standard CAD models. In each iteration, for each point... ,exist Search for the nearest point Construct a point-to-plane distance error function ,in Let be a rotation matrix. It is a translation vector. for The normal vector of the surface element. The Levenberg-Marquardt algorithm is used to solve for minimizing... of and The iteration termination condition is that the change in the root mean square error between two consecutive iterations is less than... The value should be in millimeters, or the number of iterations should reach 50. The output is a 4×4 homogeneous transformation matrix. Its form is .

[0046] Optionally, threaded fasteners exhibit approximately rotational symmetry about their axis, which may cause ambiguity in registration regarding the axial rotation angle. This ambiguity is resolved by extracting the starting end features of the thread helix from the point cloud data and aligning them with the thread's starting position in the standard CAD model, thus locking the axial rotational degree of freedom. After registration, each measurement point in the point cloud can be accessed via... Mapped to the corresponding surface position in the standard CAD model coordinate system.

[0047] In step S12, based on the thread geometry parameters of the CAD model, the angle between the scanning direction and the normal of each facet is calculated. Ray projection is performed on the facets in the thread region along the scanning direction to determine whether they are occluded by adjacent tooth crests. Based on the angle and the occlusion determination result, each facet is marked as a three-dimensional reliable region or a three-dimensional unreliable region.

[0048] The threaded area of ​​precision fasteners has a helical groove structure. When viewed from a specific scanning direction, the crests of adjacent tooth profiles can geometrically obstruct the root area, preventing structured light from fully illuminating the root surface. Simultaneously, in areas where the angle between the surface normal and the scanning direction is too large, the fringe projection is blurred, the phase calculation accuracy is low, and the point cloud quality is significantly degraded. These two factors lead to unreliable 3D data for the thread root and the lower part of the thread lateral surface. In traditional inspection schemes, these areas are either completely skipped, resulting in missed detections of critical parts, or rely on hardware solutions that increase the scanning angle to improve coverage, which is costly and time-consuming. This embodiment pre-assesses the reliability of the 3D data for each region using the precise thread geometry information of a standard CAD model after registration, and then allocates different inspection strategies accordingly, thereby covering the inspection blind spots without increasing hardware.

[0049] Please see Figure 2 , Figure 2 A flowchart illustrating the identification process of a three-dimensional reliable region according to an embodiment of this disclosure is shown, such as... Figure 2 As shown, in step S1201, the angle between the scanning direction and the normal is calculated. In this embodiment, the surface of the standard CAD model is represented by a triangular mesh, with each triangular facet constituting a face element. The unit vector of the outward normal direction at the center point of each face element is calculated. Based on the scanner's position in the scanning coordinate system and the center diameter of the surface element... The coordinates after inverse transformation are used to calculate the direction from the center of the facet cell to the scanner and then normalized to obtain the unit vector of the scanning direction. The angle between the two is:

[0050]

[0051] in and All are unit direction vectors, and the dot product result is a scalar. The unit is radians. Angle Reflecting the geometric conditions when structured light is incident on this surface element: The larger the value, the blurrier the fringe projection, and the greater the 3D measurement error.

[0052] Among them, an angle threshold is required in the face element labeling. To distinguish between reliable and unreliable. The calibration test determined that: a standard planar reference part of the same material as the fastener being tested should be used. Point clouds should be collected at different incident angles under the same scanning conditions. The root mean square error of the distance between the point cloud and the reference plane should be calculated for each angle. The incident angle at which the root mean square error first exceeds the detection accuracy requirement should be recorded as follows: As an example, when the required detection accuracy is 10 micrometers, if the root mean square error in the calibration test first exceeds 10 micrometers at an incident angle of approximately 75°, then... Taking 75°, which is approximately 1.31 radians. In practical applications... The typical value range is between 70° and 80°, and the specific value depends on the optical characteristics of the scanner used and the reflective characteristics of the surface of the workpiece being measured.

[0053] In step S1202, it is determined whether a surface element is occluded by adjacent tooth crests based on ray projection. This is done solely based on the included angle. This is insufficient to fully determine the reliability of the data for the threaded region cells; it is also necessary to determine whether the cells are geometrically occluded by adjacent tooth crests. Occlusion determination is performed using a ray casting method: from the cell center along the scanning direction... A ray is emitted towards one side of the scanner, and it is detected whether the ray intersects with a facet in the adjacent tooth crest region of the standard CAD model. If the ray intersects with a facet and the intersection point is located between the center of the starting facet and the scanner, the starting facet is determined to be occluded.

[0054] The intersection detection of rays and triangular facets employs the Möller-Trumbore algorithm, a classic algorithm in computer graphics for finding intersections between rays and triangles. Starting from the ray origin... Direction vector and the three vertices of the triangular face Using the input as input, the system calculates the simultaneous solution of the ray parametric equation and the triangle centroid coordinate equation to determine whether the ray passes through the interior of the triangle and provides the intersection point position parameters.

[0055] For example, the input is the ray origin. (i.e., coordinates of the center of the surface element), unit vector of the ray direction (i.e., scanning direction) ), and the three vertices of the triangular face. Algorithm for solving ray parametric equations ,in For ray parameters, These are the barycenter coordinate parameters. This is the third barycentric coordinate component. If and , , If they intersect, then determine that they are intersecting and output the position of the intersection point. and parameters Value; otherwise, they are considered non-intersecting. When the intersection point is located between the center of the facet and the scanner (i.e., ... If the distance from the center of the face element to the scanner is less than the distance from the center of the face element, the face element is determined to be occluded.

[0056] Optionally, to control computational load, the search range for intersection detection is limited to ±1.5 times the pitch from the center of the current facet along the fastener axis. Surface elements within this range are excluded because surface elements outside this range cannot geometrically constitute occlusion. For example, for an M10×1.5 bolt, The search range is 1.5 mm, which is the set of facets within 2.25 mm above and below the current facet's axial position. Optionally, the triangular facets of the threaded region of the standard CAD model can be pre-organized into a bounding box hierarchy (BVH) to accelerate raycasting calculations.

[0057] In step S1203, the three-dimensional reliable region and the three-dimensional unreliable region are marked based on the combined results of the included angle and occlusion determination. Combining the above two determinations, a reliability label is assigned to each surface element of the standard CAD model. Specifically, the labeling rule is that a surface element is unoccluded and the included angle... Less than the included angle threshold The time marker is designated as a three-dimensional reliable region; the surface element is occluded or has an angle. Not less than the included angle threshold The time marker is designated as a three-dimensional unreliable region.

[0058] After marking, the surface of the standard CAD model forms two complementary regions: the three-dimensional reliable region mainly covers the head, shank, and thread crest and the lateral surface near the crest; the three-dimensional unreliable region is mainly concentrated at the thread root and the lower part of the lateral surface away from the crest. Taking an M10×1.5 standard hex bolt as an example, when the scanning direction is perpendicular to the bolt axis, approximately 40% to 60% of the face elements in each thread root area are marked as three-dimensional unreliable regions. The proportion is related to the relative relationship between the scanning direction and the helix initiation angle.

[0059] For precision fasteners of the same specification at the same scanning station, the marking results are only related to the fastener specifications and scanning direction, and are not related to the individual parts being tested. Therefore, after the initial calculation is completed, the marking results are stored and can be directly reused for subsequent testing of fasteners of the same specification.

[0060] In step S13, the deviation between the point cloud data in the three-dimensional reliable area and the surface of the CAD model is calculated, and the defect is determined based on whether the deviation exceeds the tolerance limit.

[0061] Specifically, deviation detection is performed on the surface area marked as a three-dimensional reliable region. For each measurement point in the registered point cloud data falling within the three-dimensional reliable region, the signed distance to the nearest facet on the standard CAD model surface is calculated. A positive value indicates that the measurement point is located outside the facet normal (convex), and a negative value indicates that it is located inside the normal (concave). The deviation values ​​of each measurement point are summarized into a deviation distribution according to their corresponding positions on the standard CAD model. When the absolute value of the deviation exceeds the tolerance limit corresponding to the region, it is marked as a candidate defect area. The tolerance limit is specified by the product drawings or inspection standards, and this disclosure does not impose any limitations on it.

[0062] In step S14, based on the transformation matrix and pre-calibrated camera parameters, perspective projection and grayscale rendering are performed on the surface elements of the three-dimensional unreliable region in the CAD model to generate a synthetic reference image.

[0063] The point cloud data quality of the three-dimensional unreliable region is insufficient to support deviation analysis. Therefore, this embodiment utilizes two-dimensional images for defect detection. Two-dimensional detection requires a reference standard to determine which regions in the image contain anomalies. This embodiment uses a standard CAD model as the source for generating the reference image, i.e., based on the transformation matrix obtained from registration. Using pre-calibrated camera parameters, starting from the surface elements of the unreliable 3D region in the standard CAD model, a grayscale image reflecting the ideal appearance of the region under defect-free conditions is generated through graphics rendering; this is the synthetic reference image. Since the generation of the synthetic reference image depends on the precise pose provided by the registration results and the ideal geometry provided by the standard CAD model, both must work together to produce a reference that matches the actual viewpoint.

[0064] Please see Figure 3 , Figure 3 A flowchart illustrating the process of generating a composite reference image according to an embodiment of this disclosure is shown, such as... Figure 3 As shown, in step S1401, the perspective projection of the three-dimensional unreliable region surface elements is performed. All surface elements marked as three-dimensional unreliable regions are extracted from the standard CAD model, and a transformation matrix is ​​used... inverse matrix Transform the vertex coordinates of the face elements from the standard CAD model coordinate system to the scan coordinate system; then, based on the camera extrinsic parameters, i.e., the rotation matrix... Translation vector The vertex coordinates are further transformed to the camera coordinate system; finally, the camera intrinsic parameter matrix is ​​used. Perform perspective projection and calculate the pixel coordinates of each vertex on the image plane. , and All results were obtained through calibration during the installation of the testing system. It includes parameters such as focal length and principal point coordinates.

[0065] The three vertices of each facet are projected onto the image plane to form a triangular region. Rasterization is used to perform color calculations on each pixel within the triangle. During projection, a depth buffer is used to handle the occlusion relationships between facets, retaining only the facets visible from the camera's perspective.

[0066] In step S1402, grayscale rendering is performed based on the Lambertian reflection model. After rasterization, the expected grayscale value is calculated for each visible pixel. In one example, grayscale rendering uses the Lambertian reflection model:

[0067]

[0068] in To synthesize the reference image at pixel location The grayscale value at that location is an integer ranging from 0 to 255. Used as a reference scalar for light intensity; For pixels The unit outward normal vector of the corresponding surface element; Let be the unit direction vector pointing from the surface element to the light source. All the above vectors are unit vectors, and their dot product is summed. The result of the operation is always a scalar, multiplying by a scalar The resulting grayscale values ​​have consistent dimensions. The operation truncates negative values ​​to 0, indicating that the backlit side does not receive direct sunlight.

[0069] Where parameters and The image is obtained through illumination calibration. Specifically, under the same illumination conditions, images of a reference object with a known geometric shape (e.g., a standard sphere or standard cylinder) are acquired. Using the known normal directions of each surface point of the reference object and the actual captured grayscale values, the Lambertian reflection model described above is fitted using the least squares method. The solution that minimizes the sum of squared fitting residuals is then obtained. and Specifically, grayscale values ​​of multiple regions with different orientations on the surface of the reference part are collected. and corresponding normal Substitute the known quantities into the model expression to construct an overdetermined system of equations, and solve it using the least squares method. After calibration... and It will be used for all subsequent tests.

[0070] The synthesized reference image is only valid within the pixel range corresponding to the unreliable 3D region; the remaining pixels are marked as invalid regions and do not participate in subsequent comparisons. Optionally, a Gaussian smoothing filter is applied to the synthesized reference image to simulate the imaging blur characteristics of an actual camera. The standard deviation of the Gaussian kernel can be selected between 0.5 and 1.5 pixels, determined by comparing the spectral characteristics of the synthesized image and the actual image in the same region on the reference image.

[0071] In step S15, the region corresponding to the three-dimensional unreliable area in the two-dimensional grayscale image is compared pixel by pixel with the synthesized reference image, and the defects are identified based on the difference distribution.

[0072] After the synthetic reference image is generated, the actual acquired 2D grayscale image is compared pixel by pixel with the synthetic reference image within the corresponding pixel range of the 3D unreliable region. Defects are identified by analyzing the image differences. The synthetic reference image represents the ideal appearance of the region under defect-free conditions. In the actual image, local morphological deviations caused by surface defects, such as cracks, pits, burrs, or poor forming, will manifest as local grayscale differences during the comparison.

[0073] Please see Figure 4 , Figure 4 A flowchart illustrating the process of identifying defects based on difference distribution according to an embodiment of the present disclosure is shown, such as... Figure 4 As shown, in step S1501, local mean normalization is performed. There may be an overall brightness deviation between the actual image and the synthesized reference image, caused by factors such as fluctuations in light source power and differences between the surface reflectivity of fasteners and the Lambertian model. Before comparison, the influence of this deviation needs to be eliminated; local mean normalization is performed on both the two-dimensional grayscale image and the synthesized reference image.

[0074] Specifically, for each pixel position in the image ,by Take the side length as the center. A square local window is used to calculate the grayscale mean within the window. and grayscale standard deviation The normalization formula is:

[0075]

[0076] in The grayscale value before normalization. This represents the average grayscale value of a local window. The standard deviation of grayscale in a local window. To prevent division by zero of extremely small positive numbers (e.g., 1.0). This is the normalized grayscale value. Normalization eliminates brightness gradients within the window range while preserving local grayscale abrupt changes caused by imperfections.

[0077] The local window side length The selected value should be larger than the typical size of the expected defect area but smaller than the projection range of a single curved surface area on the image. Alternatively, it can be determined based on the number of projected pixels of the pitch on the image. Take the pitch as 0.3 to 0.5 times the number of projected pixels. For example, a pitch of 1.5 mm corresponds to approximately 150 pixels in an image. Take 45 to 75 pixels.

[0078] The above normalization is performed on both the 2D grayscale image and the synthetic reference image to obtain the normalized 2D grayscale image. and the normalized synthetic reference image .

[0079] In step S1502, the specular reflection risk area is marked. In addition to the diffuse reflection component, the surface of the metal fastener also has a specular reflection component. When the normal direction of the surface element happens to make the specular reflection direction of the incident light close to the camera's viewing direction, a bright spot appears in this area in the actual image, resulting in a significant deviation from the predicted grayscale value of the Lambertian model. This deviation is not caused by a defect, and if left untreated, it will be misjudged as a defect.

[0080] Specifically, the determination of specular reflection risk areas is accomplished using the geometric relationships between the surface normals of standard CAD models, the light source direction, and the camera direction. For each surface element in the 3D unreliable region projected onto the image, a half-angle vector, i.e., the unit vector of the light source direction, is calculated. The unit direction vector from the center of the surface element to the optical center of the camera. Vector and normalized result:

[0081]

[0082] and All are unit direction vectors. This is the normalized unit vector. If the surface element normal... with half-angle vector The dot product (i.e., the cosine of the angle between the two) exceeds the cosine value corresponding to the mirror reflection angle threshold. If so, the pixel is marked as a specular reflection risk area.

[0083] Specular reflection angle threshold The determination was made through experimental calibration. Images were acquired on the smooth planar area of ​​the sample fastener under the same illumination conditions, and the number of surface elements corresponding to the pixels that actually showed bright spots was statistically analyzed. The value distribution is used, and the angle corresponding to the 95th quantile of the distribution is taken as... Metal fastener surface The typical value range is 5° to 15°.

[0084] In step S1503, adaptive thresholding based on the difference distribution is performed. The normalized two-dimensional grayscale image is then calculated. With the normalized synthetic reference image absolute difference per pixel between:

[0085]

[0086] and All are dimensionless grayscale values ​​after normalization, and the absolute difference is... It is also a dimensionless scalar.

[0087] The mean of the absolute differences of all valid pixels corresponding to the three-dimensional unreliable region. and standard deviation Based on this, a defect judgment threshold is set:

[0088]

[0089] in For threshold coefficient, , and All are dimensionless quantities. It is also a dimensionless quantity. The determination is based on the normality deviation criterion, that is, taking This corresponds to a 99.7% confidence level under a normal distribution, meaning that pixels with differences exceeding this threshold have a probability of less than 0.3% appearing in normal products. The typical value range is 2.5 to 4.0. It is selected during the initial deployment of the detection system through comparative tests on a set of known qualified products and known defective products to achieve a target balance between false alarm rate and false negative rate.

[0090] For pixels marked as specular reflection risk areas in the aforementioned steps, an increased threshold coefficient is applied. ,in This is the specular reflection compensation amount. The method for determining this is to statistically analyze the distribution of absolute differences in the specular reflection areas of the sample fasteners during the calibration phase, and then... Start from 0 and gradually increase until the difference in the region exceeds [a certain value]. The increment required to reduce the pixel ratio to below 0.3% is... In practical applications The typical value ranges from 1.0 to 2.0.

[0091] For pixels in non-specular reflection risk areas, the absolute difference Exceed Pixels marked as candidate defects; for pixels in specular reflection risk areas, the absolute difference exceeds [a certain threshold]. The marked pixels are candidate defective pixels.

[0092] In step S1504, connected component analysis and final defect determination are performed. Specifically, eight-neighbor connected component labeling is performed on candidate defect pixels, grouping spatially adjacent candidate defect pixels into the same connected component, and the area (in pixels) of each connected component is calculated. A defect is considered defective if its area is less than a minimum area threshold. Connected regions are considered isolated outliers caused by measurement noise and are filtered out.

[0093] in The method for determining this is as follows: let the minimum actual size of the defect that needs to be detected be... As specified in the inspection standards, the unit is micrometers; the actual size corresponding to a single pixel is... The value is obtained from camera calibration and is expressed in micrometers per pixel. Pick and The square of the ratio, that is .by micrometers Taking micrometers per pixel as an example, It has 36 pixels. The area reaches The connected regions are retained as the final detected defective regions.

[0094] Optionally, shape features are extracted from the preserved connected regions to classify the defect types. For example, the aspect ratio of the connected regions is calculated. Slender regions with an aspect ratio greater than 5 usually correspond to cracks, while compact regions with an aspect ratio close to 1 usually correspond to pits or burrs.

[0095] In step S16, the defect detection results of the three-dimensional reliable region and the three-dimensional unreliable region are merged into the CAD model surface coordinate system to generate a defect distribution map.

[0096] Specifically, candidate defect areas in the 3D reliable region are directly recorded using deviation values ​​and their positions on the standard CAD model surface; defect areas detected in the 3D unreliable region are located to their corresponding positions on the standard CAD model surface via inverse mapping of image pixel coordinates to the camera projection model. Each defect record includes the defect's position coordinates on the standard CAD model, its functional region, and the source of detection.

[0097] For example, during the rasterization process, the corresponding 3D polygon index is recorded for each rendered pixel. And the coordinates of the sampled 3D points on this surface element (in the CAD model coordinate system). Generate a synthetic reference image. At the same time, a corresponding face element index map is generated. Each pixel value stores its corresponding face index number; and a depth map. This stores the distance (in millimeters) from the 3D point corresponding to the pixel to the camera's optical center. During subsequent inverse mapping, candidate defect pixels detected in the 2D image are processed by... Find the element to which it belongs, and obtain its precise three-dimensional position on the surface of the CAD model by interpolating the coordinates of its three vertices.

[0098] During merging, if a defective region detected in the 3D reliable region and a defective region detected in the 3D unreliable region are adjacent on the surface of the standard CAD model and the boundary distance is less than a preset distance threshold, such as 2 facet sizes, then the two will be merged into the same defect record. The location and range will be the union of the two, so as to avoid the same defect being split into two independent records due to crossing the region boundary.

[0099] Therefore, by utilizing the thread geometry information of the standard CAD model to predict the reliability of the 3D data for each surface region, and based on the reliability marking results, region-level allocation is performed between the two detection modes of 3D deviation analysis and 2D image comparison, enabling detection coverage of traditional 3D scanning blind areas such as thread roots to be obtained based on synthetic reference images. The synthetic reference image is generated by rendering the standard CAD model according to the actual registration pose and camera parameters, and its geometric correspondence with the measured part is clear. The detection results of the 3D reliable area and the 3D unreliable area are merged using the surface coordinates of the standard CAD model as a unified benchmark to form a complete defect distribution map.

[0100] Figure 5 A block diagram is shown of a precision fastener surface defect detection system based on visual recognition and 3D model comparison, according to an embodiment of the present disclosure. Figure 5 As shown, the system includes:

[0101] The data acquisition and registration module 501 is used to perform three-dimensional scanning of the precision fastener to obtain surface point cloud data and simultaneously acquire two-dimensional grayscale images; and to perform rigid body registration of the point cloud data with the standard CAD model of the precision fastener to obtain a transformation matrix.

[0102] The reliability partitioning module 502 is used to calculate the angle between the scanning direction and the normal of each face element according to the thread geometry parameters of the CAD model, perform ray projection on the face elements in the thread area along the scanning direction to determine whether they are blocked by adjacent tooth crests, and mark each face element as a three-dimensional reliable area or a three-dimensional unreliable area according to the angle and the occlusion determination result.

[0103] The three-dimensional defect detection module 503 is used to calculate the deviation between the point cloud data in the three-dimensional reliable area and the surface of the CAD model, and to determine the defect based on whether the deviation exceeds the tolerance limit.

[0104] The reference image generation module 504 is used to perform perspective projection and grayscale rendering on the surface elements of the three-dimensional unreliable area in the CAD model according to the transformation matrix and the pre-calibrated camera parameters to generate a synthetic reference image.

[0105] The two-dimensional defect detection module 505 is used to compare the region corresponding to the three-dimensional unreliable area in the two-dimensional grayscale image with the synthetic reference image pixel by pixel, and identify defects based on the difference distribution.

[0106] The result fusion module 506 is used to merge the defect detection results of the three-dimensional reliable region and the three-dimensional unreliable region into the CAD model surface coordinate system to generate a defect distribution map.

[0107] In some embodiments, the system provided in this disclosure may have functions or include modules that can be used to execute the methods described in the above method embodiments. The specific implementation of these methods can be referred to the description in the above method embodiments, and for the sake of brevity, they will not be repeated here.

[0108] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0109] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for detecting surface defects in precision fasteners based on visual recognition and 3D model comparison, characterized in that, include: Three-dimensional scanning of precision fasteners is used to obtain surface point cloud data, and two-dimensional grayscale images are acquired simultaneously. The point cloud data is rigidly registered with the standard CAD model of the precision fastener to obtain the transformation matrix. Based on the thread geometry parameters of the CAD model, calculate the angle between the scanning direction and the normal of each facet element, perform ray projection along the scanning direction to determine whether the facet elements in the thread region are occluded by adjacent tooth crests, and mark each facet element as a three-dimensional reliable region or a three-dimensional unreliable region based on the angle and occlusion determination results. For the point cloud data within the three-dimensional reliable region, calculate the deviation from the surface of the CAD model, and determine the defect based on whether the deviation exceeds the tolerance limit; Based on the transformation matrix and pre-calibrated camera parameters, perspective projection and grayscale rendering are performed on the facets of the three-dimensional unreliable region in the CAD model to generate a synthetic reference image; The region corresponding to the three-dimensional unreliable area in the two-dimensional grayscale image is compared pixel by pixel with the synthetic reference image, and the defects are identified based on the difference distribution. The defect detection results of the three-dimensional reliable region and the three-dimensional unreliable region are merged into the coordinate system of the CAD model surface to generate a defect distribution map. The step of marking each surface element as a three-dimensional reliable region or a three-dimensional unreliable region includes: marking unobstructed surface elements with an included angle less than the included angle threshold as three-dimensional reliable regions, and marking obstructed surface elements or surface elements with an included angle not less than the included angle threshold as three-dimensional unreliable regions; the included angle threshold is determined by calibration test: point clouds are collected at different incident angles on a standard planar reference part of the same material as the fastener being tested, the root mean square error of the point cloud distance at each incident angle is calculated, and the incident angle corresponding to the first time the root mean square error exceeds the detection accuracy requirement is taken as the included angle threshold; The ray projection includes: emitting a ray from the center of the face element along the scanning direction toward one side of the scanner, and detecting whether the ray intersects with the face elements of the adjacent tooth crest region in the CAD model; the search range for the intersection detection is limited to face elements within ±1.5 times the pitch from the center of the current face element along the fastener axis direction; The rigid body registration includes coarse registration and fine registration: coarse registration determines the initial pose based on the macroscopic geometric features of the fastener, and fine registration uses an iterative nearest-point algorithm to minimize the distance error between the point cloud data and the surface of the CAD model; during the registration process, the starting end features of the thread helix are extracted from the point cloud data and aligned with the starting position of the thread in the CAD model to lock the axial rotational degree of freedom.

2. The method according to claim 1, characterized in that, The grayscale rendering uses the Lambertian reflection model, and the expected grayscale value of each pixel is calculated based on the cosine relationship between the normal direction of each surface element and the pre-calibrated light source direction. The light source direction and reference illumination intensity are obtained by acquiring images of a reference object with known geometric shape and fitting the Lambertian reflection model parameters using the least squares method.

3. The method according to claim 1, characterized in that, Before the pixel-by-pixel comparison, local mean normalization processing is performed on the two-dimensional grayscale image and the synthetic reference image respectively: a local window is taken with each pixel as the center, the grayscale mean and grayscale standard deviation within the window are calculated, and the grayscale mean is subtracted from the original grayscale value and then divided by the grayscale standard deviation to obtain the normalized grayscale value; the pixel-by-pixel comparison is performed based on the normalized grayscale value.

4. The method according to claim 1, characterized in that, The step of identifying defects based on the difference distribution includes: calculating the pixel-by-pixel absolute difference between the two-dimensional grayscale image and the synthetic reference image at corresponding pixels in the three-dimensional unreliable region; calculating the mean and standard deviation of the absolute difference; using the sum of the product of the mean, the threshold coefficient, and the standard deviation as the defect judgment threshold; and marking pixels whose absolute difference exceeds the defect judgment threshold as candidate defect pixels.

5. The method according to claim 4, characterized in that, The half-angle vector of each pixel is calculated based on the normal direction, light source direction, and camera direction of the CAD model surface element. When the cosine value of the surface element normal and the half-angle vector exceeds the cosine value corresponding to the specular reflection angle threshold, the pixel is marked as a specular reflection risk area. The increased threshold coefficient is used to determine the candidate defective pixels in the specular reflection risk area. The specular reflection angle threshold is determined by the angle distribution between the normal and the half-angle vector when a bright spot actually appears on the sample fastener.

6. The method according to claim 5, characterized in that, Connectivity labeling is performed on the candidate defective pixels to group spatially adjacent candidate defective pixels into the same connected region. Connective regions with an area smaller than the minimum area threshold are filtered out, and the remaining connected regions are used as the final detected defective regions. The minimum area threshold is determined based on the square of the ratio of the minimum detectable defect size to the actual size of the single pixel.

7. A precision fastener surface defect detection system based on visual recognition and 3D model comparison, used to implement the method as described in any one of claims 1-6, characterized in that, include: The data acquisition and registration module is used to perform 3D scanning of precision fasteners to obtain surface point cloud data and simultaneously acquire 2D grayscale images. The point cloud data is rigidly registered with the standard CAD model of the precision fastener to obtain the transformation matrix. The reliability partitioning module is used to calculate the angle between the scanning direction and the normal of each facet element according to the thread geometry parameters of the CAD model, perform ray projection on the facet elements in the thread area along the scanning direction to determine whether they are occluded by adjacent tooth crests, and mark each facet element as a three-dimensional reliable area or a three-dimensional unreliable area according to the angle and occlusion determination results. The three-dimensional defect detection module is used to calculate the deviation between the point cloud data in the three-dimensional reliable area and the surface of the CAD model, and to determine the defect based on whether the deviation exceeds the tolerance limit. The reference image generation module is used to perform perspective projection and grayscale rendering on the surface elements of the three-dimensional unreliable area in the CAD model according to the transformation matrix and the pre-calibrated camera parameters to generate a synthetic reference image. The two-dimensional defect detection module is used to compare the region corresponding to the three-dimensional unreliable area in the two-dimensional grayscale image with the synthetic reference image pixel by pixel, and identify defects based on the difference distribution. The result fusion module is used to merge the defect detection results of the three-dimensional reliable region and the three-dimensional unreliable region into the CAD model surface coordinate system to generate a defect distribution map.

Images