Machine vision-based high-strength threaded coupling surface defect detection method and system

Abstract

The application discloses a kind of based on machine vision high-strength threaded coupling surface defect detection method, system, the method comprises: in the process of rotating the workpiece to be detected, utilize fixed light source to irradiate and by fixed camera acquisition multiple frames of images, rotation angle is recorded simultaneously;According to the effective illumination azimuth angle of each surface point calculated from thread geometry parameter, rotation angle and light source position, the corresponding relationship of multiple frames of image pixels and thread surface point is established, and the brightness-azimuth observation sequence of each surface point is constructed;Harmonic fitting is carried out to the observation sequence, and the angle response anisotropy and strongest response azimuth angle are extracted, angle response feature map is constructed and input into classification network, and the defect classification result of each surface point is obtained.The application can utilize the response difference of surface feature with the change of illumination azimuth, improve the identification ability of threaded coupling surface defect.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, specifically to a method and system for detecting surface defects in high-strength threaded connectors based on machine vision. Background Technology

[0002] High-strength threaded fasteners are widely used in vehicles, construction machinery, rail transportation, and various load-bearing connection structures, and their surface quality directly affects the reliability of the connection. Bolts, screws, and studs may develop surface defects such as cracks and folds during cold heading, thread rolling, heat treatment, and surface treatment. Therefore, surface condition inspection is necessary during production. Currently, manual visual inspection and magnetic particle inspection are still in practical use. Meanwhile, with the development of automated production lines, machine vision inspection based on industrial cameras is increasingly being used for online identification of surface defects in threaded parts. Machine vision has the advantages of being non-contact, having high inspection speed, and being easy to integrate with conveying and rotating mechanisms. However, its inspection results are easily affected by the workpiece surface condition, thread geometry, and lighting conditions.

[0003] For high-strength threaded fasteners that have undergone quenching, tempering, phosphating, or blackening treatments, their surfaces are typically matte dark, and the thread flanks have continuously varying spatial orientations, causing the same defect to exhibit different grayscale characteristics under different locations and lighting directions. In particular, grain boundary textures in the phosphating layer and microcracks may both appear as small-scale, near-contrast dark lines in a single grayscale image. When using classification methods based on grayscale, edges, textures under fixed lighting conditions or based on the original image, normal surface textures are easily misidentified as cracks, or low-contrast microcracks may be missed. Therefore, how to improve the ability to distinguish between microcracks and normal surface textures while adapting to the characteristics of threaded curved surfaces and dark surfaces remains a problem that current machine vision inspection technologies need to solve. Summary of the Invention

[0004] In view of this, this disclosure proposes a method and system for detecting surface defects in high-strength threaded fasteners based on machine vision.

[0005] According to one aspect of this disclosure, a method for detecting surface defects in high-strength threaded fasteners based on machine vision is provided, including:

[0006] During the rotation of the workpiece around its own axis, the workpiece surface is illuminated by a light source at a fixed position, and multiple frames of images are captured by a camera at a fixed position, while the rotation angle corresponding to each frame of image is recorded simultaneously.

[0007] The normal vector at each point on the thread surface is calculated based on the thread geometry parameters of the workpiece to be inspected. Combined with the rotation angle and light source position of each frame, the effective illumination azimuth angle of each surface point in each frame is calculated. The effective illumination azimuth angle is the angle between the projection of the incident light direction on the local tangent plane of the surface and the tangent direction of the helix.

[0008] Based on the thread geometry parameters and the rotation angle of each frame, a correspondence between pixels and thread surface points in multiple frames of images is established. The gray value of the same surface point in each visible frame and the corresponding effective illumination azimuth angle are extracted to form a brightness-azimuth angle observation sequence.

[0009] Harmonic fitting was performed on the brightness-azimuth observation sequence of each surface point to obtain the angular response anisotropy and the strongest response azimuth.

[0010] The anisotropy of the angular response and the azimuth of the strongest response are used to construct an angular response feature map, which is then input into a classification network to output the defect classification results for each surface point.

[0011] In one possible implementation, the harmonic fitting uses the least squares method to fit a first-order cosine model to the brightness-azimuth observation sequence to obtain the DC component, fundamental frequency amplitude, and fundamental frequency phase; the angular response anisotropy is the ratio of the fundamental frequency amplitude to the DC component; and the strongest response azimuth is the fundamental frequency phase.

[0012] In one possible implementation, the method further includes: calculating the response sharpness based on the difference between the maximum and minimum grayscale values ​​in the brightness-azimuth observation sequence and the fundamental frequency amplitude; and adding the response sharpness as an additional channel to the angle response feature map.

[0013] In one possible implementation, the strongest response azimuth angle in the angular response feature map is stored as two independent channels: a cosine value and a sine value.

[0014] In one possible implementation, the method further includes: calculating the azimuth angle of the helical tangent direction at each surface point relative to the circumferential reference axis based on the thread geometry parameters, converting the azimuth angle into cosine and sine values, and splicing them as two additional channels to the angle response feature map, with the spliced ​​result serving as the input to the classification network.

[0015] In one possible implementation, the classification network is a fully convolutional network, comprising a first convolutional layer, a second convolutional layer, a third convolutional layer, and an output convolutional layer connected in sequence; the first, second, and third convolutional layers all use square convolutional kernels with a side length of 3, and each layer is followed by a batch normalization layer and an activation function; the output of the second convolutional layer is added element-wise to the output of the third convolutional layer and then input into the output convolutional layer; the output convolutional layer uses a convolutional kernel with a side length of 1, and the number of output channels is equal to the number of classification categories.

[0016] In one possible implementation, the classification network is trained using a composite loss function, which consists of a weighted sum of a weighted cross-entropy loss term and a spatial continuity loss term. In the weighted cross-entropy loss term, the weight of each category is inversely proportional to the total number of pixels of the corresponding category in the training data. The spatial continuity loss term penalizes defective predicted pixels whose number of similar predicted pixels in the neighborhood is less than a set threshold.

[0017] In one possible implementation, in the spatial continuity loss term, for each pixel predicted as a defect category, the number of predicted pixels belonging to the same defect category within its preset neighborhood window is counted. When the number is less than the set threshold, the difference between the set threshold and the number is accumulated as the penalty value for that pixel. The set threshold is determined based on the minimum number of consecutive pixels along the extension direction of the defect feature.

[0018] In one possible implementation, the method further includes post-processing of the defect classification results: performing connected component analysis on the regions identified as crack categories, and removing regions whose connected component length is less than a preset minimum reportable defect length.

[0019] According to another aspect of this disclosure, a machine vision-based surface defect detection system for high-strength threaded fasteners is provided, comprising:

[0020] The rotation acquisition module is used to illuminate the surface of the workpiece with a light source at a fixed position and acquire multiple frames of images with a camera at a fixed position while the workpiece rotates around its own axis, and to simultaneously record the rotation angle corresponding to each frame of image.

[0021] The effective illumination azimuth angle calculation module is used to calculate the normal vector at each point on the thread surface based on the thread geometry parameters of the workpiece to be inspected, and to calculate the effective illumination azimuth angle of each surface point in each frame by combining the rotation angle and the position of the light source in each frame; the effective illumination azimuth angle is the angle between the projection of the incident light direction on the local tangent plane of the surface and the tangent direction of the helix.

[0022] The observation sequence construction module is used to establish the correspondence between pixels and thread surface points in multiple frames of images based on the thread geometric parameters and the rotation angle of each frame, extract the gray value of the same surface point in each visible frame and the corresponding effective illumination azimuth angle, and form a brightness-azimuth angle observation sequence.

[0023] An angle response feature extraction module is used to perform harmonic fitting on the brightness-azimuth observation sequence of each surface point to obtain the angle response anisotropy and the strongest response azimuth.

[0024] The defect classification module is used to construct an angle response feature map from the angle response anisotropy degree and the strongest response azimuth angle, input the angle response feature map into the classification network, and output the defect classification results of each surface point.

[0025] This application utilizes the relative illumination direction change caused by the workpiece rotating around its own axis during the inspection of high-strength threaded fasteners. Under fixed light source and camera conditions, it establishes a correspondence between the grayscale changes of the same thread surface point at different rotation angles and its effective illumination azimuth angle. Furthermore, it extracts the anisotropy of the angular response and the strongest response azimuth angle through harmonic fitting, and further constructs an angular response feature map with clear physical meaning for defect classification. This transforms the detection basis from local grayscale, edge, or texture features in a single image to the directional response of surface features as the illumination azimuth changes. This method can leverage the differences in spatial morphology and illumination response between deep groove defects such as microcracks and shallow surface textures such as phosphate layer grain boundaries to reduce the interference of dark surfaces, threaded curved surfaces, and surface treatment textures on defect identification. While maintaining online non-contact inspection capabilities, it improves the distinguishability and detection stability of small surface defects and gives the classification results better physical interpretability.

[0026] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0027] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0028] Figure 1 A flowchart is shown for a machine vision-based method for detecting surface defects in high-strength threaded connectors according to an embodiment of the present disclosure.

[0029] Figure 2 A schematic diagram illustrating the effective illumination azimuth geometry according to an embodiment of the present disclosure is shown.

[0030] Figure 3A schematic diagram showing the brightness-effective illumination azimuth response of microcracks and phosphate layer grain boundaries according to an embodiment of the present disclosure.

[0031] Figure 4 A schematic diagram of a classification network structure according to an embodiment of the present disclosure is shown.

[0032] Figure 5 A schematic diagram of a machine vision-based surface defect detection system for high-strength threaded connectors is shown according to an embodiment of the present disclosure.

[0033] Figure 6 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0034] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0035] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0036] This embodiment provides a machine vision-based method for detecting surface defects in high-strength threaded fasteners. This method is particularly suitable for online surface quality inspection of threaded fasteners with a strength grade of 10.9 and above. The workpieces under inspection typically undergo phosphating or blackening surface treatment after quenching and tempering heat treatment, resulting in a matte dark surface. The inspection line is equipped with a linear or area scan camera. The workpiece rotates uniformly around its axis at the inspection station, for example, at 1 to 3 revolutions per second. Illumination uses a fixed-position strip LED light source mounted on the same side as the camera, with the angle between the light source's optical axis and the workpiece's axis, for example, 30 to 60 degrees. A rotary encoder is synchronously triggered with the camera, ensuring that each frame corresponds to a specific rotation angle value; the encoder accuracy is, for example, 0.1 degrees. The thread specifications of the workpiece are pre-configured by the inspection system, including pitch, thread angle, major diameter, and minor diameter.

[0037] Phosphate layer grain boundary textures and quenching microcracks exhibit similar scale and contrast dark line features in grayscale images under conventional single illumination conditions, making them difficult to distinguish through morphological analysis of a single image. The core idea of ​​this method is to utilize the natural changes in the illumination azimuth angle of each surface point relative to a fixed light source caused by the workpiece's rotational motion at the inspection station, extracting the response features of the brightness of each surface point as a function of the azimuth angle, and thereby distinguishing between two types of surface features with significant differences in aspect ratio.

[0038] The following is an explanation of the terms that may appear in this article.

[0039] The implementation process of the method described in this application will be described in detail below with reference to specific embodiments. It should be noted that this embodiment is only used to explain this application and is not intended to limit the scope of protection of this application. Conventional adjustments or substitutions of each step by those skilled in the art without departing from the concept of this application should be included in the scope of protection of this application.

[0040] Figure 1 A flowchart illustrating a machine vision-based method for detecting surface defects in high-strength threaded fasteners according to an embodiment of this disclosure is shown. Figure 1 As shown, the method may include:

[0041] In step S11, multiple frames of images are acquired and the rotation angle is recorded. In this embodiment, the workpiece is driven by a servo motor to rotate uniformly around its own axis, and the camera continuously acquires images at a fixed frame rate. The number of frames acquired within a complete rotation cycle is determined by the ratio of the frame rate to the rotation speed; for example, when the rotation speed is 2 revolutions per second and the frame rate is 2000 frames per second, 1000 frames are acquired per cycle. The position and intensity of the light source remain constant throughout the acquisition process. The encoder outputs rotation angle values ​​corresponding one-to-one with each frame of image. The acquired data consists of an image sequence and a rotation angle sequence corresponding to each frame.

[0042] In step S12, the normal vector at each point on the thread surface is calculated based on the thread geometry parameters, and the effective illumination azimuth angle is calculated by combining the rotation angle of each frame and the position of the light source. The effective illumination azimuth angle is the angle between the projection of the incident light direction onto the local tangent plane of the surface and the tangent direction of the helix.

[0043] In this embodiment, the threaded surface is a helical surface that can be analytically described by parameters. In the workpiece fixed coordinate system, any point on the threaded surface is determined by two parameters: the circumferential angle. (Values ​​range from 0 to...) ) and tooth position parameters (Values ​​range from 0 to 1, where 0 corresponds to the root of the thread and 1 corresponds to the crest of the thread). Given pitch. , large diameter , paths and tooth shape half angle The radial coordinates of the surface points are:

[0044]

[0045] in The distance from this point to the workpiece axis is given by the following spatial coordinates of the surface point in the workpiece coordinate system:

[0046]

[0047] Find the expression for each of the above expressions. and The partial derivatives of the two vectors are used to determine the direction of their cross product, which is the direction of the outward normal vector at that point. The specific calculation is a standard operation in spiral differential geometry; normalizing the cross product yields the unit outward normal vector at that point. .

[0048] In one implementation, the effective illumination azimuth angle refers to the angle between the projection of the incident light direction onto the local tangent plane of a surface point and the direction of the helical tangent at that point. When the workpiece rotates to the desired angle... At that time, surface point The normal vector in the laboratory coordinate system becomes ,in Rotation about the workpiece axis The standard rotation matrix of the angle.

[0049] Figure 2 A schematic diagram illustrating the effective illumination azimuth geometry according to an embodiment of the present disclosure is shown, such as... Figure 2 As shown, for any surface point P on the threaded surface, the local tangent plane and unit outward normal vector of that point are determined based on the thread's geometric parameters. When the workpiece rotates around its own axis to the corresponding rotation angle, the incident light direction of the fixed light source and the spatial orientation of the surface point jointly determine the current illumination conditions. The incident light direction is projected onto the local tangent plane of surface point P. Within this tangent plane, the helical tangent direction is used as the reference direction, and the angle between the projected direction and the helical tangent direction is the effective illumination azimuth angle. As the workpiece rotation angle changes, the effective illumination azimuth angle corresponding to the same surface point changes continuously, thus providing a geometric basis for subsequently establishing a brightness-azimuth angle observation sequence and extracting angle response features.

[0050] Specifically, let the unit vector of the light source direction be... (Determined by the installation position of the light source, and approximately the same within a local area of ​​the workpiece surface). The projection vector of the incident light onto the local tangent plane of the surface is:

[0051]

[0052] in Let the dot product of vectors be the unit vector of the helical tangent direction at this surface point in the laboratory coordinate system. ,and The unit vector that is orthogonal and lies in the tangent plane is The effective lighting azimuth angle Calculate using the following formula:

[0053]

[0054] The range of values ​​is The unit is radians. As the workpiece rotates... Changes in the effective illumination azimuth of the same surface point This also changes accordingly, and this change forms the physical basis for subsequent angular response analysis. Because microcracks have a narrow, deep, groove-like characteristic, when... When perpendicular to the crack direction, a deep shadow is produced with the lowest brightness; when... When the light is parallel to the crack direction, it can enter the crack and significantly increase the brightness; while the phosphate layer grain boundary is a shallow surface relief, and the brightness response to changes in azimuth angle is gradual and has no clear directional preference.

[0055] Figure 3 A schematic diagram showing the brightness-effective illumination azimuth response of microcracks and phosphate layer grain boundaries according to an embodiment of the present disclosure. Figure 3 The differences in typical brightness response between microcracks and phosphate layer grain boundaries under varying effective illumination azimuth angles are illustrated. As the workpiece rotates, the effective illumination azimuth angle of the same threaded surface point relative to a fixed light source changes. Microcracks, being narrow and deep groove-like features, exhibit a significant change in their illumination state with the illumination direction, resulting in a large peak-valley variation in their grayscale response, demonstrating high angular response anisotropy and a sharp response characteristic; the illumination direction corresponding to the maximum brightness response is the strongest response azimuth angle. In contrast, phosphate layer grain boundaries, being shallow relief features, show less variation in grayscale value with the effective illumination azimuth angle, a smoother overall response, and lower angular response anisotropy. Therefore, the directional brightness response differences between different surface features can be utilized.

[0056] For example, for an M12 bolt ( =1.75mm, =12mm, =10.106mm, =30°), at a point on the tooth surface =0.5, At a position of 1.0 rad, when the rotation angle changes from 0 to... / 2, effective illumination azimuth angle The value changes from approximately 0.3 rad to approximately 1.9 rad, a range of approximately 1.6 rad, which is sufficient to cover the azimuth range required for angular response analysis.

[0057] In step S13, the correspondence between pixels and thread surface points in multiple frames of images is established, gray values ​​and effective illumination azimuth angles are extracted, and a brightness-azimuth angle observation sequence is formed.

[0058] In this embodiment, given the known thread geometry parameters and the rotation angles of each frame, the thread surface coordinates corresponding to each pixel in the image are... The camera's intrinsic and extrinsic parameters are obtained through inverse calculation using a camera imaging model, and are pre-calibrated. For the same surface point... The position of its projected pixel in each frame of the image can be calculated based on the rotation angle and imaging geometry.

[0059] For example, the coordinates of the threaded surface corresponding to each pixel in the image. It is obtained through inverse calculation using a pinhole camera imaging model. Specifically, let the homogeneous coordinates of the spatial point in the world coordinate system be... via extrinsic parameter matrix ( It is a 3×3 rotation matrix. Transformed to the camera coordinate system (using a 3×1 translation vector), and then processed by the intrinsic parameter matrix. (in For equivalent focal length, The coordinates of the principal point are projected onto the normalized image plane to obtain the corresponding pixel coordinates. The aforementioned intrinsic and extrinsic parameters can be obtained in advance using the Zhang Zhengyou calibration method. The calibration target is placed at the workpiece axis position in the detection station for execution. Based on this imaging model, for any thread surface point with known spatial coordinates... Its projected pixel position can be uniquely calculated; conversely, given the pixel position and the tooth profile position of that point, The corresponding circumferential angle can be obtained by inverse solution. .

[0060] The visibility criterion is that when the surface normal vector... A point is considered visible if the angle between its position and the camera's line of sight is less than 90 degrees and it is not obscured by adjacent teeth. Occlusion determination is based on the geometric occlusion relationship between the spatial position of adjacent tooth cusps and the line of sight, and is a standard calculation method involving ray-triangle intersection.

[0061] After the above processing, a set of multi-frame observation records is established for each discrete sampling point on the thread surface. Each record contains the grayscale value read from that point in each visible frame. and the effective illumination azimuth angle of the corresponding frame ,in ; The visible frame number for this point depends on the camera field of view and the thread geometry, typically ranging from 8 to 20. This set of paired data constitutes the brightness-azimuth observation sequence for this surface point, denoted as [missing value]. .

[0062] In step S14, harmonic fitting is performed on the brightness-azimuth observation sequence to obtain the angular response anisotropy and the strongest response azimuth.

[0063] According to embodiments of this disclosure, a first-order cosine model is used to perform least-squares fitting on the brightness-azimuth observation sequence of each surface sampling point. Specifically, the fitting model is as follows:

[0064]

[0065] in The DC component represents the average brightness of that point, measured in grayscale values ​​(integers or floating-point numbers ranging from 0 to 255). and This represents the fundamental frequency harmonic coefficient. (The remaining text appears to be incomplete and requires further context.) Substituting the observed data into the above equation, we construct an overdetermined system of linear equations. The design matrix is ​​then... Behavior The observation vector is The parameter vector to be determined is Solving this least squares problem using the normal equation method is equivalent to finding the solution. ,in To design the matrix, For parameter vectors, Let be the observation vector. The normal equation method is a standard solution for linear least squares, and it has extremely low computational cost for the case of three unknowns.

[0066] Seek , , Then, calculate the fundamental frequency amplitude:

[0067]

[0068] Dimensions and They are consistent, both being grayscale values.

[0069] In one implementation, the anisotropy of the angular response is... Defined as the ratio of the fundamental frequency amplitude to the DC component:

[0070]

[0071] This characterizes the relative magnitude of the surface point brightness variation with the illumination azimuth angle. For shallow relief features such as phosphate layer grain boundaries, Typically below 0.15; for deep groove-type features such as microcracks, Typically, it is above 0.3. This threshold range is determined through statistical analysis of known defect samples and normal samples, wherein during the calibration phase, at least 50 confirmed crack points and 200 normal coating points are collected respectively. The value is determined by taking the optimal separation point between the two distributions as the initial reference threshold.

[0072] Specifically, this threshold range is determined through statistical analysis of known defect samples and normal samples. During the calibration phase, at least 50 confirmed crack points and 200 normal coating points are collected respectively. Values. The two types of samples... A histogram of the values ​​was plotted, and the optimal separation threshold was determined using the maximization criterion of the Youden index under the ROC curve. ,in The candidate threshold is used, and the sensitivity and specificity are calculated from the confusion matrix. A threshold is selected that... The largest The value is used as the initial reference threshold.

[0073] The strongest response azimuth angle is defined as the illumination azimuth direction when the brightness response reaches its maximum value:

[0074]

[0075] The range of values ​​is The unit is radians. For crack-like features, When the light is directed in a direction parallel to the crack's direction, it can penetrate deep into the crack, resulting in the highest brightness.

[0076] As an example, the observation sequence of a certain surface point contains 10 frames of data, which are obtained through least squares fitting. , , (All are grayscale values); then ; ; rad. Normal phosphating surface points on the same workpiece, obtained by fitting. , , ;but ; The anisotropy of their angular responses differs by approximately four times, a significant difference.

[0077] In one implementation, based on the obtained harmonic fitting results, the response sharpness is further calculated using the difference between the maximum and minimum grayscale values ​​in the brightness-azimuth observation sequence, as well as the fundamental frequency amplitude. Response sharpness Defined as the ratio of the measured brightness range to the fundamental frequency fitting range:

[0078]

[0079] in and These are the maximum and minimum gray values ​​in the observed sequence. This represents the theoretical peak-to-valley difference in a first-order cosine model. When the brightness varies strictly with azimuth angle according to a sine law... The crack's step-shading effect, caused by its narrow slit structure, makes the actual response sharper than a sine wave, typically ranging from 1.2 to 1.8; the phosphate layer grain boundaries... It is usually between 0.9 and 1.1.

[0080] Following the previous example, the actual measurement of crack points , , Actual measurement of normal coating points , , Response sharpness is an auxiliary discriminative feature, used in conjunction with angular response anisotropy.

[0081] In step S15, an angle response feature map is constructed and input into the classification network to output the defect classification result.

[0082] According to an embodiment of this disclosure, the threaded surface is unfolded into a two-dimensional plane along the helical direction, with the horizontal axis corresponding to the circumferential direction and the vertical axis corresponding to the axial direction. The unfolded surface is discretized as follows: The pixel grid has a resolution of, for example, 5 to 10 micrometers per pixel, representing the actual surface length.

[0083] Specifically, the aforementioned calculated characteristic value is assigned to each grid point. To eliminate the periodic discontinuity of the azimuth value, the strongest response azimuth is selected. It is converted into two components: cosine and sine. The angle response feature map contains four channels: Channel 1 stores the anisotropy of the angle response. Channel 2 storage Channel 3 storage Channel 4 storage response sharpness All channels use floating-point numbers as their data type, and all have a spatial dimension of [missing information]. .

[0084] In one implementation, the azimuth angle of the helical tangent direction relative to the circumferential reference axis at each grid point on the unfolded surface is calculated based on the thread geometry parameters. Its value is equal to the helix angle at that point:

[0085]

[0086] in The diameter corresponding to the tooth profile position of this grid point (derived from tooth profile parameters). (Determined by interpolation of major and minor diameters), with the dimension of length; The pitch is measured in units of length. The dimension of is radians. Convert to and The two components are concatenated as two additional channels to the angle response feature map, forming a 6-channel input feature map with dimensions of [dimension number missing]. The thread direction coding provides a local helical direction reference for the classification network: quenching cracks... The direction is usually close to the circumferential direction, with twisted and folded threads. The direction is usually close to the tangent direction of the spiral. Direction encoding allows the network to compare the direction of the angular response with the pattern of the defect direction.

[0087] In one implementation, the classification network employs a fully convolutional network (FCN) structure. The network input consists of local blocks cropped from the full 6-channel feature map, with block spatial dimensions, for example... Each pixel has 6 channels. The network output is a pixel-by-pixel classification probability map of the same spatial size, with 3 channels, corresponding to the three categories of normal surface, crack, and fold.

[0088] Figure 4 A schematic diagram of the classification network structure according to an embodiment of the present disclosure is shown. A complete six-channel feature map is cropped into local feature blocks by a sliding window and then input into a fully convolutional integral classification network. The classification network is configured with a first convolutional layer, a second convolutional layer, and a third convolutional layer, all three layers using 3×3 convolutional kernels. The outputs of the second and third convolutional layers are added element-wise and then fed into the output convolutional layer. The output convolutional layer uses a 1×1 convolutional kernel and outputs three category channels. Softmax is applied to obtain the pixel-by-pixel classification probabilities for normal surfaces, cracks, and folds. During inference, blocks are slid across the complete feature map with a stride of 32. The average of multiple inference probabilities for overlapping regions is taken, and the category with the highest probability is selected for each pixel, ultimately forming the defect classification result for the threaded surface.

[0089] Specifically, the network consists of four convolutional layers connected in sequence:

[0090] The first convolutional layer uses a square convolutional kernel with a side length of 3, a stride of 1, padding of 1, and 16 output channels; followed by a batch normalization layer and a ReLU activation function. The input dimension is... The output dimension is .

[0091] The second convolutional layer also uses a square convolutional kernel with a side length of 3, a stride of 1, padding of 1, and 32 output channels; followed by a batch normalization layer and a ReLU activation function. Input dimension Output dimension .

[0092] The third convolutional layer uses a square kernel with sides of length 3, a stride of 1, padding of 1, and 32 output channels; followed by a batch normalization layer and a ReLU activation function. Input dimensions... Output dimension The outputs of layer 2 and layer 3 are added element-wise to form a residual connection; the dimension of the sum remains the same. This serves as the input to the fourth layer. Residual connections allow gradients to bypass the third layer and propagate directly to the second layer, which helps improve training stability when the network is shallow.

[0093] The fourth layer is the output convolutional layer: a convolutional kernel with a side length of 1, a stride of 1, and 3 output channels (equal to the number of classification categories); it is followed by a Softmax function to output the probability distribution of each pixel across the 3 categories. The final output dimension is... .

[0094] The inference phase processing flow is the same as the training phase. A sliding window with a step size of 32 is used to extract blocks from the complete thread surface feature map and feed them into the network one by one. For overlapping areas, the average probability of each inference iteration is taken, and the class with the highest probability for each pixel is used as the classification result.

[0095] The model input data (6-channel feature map blocks) corresponds exactly to the result of stitching together the aforementioned angle response feature maps and encoding the thread direction. The model output data (pixel-by-pixel 3-class probability maps) is directly used to determine the defect category at each surface location. The 6-channel feature map encodes the angle response characteristics and thread direction information of each surface point. The network establishes a mapping relationship from features to classification by learning the angle response patterns of different defect types in the training data.

[0096] In one implementation, for model training, the labels for the training data are obtained as follows: Rotational scanning images of high-strength threaded connectors from the same batch are first acquired, and angular response feature maps are calculated. Then, magnetic particle inspection (MPI) is performed on the same batch of workpieces. The locations of cracks detected by MPI are mapped to the corresponding coordinate positions in the angular response feature maps and labeled as crack categories. Folding defects are labeled based on metallographic cross-sectional inspection results. Unlabeled areas are labeled as normal surfaces.

[0097] Specifically, the training data should cover different phosphating batches and workpieces of different specifications, such as the M8 to M16 range. Before training, Z-score standardization is performed on all six input channels. The standardization parameters (mean and standard deviation) are calculated from the training set. The validation and test sets are standardized using the same training set parameters to ensure data distribution consistency. The labeled dataset is randomly divided into training, validation, and test sets in a 7:1.5:1.5 ratio, maintaining a relatively consistent proportion of defect pixels for each category across the subsets (stratified sampling).

[0098] In one implementation, the loss function consists of a weighted sum of a weighted cross-entropy loss term and a spatial continuity loss term:

[0099]

[0100] in The weighting coefficients are used to balance the two losses.

[0101] The weighted cross-entropy loss term is calculated as follows:

[0102]

[0103] in and The spatial dimension of the block (e.g., all are 64); For pixels Category The true label (unique hot encoding); The category of the network's output for this pixel. Predicting probabilities; For category The weight.

[0104] Specifically, category weights It is inversely proportional to the total number of pixels of the corresponding category in the training set. It is determined by counting the total number of pixels in the training set for the three categories: normal surface, crack, and fold, denoted as [missing information]. , , ,Pick ,in It is the maximum value among the three. For example, when normal pixels account for 97% of the total, cracks account for 2%, and folds account for 1%, it is the maximum value among the three. , , In practical applications, to avoid the impact of extreme weights on training stability, an upper limit can be set for the weights, such as truncating them to no more than 10.

[0105] Among them, the spatial continuity loss term penalizes defective predicted pixels whose number of similar predicted pixels in the neighborhood is less than a set threshold:

[0106]

[0107] in This is the set of pixels in the current block that are predicted to be of the defect category. for Total number of pixels (when) hour ); For pixels In its preset neighborhood window (e.g.) The number of predicted pixels belonging to the same defect category within a given area; To set a threshold.

[0108] Understandably, the physical basis for this loss term lies in the fact that real cracks and folds are characterized by continuous extension along a certain direction, and do not appear as isolated points. The minimum number of consecutive pixels along the extension direction of the defect feature is determined by calculating the minimum pixel span corresponding to the system resolution and the minimum reportable defect length, and taking half of it as the threshold. For example, when the resolution is 5 micrometers / pixel and the minimum defect length is 50 micrometers, the corresponding span is 10 pixels. The threshold is a suitable value between 3 and 5, for example, 3.

[0109] Among them, the weighting coefficient Ten candidate values ​​were selected in a geometric progression within the range of 0.01 to 1.0 using a grid search on the validation set. After training the model for each candidate value, the crack detection rate and false alarm rate were evaluated. The model with the lowest false alarm rate was selected while maintaining a crack detection rate of at least 95%. Value. In actual testing. The typical value ranges from 0.05 to 0.2.

[0110] The training uses the Adam optimizer, with an initial learning rate set to, for example, 0. The learning rate adjustment strategy is cosine annealing, which decays from the initial value according to a cosine function to... The training run is 100 to 200 epochs, stopping when the loss on the validation set no longer decreases after 20 consecutive epochs. Batch size is, for example, 16 to 32. A Dropout operation with a dropout rate of 0.2 is added between layers 2 and 3 to prevent overfitting; this operation is only active during training and disabled during inference.

[0111] In one implementation, the defect classification results output by the classification network are post-processed. Specifically, connected component analysis is performed on the regions identified as cracks. The extension length of each connected component along its principal direction is calculated (the physical length is obtained by multiplying the number of pixels by the resolution), and connected components with a length less than the preset minimum reportable defect length are removed. The minimum reportable defect length is determined according to product standards or customer requirements, for example, 50 micrometers. The connected component analysis uses the 8-connectivity criterion, which is a standard operation in image processing.

[0112] The above method utilizes the existing motion of workpiece rotation on the production line, including the detection of changes in the illumination azimuth angle caused by threaded connections, to extract the angular response features of each surface point from multiple frames of images. It then distinguishes between different angular response patterns caused by the physical differences in aspect ratio between microcracks and phosphate layer grain boundaries. Compared to grayscale analysis methods using single images, this approach is more adaptable to batch variations in the phosphate layer. The classification network uses angular response features calculated based on physical principles as input instead of the original images, reducing the need for training data and making the detection results physically interpretable.

[0113] Figure 5 A block diagram of a machine vision-based surface defect detection system for high-strength threaded fasteners according to an embodiment of the present disclosure is shown. Figure 5 As shown, the system includes:

[0114] The rotation acquisition module 401 is used to illuminate the surface of the workpiece with a light source at a fixed position and acquire multiple frames of images with a camera at a fixed position during the rotation of the workpiece around its own axis, and to simultaneously record the rotation angle corresponding to each frame of images.

[0115] The effective illumination azimuth angle calculation module 402 is used to calculate the normal vector at each point on the thread surface based on the thread geometry parameters of the workpiece to be inspected, and to calculate the effective illumination azimuth angle of each surface point in each frame by combining the rotation angle and the position of the light source in each frame; the effective illumination azimuth angle is the angle between the projection of the incident light direction on the local tangent plane of the surface and the tangent direction of the helix.

[0116] The observation sequence construction module 403 is used to establish the correspondence between pixels and thread surface points in multiple frames of images based on the thread geometric parameters and the rotation angle of each frame, extract the gray value of the same surface point in each visible frame and the corresponding effective illumination azimuth angle, and form a brightness-azimuth angle observation sequence.

[0117] Angle response feature extraction module 404 is used to perform harmonic fitting on the brightness-azimuth observation sequence of each surface point to obtain the angle response anisotropy and the strongest response azimuth.

[0118] The defect classification module 405 is used to construct an angle response feature map by combining the angle response anisotropy degree and the strongest response azimuth angle, input the angle response feature map into the classification network, and output the defect classification results for each surface point.

[0119] In some embodiments, the system provided in this disclosure may have functions or include modules that can be used to execute the methods described in the above method embodiments. The specific implementation of these methods can be referred to the description in the above method embodiments, and for the sake of brevity, they will not be repeated here.

[0120] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. (Refer to...) Figure 6 Electronic devices can be provided as servers or terminal devices. (See reference...) Figure 6 The electronic device includes a processing component 601, which further includes one or more processors, and memory resources represented by memory 602 for storing instructions, such as application programs, that can be executed by the processing component 601. The application programs stored in memory 602 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 601 is configured to execute instructions to perform the methods described above.

[0121] The electronic device may also include a power supply component 603 configured to perform power management of the electronic device, a wired or wireless network interface 604 configured to connect the electronic device to a network, and an input / output interface 605 (I / O interface). The electronic device can operate on an operating system stored in memory 602.

[0122] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 602 including computer program instructions that can be executed by a processing component 601 of an electronic device to perform the above-described method.

[0123] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for detecting surface defects in high-strength threaded fasteners based on machine vision, characterized in that, include: During the rotation of the workpiece around its own axis, the workpiece surface is illuminated by a light source at a fixed position, and multiple frames of images are captured by a camera at a fixed position, while the rotation angle corresponding to each frame of image is recorded simultaneously. The normal vector at each point on the thread surface is calculated based on the thread geometry parameters of the workpiece to be inspected. Combined with the rotation angle and light source position of each frame, the effective illumination azimuth angle of each surface point in each frame is calculated. The effective illumination azimuth angle is the angle between the projection of the incident light direction on the local tangent plane of the surface and the tangent direction of the helix. Based on the thread geometry parameters and the rotation angle of each frame, a correspondence between pixels and thread surface points in multiple frames of images is established. The gray value of the same surface point in each visible frame and the corresponding effective illumination azimuth angle are extracted to form a brightness-azimuth angle observation sequence. Harmonic fitting was performed on the brightness-azimuth observation sequence of each surface point to obtain the angular response anisotropy and the strongest response azimuth. The anisotropy of the angular response and the azimuth of the strongest response are used to construct an angular response feature map, which is then input into a classification network to output the defect classification results for each surface point.

2. The method according to claim 1, characterized in that, The harmonic fitting uses the least squares method to fit a first-order cosine model to the brightness-azimuth observation sequence to obtain the DC component, fundamental frequency amplitude, and fundamental frequency phase; the angular response anisotropy is the ratio of the fundamental frequency amplitude to the DC component; the strongest response azimuth is the fundamental frequency phase.

3. The method according to claim 2, characterized in that, Also includes: The response sharpness is calculated based on the difference between the maximum and minimum gray values ​​in the brightness-azimuth observation sequence and the fundamental frequency amplitude. The response sharpness is added as an additional channel to the angle response feature map.

4. The method according to claim 1, characterized in that, In the angle response feature map, the strongest response azimuth angle is stored as two independent channels: the cosine value and the sine value.

5. The method according to claim 4, characterized in that, Also includes: The azimuth angle of the helical tangent direction at each surface point relative to the circumferential reference axis is calculated based on the thread geometry parameters. The azimuth angle is converted into cosine and sine values, which are then used as two additional channels and spliced ​​to the angle response feature map. The spliced ​​result is used as the input to the classification network.

6. The method according to claim 1, characterized in that, The classification network is a fully convolutional network, comprising a first convolutional layer, a second convolutional layer, a third convolutional layer, and an output convolutional layer connected in sequence. The first, second, and third convolutional layers all use square convolutional kernels with a side length of 3. Each layer is followed by a batch normalization layer and an activation function. The output of the second convolutional layer is added element-wise to the output of the third convolutional layer and then input into the output convolutional layer. The output convolutional layer uses a convolutional kernel with a side length of 1, and the number of output channels is equal to the number of classification categories.

7. The method according to claim 6, characterized in that, The classification network is trained using a composite loss function, which consists of a weighted sum of a weighted cross-entropy loss term and a spatial continuity loss term. In the weighted cross-entropy loss term, the weight of each category is inversely proportional to the total number of pixels of the corresponding category in the training data. The spatial continuity loss term penalizes defective predicted pixels whose number of similar predicted pixels in the neighborhood is less than a set threshold.

8. The method according to claim 7, characterized in that, In the spatial continuity loss term, for each pixel predicted as a defect category, the number of predicted pixels belonging to the same defect category within its preset neighborhood window is counted. When the number is less than the set threshold, the difference between the set threshold and the number is accumulated as the penalty value for that pixel. The set threshold is determined based on the minimum number of consecutive pixels along the extension direction of the defect feature.

9. The method according to claim 1, characterized in that, It also includes post-processing of the defect classification results: performing connected component analysis on the regions identified as cracks, and removing regions whose connected component length is less than the preset minimum reportable defect length.

10. A machine vision-based surface defect detection system for high-strength threaded fasteners, characterized in that, include: The rotation acquisition module is used to illuminate the surface of the workpiece with a light source at a fixed position and acquire multiple frames of images with a camera at a fixed position while the workpiece rotates around its own axis, and to simultaneously record the rotation angle corresponding to each frame of image. The effective illumination azimuth angle calculation module is used to calculate the normal vector at each point on the thread surface based on the thread geometry parameters of the workpiece to be inspected, and to calculate the effective illumination azimuth angle of each surface point in each frame by combining the rotation angle and the position of the light source in each frame; the effective illumination azimuth angle is the angle between the projection of the incident light direction on the local tangent plane of the surface and the tangent direction of the helix. The observation sequence construction module is used to establish the correspondence between pixels and thread surface points in multiple frames of images based on the thread geometric parameters and the rotation angle of each frame, extract the gray value of the same surface point in each visible frame and the corresponding effective illumination azimuth angle, and form a brightness-azimuth angle observation sequence. An angle response feature extraction module is used to perform harmonic fitting on the brightness-azimuth observation sequence of each surface point to obtain the angle response anisotropy and the strongest response azimuth. The defect classification module is used to construct an angle response feature map from the angle response anisotropy degree and the strongest response azimuth angle, input the angle response feature map into the classification network, and output the defect classification results of each surface point.

Images