Fastener size measurement method and system based on image recognition

Abstract

This invention discloses a method and system for measuring fastener dimensions based on image recognition, belonging to the field of image recognition technology. The method includes: extracting observed edge points; inputting the outer radius deviation features, candidate angles, and thread parameters of the observed edge points into a physical encoder to generate multiple sets of theoretical edge points; matching the observed edge points with the theoretical edge points and calculating the difference feature vector; calculating local attention weights based on the local image features and physical model description of the matched point pairs; determining the optimal matched point pair based on the local attention weights; fusing the difference features and local attention weights of the optimal matched point pair into a decoupled network, outputting an attitude correction amount, a two-dimensional position correction vector, and edge confidence to obtain the cleaned edge points; updating the rotation angle using the attitude correction amount, and selecting high-confidence edge points to fit a circle and calculate the outer diameter dimension. This invention can improve the accuracy of fastener dimension measurement.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a fastener size measurement method and system based on image recognition. Background Technology

[0002] Existing online fastener dimensional inspection methods mostly employ backlit contour measurement based on optical screening machines. This method involves passing the fastener vertically or horizontally through a detection area. A backlight source is positioned on one side of the detection area to emit parallel light, while an industrial camera on the other side captures the contour shadow image of the fastener. Because the light is blocked by the fastener itself, the fastener area appears dark in the image acquired by the camera, while the background area appears bright, creating a high-contrast light-dark boundary at the fastener's contour edge. The system binarizes this contour shadow image, extracts the pixel coordinates of the fastener contour, calculates feature dimensions such as rod diameter, head diameter, and total length based on these pixel coordinates, and determines whether the fastener is qualified according to a preset dimensional tolerance threshold.

[0003] However, this method relies on the fastener maintaining a stable upright or horizontal posture during transport, and under backlighting, it can only acquire a two-dimensional projected profile, limiting its ability to measure three-dimensional structural features such as thread helical characteristics. When it is necessary to measure thread feature dimensions such as the major diameter and pitch, if the fastener rotates during transport, the projection shape of its helical profile in the two-dimensional image changes accordingly. The correspondence between the directly extracted profile edge pixels and the actual thread major diameter is unclear, leading to measurement deviations. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a fastener dimension measurement method and system based on image recognition, thereby resolving the issues present in the background art.

[0005] To achieve the aforementioned objectives, this invention proposes a fastener dimension measurement method based on image recognition, comprising: Acquire fastener image frames, extract edge points from the fastener image frames, and construct a set of observed edge points containing relative coordinates and gradient directions; The outer radius observation value is obtained by calculating the average distance from the edge point in the observation edge point set to the geometric center of the fastener. The outer radius deviation feature is obtained by calculating the difference between the outer radius observation value and the standard outer radius of the fastener. A physical embedding vector is generated based on the outer radius deviation feature and the standard thread parameters. Multiple sets of theoretical edge points are generated based on the physical embedding vector. The observed edge point set and the theoretical edge point set are matched to generate matching point pairs. The difference feature vector of the matching point pairs is calculated based on the relative coordinates and gradient direction. The local attention weights are calculated based on the gradient direction and physical embedding vector of the matching point pair, and the pair with the largest local attention weights is determined as the optimal matching point pair. The differential features of the optimal matching point pair and the local attention weights are input into the decoupled network, and the outputs the pose correction, correction vector and edge confidence. The coordinates of the observed edge points are corrected based on the correction vector to obtain the set of decontamination edge points. The outer diameter is calculated based on the reliable edge points selected from the set of cleaned edge points based on the edge confidence.

[0006] The present invention also provides a fastener dimension measurement method and system based on image recognition, the system being used to implement the above-described method, the system comprising: The edge extraction module acquires fastener image frames, extracts edge points from the fastener image frames, and constructs a set of observed edge points containing relative coordinates and gradient directions. The model encoding module calculates the average distance from the edge points in the observation edge point set to the geometric center of the fastener to obtain the outer radius observation value, calculates the difference between the outer radius observation value and the standard outer radius of the fastener to obtain the outer radius deviation feature, generates a physical embedding vector based on the outer radius deviation feature and standard thread parameters, and generates multiple sets of theoretical edge point sets based on the physical embedding vector. The feature matching module matches the observed edge point set and the theoretical edge point set to generate matching point pairs, and calculates the difference feature vector of the matching point pairs based on relative coordinates and gradient direction; The attention evaluation module calculates local attention weights based on the gradient direction and physical embedding vector of the matching point pair, and determines the pair with the largest local attention weights as the optimal matching point pair. The decoupling correction module inputs the difference features of the optimal matching point pair and the local attention weights into the decoupling network, and outputs the pose correction amount, correction vector and edge confidence. Based on the correction vector, the coordinates of the observed edge points are corrected to obtain the set of decontamination edge points. The size calculation module selects reliable edge points from the set of cleaned edge points based on edge confidence, and calculates the outer diameter based on the reliable edge points.

[0007] The beneficial effects of this invention are as follows: This invention first fuses outer radius deviation features and candidate angles using a physical encoder to generate multiple sets of theoretical edge points containing topological adjacency points, thereby establishing a physical geometric model that can adaptively compensate for individual manufacturing tolerances and solving the matching failure problem between the standard thread model and the measured part size deviation. Then, an attention weighting mechanism is used to accurately assess the reliability of matching point pairs in motion-blurred scenarios, eliminating attitude misjudgments caused by conveyor belt trails. Subsequently, a decoupled network fuses difference features with local attention weights, outputting position correction vectors and edge confidence in parallel. Point-by-point coordinate correction and confidence quantification are performed on the cleaned edge points. Finally, based on high-confidence correction points, circle fitting is performed to obtain the outer diameter, achieving high-precision measurement under conditions of both motion blur and local occlusion. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the steps of the image recognition-based fastener dimension measurement method of the present invention. Figure 2 This is a schematic diagram of the fastener size measurement system based on image recognition according to the present invention. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0010] like Figure 1 As shown, a fastener dimension measurement method based on image recognition includes: S1: Obtain the fastener image frame, extract the edge points in the fastener image frame, and construct a set of observed edge points containing relative coordinates and gradient direction.

[0011] Multiple fasteners are sequentially fed into the field of view of the camera via a conveyor belt. The conveyor belt runs at a constant speed, and the industrial camera continuously captures images at a fixed frame rate. The color images are converted into grayscale images, and then the distortion of the images is corrected according to the camera's intrinsic parameter matrix. The corrected images are then used as standard images.

[0012] Then, YOLOv5 and other object detection algorithms were used to identify the axial contour of the fastener, identifying its head and tail feature regions. The middle section containing the thread helix feature was selected as the region of interest, and pixels in non-threaded areas were removed. The gray-level gradient of each pixel within the region of interest of the standard image was calculated using the Sobel operator, calculating the gray-level changes in the horizontal and vertical directions to determine the gradient magnitude and direction of each pixel. Based on the gradient magnitude and direction, the gradient was embedded into the standard image to obtain a gradient magnitude map. An initial threshold was set, and pixels with gradient magnitudes greater than the initial threshold were selected as the first pixel. The geometric center of the region of interest of the current fastener was calculated and used as the local dynamic origin of the current fastener. The coordinates of the first pixel were converted to relative coordinates based on the local dynamic origin. The gradient magnitude, gradient direction, and relative coordinates of the first pixel were combined to form the observation edge point set of the current fastener. Therefore, the edge points in the observation edge point set come from the boundary region between the thread contour of the part and the background.

[0013] S2: Calculate the average distance from the edge points in the observation edge point set to the geometric center of the fastener to obtain the outer radius observation value, calculate the difference between the outer radius observation value and the standard outer radius of the fastener to obtain the outer radius deviation feature, generate a physical embedding vector based on the outer radius deviation feature and the standard thread parameters, and generate multiple sets of theoretical edge points based on the physical embedding vector.

[0014] Obtain the standard pitch and standard outer radius of the fastener, and concatenate them to form the standard thread parameters. From the set of observed edge points, calculate the average distance of all observed edge points from the local dynamic origin, and use this average as the observed outer radius value of the current part. Calculate the difference between the observed outer radius value and the standard outer radius, and use this difference as the outer radius deviation feature. The outer radius deviation feature reflects the degree of deviation between the actual size of the current part and the standard size.

[0015] In this embodiment, generating multiple sets of theoretical edge point sets based on physical embedding vectors includes: Candidate angles are constructed by concatenating the outer radius deviation features, candidate angles, and standard thread parameters into an input vector. The input vector is then converted into a physical embedding vector corresponding to each candidate angle based on a physical encoder. The physical embedding vector is input into a linear mapping layer to calculate the modulation coefficient. Based on the helical curve equation containing the modulation coefficient, the coordinates of the theoretical edge points of the current sampling radius and their theoretical gradient directions are obtained. The theoretical edge points are then combined into a set of theoretical edge points.

[0016] In this embodiment, the candidate angle is constructed, including: Obtain the initial value of the current rotation angle of the current image frame, set the angle offset, add the angle offset to the initial value of the current rotation angle, subtract the angle offset, and keep it unchanged to obtain the candidate angle.

[0017] An angle offset is set, determined based on the expected attitude estimation error range. In this embodiment, the angle offset is set to 1 degree. The initial value of the current rotation angle is then increased by the angle offset, decreased by the angle offset, or left unchanged to obtain three candidate angles. The initial value of the current rotation angle is a predicted value of the unfolded attitude angle of the fastener's helical texture in the current image frame. For the first image frame, this value is obtained by detecting the principal direction of the thread using a Hough transform. For subsequent image frames, since the fastener translates with the conveyor belt and its own attitude is fixed, this value is equal to the corrected rotation angle of the previous frame; the specific correction method will be described later. However, due to the horizontal motion blur caused by the conveyor belt translation and the jitter of the local dynamic origin during tracking, the actual observed gradient direction of the thread edge in the current frame will still undergo a slight deflection. Therefore, this step still generates multiple candidate angles within the neighborhood of the initial value of the current rotation angle to cover and correct the aforementioned observation errors.

[0018] To address this bias, this embodiment generates three sets of theoretical edge points covering different angle assumptions. Subsequently, the edge points observed in the image are compared with these three sets of theoretical edge points, and the set with the highest matching degree is selected. The candidate angle corresponding to this set is the assumption closest to the actual rotation angle.

[0019] The thread parameter vector, outer radius deviation features, and each candidate angle are concatenated into an input vector, which is then input to the physical encoder. The physical encoder outputs a physical embedding vector corresponding to each candidate angle. The physical embedding vector encodes the geometric state at a specific rotation angle, combined with the current part's outer diameter deviation information. In this embodiment, the physical encoder includes an input layer, three hidden layers, and an output layer. The first fully connected layer maps the input dimension to 64 neurons, the second layer to 128 neurons, and the third layer to 64 neurons. Each fully connected layer is followed by a ReLU activation function. The output layer maps the 64-dimensional features of the third layer to a fixed-dimensional physical embedding vector. The physical encoder is further connected to a linear mapping layer, which is a single-layer fully connected network. The input is the physical embedding vector, and the output is modulation coefficients. Since the physical embedding vector contains the outer radius deviation features, the linear mapping layer can output corresponding modulation coefficients based on these deviation features. The modulation coefficients correct the standard helix formula and compensate for manufacturing deviations in the current part. When the outer radius deviation characteristic indicates that the actual outer diameter of the current part is greater than the standard outer radius, the modulation coefficient will increase the spiral unfolding speed accordingly, causing the generated theoretical edge point to expand outward, thus theoretically approaching the actual edge position of the current part.

[0020] The aforementioned model employs an end-to-end training approach. Specifically, SolidWorks is used to randomly generate multiple sets of 3D fastener models with different standard outer radii, pitches, and rotation angles within a preset range. These 3D models are then projected and rendered from a virtual camera perspective to obtain the coordinates of their standard theoretical edge points on a 2D plane. Each training sample includes randomly set thread parameters, outer diameter deviations, candidate angles, and corresponding standard theoretical edge point coordinates. These input parameters are concatenated and input to a physical encoder, which outputs a predicted physical embedding vector. This abstract vector is then input to a linear mapping layer, which outputs modulation coefficients. Using these modulation coefficients in conjunction with the input candidate angles, the theoretical edge point coordinates predicted by the current network are calculated using a spiral curve equation. By calculating the geometric reconstruction error between the predicted coordinates and the ground truth labels, a gradient descent optimizer is used for backpropagation iterative updates, thus completing the model training.

[0021] Then, within the edge pixel area covered by the fastener thread, N sampling radii are uniformly selected along the radial direction of the fastener. For each sampling radius... and candidate angles The image coordinates of the theoretical edge points are calculated based on the following spiral curve equation: , ,in, and Let i be the image coordinates of the i-th theoretical edge point. and The coordinates of the local dynamic origin of the current fastener are given. Let be the i-th sampling radius. Here, p represents the candidate angle, p is the standard pitch, and d is the standard outer radius. and is the modulation coefficient.

[0022] For each candidate angle, calculate the relative coordinates of N theoretical edge points according to the formula above. Simultaneously, calculate the theoretical gradient direction for each theoretical edge point. The theoretical gradient direction is the direction perpendicular to the tangent direction of the helical curve at that point. Combine the relative coordinates and theoretical gradient direction of the N theoretical edge points corresponding to each candidate angle to form a set of theoretical edge points. Since three candidate angles are generated, three sets of theoretical edge points are obtained.

[0023] Through the above steps, the system generates multiple hypotheses within the neighborhood of the current rotation angle estimate. Each hypothesis corresponds to a set of theoretical edge points, which is generated not only based on standard thread parameters but also incorporates the coarse observation bias of the current part, representing the theoretical reference position of the thread edge in the image at that hypothetical angle.

[0024] Since the thread structure of a fastener is a standard helix wrapped around the surface of a cylinder, substituting the sampling radius, candidate angle, and physical parameters into the helical curve equation yields the theoretical pixel coordinates that the actual thread edge should occupy in the camera's field of view under specific orientation and size assumptions. Finally, the theoretical edge points, along with their corresponding coordinates and gradient directions, form a set of theoretical edge points.

[0025] S3: Match the observed edge point set and the theoretical edge point set to generate matching point pairs, and calculate the difference feature vector of the matching point pairs based on relative coordinates and gradient direction.

[0026] In this embodiment, the difference feature vector of the matching point pair is calculated based on relative coordinates and gradient direction, including: Calculate the Euclidean distance between the relative coordinates of the observed edge points and the relative coordinates of the theoretical edge points in the matched point pair to obtain the positional deviation. Calculate the absolute value of the angle between the gradient direction of the observed edge points and the theoretical gradient direction of the theoretical edge points in the matched point pair to obtain the directional deviation. Concatenate the positional deviation and directional deviation along the feature dimension to obtain the difference features. Arrange the difference features of all matched point pairs corresponding to the same set of theoretical edge points to obtain the difference feature vector.

[0027] For each set of theoretical edge points, calculate the Euclidean distance between each observed edge point in the observed edge point set and all theoretical edge points in that set. Select the theoretical edge point with the smallest Euclidean distance and form a matching point pair with it. Repeat the above operation for all observed edge points in the observed edge point set to obtain all matching point pairs corresponding to that set of theoretical edge points.

[0028] For each pair of matched points, the Euclidean distance between the coordinates of the observed edge point and the coordinates of the theoretical edge point is calculated, and this Euclidean distance is used as the positional deviation. The positional deviation reflects the degree of spatial deviation between the actual detected edge position in the image and the edge position predicted by the physical model. The larger the positional deviation, the lower the match between the observed edge point and the theoretical prediction, which may be due to motion blur causing the edge position to shift.

[0029] Calculate the absolute value of the angle between the gradient direction of the observed edge point and the theoretical gradient direction of the theoretical edge point in the matching point pair. This absolute value is used as the direction deviation. The direction deviation reflects the degree of angular deviation between the observed edge direction and the edge direction predicted by the thread geometry. When the direction deviation is small, it indicates that the direction of grayscale change at the observed edge point is consistent with the normal direction of the thread profile at that location, and the observed point is likely from the actual thread edge.

[0030] Positional and directional deviations are concatenated to form the difference features of the matched point pairs. The difference features of all matched point pairs corresponding to the same set of theoretical edge points are then arranged to obtain the difference feature vector of the observed edge point set. Since there are three sets of theoretical edge points, the above matching process is performed separately for each set, ultimately obtaining three sets of matched point pairs and their corresponding three sets of difference feature vectors.

[0031] S4: Calculate the local attention weights based on the gradient direction and physical embedding vector of the matching point pair, and determine the pair with the largest local attention weights as the optimal matching point pair.

[0032] In this embodiment, the local attention weights are calculated based on the gradient direction of the matched point pairs and the physical embedding vector, including: Centered on the coordinates of the observed edge points in the matching point pair, a rectangular region of a preset size is extracted from the fastener image frame. The gradient magnitude of all pixels within the rectangular region is flattened into a one-dimensional vector and mapped to a query vector through a single-layer fully connected network. The physical embedding vector corresponding to the theoretical edge points in the matching point pair is extracted and mapped to a key vector through a single-layer fully connected network. Local attention weights are calculated based on the query vector and the key vector.

[0033] In the previous step, three sets of matching point pairs and their corresponding three sets of differential feature vectors were obtained. Each set of matching point pairs corresponds to a candidate angle hypothesis. The purpose of this step is to evaluate the overall degree of agreement between the observed edge and the theoretical edge under each hypothesis, and select the set with the highest degree of agreement from the three sets as the basis for subsequent edge correction and size measurement.

[0034] For each matching point pair, a rectangular region of a preset size is extracted from the gradient magnitude map of the standard image, centered on the coordinates of the observed edge point. The gradient magnitudes of all pixels within the rectangular region are flattened into a one-dimensional vector. To accurately evaluate the matching degree between the observed image features and the physical theoretical features, this invention introduces a cross-attention mechanism based on the Transformer architecture. Specifically, the Transformer architecture includes a single-layer fully connected network for mapping query vectors and key vectors. This single-layer fully connected network maps the flattened one-dimensional vector of gradient magnitudes into a query vector of fixed dimensions. The query vector encodes the image gradient distribution information around the observed edge point.

[0035] Extract the physical embedding vectors corresponding to the theoretical edge points in the matching point pair. Map these physical embedding vectors to fixed-dimensional key vectors using another single-layer fully connected network. The key vectors encode the physical model's description of the thread geometry under the candidate angle assumption.

[0036] Calculate the local attention weights for the matched point pair, based on the following formula: ,in, Let be the local attention weights for the i-th matching point pair. Let i be the query vector for the i-th matching point pair. Let D be the key vector of the i-th matching point pair, where D is the vector dimension.

[0037] Local attention weights reflect the degree of matching between local image features and the physical model description in a single matching point pair. When the gradient distribution around an observed edge point matches the spiral edge pattern expected by the physical model, the directions of the query vector and the key vector tend to be consistent, the dot product is larger, and the corresponding local attention weight is higher. When the observed edge point comes from a noisy region or background texture, the gradient distribution around it does not match the spiral pattern, the dot product is smaller, and the corresponding local attention weight is lower.

[0038] After obtaining the local attention weights for all matching point pairs within each group, the local attention weights within the same group are summed to obtain the group confidence score. The group confidence score represents the overall degree of agreement between the observed edge and the theoretical edge under a given candidate angle assumption. A higher group confidence score indicates that, under that candidate angle, the local image features of most observed edge points are consistent with the physical model description, and the candidate angle is closer to the true rotation angle of the fastener.

[0039] Compare the group confidence scores of the three groups, and determine the group with the highest group confidence score as the optimal theoretical edge group. The candidate angle corresponding to the optimal theoretical edge group is the hypothesis that is closest to the true rotation angle among the three candidate angles. All matching point pairs corresponding to the optimal theoretical edge group are determined as optimal matching point pairs, and the differential feature vector corresponding to the optimal theoretical edge group is determined as the optimal differential feature vector.

[0040] In one alternative embodiment, calculating local attention weights based on the gradient directions of the matched point pairs and the physical embedding vectors further includes: The absolute value of the angle between the gradient direction of the observed edge point in the matched point pair and the direction of the conveyor belt movement is obtained as the motion blur interference angle. If the motion blur interference angle is less than the blur interference threshold, the horizontal blur length is calculated based on the conveyor belt speed, camera exposure time, and camera intrinsic parameters. The horizontal blur length, direction deviation, and position deviation are concatenated into a blur interference vector. The blur interference vector is passed through a fully connected network and a Sigmoid activation function to obtain the blur suppression gate value. The theoretical gradient direction vector of the theoretical edge point is extracted. The scaled dot product of the key vector and the theoretical gradient direction vector is calculated and passed through the hyperbolic tangent function to obtain the self-consistency score. The blur suppression gate value is multiplied by the self-consistency score to obtain the local attention weight of the matched point pair.

[0041] In this embodiment, since the fastener moves at a constant speed horizontally along the conveyor belt, the motion blur generated in the image frame has strict directionality, meaning it only produces a trailing effect along the horizontal direction of the image coordinate system, while the vertical direction remains unaffected. Simultaneously, the projection of the fastener's spiral texture into the two-dimensional image results in different angles between the edge tangent direction and the horizontal movement direction at different locations. When the thread edge approaches horizontal, the gradient features at that edge are severely contaminated by motion blur; when the thread edge approaches vertical, the gradient features at that edge are almost unaffected by motion blur. Therefore, using the same dot product formula to calculate the local attention weights for all matching point pairs will lead to systematic evaluation bias due to the varying degrees of motion blur interference experienced by different matching point pairs.

[0042] To address the aforementioned issues, in one optional embodiment, the gradient direction of the observed edge points in the matched point pair is obtained, and the absolute value of the angle between the gradient direction and the conveyor belt's movement direction is calculated. This absolute value is used as the motion fuzzy interference angle. A fuzzy interference threshold is set; if the motion fuzzy interference angle is greater than the threshold, the matched point pair is determined to be in a low-fuzzy interference state. In the low-fuzzy interference state, the angle between the gradient direction of the observed edge points and the horizontal movement direction is large, indicating that the threaded edge at that location deviates from the horizontal direction. The trailing effect generated by the horizontal movement of the conveyor belt has less interference with the gradient magnitude and gradient direction of this edge, and the observed image gradient features have high fidelity. At this time, the local attention weight is still calculated using the dot product of the query vector and the key vector as described above.

[0043] If the motion blur interference angle is less than or equal to the blur interference threshold, the matching point pair is determined to be in a high blur interference state. In the high blur interference state, the angle between the gradient direction of the observed edge point and the horizontal motion direction is small, indicating that the thread edge at that location is close to the horizontal direction and nearly parallel to the conveyor belt's motion direction. At this time, the horizontal tailing effect generated by the uniform motion of the conveyor belt stretches and mixes the grayscale value of the pixel along the horizontal direction, causing both the extracted gradient magnitude and gradient direction to deviate from the true values. Under this disclosure, the horizontal blur length is estimated based on the conveyor belt speed and camera exposure time. The conveyor belt speed and camera exposure time are both system preset parameters, and the horizontal blur length is... Calculated based on the following formula: Where v is the conveyor belt speed, measured in millimeters per second. This refers to the camera exposure time, in seconds. This is the focal length parameter in the horizontal direction of the camera intrinsic parameter matrix, expressed in pixels per millimeter. The physical meaning is the number of pixels in the image corresponding to the horizontal displacement of the fastener during a single frame exposure time, which is the horizontal tail length of the motion blur kernel.

[0044] Next, the directional and positional deviations of the matched point pair are extracted, and the horizontal blur length, directional deviation, and positional deviation are concatenated into a blur interference vector. The theoretical gradient directions of the theoretical edge points in the matched point pair are extracted, and these theoretical gradient directions are mapped to theoretical gradient direction vectors with the same dimension as the key vectors through a single-layer fully connected network. The local attention weights are then calculated based on the following formula: ,in, Let be the local attention weights for the i-th matching point pair. It is the Sigmoid activation function. The weight matrix is ​​a learnable matrix. For learnable bias vectors, For fuzzy interference vectors, Let i be the key vector of the i-th matching point pair. Let D be the theoretical gradient direction vector of the theoretical edge point in the i-th matching point pair, where D is the vector dimension.

[0045] In this formula, the first factor For fuzz suppression gating, when the horizontal fuzz length Large and directional deviation When the value is large, it indicates that motion blur has a significant impact on the point and the observed gradient direction has deviated significantly from the theoretical direction. The output value of the blur suppression gate approaches 0, thus greatly reducing the upper limit of the local attention weight of the matched point pair and preventing contaminated observation features from obtaining false high confidence. The second factor... The theoretical gradient direction information encoded by the physical embedding vector is used to replace the contaminated image observation features in the evaluation. The geometric self-consistency of the physical model is measured by the scaled dot product of the key vector and the theoretical gradient direction vector. When the prediction of the physical model at this position is highly consistent with the theoretical gradient direction, the factor approaches 1.

[0046] The learnable parameters in the aforementioned conditional attention computation mechanism, along with the original single-layer fully connected network and decoupled network used for mapping query vectors and key vectors, are trained end-to-end as a whole model to obtain optimal parameters. During training, in a simulated image training set that simulates conveyor belt motion blur, training samples with different levels of motion blur are generated by setting different conveyor belt speeds and exposure time parameters, ensuring that the network can learn to correctly evaluate attention weights under different blur interference states.

[0047] S5: Input the difference features of the optimal matching point pair and the local attention weights into the decoupled network, and output the pose correction, correction vector and edge confidence. Based on the correction vector, correct the coordinates of the observed edge points to obtain the decontamination edge point set.

[0048] In this embodiment, the output of attitude correction, correction vector, and edge confidence includes: The difference features of the best matching point pairs are concatenated with the local attention weights to form a fusion feature. The fusion features of all the best matching point pairs are arranged into a fusion feature vector. The decoupled network includes a Transformer encoder. The fusion feature vector is input into the Transformer encoder, which extracts global geometric constraint features. The Transformer encoder includes a first output head, a second output head, and a third output head. The first output head obtains the pose correction amount based on the position deviation of all the best matching point pairs and the corresponding local attention weights. The second output head calculates the correction vector based on its own weight matrix and global geometric constraint features. The third output head calculates the edge confidence based on global geometric constraint features and bias scalar.

[0049] Extract the differential features and corresponding local attention weights for each of the optimal matching point pairs. Concatenate the differential features and local attention weights along the feature dimension to obtain the fused features. Arrange the fused features of all optimal matching point pairs into a fused feature vector.

[0050] The fused feature vector is input into the decoupled network. The decoupled network consists of a two-layer Transformer encoder structure, with each Transformer layer containing a multi-head self-attention mechanism and a feedforward neural network. After the fused feature vector passes through the first Transformer layer, the network learns the spatial correlation between different matching point pairs. The second Transformer layer further extracts edge features under global geometric constraints. The output of the two Transformer layers is fed into three parallel output heads, each of which is a single-layer fully connected network.

[0051] The first output head outputs a pose correction, representing the deviation between the initial value of the current rotation angle and the actual rotation angle. The first output head is a single-layer fully connected network. During training, the loss function of this output head is the weighted sum of the positional deviations of all matched point pairs, where the weights are determined by the corresponding local attention weights. Matching point pairs with higher local attention weights contribute more to the loss function, thus guiding the network to learn an accurate pose deviation mapping.

[0052] The second output head outputs a position correction vector for each optimal matching point pair. The second output head is a single-layer fully connected network, and the two-dimensional position correction vector is calculated based on the following formula. ,in, Let be the correction vector for the position of the i-th matching point pair. This is the weight matrix for the second output head. This is the bias vector for the second output head. This represents the global geometric constraint features for the i-th matching point pair. The weight matrix of the second output head is learned and determined during the training phase. The position correction vector represents the amount of displacement that needs to be applied to the observed edge points in the image coordinate system; the direction and magnitude of this displacement are learned by the decoupling network based on the fusion features. The role of the two-dimensional position correction vector is to adjust the position of the observed edge points affected by motion blur back to their true positions.

[0053] The third output head outputs the edge confidence for each optimal matching point pair. The third output head is a single-layer fully connected network, and the edge confidence is calculated based on the following formula: ,in, Let be the edge confidence of the i-th matching point pair. It is the sigmoid activation function. This is the weight vector of the third output head. This is the bias scalar for the third output head. The global geometric constraint features of the i-th matching point pair after two layers of Transformer encoding. Edge confidence reflects the degree of confidence of the decoupled network that the observed edge point originates from the real thread edge. When the positional deviation, directional deviation, and local attention weight of an observed edge point are small, the edge confidence of that point is close to 1.

[0054] The coordinates of the observed edge points in each optimal matching point pair are obtained. These coordinates are then added to the corresponding position correction vector to obtain the coordinates of the cleaned edge points. A set of cleaned edge points is formed by the coordinates of all cleaned edge points and their corresponding edge confidence scores. Each point in the cleaned edge point set has coordinates corrected based on physical model constraints compared to its corresponding point in the observed edge point set, making it closer to the true position of the fastener thread edge.

[0055] It is important to emphasize that the single-layer fully connected network used to map query vectors and key vectors is trained as an integral model with the decoupled network, employing an end-to-end joint training approach to obtain optimal parameters. Specifically, a training set of simulated images is first constructed, encompassing various dimensional tolerances, rotational poses, and complex conditions such as conveyor belt motion blur and local occlusion. The true rotation angle and uncontaminated, reliable edge point coordinates for each image are then obtained. Subsequently, during the model's forward propagation, the extracted image gradient features and physical embedding vectors are input into the network to calculate initial local attention weights. These weights are then fused with the difference features and input into the decoupled network, outputting the current pose correction, two-dimensional position correction vector, and edge confidence.

[0056] Subsequently, a multi-task loss function is constructed for backpropagation. This function includes pose regression loss, position correction loss, and edge classification loss. The pose regression loss uses an L1 loss function to ensure the predicted pose correction is consistent with the actual pose deviation. The position correction loss uses a mean squared error loss function to penalize the error between the network's output correction vector and the actual offset from the observation point to the true standard point. The edge classification loss uses a binary cross-entropy loss function, using whether a point is a true edge as the ground truth label to supervise the edge confidence of the network output. Finally, the Adam optimizer is used to iteratively update the physical encoder, fully connected mapping layer, and the weight matrix and bias terms within the decoupled network using gradient descent. This is achieved through joint training.

[0057] S6: Calculate the outer diameter based on the reliable edge points selected from the set of cleaned edge points according to the edge confidence.

[0058] The initial value of the current rotation angle is added to the attitude correction amount to obtain the corrected rotation angle corresponding to the current image frame. This corrected rotation angle is then used as the initial value of the current rotation angle for the next image frame. Edge points with an edge confidence greater than a preset confidence threshold are selected from the set of cleaned edge points, forming a set of reliable edge points. In this embodiment, the confidence threshold is set to 0.6. Least-squares circle fitting is performed on the relative coordinates of all edge points in the set of reliable edge points to find the center coordinates and pixel radius of the circle that minimizes the sum of the squares of the distances from all reliable edge points to the fitted circle. The specific fitting method is existing technology and will not be described here. Based on the camera calibration parameters, the pixel radius is converted into a physical size, and the physical size is multiplied by 2 to obtain the outer diameter of the fastener.

[0059] like Figure 2 As shown, the present invention also provides a fastener dimension measurement method and system based on image recognition. This system is used to implement the method described above, and includes: The edge extraction module acquires fastener image frames, extracts edge points from the fastener image frames, and constructs a set of observed edge points containing relative coordinates and gradient directions.

[0060] The model encoding module calculates the average distance from the edge points in the observation edge point set to the geometric center of the fastener to obtain the outer radius observation value, calculates the difference between the outer radius observation value and the standard outer radius of the fastener to obtain the outer radius deviation feature, generates a physical embedding vector based on the outer radius deviation feature and standard thread parameters, and generates multiple sets of theoretical edge point sets based on the physical embedding vector.

[0061] The feature matching module matches the observed edge point set and the theoretical edge point set to generate matching point pairs, and calculates the difference feature vector of the matching point pairs based on relative coordinates and gradient direction.

[0062] The attention evaluation module calculates local attention weights based on the gradient direction and physical embedding vector of the matching point pair, and determines the pair with the largest local attention weight as the optimal matching point pair.

[0063] The decoupling correction module inputs the difference features of the optimal matching point pair and the local attention weights into the decoupling network, and outputs the pose correction amount, correction vector and edge confidence. Based on the correction vector, the coordinates of the observed edge points are corrected to obtain the set of decontamination edge points.

[0064] The size calculation module selects reliable edge points from the set of cleaned edge points based on edge confidence, and calculates the outer diameter based on the reliable edge points.

[0065] Finally, it should be emphasized that the various thresholds, algorithm model hyperparameters, and normalization methods mentioned in the above embodiments of the present invention, as well as those involved in the feature vector processing, are all conventional techniques used by those skilled in the art. When applying the present invention in practice, those skilled in the art can adjust, test, and empirically set the most suitable values ​​or methods for the current application scenario through limited conventional experiments, based on specific working conditions (such as the specifications of fasteners, different conveyor belt speeds, and the specific resolution and exposure parameters of industrial cameras).

Claims

1. A method for measuring the dimensions of fasteners based on image recognition, characterized in that, include: Acquire fastener image frames, extract edge points from the fastener image frames, and construct a set of observed edge points containing relative coordinates and gradient directions; The outer radius observation value is obtained by calculating the average distance from the edge point in the observation edge point set to the geometric center of the fastener. The outer radius deviation feature is obtained by calculating the difference between the outer radius observation value and the standard outer radius of the fastener. A physical embedding vector is generated based on the outer radius deviation feature and the standard thread parameters. Multiple sets of theoretical edge points are generated based on the physical embedding vector. The observed edge point set and the theoretical edge point set are matched to generate matching point pairs. The difference feature vector of the matching point pairs is calculated based on the relative coordinates and gradient direction. The local attention weights are calculated based on the gradient direction and physical embedding vector of the matching point pair, and the pair with the largest local attention weights is determined as the optimal matching point pair. The differential features of the optimal matching point pair and the local attention weights are input into the decoupled network, and the outputs the pose correction, correction vector and edge confidence. The coordinates of the observed edge points are corrected based on the correction vector to obtain the set of decontamination edge points. The outer diameter is calculated based on the reliable edge points selected from the set of cleaned edge points based on the edge confidence.

2. The method according to claim 1, characterized in that, To generate multiple sets of theoretical edge points based on physical embedding vectors, the process includes: constructing candidate angles, concatenating the outer radius deviation features, candidate angles, and standard thread parameters into an input vector, converting the input vector into a physical embedding vector corresponding to each candidate angle based on a physical encoder, inputting the physical embedding vector into a linear mapping layer, calculating and obtaining the modulation coefficient, obtaining the coordinates of the theoretical edge points of the current sampling radius and their theoretical gradient directions based on the helical curve equation containing the modulation coefficient, and forming a set of theoretical edge points.

3. The method according to claim 2, characterized in that, Constructing candidate angles includes: obtaining the initial value of the current rotation angle of the current image frame, setting the angle offset, adding the angle offset to the initial value of the current rotation angle, subtracting the angle offset, and keeping it unchanged to obtain candidate angles.

4. The method according to claim 1, characterized in that, The difference feature vectors of matching point pairs are calculated based on relative coordinates and gradient direction, including: Calculate the Euclidean distance between the relative coordinates of the observed edge points and the relative coordinates of the theoretical edge points in the matching point pair to obtain the positional deviation. Calculate the absolute value of the angle between the gradient direction of the observed edge points and the theoretical gradient direction of the theoretical edge points in the matching point pair to obtain the directional deviation. Concatenate the positional deviation and directional deviation along the feature dimension to obtain the difference features. Arrange the difference features of all matching point pairs corresponding to the same set of theoretical edge points to obtain the difference feature vector.

5. The method according to claim 4, characterized in that, Local attention weights are calculated based on the gradient directions of the matched point pairs and the physical embedding vectors, including: Centered on the coordinates of the observed edge points in the matching point pair, a rectangular region of a preset size is extracted from the fastener image frame. The gradient magnitude of all pixels within the rectangular region is flattened into a one-dimensional vector and mapped to a query vector through a single-layer fully connected network. The physical embedding vector corresponding to the theoretical edge points in the matching point pair is extracted and mapped to a key vector through a single-layer fully connected network. Local attention weights are calculated based on the query vector and the key vector.

6. The method according to claim 5, characterized in that, The calculation of local attention weights based on the gradient direction of the matched point pairs and the physical embedding vector also includes: The absolute value of the angle between the gradient direction of the observed edge point in the matched point pair and the direction of the conveyor belt movement is obtained as the motion blur interference angle. If the motion blur interference angle is less than the blur interference threshold, the horizontal blur length is calculated based on the conveyor belt speed, camera exposure time, and camera intrinsic parameters. The horizontal blur length, direction deviation, and position deviation are concatenated into a blur interference vector. The blur interference vector is passed through a fully connected network and a Sigmoid activation function to obtain the blur suppression gate value. The theoretical gradient direction vector of the theoretical edge point is extracted. The scaled dot product of the key vector and the theoretical gradient direction vector is calculated and passed through the hyperbolic tangent function to obtain the self-consistency score. The blur suppression gate value is multiplied by the self-consistency score to obtain the local attention weight of the matched point pair.

7. The method according to claim 5, characterized in that, Output attitude correction, correction vector, and edge confidence, including: The difference features of the best matching point pairs are concatenated with the local attention weights to form a fusion feature. The fusion features of all the best matching point pairs are arranged into a fusion feature vector. The decoupled network includes a Transformer encoder. The fusion feature vector is input into the Transformer encoder, which extracts global geometric constraint features. The Transformer encoder includes a first output head, a second output head, and a third output head. The first output head obtains the pose correction amount based on the position deviation of all the best matching point pairs and the corresponding local attention weights. The second output head calculates the correction vector based on its own weight matrix and global geometric constraint features. The third output head calculates the edge confidence based on global geometric constraint features and bias scalar.

8. A fastener dimension measurement system based on image recognition, used to implement the method as described in any one of claims 1-7, characterized in that, The system includes: The edge extraction module acquires fastener image frames, extracts edge points from the fastener image frames, and constructs a set of observed edge points containing relative coordinates and gradient directions. The model encoding module calculates the average distance from the edge points in the observation edge point set to the geometric center of the fastener to obtain the outer radius observation value, calculates the difference between the outer radius observation value and the standard outer radius of the fastener to obtain the outer radius deviation feature, generates a physical embedding vector based on the outer radius deviation feature and standard thread parameters, and generates multiple sets of theoretical edge point sets based on the physical embedding vector. The feature matching module matches the observed edge point set and the theoretical edge point set to generate matching point pairs, and calculates the difference feature vector of the matching point pairs based on relative coordinates and gradient direction; The attention evaluation module calculates local attention weights based on the gradient direction and physical embedding vector of the matching point pair, and determines the pair with the largest local attention weights as the optimal matching point pair. The decoupling correction module inputs the difference features of the optimal matching point pair and the local attention weights into the decoupling network, and outputs the pose correction amount, correction vector and edge confidence. Based on the correction vector, the coordinates of the observed edge points are corrected to obtain the set of decontamination edge points. The size calculation module selects reliable edge points from the set of cleaned edge points based on edge confidence, and calculates the outer diameter based on the reliable edge points.

Images