A machine vision-based fastener assembly anomaly detection method

Abstract

The application discloses a kind of fastener assembly abnormality detection methods based on machine vision, comprising: collecting the image of correct assembly fastener, and obtaining template image set after pre-processing;Geometric frame segmentation labels each component part of fastener, and calculates its component ratio and relative fastener main body distance, constructs detection template library;Collect the assembly image of fastener to be measured, and after pre-processing, input into trained semantic segmentation model, and calculate the component ratio R of each component part i With relative fastener main body distance D i ;The component ratio and relative fastener main body distance distribution corresponding to each component part contained in fastener to be measured in detection template library are counted, and distribution range is set;Whether R i And D i Whether in distribution range, if yes, then determine that the fastener to be measured is assembled normally, if no, then determine that the fastener to be measured is assembled abnormally.The application improves the robustness and accuracy of fastener assembly abnormality detection.

Description

Technical Field

[0001] This invention relates to the field of machine vision and image processing technology, and in particular to a method for detecting fastener assembly anomalies based on machine vision. Background Technology

[0002] With the increasing demands for industrial automation and assembly precision, fasteners are widely used in machinery, electronics, and transportation. The correctness of fastener assembly directly affects the safety and stability of equipment operation. Traditional inspection methods mainly rely on manual inspection, which suffers from low efficiency, high cost, and poor consistency. Therefore, there is a need for an efficient and automated method for detecting fastener assembly anomalies.

[0003] Machine vision technology is an interdisciplinary field involving artificial intelligence, neurobiology, psychophysics, computer science, image processing, pattern recognition, and many other areas. Machine vision primarily uses computers to simulate human visual functions, extracting information from images of objective objects, processing and understanding it, and ultimately using it for practical detection, measurement, and control, meeting the needs of efficient and automated inspection. In existing technologies, computer vision technology has been used for fastener inspection. However, most methods are based on simple template matching, which is easily limited by lighting, shooting angle, and background interference, resulting in insufficient detection accuracy. Therefore, there is an urgent need for a high-precision inspection method that can automatically detect the correct assembly of fasteners under different shooting conditions.

[0004] On the other hand, fasteners are key connecting components in mechanical assembly, and their correct assembly directly affects the operational safety of equipment. Existing machine vision-based inspection methods mainly focus on low-dimensional semantic information such as whether bolts are loose or damaged, and give little consideration to the high-dimensional logical relationships of fastener assembly. Therefore, there is a need for an inspection method that can accurately understand whether the overall fastener assembly is correct. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a machine vision-based method for detecting fastener assembly anomalies, which improves the accuracy of fastener assembly anomaly detection under varying lighting and viewing angles.

[0006] The specific technical solution is as follows:

[0007] A machine vision-based method for detecting fastener assembly anomalies includes the following steps:

[0008] S1: Collect images of correctly assembled fasteners from different angles and under different lighting conditions, preprocess the images to obtain template images, and the collection of all template images forms a template image set;

[0009] S2: Perform geometric bounding box segmentation and annotation on each component of the fastener in the template image set to obtain a semantic segmentation dataset; calculate the component proportion and relative distance of each component to the fastener body to construct a detection template library; the component proportion is the ratio of the pixel area of ​​a single component segmentation region to the pixel area of ​​all component segmentation regions in the corresponding template image.

[0010] S3: Build and use a semantic segmentation dataset to train a semantic segmentation model for segmenting the components of a fastener;

[0011] S4: Acquire assembly images of the fasteners to be tested and preprocess them, then input them into the trained semantic segmentation model to segment each component.

[0012] S5: Calculate the component proportion R of each part based on the segmentation results. i Distance D from the relative fastener body i ;

[0013] S6: Filter the component data from the test template library to find components of the same type as those in the fastener to be tested, and calculate the component proportion and relative distance to the fastener body for each component. Based on the statistical values, set the distribution range of component proportion and relative distance to the fastener body. Judge the R calculated in S5. i and D i If all components are within the set distribution range, the fastener under test is determined to be assembled normally; otherwise, it is determined to be assembled abnormally.

[0014] Furthermore, in S1, the preprocessing operations include: ROI region cropping, noise reduction, and edge enhancement.

[0015] Furthermore, the expression for the component proportions of the constituent parts is as follows:

[0016]

[0017] In the formula, R i A represents the component percentage of the i-th component. i N represents the total pixel area of ​​the segmented region of the i-th component. C A represents the number of segmented regions in the corresponding template image. M This represents the total pixel area of ​​the segmented regions of all constituent parts in the corresponding template image;

[0018] The expression for the relative fastener body distance is as follows:

[0019]

[0020] In the formula, D iC represents the distance between the i-th component and the fastener body. i C represents the centroid coordinates of the segmented region of the i-th component. M This represents the centroid coordinates of the segmented regions of all constituent parts in the corresponding template image, and ||·| represents the Euclidean distance.

[0021] Further, step S2 includes the following sub-steps:

[0022] S2.1: Traverse each template image in the template image set, perform geometric bounding box segmentation and annotation on various component parts related to fastener assembly in the template images, and store the annotated template images into the semantic segmentation dataset; the individual part regions segmented by the geometric bounding boxes are the segmentation regions;

[0023] S2.2: Based on the geometric box segmentation and annotation results, calculate the component proportion of each component in the corresponding template image and the distance to the fastener body;

[0024] S2.3: Store each component part, its corresponding component ratio, and the distance to the main fastener body in the detection template library.

[0025] Further, step S3 includes the following sub-steps:

[0026] S3.1: Construct the initial semantic segmentation model, including a feature extractor and a segmentation predictor;

[0027] S3.2: Configure a multi-stage hybrid cross loss function and train an initial semantic segmentation model using data from the semantic segmentation dataset to obtain a trained semantic segmentation model;

[0028] The expression for the multi-stage hybrid crossover loss function is as follows:

[0029] L=λ1L Dice +λ2L CE +λ3L APC

[0030]

[0031] In the formula, L represents the multi-stage hybrid crossover loss function, L Dice L represents the similarity loss function, which participates in all stages of training; CE L represents the cross-loss entropy function, which participates in all stages of training; APC λ1, λ2, and λ3 represent the loss functions of the component parts; λ1, λ2, and λ3 represent the hyperparameters of the corresponding preset loss functions. α represents the total pixel area of ​​the segmented region of the i-th component predicted by the initial semantic segmentation model, and α represents the preset adjustment ratio coefficient.

[0032] Furthermore, the feature extractor is a pre-trained DenseNet201 network, whose features {S1, S2, S3} in the first three stages are used as multi-level features.

[0033] The segmentation predictor includes a Conv3×3-BN-ReLU module, a bilinear upsampling module, and an upsampling-Softmax module, used to process the multi-level features to output the segmented regions of each component.

[0034] The third-level feature S3 is processed by a Conv3×3-BN-ReLU module and a bilinear upsampling module, and then fused with the second-level feature S2 through a concatenation operation. The fused feature is then processed by another Conv3×3-BN-ReLU module and a bilinear upsampling module, and then fused with the first-level feature S1 through a concatenation operation. Finally, the fused feature is processed by an upsampling-Softmax module to output a segmentation prediction map.

[0035] Furthermore, random image enhancement operations are introduced before each training iteration to train the semantic segmentation model; the random image enhancement operations include: image rotation, scaling, offsetting, and mirroring.

[0036] Furthermore, in step S6, the statistical values ​​include the distribution mean and standard deviation, and the distribution mean of the component proportion corresponding to the i-th component is calculated. and standard deviation Mean of distance distribution relative to fastener body and standard deviation The distribution range of the component proportions is set as follows The distribution range of the distance between the fastener body and the fastener body is as follows

[0037] When making a judgment, if the following conditions are met and If the fastener is assembled correctly, it is determined that the fastener is assembled normally; otherwise, it is determined that the fastener is assembled abnormally.

[0038] Furthermore, in S1, an industrial camera is used to capture images of correctly assembled fasteners from different angles and under different lighting conditions. The industrial camera used has a resolution of 1920×1200 and a color gamut of three-channel RGB.

[0039] The beneficial effects of this invention are:

[0040] (1) The present invention collects images of fastener assembly to be tested, and determines whether there are any missing or incorrectly assembled components of the fastener under the correct assembly condition of the corresponding type of fastener in the constructed detection template library. The detection is carried out from the perspective of the overall assembly logic, which improves the robustness and accuracy of fastener assembly anomaly detection.

[0041] (2) In the process of constructing the detection template library, this invention improves the diversity of fastener assembly instance distribution and reduces the false detection rate by collecting images of correctly assembled fasteners from different perspectives and under different lighting conditions.

[0042] (3) The present invention extracts the location of each component in the fastener assembly by semantic segmentation model and filters out areas that are not related to the fastener assembly, which greatly reduces environmental interference during the detection process and improves the robustness of fastener assembly anomaly detection. Attached Figure Description

[0043] Figure 1 This is a flowchart of the steps of the fastener assembly anomaly detection method based on machine vision in an embodiment of the present invention.

[0044] Figure 2 This is a sub-flowchart of step S2 in an embodiment of the present invention.

[0045] Figure 3 This is a schematic diagram of the organization of a detection template library provided in an embodiment of the present invention.

[0046] Figure 4 This is a schematic diagram of the structure of the semantic segmentation model in this embodiment of the invention and the entire process of a fastener assembly anomaly detection method provided. Detailed Implementation

[0047] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The objectives and effects of the present invention will become clearer as a result. The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0048] like Figure 1 As shown, a fastener assembly anomaly detection method based on machine vision includes the following steps:

[0049] S1: Use an industrial camera to capture images of correctly assembled fasteners from different angles and under different lighting conditions. After preprocessing the images, template images are obtained. All template images constitute a template image set for normal assembly.

[0050] In this embodiment, an industrial camera with a resolution of 1920×1200 and a three-channel RGB color gamut is used. Preprocessing includes ROI region cropping, noise reduction, and edge enhancement to eliminate uneven lighting and noise interference, ultimately obtaining a clear fastener image. The preprocessed images constitute a normal assembly template image set, serving as standard reference data for subsequent inspection. Furthermore, this embodiment focuses on assemblies of the same type of fastener, defined here as Class A fasteners.

[0051] S2: Perform geometric bounding box segmentation and annotation on each component of the fastener in each template image in the template image set to obtain a semantic segmentation dataset, and calculate the component proportion and relative distance of each component to the fastener body to construct a detection template library.

[0052] like Figure 2 As shown, S2 is implemented through the following sub-steps:

[0053] S2.1: Perform geometric bounding box segmentation and annotation on various component parts related to fastener assembly in each template image of the template image set. The annotation information includes the type of part, etc.; store the annotated template images in the semantic segmentation dataset. Hereinafter, the single part region segmented by a single geometric bounding box is referred to as the segmentation region.

[0054] In this embodiment, the various components related to fastener assembly include bolts, nuts, washers, cotter pins, etc. Labelme software is used to draw polygonal bounding boxes to segment and label the area of ​​each component, with other unlabeled image areas serving as the background. After labeling, Labelme software generates a JSON file containing all the labeling information; this JSON file is then converted to PASCAL VOC dataset format. The labeled template image and the corresponding PASCAL VOC file are used as a semantic segmentation dataset, which is proportionally divided into training, validation, and test sets for training subsequent semantic segmentation models.

[0055] S2.2: Based on the above geometric box segmentation and annotation results, calculate the component proportion of each component in the corresponding template image and the distance to the fastener body.

[0056] The proportion of a component in the corresponding template image, which is the ratio of the total pixel area of ​​the segmented region of that component to the total pixel area of ​​the segmented regions of all components in the corresponding template image, is expressed as follows:

[0057]

[0058] In the formula, R i A represents the component percentage of the i-th component. i N represents the total pixel area of ​​the segmented region of the i-th component. C A represents the number of segmented regions in the corresponding template image. M This represents the total pixel area of ​​the segmented regions of all constituent parts in the corresponding template image.

[0059] The distance between the component and the fastener body is calculated by comparing the centroid coordinates of the segmented regions of each component with the overall centroid of all segmented regions of the corresponding component in the template image. The expression is as follows:

[0060]

[0061] In the formula, D i C represents the distance between the i-th component and the fastener body. i C represents the centroid coordinates of the segmented region of the i-th component. M This represents the centroid coordinates of the segmented regions of all constituent parts in the corresponding template image, and ||·| represents the Euclidean distance.

[0062] S2.3: Store each component part, its corresponding component ratio, and the distance to the main fastener body in the detection template library.

[0063] In this embodiment, the organization of the detection template library corresponding to Class A fasteners is as follows: Figure 3 As shown, its organization includes: fastener assembly inspection template library - template images - component parts - component proportion + distance from the fastener body.

[0064] S3: Construct and train semantic segmentation models for each component of the fastener based on the data in the semantic segmentation dataset. Preferably, S3 is implemented through the following sub-steps:

[0065] S3.1: Construct the initial semantic segmentation model, such as Figure 4 As shown, the model includes a feature extractor and a segmentation predictor.

[0066] Preferably, a pre-trained DenseNet201 network is introduced as the feature extractor of the semantic segmentation model, and the features of its first three stages {S1, S2, S3} are selected as multi-level features.

[0067] A segmentation predictor is formed by introducing a Conv3×3-BN-ReLU module, a bilinear-upsampling module, and an upsampling-softmax module to process multi-level features and output segmented regions for each component.

[0068] Specifically, in this embodiment, after the input image is extracted by the feature extractor to obtain multi-level features {S1, S2, S3}, the third-level feature S3 is processed by a Conv3×3-BN-ReLU module and a Bilinear-Upsampling module, and then fused with the second-level feature S2 through a concatenation operation. The fused feature is then processed by another Conv3×3-BN-ReLU module and a Bilinear-Upsampling module, and then fused with the first-level feature S1 through a concatenation operation. Finally, the fused feature is processed by an UpSampling-Softmax module to output a segmentation prediction map.

[0069] S3.2: Configure a multi-stage hybrid cross-loss function and train an initial semantic segmentation model using data from the semantic segmentation dataset to obtain a trained semantic segmentation model; the training process of the semantic segmentation model is as follows: Figure 4 As shown.

[0070] Specifically, a multi-stage hybrid cross-loss function is used to calculate the loss of the segmentation model during the training phase. In this embodiment, the training phase consists of 100 iterations, divided into two phases, each including 50 iterations. The multi-stage hybrid cross-loss function is configured as follows:

[0071] L=λ1L Dice +λ2L CE +λ3L APC

[0072]

[0073] In the formula, L represents the multi-stage hybrid crossover loss function, L Dice L represents the similarity loss function. CE L represents the cross-loss entropy function. APC L represents the component loss function of the part; Dice With L CE Participating in all training phases (i.e., 100 iterations), L APC It only participates in the second phase of training (i.e., the last 50 iterations); λ1, λ2, and λ3 represent the hyperparameters of the corresponding loss functions (fixed values ​​set manually); α represents the total pixel area of ​​the segmented region of the i-th component predicted by the initial semantic segmentation model, and α represents the preset adjustment ratio coefficient.

[0074] Furthermore, in this embodiment, random image enhancement operations are introduced before each training iteration to train the semantic segmentation model. The random image enhancement operations include image rotation, scaling, offsetting, and mirroring to improve the generalization of the semantic segmentation model.

[0075] In the process of training the semantic segmentation model, this invention improves the segmentation accuracy of the semantic segmentation model by using a multi-stage hybrid cross loss function and random image enhancement operations, thereby ensuring the feasibility of extracting segmented regions of each component of the fastener and ultimately improving the accuracy of fastener assembly anomaly detection.

[0076] S4: Acquire the assembly image of the fastener to be tested and perform preprocessing operations on it. Input the image into the trained semantic segmentation model to extract the segmentation regions of each component of the fastener to be tested.

[0077] Specifically, in this embodiment, an industrial camera is used to acquire assembly images of the fasteners to be tested. The preprocessing operation is the same as in S1, and ROI region cropping, noise reduction and edge enhancement operations are performed on the acquired assembly images of the fasteners to be tested. Then, the preprocessed image is input into the trained semantic segmentation model to obtain the segmentation regions of each component related to the assembly of the fasteners to be tested.

[0078] S5: Based on the segmented regions of each component in the image of the fastener to be tested, the component proportion R of each component is calculated using the method in S2. i Distance D from the relative fastener body i .

[0079] S6: Statistically analyze the component proportions and relative distances to the fastener body of each component in the test template library (corresponding to the mean μ and standard deviation σ). Combine this with the component proportions and relative distances to the fastener body calculated in S5 to perform multi-dimensional analysis and output the fastener assembly judgment result. S6 includes the following sub-steps:

[0080] S6.1: Filter the data corresponding to the component parts with the same type as the component parts contained in the fastener to be tested from the test template library, and calculate the mean of the component proportion distribution corresponding to the i-th component part. with standard deviation and the corresponding average distribution of relative fastener body distance with standard deviation

[0081] S6.2: Analyze each component of the fastener image to be tested sequentially from two dimensions: component proportion and distance from the fastener body. If, for the i-th component, the analysis in both dimensions does not exceed the set distribution range, that is, the component proportion R calculated in S5 is considered valid. i Within the defined distribution range, and at a distance D relative to the fastener body i If the fastener assembly is within the set distribution range, it is determined that the fastener assembly is normal; otherwise, it is determined that the assembly is abnormal.

[0082] In this embodiment, the distribution range is set to μ±3σ. If for the i-th component, its component proportion dimension satisfies... And in the dimension of relative distance to the fastener body, it satisfies If the fastener is correctly assembled, it is determined that there are no mis-assembled or missing parts; otherwise, it is determined that the fastener assembly is abnormal.

[0083] In this embodiment, the segmentation effect of the trained semantic segmentation model on the preprocessed fastener assembly image is as follows: Figure 4 As shown, during the semantic segmentation model training process, the number of training sets, validation sets, and test sets were 700, 200, and 100, respectively. During the anomaly detection process, 700 normal samples were used to construct the detection template library, and 595 images containing both normal and abnormal data were used to test the performance of the anomaly detection method of this invention. The final results of the average segmentation accuracy of the semantic segmentation model used in this invention, the anomaly detection accuracy, anomaly detection precision, and anomaly detection recall of this invention are compared with the results of existing part segmentation-based anomaly detection (PSAD) methods as shown in Table 1 below.

[0084] Table 1 Performance Test Record of Anomaly Detection Method

[0085] Average segmentation accuracy mloU (%) 81.78 74.60 Anomaly detection accuracy (AUROC%) 99.31 85.87 Anomaly detection accuracy (%) 98.85 94.00 Anomaly detection recall (%) 98.33 73.76

[0086] As shown in the table above, the semantic segmentation model constructed by the method of the present invention has a high segmentation accuracy. Compared with existing anomaly detection methods, it reduces the false detection rate and improves the robustness and accuracy of anomaly detection.

[0087] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A method for detecting fastener assembly anomalies based on machine vision, characterized in that, Includes the following steps: S1: Collect images of correctly assembled fasteners from different angles and under different lighting conditions, preprocess the images to obtain template images, and the collection of all template images forms a template image set; S2: Perform geometric bounding box segmentation and annotation on each component of the fastener in the template image set to obtain a semantic segmentation dataset; calculate the component proportion and relative distance of each component to the fastener body to construct a detection template library; the component proportion is the ratio of the pixel area of ​​a single component segmentation region to the pixel area of ​​all component segmentation regions in the corresponding template image. S3: Construct and train a semantic segmentation model using a semantic segmentation dataset to segment the components of the fastener; S3 includes the following sub-steps: S3.1: Construct an initial semantic segmentation model, including a feature extractor and a segmentation predictor; the feature extractor is a pre-trained DenseNet201 network, and the features of its first three stages {S1, S2, S3} are used as multi-level features. The segmentation predictor includes a Conv3×3-BN-ReLU module, a bilinear upsampling module, and an upsampling-Softmax module, used to process the multi-level features to output the segmented regions of each component. The third-level feature S3 is processed by a Conv3×3-BN-ReLU module and a bilinear upsampling module, and then fused with the second-level feature S2 through a concatenation operation. The fused feature is then processed by another Conv3×3-BN-ReLU module and a bilinear upsampling module, and then fused with the first-level feature S1 through a concatenation operation. Finally, the fused feature is processed by an upsampling-Softmax module to output a segmentation prediction map. S3.2: Configure a multi-stage hybrid cross loss function and train an initial semantic segmentation model using data from the semantic segmentation dataset to obtain a trained semantic segmentation model; The expression for the multi-stage hybrid crossover loss function is as follows: ； ； In the formula, L represents the multi-stage hybrid crossover loss function, L Dice L represents the similarity loss function, which participates in all stages of training; CE L represents the cross-loss entropy function, which participates in all stages of training; APC λ1, λ2, and λ3 represent the loss functions of the component parts; λ1, λ2, and λ3 represent the hyperparameters of the corresponding preset loss functions. N C This indicates the number of segmented regions in the corresponding template image. α represents the total pixel area of ​​the segmented region of the i-th component predicted by the initial semantic segmentation model, and α represents the preset adjustment ratio coefficient. S4: Acquire assembly images of the fasteners to be tested and preprocess them, then input them into the trained semantic segmentation model to segment each component. S5: Calculate the component proportion R of each part based on the segmentation results. i Distance D from the relative fastener body i ; S6: Filter the component data from the test template library to find components of the same type as those in the fastener to be tested, and calculate the component proportion and relative distance to the fastener body for each component. Based on the statistical values, set the distribution range of component proportion and relative distance to the fastener body. Judge the R calculated in S5. i and D i If all components are within the set distribution range, the fastener under test is determined to be assembled normally; otherwise, it is determined to be assembled abnormally.

2. The fastener assembly anomaly detection method based on machine vision according to claim 1, characterized in that, In S1, the preprocessing operations include: ROI region cropping, noise reduction, and edge enhancement.

3. The fastener assembly anomaly detection method based on machine vision according to claim 1, characterized in that, The expression for the component proportions of the constituent parts is as follows: ； ； In the formula, R i A represents the component percentage of the i-th component. i A represents the total pixel area of ​​the segmented region of the i-th component. M This represents the total pixel area of ​​the segmented regions of all constituent parts in the corresponding template image; The expression for the relative fastener body distance is as follows: ； In the formula, D i C represents the distance between the i-th component and the fastener body. i C represents the centroid coordinates of the segmented region of the i-th component. M This represents the centroid coordinates of the segmented regions of all constituent parts in the corresponding template image. This indicates Euclidean distance.

4. The fastener assembly anomaly detection method based on machine vision according to claim 1, characterized in that, S2 includes the following sub-steps: S2.1: Traverse each template image in the template image set, perform geometric bounding box segmentation and annotation on various component parts related to fastener assembly in the template images, and store the annotated template images into the semantic segmentation dataset; the individual part regions segmented by the geometric bounding boxes are the segmentation regions; S2.2: Based on the geometric box segmentation and annotation results, calculate the component proportion of each component in the corresponding template image and the distance to the fastener body; S2.3: Store each component part, its corresponding component ratio, and the distance to the main fastener body in the detection template library.

5. The fastener assembly anomaly detection method based on machine vision according to claim 1, characterized in that, Before each training iteration, random image enhancement operations are introduced to train the semantic segmentation model; the random image enhancement operations include: image rotation, scaling, offsetting, and mirroring.

6. The fastener assembly anomaly detection method based on machine vision according to claim 1, characterized in that, In step S6, the statistical values ​​include the distribution mean and standard deviation. The distribution mean of the component proportion corresponding to the i-th component is calculated. and standard deviation Average distribution of distances from the main body of the fastener and standard deviation ; The distribution range of the component proportions is set as follows The distribution range of the distance relative to the fastener body is ; When making a judgment, if the following conditions are met ,and If the fastener being tested is found to be properly assembled, then the fastener being tested is determined to be properly assembled. Otherwise, the fastener to be tested is judged to be assembled abnormally.

7. The fastener assembly anomaly detection method based on machine vision according to claim 1, characterized in that, In step S1, an industrial camera is used to capture images of correctly assembled fasteners from different angles and under different lighting conditions. The industrial camera used has a resolution of 1920×1200 and a color gamut of three-channel RGB.

Images