A method for detecting misloading of fasteners in a multi-stage aircraft on-board assembly fixture
By constructing a benchmark image set of airborne assembly test pieces and using data augmentation methods, and optimizing the YOLOv9 model, the problems of high false alarm rate and low accuracy in the detection of mis-assembly and omission of aerospace parts were solved, and efficient fastener detection was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIANGXI HONGDU AVIATION IND GRP
- Filing Date
- 2025-10-21
- Publication Date
- 2026-06-02
AI Technical Summary
Existing methods for detecting defects in aerospace parts suffer from high false alarm rates and low detection accuracy in complex assembly environments. Furthermore, there is a lack of publicly available datasets of defects in aerospace parts, making model training difficult and resulting in low levels of automation in image acquisition.
By designing experimental components for airborne accessories, collecting a set of benchmark images and manually annotating them, a dataset was constructed. The YOLOv9 model was trained, and high-confidence samples were generated by combining data augmentation and a denoising diffusion probability model. The model was optimized to improve robustness and accuracy, and a misassembly/missing component detection model was used for identification.
It improves the automation of detection in complex environments, reduces the false alarm rate, significantly improves the accuracy and recall of small target detection, and enhances the model's adaptability to complex backgrounds.
Smart Images
Figure CN122134609A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of testing technology in the manufacturing process of aviation equipment, and in particular to a method for detecting misinstallation or omission of fasteners in multi-stage airborne components. Background Technology
[0002] Object detection is a computer vision technique designed to identify and locate specific targets (such as defects, cracks, etc.) in images. It is typically based on deep learning models, such as convolutional neural networks (CNNs), which are trained to identify and label regions of interest in images. However, current research in object detection has not yet met industrial requirements, especially when dealing with objects with small differences, resulting in significant errors. Furthermore, due to the high fit and cost of some fasteners in airborne components, close-range data acquisition is not feasible to avoid damage.
[0003] Image reconstruction is a method that uses sensors and mathematical algorithms to reconstruct and detect the surface of a target. However, its accuracy cannot be significantly improved due to the difficulty in applying prior knowledge from real-world industrial applications, and its reasoning speed is relatively slow.
[0004] Current methods for detecting defects in aerospace parts mainly rely on intelligent detection technologies, encompassing various methods such as computer vision and machine learning, to detect and analyze the assembly process of parts. For example, patent number 202210994360.2 discloses an intelligent detection system and method for aero-engine bolt defects based on key point detection. This system uses a mobile inspection robot to control and acquire images of aerospace parts, and utilizes a key point detection model based on an attention mechanism to obtain the positional information of the parts in the images. Key points in the attention enhancement region are detected through key point detection branches, and finally, the system is compared with standard parts to determine if defects exist. Patent number 202410425461.7 discloses a method for detecting aircraft skin defects based on a NAM-DSSD model. First, images of the aircraft skin are captured using an industrial camera. A high-resolution image is obtained using an orthogonal matching tracking algorithm. The high-resolution image is then combined with multiple images using data augmentation methods to obtain an enhanced dataset. Finally, this data is input into the model for training to identify and classify the types of defects in the parts.
[0005] The above-mentioned patents have achieved good results in practical applications, but there are still many problems: 1) The automation of image acquisition is low, and manual operation of the acquisition device is required to obtain data; 2) They perform well in specific scenarios, but there are problems such as false alarm rate and low detection accuracy when facing complex assembly environments; 3) The dimensions of aerospace parts vary greatly, making stable identification difficult; 4) There is a lack of publicly available datasets of aerospace part defects, making model training difficult. Summary of the Invention
[0006] The technical problem solved by this invention is to provide a method for detecting mis-installation or omission of fasteners in multi-stage airborne components, so as to solve the problems in the background art mentioned above.
[0007] The technical problem solved by this invention is achieved by the following technical solution:
[0008] A multi-stage method for detecting mis-installed or missing fasteners in airborne components, comprising the following steps:
[0009] S1) Data acquisition is performed on the fasteners of airborne components based on the data acquisition device, and an airborne component test piece is designed. Then, according to the acquired airborne component fastener installation specifications, the fasteners are installed on the airborne component test piece and the preliminary experiment is completed. Then, fastener data samples of the airborne component test piece are collected to finally form a reference image set.
[0010] S2) Manually annotate the images in the reference image set obtained in step S1). Use the labelimg annotation tool to annotate the location and label of all fasteners in each image to construct a part positioning detection dataset, a missing part identification dataset, and a mis-assembly identification dataset. Then, perform data augmentation on the data in the obtained part positioning detection dataset, missing part identification dataset, and mis-assembly identification dataset.
[0011] S3) Model training: Based on the part positioning detection dataset, missing part identification dataset, and mis-assembly identification dataset constructed in step S2), YOLOv9 is selected as the baseline to complete the training of the part positioning detection model, missing part identification model, and mis-assembly identification model.
[0012] S4) Building the error and omission detection model: Combine the part positioning detection model, omission identification model and error identification model trained in step S3) to form the error and omission detection model. Then, input the cropped image into the error and omission detection model to identify fastener errors and omissions.
[0013] In this invention, the reference image set formation step S1) is as follows:
[0014] First, we investigated and measured the fasteners of actual airborne accessories to obtain their dimensions, installation specifications, and error-prone parameters. Then, we drew CAD images of experimental airborne accessories and customized experimental airborne accessories made of aluminum alloy.
[0015] Secondly, background images were added to the test specimens of airborne accessories to achieve the effect of background complexity. Then, fasteners were installed on the test specimens of airborne accessories according to the fastener installation specifications obtained from the survey.
[0016] Finally, data samples of fasteners from airborne component test pieces were collected using industrial lenses at different time periods, angles, and distances, ultimately forming a baseline image set.
[0017] In this invention, the data augmentation steps in step S2) are as follows:
[0018] i. Random rotation and flipping: Rotate the image at random angles and flip it horizontally / vertically to enhance the model's robustness to rotation and mirror transformations;
[0019] ii. Random Occlusion: Randomly occludes a portion of the image to simulate target recognition under occlusion conditions;
[0020] iii. Contrast Enhancement: Adjusts the contrast of the image to make the features in the image clearer;
[0021] iv. Brightness and Hue: Data augmentation based on color space transformation adjusts brightness and hue to eliminate differences in illumination, color, and brightness between the training and test sets.
[0022] In this invention, during step S2), when constructing the part positioning detection dataset, the missing part identification dataset, and the mis-assembly identification dataset, negative sample data collected from the airborne accessory test piece are used, and a denoising diffusion probability model is adopted to generate high-confidence sample data.
[0023] In this invention, in step S3), YOLOv9 is selected as the baseline to complete the training process of the part positioning detection model, the missing part identification model, and the mis-assembly identification model, and network pruning and cross-platform processing are performed.
[0024] Network pruning involves two steps: assessing the importance of parameters and removing redundant parameters.
[0025] Cross-platform processing: Optimize the trained YOLOv9 model and convert it to ONNX format. The steps are as follows:
[0026] a. Reparameterization and optimization: Using the reparameterization code in YOLOv9, the PGI module is optimized to reduce the number of model parameters and improve inference speed, resulting in the yolov9.pt file;
[0027] b. Convert the model to ONNX format: After completing the model optimization, install the third-party ONNX and ONNX runtime dependency libraries; then input the reparameterized yolov9.pt file into the torch.onnx.export function to convert it into a model with the ONNX suffix. The ONNX model is output in ONNX format.
[0028] In this invention, in step S3), the acquired images are processed using a fastener detection model based on YOLOv9 to achieve preliminary positioning of the fasteners. Then, the images are cropped based on the positioning results to ensure that each fastener exists only in a cropped image.
[0029] In this invention, in step S4), the error / missing assembly detection model uses WS-DAN as the base model, and is optimized based on WS-DAN:
[0030] A. Optimize using spatial attention mechanisms to accurately locate key features;
[0031] B. Introduce the AutoAugment data augmentation method to improve generalization ability and reduce the risk of overfitting;
[0032] C. Input the cropped image into the error and omission detection model, and then vote based on the recognition results of each model to determine whether there is an omission. If all the omission detection models fail to identify the corresponding fastener, the result is directly output, indicating that there is an omission. If a model identifies the corresponding fastener, the corresponding error detection model is called from the error and omission detection model to check for error based on the recognition result.
[0033] D. Based on the identification results of the misinstallation identification model, determine whether the fastener under test is misinstalled. If the missing fastener is identified by the missing fastener identification model, and the misinstallation detection model can also correctly identify the fastener and determine that it is correctly installed, then it means that there is no misinstallation; otherwise, if the misinstallation detection model determines that it is misinstalled, then it means that there is misinstallation.
[0034] Beneficial effects:
[0035] 1) This invention utilizes data augmentation to acquire more diverse data for model training, thereby improving the model's adaptability to input data; in the face of complex aviation environments, incorporating data augmentation training enhances the model's robustness; the data augmentation effect is as follows: Figure 16 As shown;
[0036] 2) This invention uses negative sample data collected from airborne accessory test pieces and employs a denoising diffusion probability model to generate high-confidence sample data, with the generation effect as follows: Figure 17 As shown;
[0037] 3) This invention uses data augmentation and diffusion model processing to provide richer and higher quality data support for the model. At the same time, it uses the YOLOv9 model for detection to detect five fasteners in the finished throttle body in a complex environment. Compared with ordinary detection systems, it not only improves the automation of image acquisition, but also effectively reduces the false alarm rate in complex environments, and significantly improves the accuracy and recall rate of small target detection. Attached Figure Description
[0038] Figure 1 This is a schematic diagram of the image generation steps of the denoising diffusion probability model in a preferred embodiment of the present invention.
[0039] Figure 2 This is a schematic diagram of the UNet network structure in a preferred embodiment of the present invention.
[0040] Figure 3 This is a schematic diagram of the Programmable Gradient Information (PGI) related structure in a preferred embodiment of the present invention.
[0041] Figure 4 This is a schematic diagram of the residual structure in a preferred embodiment of the present invention.
[0042] Figure 5 This is a schematic diagram of the BAP processing procedure in a preferred embodiment of the present invention.
[0043] Figure 6 This is a schematic diagram of the spatial attention structure in a preferred embodiment of the present invention.
[0044] Figure 7 This is a schematic diagram of an airborne accessory test piece in a preferred embodiment of the present invention.
[0045] Figure 8 This is a schematic diagram of an airborne accessory test piece under a complex background in a preferred embodiment of the present invention.
[0046] Figure 9 This is a schematic diagram of an industrial lens in a preferred embodiment of the present invention.
[0047] Figure 10 This is a schematic diagram of fastener samples at different angles in a preferred embodiment of the present invention.
[0048] Figure 11 This is a schematic diagram of fastener samples at different distances in a preferred embodiment of the present invention.
[0049] Figure 12 This is a schematic diagram illustrating the use of labelimg to label all fasteners in a preferred embodiment of the present invention.
[0050] Figure 13This is a schematic diagram illustrating data enhancement in a preferred embodiment of the present invention.
[0051] Figure 14 This is a schematic diagram of the anchor regression process guided by a common IoU-based loss function.
[0052] Figure 15 This is a schematic representation of the data generated by the denoising diffusion probability model in a preferred embodiment of the present invention.
[0053] Figure 16 This is a schematic diagram illustrating the data enhancement effect in a preferred embodiment of the present invention.
[0054] Figure 17 This is a schematic diagram illustrating the generation effect of the denoising diffusion probability model in a preferred embodiment of the present invention. Detailed Implementation
[0055] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.
[0056] A multi-stage method for detecting mis-installed or missing fasteners in airborne components, comprising the following steps:
[0057] S1) Data acquisition is performed on the airborne component fasteners based on the data acquisition device, and an airborne component test piece is designed. Then, according to the acquired airborne component fastener installation specifications, the fasteners are installed on the airborne component test piece and the preliminary experiment is completed, and finally a reference image set is formed.
[0058] First, investigate and measure the fasteners of actual airborne component parts to obtain dimensions, installation specifications, and error-prone parameters; then, draw CAD images of experimental airborne component parts and customize experimental parts made of aluminum alloy, such as... Figure 7 As shown;
[0059] Due to the diverse backgrounds of real-world scenarios, this embodiment adds background images to the airborne accessory test specimen to achieve a more complex background effect. Then, based on the investigated airborne accessory fastener installation specifications, fasteners are installed on the airborne accessory test specimen, such as... Figure 8 As shown;
[0060] For data acquisition of airborne component test specimens, the Hikvision MVL-MF0828M-8MP industrial lens was selected, such as... Figure 9 As shown, considering the complex factors such as lighting changes, occlusion, and noise in the actual working environment, data samples of fasteners from airborne assembly test parts were collected at different time periods, angles, and distances during the data acquisition process. Figure 10 , 11As shown, this ultimately forms a baseline image set;
[0061] S2) Manually annotate the images in the reference image set obtained in step S1). Use the labelimg annotation tool to annotate the location and labels of all fasteners in each image. The annotation process is strictly carried out according to standard operating procedures to ensure the consistency and accuracy of the annotated data. Figure 12 As shown;
[0062] Finally, three datasets were constructed: a parts location detection dataset, a missing parts identification dataset, and a mis-assembly identification dataset. Data augmentation was then performed on these datasets to improve training performance. The main functions of data augmentation included:
[0063] Increase the number of training samples: By generating new samples, the training set is expanded, providing more samples for the model to learn, thereby alleviating the overfitting problem;
[0064] Enhancing the model's generalization ability: By introducing randomness, data augmentation can help the model learn more general features, making it more capable of generalizing to new samples.
[0065] Enhanced model robustness: Models that have undergone data augmentation have a certain degree of resistance to interference from changes in input data and noise;
[0066] Addressing the data imbalance problem: In classification tasks, data augmentation can balance the number of samples in each class, helping to improve the model's ability to identify the minority class. This embodiment performs data augmentation, such as... Figure 13 As shown:
[0067] Random rotation and flipping: The image is rotated at random angles and flipped horizontally / vertically to enhance the model's robustness to rotation and mirror transformations.
[0068] Random occlusion: Randomly occludes a portion of the image to simulate target recognition under occlusion conditions;
[0069] Contrast Enhancement: Adjusts the contrast of the image to make features in the image clearer;
[0070] Brightness and Hue: Data augmentation based on color space transformation adjusts brightness and hue, which can eliminate differences in illumination, color, and brightness between the training and test sets;
[0071] Considering the scarcity of negative sample data during the manufacturing process of airborne accessory components, which can easily lead to an imbalanced training dataset and severely impact model performance, it is necessary to expand the negative sample data. This embodiment primarily uses negative sample data collected from experimental airborne accessory components and employs a denoising diffusion probability model to generate high-confidence sample data. The denoising diffusion probability model teaches the model how to gradually generate negative sample data from noise. The generation steps are as follows: Figure 1 As shown, the denoising diffusion probability model requires T steps to complete image generation. This involves adding noise progressively from right to left from the images in the reference image set. q(x) t |x t-1 The description involves progressively adding Gaussian noise during the model training process. After T additions, the original image will approximate a Gaussian distribution. This supports diffusion in reconstructing the desired image from the Gaussian noise. This added Gaussian noise is the knowledge that the model needs to learn. θ (x t-1 |x t This describes the noise distribution predicted from left to right during the model's learning phase, simulating the distribution of noise gradually added during the training phase. It reveals the noise distribution at each step from 0 to T. Finally, the denoising diffusion probability model generates an image whose distribution approximates the original training data. After the model is trained, the required negative sample data is gradually generated from Gaussian noise. The denoising diffusion probability model uses the UNet model as its network structure, as shown below. Figure 2 As shown in Equation 1, the loss function constructed by the UNet model aims to reduce the gap between the noise predicted by the model and the noise actually added.
[0072]
[0073] The denoising diffusion probability model is based on samples generated from random noise, thus producing rich, high-quality samples. These samples are also more stable and clearer than those generated by common methods. Ultimately, various detection categories and types of negative sample data are generated using negative sample data collected from airborne accessory experimental components. The specific number of generated samples is summarized as follows: Figure 15 As shown;
[0074] S3) Model training: Based on the part positioning detection dataset, missing part identification dataset, and mis-assembly identification dataset constructed in step S2), complete the training of the part positioning detection model, missing part identification model, and mis-assembly identification model.
[0075] YOLOv9 was selected as the baseline, and relevant improvements were made based on the actual situation:
[0076] 1. Network Pruning: Since the trained model is large and only needs to locate fasteners, the model size can be reduced by pruning redundant parameters and connections. The basic principle is to prune the network weights, removing channels that contribute less to the model, thereby reducing its size. For example... Figure 3 As shown, the inference speed of the model can be improved without affecting its accuracy. Figure 3 The Multi-level Auxiliary Information module and Auxiliary ReversibleBranch are not used in the inference stage. Therefore, in order to improve the overall inference speed of the framework, the above two parts are removed by the reparameterization method of YOLOv9. As can be seen from the figure, removing these two parts does not affect the inference part of the model.
[0077] This embodiment performs pruning through the following two steps:
[0078] Parameter importance assessment: Evaluate the importance of parameters and channels in the model. The importance of parameters is evaluated based on their sensitivity, such as the gradient magnitude of the parameter with respect to the loss function. Based on the evaluation results, it can be determined which channels or parameters are redundant.
[0079] Redundant parameters are removed: Based on the parameter importance assessment results, some redundant parameters are removed. Removal may lead to a performance decrease, so the entire model needs to be fine-tuned to restore the model's performance as much as possible.
[0080] 2. Cross-platform processing: Due to the uncertainties of real-world production environments, the platforms used may differ from those used in laboratories. Therefore, the trained YOLOv9 model is optimized and converted to ONNX format. The steps are as follows:
[0081] Reparameterization and optimization: Using the reparameterization code in YOLOv9, the PGI module is optimized to reduce the number of model parameters and improve inference speed, resulting in the yolov9.pt file;
[0082] Model conversion to ONNX format: After model optimization, install the third-party ONNX and ONNX runtime dependency libraries; then input the reparameterized yolov9.pt file into the torch.onnx.export function to convert it into a model with the ONNX suffix, and the ONNX model is output in ONNX format;
[0083] 3. Loss Function Optimization: Currently, most IOU-based loss functions (such as GIoU, DIoU, CIoU, EIoU, etc.) are affected by unreasonable penalty factors, which can easily lead to anchor inflation during the regression process, significantly slowing down the convergence speed. This embodiment uses a powerful IOU (PIoU v2) loss function that combines an adaptive penalty factor based on target size and a gradient adjustment function based on anchor quality to guide anchors to regress along an effective path. Simultaneously, PIoU v2 enhances the ability to focus on medium-quality anchors, enabling the model to pay attention to higher-quality anchors, such as... Figure 14 As shown, this diagram illustrates the anchor regression process guided by several common IoU-based loss functions. The colored boxes represent anchor boxes guided by different loss functions during the regression process. Clearly, anchor boxes guided by IoU loss regress to the closest target box the fastest. Furthermore, except for IoU loss, all other loss functions guide anchor boxes with an enlarged area, while those guided by IoU loss do not. Therefore, to improve the model's localization ability, Powerful-IoU, which focuses more on high-quality anchors, is used instead of the traditional CIOU. Its specific formula is shown below:
[0084]
[0085] Compared to traditional IoU calculation methods, Powerful-IoU uses quantity as the numerator instead of the area of the intersection region. This allows for a greater focus on the model's ability to correctly locate the detected target. By using Powerful-IoU as the loss function, the model can be encouraged to focus more on improving the localization accuracy of high-quality anchors, thereby enhancing the performance of target detection.
[0086] This step uses a fastener detection model based on YOLOv9 to process the acquired images and achieve preliminary positioning of fasteners. Then, the images are cropped based on the positioning results to ensure that each fastener exists only in a cropped image. The purpose of this is to extract each fastener individually for subsequent missing fastener detection and misinstallation inspection.
[0087] ResNet: For datasets grouped by various fastener types, ResNet is first used as the basic architecture to perform missing fastener detection for each type, distinguishing whether a particular fastener exists. ResNet uses residual connections to preserve original features, making the network learning smoother and more stable, further improving the model's accuracy and generalization ability. Its residual structure is as follows: Figure 4 As shown;
[0088] WS-DAN: Since some misinstalled fasteners have similar morphological features to normal samples, a model architecture capable of capturing more detailed features is needed for their identification. This embodiment uses WS-DAN as the base model. First, ResNet is used as the backbone network as a feature extractor to extract a 2048-dimensional feature map. Then, the first few parts_nums are compared with the extracted feature map using a BAP operation, such as... Figure 5 As shown, the output of the BAP module is used as the feature matrix for category prediction;
[0089] To extract more detailed features that are helpful for classification, and to prevent the model from over-relying on any one type of feature, a feature map from the first `parts_nums` is randomly selected, and its class activation map is obtained. From the class activation map, the local region that the model is more interested in is identified; this local region is the key feature region. The key feature region is then cropped, and a scaling operation is performed on it using linear interpolation to obtain the key features at the original image size. The processed key feature map is then re-inputted into the network model for two-stage fine-grained classification. At the same time, to improve the model's ability to grasp features, the original image with the key features cropped is also input into the original model for prediction, forcing the model to grasp other features.
[0090] The loss function of this model consists of two parts: class loss and feature loss. The class loss is calculated by taking the mean of the original image prediction result and the classification result of key features, and then calculating the cross-entropy loss by taking the mean of the classification result after removing key features and the true label. The detailed formula for the loss is shown below:
[0091] L cross-entropy =-∑y i *log(p i ) Formula (2)
[0092]
[0093] L class-loss =L cross-entropy (P mean ,Y) Formula (4)
[0094] Among them, y i Let p be the label of the i-th sample. i P represents the prediction result for the i-th sample; total P is the predicted score for the original image. part P represents the predicted score for the key region. drop Y represents the predicted score after removing key regions from the original image, where Y is the label value of all samples in a given epoch.
[0095] For feature loss, initially all key features of all categories are set to 0. During training, the values of key features are dynamically adjusted using the following formula, while the feature loss is calculated:
[0096] C k =C k-1 +β(f k -C k-1 ) Formula (5)
[0097]
[0098] Among them, C k The key features that need to be adjusted are fk, which is the key feature that the model considers to be in this training, β is a hyperparameter, which is set to 0.05 based on experience in this embodiment, and M is the part_nums value set during training.
[0099] The final model loss is the sum of the class loss and the feature loss;
[0100]
[0101] Based on WS-DAN, this embodiment addresses the problems encountered in practical applications as follows:
[0102] Spatial attention mechanism: The structure of the spatial attention mechanism is as follows Figure 6 As shown, the input feature map is first subjected to global max pooling and global average pooling based on the channel, resulting in two H×W×1 feature maps. These two feature maps are then concatenated along the channel dimension. A 7*7 convolution is then used to perform feature fusion on the concatenated feature map, reducing the dimensionality to one channel (H×W×1). A spatial attention map is then generated using the sigmoid activation function. Finally, this feature map is multiplied by the input feature of the module to obtain the final generated feature. The calculation formula for the spatial attention mechanism is shown below:
[0103] Ms(F)=σ(f 7×7 ([AvgPool(F);MaxPool(F)]))
[0104] =σ(f 7×7 ([F s avg;F s max])) Formula (8)
[0105] In equation (8) above, σ is the sigmoid function, f 7*7 This is a convolution operation with a filter size of 7*7;
[0106] S4) Building the error and omission detection model: Combine the part positioning detection model, omission identification model and error identification model trained in step S3) to form the error and omission detection model.
[0107] Since the visualization results of misassembled fasteners are not significantly different from normal samples, it is necessary to capture more detailed features for identification. This embodiment uses WS-DAN as the base model, which has the advantage of capturing subtle features, thereby improving the recognition accuracy of misassembled fasteners. WS-DAN combines attention mechanism and weakly supervised learning, enabling it to automatically learn key features during training. By introducing the attention mechanism, WS-DAN can focus on the most informative region in the image, thereby improving the sensitivity to misassembled fasteners. Weakly supervised learning allows WS-DAN to learn from data with less label information without relying on a large amount of labeled data. This is particularly useful for fastener misassembly detection scenarios, as it is usually difficult to obtain a large amount of labeled data for misassembly cases.
[0108] This embodiment optimizes the traditional WS-DAN as follows:
[0109] A. Optimize using spatial attention mechanism. Spatial attention mechanism can help the model better focus on important regions in the input feature map, thereby improving the ability to locate key features; this optimization can help the model better understand the input data and more accurately locate key features.
[0110] B. Introducing the AutoAugment data augmentation method: Traditional data augmentation methods usually require manual specification of augmentation strategies, while AutoAugment can automatically search for and improve data augmentation strategies to improve the generalization performance of the model. AutoAugment automatically designs the best data augmentation strategy based on the specific dataset and task, enabling the model to better adapt to the diversity of various samples, improve generalization ability and reduce the risk of overfitting.
[0111] C. Input the cropped image into the error and omission detection model, and then vote based on the recognition results of each model to determine whether there is an omission. If all the omission detection models fail to identify the corresponding fastener, the result is directly output, indicating that there is an omission. If a model identifies the corresponding fastener, the corresponding error detection model is called from the error and omission detection model to check for error based on the recognition result.
[0112] D. Based on the identification results of the misinstallation identification model, determine whether the fastener under test is misinstalled. If the missing fastener is identified by the missing fastener identification model, and the misinstallation detection model can also correctly identify the fastener and determine that it is correctly installed, then it means that there is no misinstallation; otherwise, if the misinstallation detection model determines that it is misinstalled, then it means that there is misinstallation.
[0113] The specific embodiments described herein are merely illustrative examples of the present invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or adopt similar methods to replace them, but without departing from the spirit of the present invention or exceeding the scope defined by the appended claims.
Claims
1. A method for detecting mis-installation or omission of fasteners in multi-stage airborne component assembly, characterized in that, The specific steps are as follows: S1) Data collection is performed on the fasteners of airborne components and accessories, and an airborne component test piece is designed. Then, according to the collected airborne component fastener installation specifications, the fasteners are installed on the airborne component test piece and the preliminary experiment is completed. Then, fastener data samples of the airborne component test piece are collected to finally form a reference image set. S2) Manually annotate the images in the reference image set obtained in step S1). Use the labelimg annotation tool to annotate the location and label of all fasteners in each image to construct a part positioning detection dataset, a missing part identification dataset, and a mis-assembly identification dataset. Then, perform data augmentation on the data in the obtained part positioning detection dataset, missing part identification dataset, and mis-assembly identification dataset. S3) Based on the part positioning detection dataset, missing part identification dataset, and mis-assembly identification dataset constructed in step S2), YOLOv9 is selected as the baseline to complete the training of the part positioning detection model, missing part identification model, and mis-assembly identification model. S4) Combine the part positioning detection model, missing part identification model and mis-assembly identification model trained in step S3) to form a mis-assembly detection model. Then, input the cropped image into the mis-assembly detection model to identify the missing fasteners.
2. The method for detecting mis-installation or omission of fasteners in multi-stage airborne assembly as described in claim 1, characterized in that, In step S1), the steps for forming the reference image set are as follows: First, we investigated and measured the fasteners of actual airborne accessories to obtain their dimensions, installation specifications, and error-prone parameters. Then, we drew CAD images of experimental airborne accessories and customized experimental airborne accessories made of aluminum alloy. Secondly, background images were added to the test specimens of airborne accessories to achieve the effect of background complexity. Then, fasteners were installed on the test specimens of airborne accessories according to the fastener installation specifications obtained from the survey. Finally, data samples of fasteners from airborne component test pieces were collected using industrial lenses at different time periods, angles, and distances, ultimately forming a baseline image set.
3. The method for detecting mis-installation or omission of fasteners in multi-stage airborne assembly as described in claim 1, characterized in that, In step S2), the specific steps of data augmentation are as follows: i. Random rotation and flipping: Rotate the image at random angles and flip it horizontally / vertically to enhance the model's robustness to rotation and mirror transformations; ii. Random Occlusion: Randomly occludes a portion of the image to simulate target recognition under occlusion conditions; iii. Contrast Enhancement: Adjusts the contrast of the image to make the features in the image clearer; iv. Brightness and Hue: Data augmentation based on color space transformation adjusts brightness and hue to eliminate differences in illumination, color, and brightness between the training and test sets.
4. The method for detecting mis-installation or omission of fasteners in multi-stage airborne assembly accessories according to claim 1, characterized in that, In step S2), during the process of constructing the part positioning detection dataset, the missing part identification dataset, and the mis-assembly identification dataset, negative sample data collected from the airborne accessory test piece are used, and a denoising diffusion probability model is adopted to generate high-confidence sample data.
5. The method for detecting mis-installation or omission of fasteners in multi-stage airborne assembly accessories according to claim 1, characterized in that, In step S3), YOLOv9 is selected as the baseline to complete the training process of the part positioning detection model, the missing part identification model, and the mis-assembly identification model, and the data is pruned and processed across platforms.
6. The method for detecting mis-installation or omission of fasteners in multi-stage airborne assembly accessories according to claim 5, characterized in that, Network pruning involves two steps: assessing the importance of parameters and removing redundant parameters.
7. The method for detecting mis-installation or omission of fasteners in multi-stage airborne assembly as described in claim 5, characterized in that, Cross-platform processing: Optimize the trained YOLOv9 model and convert it to ONNX format. The steps are as follows: a. Reparameterization and optimization: Using the reparameterization code in YOLOv9, the PGI module is optimized to reduce the number of model parameters and improve inference speed, resulting in the yolov9.pt file; b. Convert the model to ONNX format: After completing the model optimization, install the third-party ONNX and ONNX runtime dependency libraries; then input the reparameterized yolov9.pt file into the torch.onnx.export function to convert it into a model with the ONNX suffix. The ONNX model is output in ONNX format.
8. The method for detecting mis-installation or omission of fasteners in multi-stage airborne assembly according to claim 1, characterized in that, In step S3), the acquired images are processed using a fastener detection model based on YOLOv9 to achieve preliminary positioning of the fasteners. Then, the images are cropped based on the positioning results to ensure that each fastener exists only in a cropped image.
9. The method for detecting mis-installation or omission of fasteners in multi-stage airborne assembly according to claim 1, characterized in that, In step S4), the error / missing item detection model uses WS-DAN as the base model, and optimizes it accordingly: A. Optimize using spatial attention mechanisms to accurately locate key features; B. Introduce the AutoAugment data augmentation method to improve generalization ability and reduce the risk of overfitting; C. Input the cropped image into the error and missing parts detection model, and then vote on the recognition results of each model to determine whether there is a missing part. If all the missing parts recognition models fail to identify the corresponding fasteners, the result is directly output, indicating that there is a missing part. If a certain model identifies the corresponding fastener, the corresponding misassembly identification model is called from the misassembly and omission detection model to perform misassembly inspection based on the identification result; D. Based on the identification results of the misinstallation identification model, determine whether the fastener under test is misinstalled. If the missing fastener is identified by the missing fastener identification model, and the misinstallation detection model can also correctly identify the fastener and determine that it is correctly installed, then it means that there is no misinstallation; otherwise, if the misinstallation detection model determines that it is misinstalled, then it means that there is misinstallation.