An aircraft body part missing and wrong installation detection method based on deep learning

Abstract

The present application relates to a kind of based on deep learning's aircraft body parts wrong and missing installation detection method and system, to solve the detection difficult problem caused by the lack of wrong and missing installation sample in prior art.The method is by constructing general image library, and a large number of similar samples and dissimilar samples are generated using data enhancement technology, to replace positive and negative samples in actual scene for model training.Specifically, convolutional neural network or twin network is used to extract image features, and the model is optimized by contrast loss function, so that the feature distance of similar samples is closer, and the feature distance of dissimilar samples is farther.In the detection stage, by calculating the feature distance between the image to be detected and the standard image, it is judged whether the parts are wrong and missing installation.The present application does not need to rely on a large number of labeled data in actual scene, significantly improves the accuracy and robustness of detection, is suitable for quality detection in aircraft assembly process, can effectively guarantee flight safety and production efficiency.

Description

Technical Field

[0001] This invention relates to the field of aviation manufacturing and testing technology, particularly to image recognition technology based on deep learning, specifically a method and system for detecting misinstallation or omission of parts in aircraft fuselage installations. Background Technology

[0002] In aircraft assembly, the detection of missing or incorrectly assembled parts is a crucial step in ensuring flight safety and assembly quality. Traditional detection methods primarily rely on manual inspection or image processing techniques based on template matching. These methods are inefficient and struggle to handle situations involving a wide variety of parts, complex installation locations, and diverse shooting angles. While deep learning technology has made significant progress in image recognition in recent years, the scarcity of such samples in real-world aircraft part assembly detection poses a significant challenge to training deep learning models. Although existing technologies have proposed some vision-based methods for detecting missing or incorrect parts, they still depend on a large number of labeled samples, making it difficult to meet the needs of real-world industrial scenarios. Summary of the Invention

[0003] This invention aims to provide a method and system for detecting misassembly or omission of aircraft body parts based on deep learning. Through an innovative network structure, it improves the accuracy and robustness of detecting misassembly or omission of parts in real-world scenarios.

[0004] This invention proposes a deep learning-based method for detecting mis-assembly or omission of aircraft body parts. Its core innovations lie in image feature fusion with camera parameters, ViT for feature extraction, information sharing between Siamese networks based on cross-attention mechanism, and image similarity calculation based on multiple feature vectors.

[0005] According to one aspect of this application, a deep learning-based method for detecting misassembly or omission of aircraft fuselage parts is provided, comprising the following steps: S1: Construct an attention-based image similarity calculation model that integrates image and camera parameters. The model includes a ViT feature extraction layer, a multilayer perceptron layer, a cross-attention layer, a self-attention layer, a contrastive feature extraction layer, and a similarity calculation layer. The images include standard images of a fully assembled and error-free aircraft fuselage and images of the aircraft fuselage to be inspected. S2: Input the images into the ViT feature extraction layer to extract multiple feature vectors corresponding to each image; S3: Input the camera parameters into the multilayer perceptron, convert the camera parameters into a vector with the same dimension as the feature vector extracted by the ViT feature extraction layer, and add this vector to each feature vector extracted by the ViT feature extraction layer; S4: Input the multiple feature vectors corresponding to each image obtained after processing S3 into the cross-attention layer for cross-attention calculation, and then add them to the multiple feature vectors corresponding to each image obtained in S2 to complete the skip connection and obtain the feature vector after skip connection. S5: Input the feature vector after the jump connection into the attention layer, and perform internal feature integration on the feature vector after the image information interaction; S6: Input the multiple feature vectors corresponding to each image obtained after processing S5 into the same feature extraction network to calculate the feature vectors used for final comparison; S6: Calculate the cosine distance between the feature vectors corresponding to each image for the final comparison, combine all the calculated distances into a vector, input it into the multilayer perceptron, and output the final similarity score. The similarity score can directly reflect the similarity of the images and further reflect the errors and omissions of the aircraft body parts.

[0006] The camera parameters include camera focal length, camera principal point, distortion correction parameters, camera position coordinates, and three camera rotation angles.

[0007] The advantages of this application are: (1) Image feature fusion camera parameters, which integrate the shooting angle, shooting position and other information corresponding to the image into the feature vector used for comparison, thereby providing necessary information on the reflective effect of the part, the size and angle of the part in the image.

[0008] (2) Image feature extraction is performed by ViT, that is, feature extraction is performed at different locations of the image and information at different locations is fused. Compared with full convolution feature extraction, this method preserves the overall information while ensuring the granularity of information.

[0009] (3) Information sharing between Siamese networks is based on a cross-attention mechanism. When ordinary Siamese networks are used for feature extraction from two images, there is no interaction between the information of the two images, making it impossible to specifically find distinguishing information. When the differences between the two images are not obvious, fine-grained distinguishing information cannot be extracted. The Siamese network designed in this patent, after shallow feature extraction and camera parameter fusion of two comparison images, fuses the information of the other image with the information of the current image, and then fuses this information with shallow features and processes it again to obtain multiple feature vectors for comparison. The average similarity of the calculated feature vectors is the similarity between the two images.

[0010] (4) Image similarity calculation based on multiple feature vectors. The output of self-attention is multiple feature vectors. The similarity is calculated for each feature vector, and finally the similarity between the two images is calculated by a multilayer perceptron. Attached Figure Description

[0011] Figure 1 This is a flowchart of this application.

[0012] Figure 2 This is the network structure diagram of the model.

[0013] Figure 3 The detailed implementation process of fusing camera parameters and image information is described.

[0014] Figure 4 This describes the implementation process of a multilayer perceptron.

[0015] Figure 5 This is a schematic diagram of the feature extraction structure for comparison.

[0016] Figure 6 This describes the process of calculating distance. Detailed Implementation

[0017] The present application is described in detail below with reference to the embodiments, but the present application is not limited to these embodiments. Example

[0018] S1: Construct an attention-based image similarity calculation model that integrates two input images with camera parameters (camera focal length, camera principal point, distortion correction parameters, camera position coordinates, and three camera rotation angles). Figure 2 The overall model construction method includes an image ViT feature extraction layer, a multilayer perceptron layer that projects camera parameters onto image ViT features, a cross-attention layer for ViT features of two images, a self-attention layer, a contrast feature extraction layer, and a similarity calculation layer.

[0019] S2: Input the image into ViT and extract multiple independent feature vectors between the images.

[0020] S3: The camera parameters are input into the MLP (composed of fully connected layers, GELU, Dropout, and a series of fully connected layers and Dropout layers), which converts the camera parameters into a vector of the same dimension as the feature vector extracted from the image. This vector is then added to each feature vector extracted by the ViT network. This step integrates information such as the shooting angle and position of the image into the feature vector used for comparison, thereby providing necessary information about the reflective effect of the part, its size, and angle in the image.

[0021] S4: The feature vectors of the two input images to the network, i.e., the output of S3, undergo cross-attention calculation, and are then added to the features extracted by ViT in S2 to complete the skip connection operation. This step fuses the features of the two images to a certain extent, optimizing the focus of attention for each image.

[0022] S5: Perform self-attention calculation on the feature vectors after the jump connection (this self-attention has the same structure as the self-attention network in ViT, but the parameters are independent, that is, the parameters are not the same as the self-attention network in ViT). The purpose of this step is to improve the features and integrate the internal features of the features after the interaction of information between the two graphs.

[0023] S6: Input the obtained feature vectors into the feature extraction network (the feature extraction network consists of an MLP, and each feature vector will be input into the same feature extraction network) to calculate the feature vectors used for the final comparison.

[0024] S7: Calculate the cosine distance between the feature vectors corresponding to the two images, combine all the calculated distances into a vector, input it into a multilayer perceptron, and use the output as the final similarity score. The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions made by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting misassembly or omission of aircraft fuselage parts based on deep learning, characterized in that, Includes the following steps: S1: Construct an attention-based image similarity calculation model that integrates image and camera parameters. The model includes a ViT feature extraction layer, a multilayer perceptron layer, a cross-attention layer, a self-attention layer, a contrastive feature extraction layer, and a similarity calculation layer. The images include standard images of a fully assembled and error-free aircraft fuselage and images of the aircraft fuselage to be inspected. S2: Input the images into the ViT feature extraction layer to extract multiple feature vectors corresponding to each image; S3: Input the camera parameters into the multilayer perceptron, convert the camera parameters into a vector with the same dimension as the feature vector extracted by the ViT feature extraction layer, and add this vector to each feature vector extracted by the ViT feature extraction layer; S4: Input the multiple feature vectors corresponding to each image obtained after processing S3 into the cross-attention layer for cross-attention calculation, and then add them to the multiple feature vectors corresponding to each image obtained in S2 to complete the skip connection and obtain the feature vector after skip connection. S5: Input the feature vector after the jump connection into the attention layer, and perform internal feature integration on the feature vector after the image information interaction; S6: Input the multiple feature vectors corresponding to each image obtained after processing S5 into the same feature extraction network to calculate the feature vectors used for final comparison; S7: Calculate the cosine distance between the feature vectors corresponding to each image for the final comparison, combine all the calculated distances into a vector, input it into the multilayer perceptron layer, and output the final similarity score. The similarity score can directly reflect the similarity of the images and further reflect the errors and omissions of the aircraft body parts.

2. The method for detecting misassembly or omission of aircraft fuselage parts based on deep learning according to claim 1, characterized in that, The camera parameters include camera focal length, camera principal point, distortion correction parameters, camera position coordinates, and three camera rotation angles.

Images