Neural network-based aircraft skin gluing quality defect detection method

Abstract

Disclosed in the present invention is a neural network-based aircraft skin gluing quality defect detection method, comprising: data acquisition: using a camera to photograph an aircraft skin to acquire picture data; preprocessing the acquired picture data; annotating the data by means of annotation software to acquire a data set for network training; establishing a feature erasing and boundary refinement-based defect detection network model, wherein the defect detection network model comprises a feature extraction network, a semantically-guided feature erasing module, a multi-scale feature fusion network, and a boundary refinement-based defect prediction network which are connected in sequence; using the data set to train the network model, and storing parameters of the trained model; and using the trained network model to detect a directly acquired skin gluing image, and outputting the detection result. Nondestructive detection of aircraft skin gluing defects can be quickly and accurately achieved, thereby promoting high-quality intelligent manufacturing processes of skins.

Description

A neural network-based method for detecting quality defects in aircraft skin gluing

[0001] This application claims the priority of the Chinese patent application submitted to the China Patent Office on June 8, 2023, the application number of which is 202310676359X, and the name of the invention is “A method for detecting quality defects of aircraft skin gluing based on neural network”, all or part of which is incorporated by reference into this application. Technical Field

[0002] The invention belongs to the technical field of aircraft skin defect detection, and in particular relates to a method for detecting aircraft skin gluing quality defects based on a neural network. Background Art

[0003] With the rapid development of science and technology in my country, aircraft play a key role in various fields such as military, transportation, and agriculture. Among them, aircraft skin parts are an important part of aircraft. Ensuring their manufacturing quality is a key factor in determining the overall performance and safe operation of the aircraft.

[0004] The primary cause of surface damage and defects in aircraft skins is the periodic expansion and contraction of the skin surface during each takeoff and landing. This cyclical expansion and contraction can lead to microcracks in the material surrounding the rivets. This can be exacerbated by harsh flight conditions, further accelerating the development of these cracks and promoting corrosion. These defects not only affect the aesthetic appearance of the aircraft skin but also compromise its structural integrity, reducing its strength and posing a serious threat to the safety of pilots and passengers.

[0005] Traditional aircraft skin defect detection often relies on visual inspection by technicians. Manual inspection is closely related to the technicians' experience and sense of responsibility, and has significant limitations. It is prone to omissions, mistakes, and omissions, and its detection efficiency is very low. With the continuous improvement of aircraft equipment performance, higher requirements are also placed on its detection technology. There is an urgent need for corresponding detection technology to be rapidly advanced in the direction of intelligence, integration, digitization, and online. Currently, most aviation manufacturing companies in my country have widely adopted digital measurement equipment such as lidar, laser trackers, and total stations for aircraft skin surface defect detection. They are gradually moving away from the previous detection methods that relied on process equipment such as mold lines and templates, but manual inspection by technicians is still the main method. To address the prominent problems of heavy reliance on manual acquisition of detection technology, poor consistency, and low efficiency, a neural network-based method for aircraft skin gluing quality defect detection is proposed.

[0006] Summary of the Invention

[0007] In response to the above technical problems, the present invention provides a method for detecting aircraft skin gluing quality defects based on a neural network.

[0008] The technical solution adopted by the present invention to solve the technical problem is:

[0009] A method for detecting aircraft skin gluing quality defects based on a neural network, the method comprising the following steps:

[0010] S100: Use a high-definition industrial camera to take pictures of the aircraft skin to obtain image data and pre-process the image data;

[0011] S200: labeling the pre-processed data using labeling software to obtain a data set for network training;

[0012] S300: Establishing a defect detection network model based on feature erasure and boundary refinement, wherein the defect detection network model includes a feature extraction network, a semantic-guided feature erasure module, a multi-scale feature fusion network, and a defect prediction network based on boundary refinement, which are connected in sequence. The feature extraction network is used to extract a multi-scale feature map; the semantic-guided feature erasure module is used to process the multi-scale feature map so that a preset region of the feature map has a preset probability of being set to zero; the multi-scale feature fusion network is used to deeply fuse the processed features of different scales to obtain a fused multi-scale feature map; the defect prediction network based on boundary refinement is used to perform prediction based on the fused multi-scale feature map to obtain a classification prediction result and a box prediction result;

[0013] S400: Using the data set to train the defect detection network model to obtain classification prediction results and box prediction results, performing back propagation to update the network weights based on the classification prediction results, box prediction results, and a preset network loss function, and obtaining a trained defect detection network model after completing the preset training rounds;

[0014] S500: Use the trained defect detection network model to detect the directly collected skin glue coating image and obtain the quality defect detection result.

[0015] Preferably, in S300, the feature extraction network is used to extract a multi-scale feature map, and the semantic-guided feature erasure module is used to process the multi-scale feature map so that a preset area of ​​the feature map has a preset probability of being set to zero, including:

[0016] S311: After the defect images in the dataset are extracted by the residual network in the feature extraction network, three input feature maps with different scales are obtained;

[0017] S312: Resize any input feature map F according to a preset size. pre Cut the blocks to get feature blocks f with the same size and number of channelsn , where n represents the number of feature blocks;

[0018] S313: Input feature map F pre Input into the global average pooling layer to obtain the global semantic feature g with global semantic feature information;

[0019] S314: Calculate each feature block f n The semantic similarity cos_sim between the global semantic feature g and the cosine distance is used as the similarity measure, and the calculation formula is as follows:

[0020] S315: Sort by semantic similarity cos_sim from large to small to obtain L cs , and take out the first K matrix blocks f with high similarity k ;

[0021] S316: f k The feature is input into the DropOut layer, and the probability of DropOut is set, i.e. f k Each element in the feature has a preset probability of being set to zero.

[0022] Preferably, the defect prediction network based on boundary refinement in S300 is used to perform prediction based on the fused multi-scale feature map to obtain classification prediction results and box prediction results, including:

[0023] S321: The fused multi-scale feature map F f Input into the defect feature enhancement network to obtain the defect shape enhanced feature F s ;

[0024] S322: Enhance the defect shape to enhance the enhanced feature F s Input into the coarse classification branch and the coarse frame prediction branch respectively to obtain the coarse classification result Cls coarse , enhanced classification features, rough frame prediction results Bbox coarse and the enhanced coarse frame prediction; the coarse classification results and the enhanced classification features are input into the boundary perception module of the coarse classification branch to obtain the refined classification features F cls_g , the coarse box prediction result and the enhanced box prediction are input into the boundary perception module of the coarse box prediction branch to obtain the refined box prediction feature F bbox_g ;

[0025] S323: Refine the classification feature F cls_g and refined box prediction feature F bbox_g The features F after defect shape enhancement s Fusion, respectively input into two 1x1 convolutional layers to obtain the final classification prediction result Clsrefine and box prediction result Bbox refine .

[0026] Preferably, S321 includes:

[0027] S3211: The fused multi-scale feature map F f Input to the defect feature enhancement network for horizontal and vertical global average pooling operations to obtain the horizontal feature g with slender shape perception ability h and longitudinal feature g w ;

[0028] S3212: Bilinear interpolation of lateral features g h and longitudinal feature g w Interpolate and obtain the fused multi-scale feature map F f Horizontal features G of uniform size w and longitudinal feature G h , specifically: G h =G inter (GAP h (F f )) G w =F inter (GAP w (F f ))

[0029] Among them, F inter Represents bilinear interpolation, GAP h and GAP w Represents the vertical and horizontal global average pooling operations respectively;

[0030] S3213: Then the horizontal feature G w and longitudinal feature G h The corresponding elements are added to obtain the fusion features with vertical and horizontal perception, and then the 1x1 convolution and Sigmoid layers are sequentially passed to obtain the fused multi-scale feature map F f Weights w of consistent size;

[0031] S3214: Combine the weight w with the fused multi-scale feature map F f Perform element-wise multiplication to obtain the enhanced feature F of the defect shape s , the calculation formula is as follows: w=sigmoid(conv1(G h +G w )) F s =F f w

[0032] Among them, Sigmoid represents the Sigmoid layer, conv1 represents 1x1 convolution, “+” represents element-wise addition, and “·” represents element-wise multiplication.

[0033] Preferably, the coarse classification branch and the coarse frame prediction branch each include four 3x3 convolutional layers and one 1x1 convolutional layer, and S322 includes:

[0034] S3221: Feature F after defect shape enhancement s Input into the coarse classification branch and the coarse frame prediction branch respectively, and obtain the enhanced classification feature F' through four 3x3 convolutional layers s_cls And the enhanced rough frame prediction feature F' s_bbox , enhanced classification feature F' s_cls The coarse classification result Cls is obtained through the 1x1 convolution layer output coarse , enhanced coarse frame prediction feature F' s_bbox The coarse frame coordinate offset ΔBbox is obtained by outputting the 1x1 convolution layer coarse , offset the rough frame coordinates by ΔBbox coarse Decode to get the predicted box coordinates bbpx coarse ;

[0035] S3222: predict the box coordinates ΔBbox coarse And the enhanced classification feature F' s_cls The boundary perception module input to the coarse classification branch obtains the boundary-aware refined classification feature F cls_g , the predicted box coordinate ΔBbox coarse And the enhanced rough frame prediction feature F' s_bbox The boundary perception module input to the coarse frame prediction branch obtains the boundary-aware refined frame prediction feature F bbox_g .

[0036] Preferably, S3222 includes:

[0037] S32221: Enhanced classification feature F' s_cls And the enhanced rough frame prediction feature F' s_bbox Input into two 3x3 convolutional layers respectively to obtain the central feature F' s_c and boundary characteristics F' s_b , the central feature F' s_c and boundary characteristics F' s_b Perform splicing to obtain the spliced ​​feature F' s_cb ;

[0038] S32222: Feature F' s_b And the rough prediction box coordinate Bbox coarseInput to the boundary alignment module, the boundary alignment module first uniformly samples N points on the four sides of the coarse prediction box, and obtains the feature map F' corresponding to each point through bilinear interpolation s_b The value of , takes the maximum value of the feature among N points as the boundary perception value of the corresponding edge, and obtains the output F' s ' _b ;

[0039] S32223: Feature F' s_cb Input into the 3x3 convolution layer and Sigmoid function at the same time to get the mask of each point: mask=Sigmoid(conv3(F' s_cb ))

[0040] F cls_g F bbox_g =conv1(mask·F' s_cb )

[0041] S32224: Mask and concatenated features F' s_cb Perform element-by-element multiplication and use it as the output F of the boundary perception module after 1x1 convolution dimension reduction. bbox_g or F cls_g .

[0042] Preferably, the calculation formula of S32222 is as follows:

[0043] Where C represents the number of channels, (i, j) represents the coordinates of the feature point, (x0, y0) represents the coordinates of the upper left corner of the coarse prediction box, (x1, y1) represents the coordinates of the lower right corner of the coarse prediction box, k represents the sampling point position (0≤k≤N-1), N represents the number of sampling points, and h and w represent the height and width of the prediction box, respectively.

[0044] Preferably, the preset network loss function includes classification loss Focal Loss and box prediction loss GIoU Loss, and the classification loss includes coarse classification loss Loss cls_coa And the final refined classification loss Loss cls_ref , box prediction loss GIoU Loss includes rough prediction loss Loss reg_coa and detailed prediction of loss reg_ref ;

[0045] Classification loss Focal Loss is calculated as follows: Focal Loss = -y(1-p) γ log(p)-(1-y)p γ log(1-p) Loss cls =Loss cls_coa +γ1·Losscls_ref

[0046] Where y represents the true label of the classification, p represents the predicted value of the coarse classification or fine classification, and γ1 is a hyperparameter used to adjust the weight between the coarse and fine classification losses;

[0047] The box prediction loss GIoULoss is calculated as follows: Loss reg =Loss reg_coa +γ2·Loss reg_ref

[0048] Among them, IoU represents the intersection-over-union ratio between the label and the predicted box, C represents the minimum closed shape, and A c represents the area of ​​C, U represents the area of ​​A and B, and γ2 is a hyperparameter used to adjust the weight between the coarse-fine box prediction loss;

[0049] The final loss function of the entire network is calculated as follows: Loss = Loss reg +γ·Loss cls

[0050] Among them, γ is a hyperparameter used to adjust the weight ratio between classification and box prediction losses. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] FIG1 is a flow chart of a method for detecting aircraft skin gluing quality defects based on a neural network according to an embodiment of the present invention;

[0052] FIG2 is a schematic diagram of the structure of a defect detection network model according to an embodiment of the present invention;

[0053] FIG3 is a schematic diagram of the structure of a semantically guided feature erasure module according to an embodiment of the present invention;

[0054] FIG4 is a schematic diagram of the structure of a defect feature enhancement network according to an embodiment of the present invention;

[0055] FIG5 is a schematic diagram of the structure of a boundary perception module in one embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below with reference to the accompanying drawings.

[0057] In one embodiment, as shown in FIG1 , a neural network-based method for detecting aircraft skin gluing quality defects includes the following steps:

[0058] S100: Use a high-definition industrial camera to take pictures of the aircraft skin to obtain image data and pre-process the image data;

[0059] S200: labeling the pre-processed data using labeling software to obtain a data set for network training;

[0060] S300: Establishing a defect detection network model based on feature erasure and boundary refinement, wherein the defect detection network model includes a feature extraction network, a semantic-guided feature erasure module, a multi-scale feature fusion network, and a defect prediction network based on boundary refinement, which are connected in sequence. The feature extraction network is used to extract a multi-scale feature map; the semantic-guided feature erasure module is used to process the multi-scale feature map so that a preset region of the feature map has a preset probability of being set to zero; the multi-scale feature fusion network is used to deeply fuse the processed features of different scales to obtain a fused multi-scale feature map; the defect prediction network based on boundary refinement is used to perform prediction based on the fused multi-scale feature map to obtain a classification prediction result and a box prediction result;

[0061] S400: Using the data set to train the defect detection network model to obtain classification prediction results and box prediction results, performing back propagation to update the network weights based on the classification prediction results, box prediction results, and a preset network loss function, and obtaining a trained defect detection network model after completing the preset training rounds;

[0062] S500: Use the trained defect detection network model to detect the directly collected skin glue coating image and obtain the quality defect detection result.

[0063] Specifically, the structural diagram of the defect detection network model is shown in FIG2 .

[0064] The above-mentioned neural network-based aircraft skin gluing quality defect detection method, based on the defect detection network model of feature erasure and boundary refinement, can quickly and accurately realize non-destructive detection of aircraft skin gluing defects and promote the high-quality intelligent manufacturing process of skin.

[0065] In one embodiment, as shown in FIG3 , the feature extraction network in S300 is used to extract a multi-scale feature map, and the semantic-guided feature erasure module is used to process the multi-scale feature map so that a predetermined region of the feature map has a predetermined probability of being set to zero, including:

[0066] S311: After the defect images in the dataset are extracted by the residual network in the feature extraction network, three input feature maps with different scales are obtained;

[0067] S312: Resize any input feature map F according to a preset size. pre Cut the blocks to get feature blocks f with the same size and number of channels n , where n represents the number of feature blocks;

[0068] S313: Input feature map F pre Input into the global average pooling layer to obtain the global semantic feature g with global semantic feature information;

[0069] S314: Calculate each feature block f n The semantic similarity cos_sim between the global semantic feature g and the cosine distance is used as the similarity measure, and the calculation formula is as follows:

[0070] S315: Sort by semantic similarity cos_sim from large to small to obtain L cs , and take out the first K matrix blocks f with high similarity k ;

[0071] S316: f k The feature is input into the DropOut layer, and the probability of DropOut is set, i.e. f k Each element in the feature has a preset probability of being set to zero

[0072] Specifically, after the defect image is extracted by the residual network, three feature maps F1, F2, and F3 with different scales are obtained. In order to enhance the robustness of the network, a semantic-guided feature erasure module is used to process the features so that some areas of the feature map have a certain probability of being set to zero. In this embodiment, the DropOut probability is set to 0.4, that is, f k Each element in the feature has a probability of 0.4 being set to 0, which can enhance the feature extraction ability of the neural network and make the extracted features more robust. k It has a high similarity with the global semantic information, so f k The feature is more discriminative than other features.

[0073] Furthermore, the feature fusion module often includes a feature pyramid FPN (Feature Pyramid Network). The purpose of the feature pyramid is to obtain feature maps with high-level semantic information and low-level positional information, and then deeply fuse features of different scales. Since the feature maps of different layers of ResNet are of different sizes, the receptive fields of the feature maps mapped back to the original image are also different, and usually high-level features are more semantic, while low-level features belong to pixel-level positional information. Feature fusion is performed through the horizontal and vertical connections of FPN, so that high-level semantic features and low-level pixel features can be effectively fused. Since the surface defects of the aircraft skin are small and the respective features between the defects and the background are not obvious, the high-level feature maps with semantic information are passed through the top-down feature fusion module, so that the underlying pixel-level features can have high-level semantic information, thereby improving the detection accuracy. To this end, a top-down feature pyramid structure is adopted to fuse the features of foreign objects. The proposed method differs from the five-layer FPN structure of the classic object detection algorithm RetinaNet by selecting only four layers of feature maps of varying sizes to construct a feature pyramid structure. These feature pyramids have 256, 512, 1024, and 2048 channels, respectively. These channels are then unified to 256 dimensions through a 1×1 convolution. This reduces the number of parameters used in the detection process while ensuring accurate detection of aircraft skin defects. This optimizes the detection network structure, reduces computing power consumption, and accelerates detection to a certain extent, saving training time.

[0074] In one embodiment, as shown in FIG2 , the defect prediction network based on boundary refinement in S300 is used to predict based on the fused multi-scale feature map to obtain classification prediction results and box prediction results, including:

[0075] S321: The fused multi-scale feature map F f Input into the defect feature enhancement network to obtain the defect shape enhanced feature F s ;

[0076] S322: Enhance the defect shape to enhance the enhanced feature F s Input into the coarse classification branch and the coarse frame prediction branch respectively to obtain the coarse classification result Cls coarse , enhanced classification features, rough frame prediction results Bbox coarse and the enhanced coarse frame prediction; the coarse classification results and the enhanced classification features are input into the boundary perception module of the coarse classification branch to obtain the refined classification features F cls_g , the coarse box prediction result and the enhanced box prediction are input into the boundary perception module of the coarse box prediction branch to obtain the refined box prediction feature F bbox_g ;

[0077] S323: Refine the classification feature Fcls_g and refined box prediction feature F bbox_g The features F after defect shape enhancement s Fusion, respectively input into two 1x1 convolutional layers to obtain the final classification prediction result Cls refine and box prediction result Bbox refine .

[0078] In one embodiment, as shown in FIG4 , S321 includes:

[0079] S3211: The fused multi-scale feature map F f Input to the defect feature enhancement network for horizontal and vertical global average pooling operations to obtain the horizontal feature g with slender shape perception ability h and longitudinal feature g w ;

[0080] S3212: Bilinear interpolation of lateral features g h and longitudinal feature g w Interpolate and obtain the fused multi-scale feature map F f Horizontal features G of uniform size w and longitudinal feature G h , specifically: G h =F inter (GAP h (F f )) G w =F inter (GAP w (F f ))

[0081] Among them, F inter Represents bilinear interpolation, GAP h and GAP w Represents the vertical and horizontal global average pooling operations respectively;

[0082] S3213: Then the horizontal feature G w and longitudinal feature G h The corresponding elements are added to obtain the fusion features with vertical and horizontal perception, and then the 1x1 convolution and Sigmoid layers are sequentially passed to obtain the fused multi-scale feature map F f Weights w of consistent size;

[0083] S3214: Combine the weight w with the fused multi-scale feature map F f Perform element-wise multiplication to obtain the enhanced feature F of the defect shape s , the calculation formula is as follows: w=Sigmoid(conv1(G h +G w)) F s =F f w

[0084] Among them, Sigmoid represents the Sigmoid layer, conv1 represents 1x1 convolution, “+” represents element-wise addition, and “·” represents element-wise multiplication.

[0085] Specifically, since most skin coating defects are slender and weak, their morphological characteristics are easily ignored by the network model. Using the defect feature enhancement network to enhance the skin coating defects can ensure the effective enhancement of their morphological characteristics.

[0086] In one embodiment, as shown in FIG2 , the coarse classification branch and the coarse frame prediction branch each include four 3x3 convolutional layers and one 1x1 convolutional layer, and S322 includes:

[0087] S3221: Feature F after defect shape enhancement s Input into the coarse classification branch and the coarse frame prediction branch respectively, and obtain the enhanced classification feature F' through four 3x3 convolutional layers s_cls And the enhanced rough frame prediction feature F' s_bbox , enhanced classification feature F' s_cls The coarse classification result Cls is obtained through the 1x1 convolution layer output coarse , enhanced coarse frame prediction feature F' s_bbox The coarse frame coordinate offset ΔBbox is obtained by outputting the 1x1 convolution layer coarse , offset the rough frame coordinates by ΔBBox coarse Decode to get the predicted box coordinates Bbox coarse ;

[0088] S3222: predict the box coordinates ΔBbox coarse And the enhanced classification feature F' s_cls The boundary perception module input to the coarse classification branch obtains the boundary-aware refined classification feature F cls_g , the predicted box coordinate ΔBbox coarse And the enhanced rough frame prediction feature F' s_bbox The boundary perception module input to the coarse frame prediction branch obtains the boundary-aware refined frame prediction feature F bbox_g .

[0089] In one embodiment, as shown in FIG5 , S3222 includes:

[0090] S32221: Enhanced classification feature F' s_cls And the enhanced rough frame prediction feature F' s_bbox Input into two 3x3 convolutional layers respectively to obtain the central feature F' s_c and boundary characteristics F's_b , the central feature F' s_c and boundary characteristics F' s_b Perform splicing to obtain the spliced ​​feature F' s_cb ;

[0091] S32222: Feature F' s_b And the rough prediction box coordinate Bbox coarse Input to the boundary alignment module, the boundary alignment module first uniformly samples N points on the four sides of the coarse prediction box, and obtains the feature map F' corresponding to each point through bilinear interpolation s_b The maximum value of the feature among N points is taken as the boundary perception value of the corresponding edge, and the output F" is obtained. s_b ;

[0092] S32223: Feature F' s_cb Input into 3x3 convolution layer and Sigmoid function at the same time to get the mask of each point: mask=Sigmoid(con3(F' s_cb )) F cls_g F bbox_g =conv1(mask·F' s_cb )

[0093] S32224: Mask and concatenated features F' s_cb Perform element-by-element multiplication and use it as the output F of the boundary perception module after 1x1 convolution dimension reduction. bbox_g or F cls_g .

[0094] Furthermore, the calculation formula of S32222 is as follows:

[0095] Where C represents the number of channels, (i, j) represents the coordinates of the feature point, (x0, y0) represents the coordinates of the upper left corner of the rough prediction box, (x1, y1) represents the coordinates of the lower right corner of the rough prediction box, k represents the sampling point position (0≤k≤N-1), N represents the number of sampling points, h and w represent the height and width of the prediction box respectively

[0096] Furthermore, according to the idea of ​​residual learning, the obtained feature F with boundary perception is bbox_g or F cls_g and the shape-enhanced feature F s Add the elements and input them into the 1x1 convolution layer to get the final classification score Cls after refinement refine Or the prediction result Bbox of the box offset refine .

[0097] Finally, the losses between the coarse classification, coarse box prediction, final classification prediction, final box prediction and the true label are calculated respectively.

[0098] In one embodiment, the preset network loss function includes classification loss Focal Loss and box prediction loss GIoU Loss, and the classification loss includes coarse classification loss Loss cls_coa And the final refined classification loss Loss cls_ref , box prediction loss GIoU Loss includes rough prediction loss Loss reg_coa and detailed prediction of loss reg_ref ;

[0099] Classification loss Focal Loss is calculated as follows: Focal Loss = -y(1-p) γ log(p)-(1-y)p γ log(1-p) Loss cls =Loss cls_coa +γ1·Loss cls_ref

[0100] Among them, y represents the true label of the classification, p represents the predicted value of coarse classification or fine classification, and γ1 is a hyperparameter used to adjust the weight between coarse and fine classification losses.

[0101] Box prediction loss GIoU Loss is calculated as follows: Loss reg =Loss reg_coa +γ2·Loss reg_ref

[0102] Among them, IoU represents the intersection-over-union ratio between the label and the predicted box, C represents the minimum closed shape, and A c represents the area of ​​C, U represents the area of ​​A and B, and γ2 is a hyperparameter used to adjust the weight between the coarse-fine box prediction loss.

[0103] The final loss function of the entire network is calculated as follows: Loss = Loss reg +γ·Loss cls

[0104] Among them, γ is a hyperparameter used to weight the loss between classification and box prediction.

[0105] Specifically, the classification loss includes the coarse classification loss Loss cls_coa And the final refined classification loss Loss cls_refIn order to alleviate the imbalance problem of positive and negative samples, Focal Loss is used as the classification loss; the box prediction loss also includes the coarse prediction loss Loss. reg_coa and detailed prediction of loss reg_ref Both,losses use GIoU Loss as the box prediction loss.

[0106] Furthermore, the neural network is trained using back propagation and stochastic gradient descent algorithms, and the training weights are saved.

[0107] First, the backpropagation algorithm calculates the gradient of the loss function with respect to each parameter, using the chain rule to traverse the network in reverse order from the output layer to the input layer. The backpropagation algorithm reuses the intermediate values ​​stored in the forward propagation to avoid repeated calculations and save computation time.

[0108] If gradient descent is used, the computational cost of each independent variable iteration is O(n ) , which grows linearly with n (the number of samples). Therefore, when the training data set is large, the computational cost of gradient descent for each iteration will be high. Using stochastic gradient descent can reduce the computational cost generated during iteration. In each iteration of stochastic gradient descent, the algorithm randomly selects a portion of samples and updates the model parameters by calculating the gradients of these samples, thereby gradually approaching the optimal solution. We randomly and uniformly sample an index i from the data sample, where i∈1,…,n, and calculate the gradient To update the weight θ:

[0109] Among them, θ n+1 is the updated value of the neural network parameters, θ n is the current parameter value, is the gradient, J(θ) is the function, and η is the learning rate.

[0110] The training process is repeated, and the steps of forward propagation, loss calculation, backpropagation, and updating weights and biases are repeated until the model converges. The convergence of the model is judged by comparing the changes in the loss function value.

[0111] Save the trained weights. Once the model is trained, save the trained weights. These weights can be used to make predictions. Save the weights to a file so that you can reload them when needed.

[0112] This neural network-based method for detecting aircraft skin gluing defects, based on a defect detection network model that utilizes feature erasure and boundary refinement, can rapidly and accurately perform nondestructive detection of aircraft skin gluing defects, promoting high-quality intelligent skin manufacturing. It addresses the existing challenges of heavily manual acquisition of detection technology, resulting in poor consistency and low efficiency.

[0113] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for detecting aircraft skin glue quality defects based on neural network, characterized in that: The method comprises the following steps: S100: photographing the aircraft skin using a high-definition industrial camera to obtain image data, and preprocessing the image data; S200: labeling the pre-processed data through labeling software to obtain a data set for network training; S300: Establishing a defect detection network model based on feature erasure and boundary refinement, wherein the defect detection network model comprises a feature extraction network, a semantically guided feature erasure module, a multi-scale feature fusion network and a defect prediction network based on boundary refinement connected in sequence, wherein the feature extraction network is used to extract a multi-scale feature map; the semantically guided feature erasure module is used to process the multi-scale feature map so that a preset area of ​​the feature map has a preset probability of being set to zero; the multi-scale feature fusion network is used to deeply fuse the processed features of different scales to obtain a fused multi-scale feature map; the defect prediction network based on boundary refinement is used to predict according to the fused multi-scale feature map to obtain a classification prediction result and a box prediction result; S400: using the data set to train the defect detection network model to obtain classification prediction results and frame prediction results, performing back propagation to update the network weights according to the classification prediction results, the frame prediction results and a preset network loss function, and obtaining a trained defect detection network model after completing preset training rounds; S500: using the trained defect detection network model to detect the directly collected skin glue coating image, and obtaining a quality defect detection result; The feature extraction network in S300 is used to extract a multi-scale feature map, and the semantic-guided feature erasure module is used to process the multi-scale feature map so that a preset area of ​​the feature map has a preset probability of being set to zero, including: S311: After the defect images in the data set are extracted by the residual network in the feature extraction network, three input feature maps with different scales are obtained; S312: Resize any of the input feature maps F according to a preset size. pre Cut the blocks to get feature blocks f with the same size and number of channels n , where n represents the number of feature blocks; S313: Input feature map F pre Input into the global average pooling layer to obtain the global semantic feature g with global semantic feature information; S314: Calculate each feature block f n The semantic similarity cos_sim between the global semantic feature g and the global semantic feature g, wherein the similarity measurement method adopts the cosine distance, and the calculation formula is as follows: S315: Sort the semantic similarity cos_sim from large to small to obtain L cs , and take out the first K matrix blocks with high similarity f k ; S316: The f k The feature is input into the DropOut layer, and the probability of DropOut is set, i.e. f k Each element in the feature has a preset probability of being set to zero.

2. The method according to claim 1, characterized in that The defect prediction network based on boundary refinement in S300 is used to predict the classification prediction results and box prediction results based on the fused multi-scale feature map, including: S321: The fused multi-scale feature map F f Input into the defect feature enhancement network to obtain the defect shape enhanced feature F s ; S322: Enhance the defect shape and the enhanced feature F s Input into the coarse classification branch and the coarse frame prediction branch respectively to obtain the coarse classification result Cls coarse , enhanced classification features, rough frame prediction results Bbox coarse and the enhanced rough frame prediction; the rough classification result and the enhanced classification feature are input into the boundary perception module of the rough classification branch to obtain the refined classification feature F cls_g ,Will The coarse frame prediction result and the enhanced frame prediction are input into the boundary perception module of the coarse frame prediction branch to obtain the refined frame prediction feature F bbox_g ; S323: Refine the classification feature F cls_g and the refined box prediction feature F bbox_g and the defect shape enhanced features F s After fusion, they are input into two 1x1 convolutional layers to obtain the final classification prediction result Cls refine And box prediction result Bbox refine .

3. The method according to claim 2, characterized in that S321 includes: S3211: The fused multi-scale feature map F f The defect feature enhancement network is input to perform global average pooling operations in the horizontal and vertical directions to obtain the horizontal feature g with slender shape perception ability. h and longitudinal characteristics g w ; S3212: The lateral feature g is interpolated by bilinear interpolation method. h and longitudinal characteristics g w Interpolate to obtain the fused multi-scale feature map F f The lateral features G of the same size w and longitudinal feature G h , specifically: G h =F inter (GAP h (F f )) G w =F inter (GAP w (F f )) Among them, F inter Represents bilinear interpolation, GAP h and GAP w Represents the vertical and horizontal global average pooling operations respectively; S3213: Then the horizontal feature G w and the longitudinal characteristic G h The corresponding elements are added to obtain the fusion features with vertical and horizontal perception, and then the 1x1 convolution and Sigmoid layers are sequentially passed to obtain the fused multi-scale feature map F f Weights w of the same size; S3214: The weight w is combined with the fused multi-scale feature map F f Multiply the elements to get the enhanced feature F of the defect shape s , the calculation formula is as follows: w=Sigmoid(conv1(G h +G w )) F s =F f ·w Among them, Sigmoid represents the Sigmoid layer, conv1 represents 1x1 convolution, "+" represents element-wise addition, and "·" represents element-by-element multiplication.

4. The method according to claim 3, characterized in that The coarse classification branch and the coarse frame prediction branch both include four 3x3 convolutional layers and one 1x1 convolutional layer, and S322 includes: S3221: Enhance the defect shape to feature F s The four 3x3 convolutional layers are used to obtain the enhanced classification feature F'. s_cls And the enhanced rough frame prediction feature F' s_bbox , the enhanced classification feature F' s_cls The coarse classification result Cls is obtained through the 1x1 convolution layer output coarse , the enhanced rough frame prediction feature F' s_bbox The rough frame coordinate offset ΔBbox is obtained through the 1x1 convolution layer output coarse , the rough frame coordinate offset ΔBbox coarse Decode to get the predicted box coordinates Bbox coarse ; S3222: The predicted box coordinates Bbox coarse and the enhanced classification feature F' s_cls The boundary perception module input to the coarse classification branch obtains the refined classification feature F with boundary perception cls_g , the predicted box coordinates Bbox coarse and the enhanced rough frame prediction feature F' s_bbox The boundary perception module input to the coarse frame prediction branch obtains the boundary-aware refined frame prediction feature F bbox_g .

5. The method according to claim 4, characterized in that S3222 includes: S32221: The enhanced classification feature F' s_cls and the enhanced rough frame prediction feature F' s_bbox Input into two 3x3 convolutional layers to get the central feature F' s_c and boundary feature F' s_b , the central feature F' s_c and the boundary feature F' s_b Perform splicing to obtain the spliced ​​feature F' s_cb ; S32222: The feature F' s_b And the predicted box coordinates Bbox coarse Input to the boundary alignment module, the boundary alignment module first uniformly samples N points on the four sides of the rough prediction box, and obtains the feature map F' corresponding to each point through bilinear interpolation s_b The maximum value of the feature among N points is taken as the boundary perception value of the corresponding edge, and the output F" is obtained. s_b ; S32223: The feature F' s_cb Input into the 3x3 convolution layer and the Sigmoid function at the same time to get the mask of each point: mask=Sigmoid(conv3(F’ s_cb )) F cls_g or F bbox_g =conv1(mask·F’ s_cb ) S32224: Combine the mask and the concatenated feature F' s_cb Element-wise multiplication is performed, and after 1x1 convolution, the dimension is reduced as the output F of the boundary perception module. bbox_g or F cls_g .

6. The method according to claim 5, characterized in that The calculation formula of S32222 is as follows: Among them, C represents the number of channels, (i, j) represents the coordinates of the feature point, (x0, y0) represents the coordinates of the upper left corner of the rough prediction box, (x1, y1) represents the coordinates of the lower right corner of the rough prediction box, k represents the sampling point position, 0≤k≤N-1, N represents the number of sampling points, h and w represent the height of the prediction box, respectively. and width.

7. The method according to claim 6, characterized in that The preset network loss function includes classification loss Focal Loss and box prediction loss GIoULoss. The classification loss includes coarse classification loss Loss cls_coa And the final refined classification loss Loss cls_ref , the box prediction loss GIoU Loss includes the rough prediction loss Loss reg_coa and detailed prediction of loss reg_ref ; The classification loss Focal Loss is calculated as follows: Focal Loss=-y(1-p) γ log(p)-(1-y)p γ log(1-p) Loss cls =Loss cls_coa +γ1·Loss cls_ref Among them, y represents the true label of the classification, p represents the predicted value of the coarse classification or fine classification, and γ1 is a hyperparameter used to adjust the weight between the coarse-fine classification loss; The box prediction loss GIoU Loss is calculated as follows: Loss reg =Loss reg_coa +γ2·Loss reg_ref Among them, IoU represents the intersection-over-union ratio between the label and the predicted box, C represents the minimum closed shape, and A c represents the area of ​​C, U represents the area of ​​A and B, and γ2 is a hyperparameter used to adjust the weight between the coarse-fine box prediction loss; Finally, the loss function of the entire network is calculated as follows: Loss=Loss reg +γ·Loss cls Among them, γ is a hyperparameter used to adjust the weight ratio between classification and box prediction losses.

Images