Unmanned aerial vehicle fault batch identification method based on image processing
By building a drone fault retrieval database and image processing technology, using a variety of convolutional neural networks and generative adversarial networks for training, the problems of low efficiency and insufficient accuracy of drone fault detection are solved, and efficient batch identification of unmanned component failures are achieved.
Patent Information
- Application Number
- CN202510341485.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-21
- Publication Date
- 2025-07-29
AI Technical Summary
The existing drone fault detection methods are inefficient, the accuracy is affected by the experience of the operator, and batch inspection cannot be carried out, and minor damage is easily ignored.
A drone fault search database is constructed, and the residual-connected convolutional neural network, composite scaling convolutional neural network and lightweight convolutional neural network are used for training. Combined with the generation of adversarial network expansion samples, the unmanned mechanism parts are identified and segmented through image processing, and the recognition model with the best performance is matched for fault identification.
It improves the efficiency and accuracy of unmanned component failure identification, and realizes batch identification of unmanned component failures.
Smart Images
Figure CN120388205A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of UAV fault identification, and specifically to a method for batch identification of UAV faults based on image processing. Background Art
[0002] With the rapid development of UAV technology, its applications in fields such as agricultural inspection, logistics transportation, and disaster rescue are becoming increasingly widespread.
[0003] However, after long-term operation of UAVs in complex operating environments, their core structural components (such as propellers, arms, fuselage frames, and screws) are prone to damage due to mechanical fatigue, collision, corrosion, or assembly defects, resulting in a decline in flight stability and even a risk of crashing.
[0004] Currently, the mainstream detection methods rely on manual visual inspection or physical detection using handheld devices (such as torque wrenches), which cannot perform batch detection of UAVs, have low efficiency, and the detection accuracy is affected by the experience of operators, and small losses are easily overlooked. Summary of the Invention
[0005] The purpose of the present invention is to provide a method for batch identification of UAV faults based on image processing, including the following steps:
[0006] 1) Construct a UAV fault retrieval database; the UAV fault retrieval database includes n types of UAV component types and the state types of each UAV component;
[0007] The UAV fault retrieval database is as follows:
[0008]
[0009] Among them, A represents the matrix of UAV component types; a n represents the nth type of UAV component type; B represents the matrix of UAV component state types; there is a mapping relationship between matrix A and matrix B; b n represents the state type vector of the nth type of UAV component; represents the mth n state type of the nth type of UAV component;
[0010] 2) Collect m ×M images of the ith type of UAV component in the state type i and label them with the state type label, thereby constructing a small sample image set, that is:
[0011]
[0012] In the formula, is the image vector of the ith type of UAV component in the state type is the Mth image of the ith unmanned aircraft component in the state type ;
[0013] 3) Divide the small sample image set into a training set and a test set; record the average resolution and size χ×φ of the small sample image set; I l is the resolution of the lth image in the small sample image set;
[0014] 4) Use the training set to train the convolutional neural network based on residual connection, the convolutional neural network based on compound scaling, and the lightweight convolutional neural network respectively, to obtain the unmanned aircraft component fault recognition model F1, the unmanned aircraft component fault recognition model F2, and the unmanned aircraft component fault recognition model F3;
[0015] The convolutional neural network based on residual connection includes an input layer, a convolutional layer, at least one residual block, and an output layer;
[0016] The convolutional neural network based on compound scaling includes an input layer, a convolutional layer, an EfficientNet module, a pooling layer, a fully connected layer, a batch normalization layer, a compound scaling layer, and an output layer;
[0017] The lightweight convolutional neural network includes an input layer, a lightweight convolutional unit, and an output layer;
[0018] 5) Use the test set to test the performance of the trained unmanned aircraft component fault recognition models F1, F2, and F3, and select the unmanned aircraft component fault recognition model with the best performance as the model to be optimized according to the performance index objective function;
[0019] The performance index objective function is as follows:
[0020] max{ω1f 11 +ω2f 12 ,ω1f 21 +ω2f 22 ,ω1f 31 +ω2f 32}(3)
[0021] In the formula, f 11 , f 21 , f 31 are the average recognition rates of the unmanned aircraft component fault recognition models F1, F2, and F3; f 12 , f 22 , f 32 are the accuracies of the unmanned aircraft component fault recognition models F1, F2, and F3; ω1 and ω2 are weights;
[0022] 6) Augment the small sample image set using a generative adversarial network to obtain an optimized image set, i.e.:
[0023]
[0024] In the formula, is the image vector of the i-th unmanned aircraft component generated by the generative adversarial network under the state type ; is the M'-th image of the i-th unmanned aircraft component generated by the generative adversarial network under the state type ; M' > M;
[0025] The generative adversarial network includes a generator and a discriminator; the generator includes an input layer, a fully connected layer, a convolutional layer, and an output layer; the discriminator includes an input layer, a convolutional layer, a fully connected layer, and an output layer;
[0026] The objective function of the generative adversarial network is as follows:
[0027]
[0028] where G represents the generator, D represents the discriminator, Y represents the state type label, p Z (Z) represents the prior noise distribution; E represents the expectation; Z represents the noise; X is the data of the small sample image set; p data (X) represents the data distribution of the small sample image set;
[0029] 7) Train the model to be optimized using the optimized image set to obtain the optimal fault identification model for the i-th unmanned aircraft component;
[0030] 8) Determine whether i > n holds. If so, go to step 9). Otherwise, set i = i + 1 and return to step 2);
[0031] 9) Write the optimal fault identification models of all unmanned aircraft components into the unmanned aircraft fault retrieval database and form a mapping with n types of unmanned aircraft components, i.e.:
[0032]
[0033] In the formula, C is the optimal fault identification model matrix corresponding to n types of unmanned aircraft components; c n is the optimal fault identification model for the n-th unmanned aircraft component;
[0034] 10) Select the type of unmanned aircraft component to be detected and call the corresponding optimal fault identification model;
[0035] 11) Obtain the image of the unmanned aircraft to be detected and perform pixel grid division to obtain the pixel grid matrix K of the unmanned aircraft image to be detected, i.e.:
[0036]
[0037] In the formula, k eg is the pixel grid serial number of the e-th row and g-th column of the UAV image to be detected; k egx , k egy is the horizontal and vertical coordinates of the lower left vertex of the pixel grid k eg ;
[0038] 12) Perform UAV component recognition and segmentation on the UAV image to be detected, obtain several UAV component sub-images, and assign codes to each UAV component sub-image;
[0039] The steps of performing UAV component recognition and segmentation include:
[0040] 12.1) Use the Canny algorithm to perform edge extraction on the UAV image to be detected to obtain an edge image;
[0041] Among them, the edge density ρ of the edge image is as follows:
[0042]
[0043] In the formula, N1 is the length of the edge image, N2 represents the width of the edge image, p dj represents the value of the pixel point (d, j) in the edge image, p dj = 1 represents the edge, p dj = 0 represents the background;
[0044] 12.2) Take the Q×Q area centered on each pixel point as a template, and calculate the edge density ρ of each pixel point dj , that is:
[0045]
[0046] 12.3) Based on the edge density ρ of each pixel point dj , perform binary image segmentation on the UAV image to be detected according to the segmentation principle, and segment the background and the UAV component area to obtain a binary image P;
[0047] The segmentation principle is as follows:
[0048]
[0049] In the formula, P(d, j) is the pixel value;
[0050] 12.4) Perform an opening operation on the binary image P to eliminate noise and smooth the boundary to obtain a binary segmentation image p', that is:
[0051]
[0052] In the formula, represents the opening operation; S is the structuring element;
[0053] 12.5) Perform masking processing on the drone image to be detected based on the binary segmentation image p', and obtain the drone component mask image;
[0054] 12.6) Crop the drone components in the drone component mask image and assign codes to construct the drone component sub-image matrix H, that is:
[0055]
[0056] In the formula, is the sth s drone component sub-image in the drone component mask image; is the code of the sth s drone component sub-image; the code is the horizontal and vertical coordinates of the lower left vertex of the central pixel grid of the drone component sub-image in the drone image to be detected;
[0057] 12) Adjust the resolution and size of each drone component sub-image so that the resolution and size of the drone component sub-image are the same as the average resolution and size of the small sample image set;
[0058] Input the adjusted drone component images into the called optimal fault recognition model in sequence for drone component fault recognition, and determine the state of each drone component in the drone image to be detected;
[0059] 13) Based on the codes of the drone components, label the drone component state recognition results on the drone image to be detected.
[0060] Furthermore, the drone components include propellers, arms, fuselage frames, and screws.
[0061] Furthermore, when the drone component is a propeller, the state types of the drone component include normal, broken, bent, and notched.
[0062] When the drone component is an arm, the state types of the drone component include normal, cracked, and notched;
[0063] When the drone component is a fuselage frame, the state types of the drone component include normal, cracked, and notched;
[0064] When the drone component is a screw, the state types of the drone component include normal, loose, and missing; loose means that the screw protrudes more than 0.5 mm from the base surface.
[0065] Furthermore, the number of images in the small sample image set ranges from 1000 to 5000;
[0066] the number of images in the optimized image set is greater than 10 6 .
[0067] Furthermore, in the convolutional neural network based on residual connection, the output of the convolutional layer is shown as follows:
[0068] D' = X' * G' (13)
[0069] In the formula, D' is the output of the convolutional layer; X' is the input; G' is the convolutional kernel;
[0070] In the convolutional neural network based on residual connection, the output of the residual block is shown as follows:
[0071] E' = w2σ(w1x') + w3x' (14)
[0072] In the formula, E' is the output of the residual block; x' is the output of the previous convolutional layer adjacent to the current residual block; w1, w2 are the residual block parameters; σ is the ReLU activation function; w3 is a linear transformation function used to ensure that the dimensions of two adjacent residual blocks are consistent.
[0073] Furthermore, when training the convolutional neural network based on residual connection, the convolutional neural network based on compound scaling, and the lightweight convolutional neural network, the loss function Loss is shown as follows:
[0074]
[0075] In the formula, P' is the number of samples; z is the true label of the sample; z' is the label output by the convolutional neural network; λ is the penalty parameter; ω p' is the connection weight.
[0076] Furthermore, in the generative adversarial network, the loss Loss of the discriminator D is shown as follows:
[0077]
[0078] In the formula, R is the output result of the fully connected layer in the discriminator; c' is the number of state type labels; z j' is the one-hot encoding form of the state type label; p j' is the output probability distribution of the previous layer network.
[0079] Furthermore, in the convolutional neural network based on compound scaling, the output of the EfficientNet module is shown as follows:
[0080] T = Sigmoid(BN(DepthwiseConv(X”)))(17)
[0081] X' = Sigmoid(BN(Conv 1×1 (X)))(18)
[0082] Wherein, Sigmoid is the activation function; BN is batch normalization; Conv 1×1 is a 1×1 convolution operation; DepthwiseConv is a depthwise separable convolution operation; X” is the input.
[0083] Furthermore, in the lightweight convolutional neural network, the steps for the lightweight convolutional unit to process the input image include:
[0084] Performing channel expansion on the input image through a pointwise convolution operation, and then grouping the channels of the input image through a depth convolution operation;
[0085] Normalizing the input of the input layer and each intermediate layer of the lightweight convolutional neural network, and then concatenating the input image with the batch-normalized image;
[0086] After concatenation, perform pointwise convolution to contract the output channels of the lightweight convolutional neural network, and use the activation function R_Hard_Swish to measure the feature map and output the result;
[0087] Among them, the activation function R_Hard_Swish is as follows:
[0088]
[0089] Among them, X represents the input image, ReLU6 represents the ReLU6 activation function, and β is a constant; the value range of β is (0,1).
[0090] Furthermore, the update process of the parameters in the generative adversarial network is as follows:
[0091] S1) Initialize the model parameters w, the exponentially weighted average E[g 2 0 = 0, and set the decay coefficient η and the learning rate α;
[0092] S2) Calculate the gradient g t of the loss function J(w t ) with respect to the parameters, that is:
[0093]
[0094] Wherein, is the gradient symbol; t is the iteration number;
[0095] S3) Update the exponentially decaying average value E[g 2 t , that is:
[0096]
[0097] S4) Update the parameter w t+1 , that is:
[0098]
[0099] In the formula, ⊙ represents element-wise multiplication; ε represents a constant term to prevent the denominator from being 0;
[0100] S5) Determine whether the maximum number of iterations is reached or the difference between the parameters of two adjacent iterations is less than a preset threshold. If so, end the iteration; otherwise, set t = t + 1 and return to step S2).
[0101] The technical effect of the present invention is beyond doubt. The present invention matches the recognition model with the optimal performance for different unmanned aircraft component fault recognition operations through a small batch of samples, and then expands the small batch of samples through a generative adversarial network, and retrains the selected recognition model with the expanded samples, improving the efficiency and accuracy of unmanned aircraft component fault recognition.
[0102] The present invention recognizes, crops, and adjusts the resolution of the unmanned aircraft components in the image, and then processes the unmanned aircraft component image using the recognition model, which can realize the batch recognition of unmanned aircraft component faults. Description of the Drawings
[0103] Figure 1 is a flowchart;
[0104] Figure 2 is a schematic diagram of the lower left vertex of the pixel grid. Detailed Embodiments
[0105] The present invention will be further described below in conjunction with embodiments, but it should not be understood that the above-mentioned subject scope of the present invention is limited to the following embodiments. Without departing from the above technical idea of the present invention, various substitutions and changes made according to ordinary technical knowledge and customary means in the art should be included in the protection scope of the present invention.
[0106] Embodiment 1:
[0107] Refer to Figures 1 to 2 , a method for batch recognition of unmanned aircraft faults based on image processing, including the following steps:
[0108] 1) Construct a database for retrieving unmanned aircraft faults; the database for retrieving unmanned aircraft faults includes n types of unmanned aircraft components and the status types of each unmanned aircraft component;
[0109] The UAV fault retrieval database is as follows:
[0110]
[0111] Among them, A represents the UAV component type matrix; a n represents the nth UAV component type; B represents the UAV component status type matrix; matrix A and matrix B form a mapping relationship; b n represents the status type vector of the nth UAV component; represents the mth n status type of the nth UAV component;
[0112] 2) Collect m ×M images of the ith UAV component under the status type i and label them with the status type label, so as to construct a small sample image set, that is:
[0113]
[0114] In the formula, is the image vector of the ith UAV component under the status type ; is the Mth image of the ith UAV component under the status type ;
[0115] 3) Divide the small sample image set into a training set and a test set; record the average resolution and size χ×φ of the small sample image set; I l is the resolution of the lth image in the small sample image set;
[0116] 4) Use the training set to train the convolutional neural network based on residual connection, the convolutional neural network based on compound scaling, and the lightweight convolutional neural network respectively to obtain the UAV component fault recognition model F1, the UAV component fault recognition model F2, and the UAV component fault recognition model F3;
[0117] The convolutional neural network based on residual connection includes an input layer, a convolutional layer, at least one residual block, and an output layer;
[0118] The convolutional neural network based on compound scaling includes an input layer, a convolutional layer, an EfficientNet module, a pooling layer, a fully connected layer, a batch normalization layer, a compound scaling layer, and an output layer;
[0119] The lightweight convolutional neural network includes an input layer, a lightweight convolutional unit, and an output layer;
[0120] 5) Use the test set to test the performance of the trained UAV component fault recognition models F1, F2, and F3, and select the UAV component fault recognition model with the best performance as the model to be optimized according to the performance index objective function;
[0121] The performance index objective function is as follows:
[0122] max{ω1f 11 +ω2f 12 ,ω1f 21 +ω2f 22 ,ω1f 31 +ω2f 32} (3)
[0123] In the formula, f 11 , f 21 , f 31 are the average recognition rates of the UAV component fault recognition models F1, F2, and F3; f 12 , f 22 , f 32 are the accuracies of the UAV component fault recognition models F1, F2, and F3; ω1 and ω2 are weights;
[0124] 6) Use the generative adversarial network to augment the small sample image set to obtain an optimized image set, that is:
[0125]
[0126] In the formula, is the image vector of the i-th UAV component in the state type generated by the generative adversarial network; is the M'-th image of the i-th UAV component in the state type generated by the generative adversarial network; M' > M;
[0127] The generative adversarial network includes a generator and a discriminator; the generator includes an input layer, a fully connected layer, a convolutional layer, and an output layer; the discriminator includes an input layer, a convolutional layer, a fully connected layer, and an output layer;
[0128] The objective function of the generative adversarial network is as follows:
[0129]
[0130] Among them, G represents the generator, D represents the discriminator, Y represents the state type label, p z (Z) represents the prior noise distribution; E represents the expectation; Z represents the noise; X is the data of the small sample image set; p data(X) represents the data distribution of the small sample image set;
[0131] 7) Use the optimized image set to train the model to be optimized, and obtain the optimal fault recognition model for the i-th type of unmanned aircraft component;
[0132] 8) Determine whether i > n holds. If so, go to step 9). Otherwise, set i = i + 1 and return to step 2);
[0133] 9) Write the optimal fault recognition models of all unmanned aircraft components into the unmanned aircraft fault retrieval database and form a mapping with n types of unmanned aircraft components, that is:
[0134]
[0135] In the formula, C is the optimal fault recognition model matrix corresponding to n types of unmanned aircraft components; c n is the optimal fault recognition model for the n-th type of unmanned aircraft component;
[0136] 10) Select the type of unmanned aircraft component to be detected and call the corresponding optimal fault recognition model;
[0137] 11) Obtain the image of the unmanned aircraft to be detected and perform pixel grid division to obtain the pixel grid matrix K of the unmanned aircraft image to be detected, that is:
[0138]
[0139] In the formula, k eg is the pixel grid serial number of the e-th row and g-th column of the unmanned aircraft image to be detected; k egx , k egy is the horizontal and vertical coordinates of the lower left vertex of the pixel grid k eg ;
[0140] 12) Perform unmanned aircraft component recognition and segmentation on the unmanned aircraft image to be detected, obtain several unmanned aircraft component sub-images, and assign codes to each unmanned aircraft component sub-image;
[0141] The steps for performing unmanned aircraft component recognition and segmentation include:
[0142] 12.1) Use the Canny algorithm to extract the edges of the unmanned aircraft image to be detected to obtain an edge image;
[0143] Among them, the edge density ρ of the edge image is as follows:
[0144]
[0145] In the formula, N1 is the length of the edge image, N2 represents the width of the edge image, p dj represents the value of the pixel point (d, j) in the edge image, pdj = 1 represents the edge, p dj = 0 represents the background;
[0146] 12.2) Using the Q×Q region centered on each pixel as a template, calculate the edge density ρ of each pixel dj , that is:
[0147]
[0148] 12.3) Based on the edge density ρ of each pixel dj , perform binary image segmentation on the drone image to be detected according to the segmentation principle, and segment the background and the drone component area to obtain a binary image P;
[0149] The segmentation principle is as follows:
[0150]
[0151] Where P(d,j) is the pixel value;
[0152] 12.4) Perform an opening operation on the binary image P to eliminate noise and smooth the boundary, obtaining a binary segmentation image p', that is:
[0153] P' = P.S(11)
[0154] Where represents the opening operation; S is the structuring element;
[0155] 12.5) Based on the binary segmentation image p', perform a masking process on the drone image to be detected to obtain a drone component mask image;
[0156] 12.6) Crop the drone components in the drone component mask image and assign codes to construct a drone component sub-image matrix H, that is:
[0157]
[0158] Where is the sth drone component sub-image in the drone component mask image s ; is the code of the sth s drone component sub-image; the code is the horizontal and vertical coordinates of the lower left vertex of the central pixel grid of the drone component sub-image in the drone image to be detected;
[0159] 12) Adjust the resolution and size of each drone component sub-image so that the resolution and size of the drone component sub-image are the same as the average resolution and size of the small sample image set;
[0160] The adjusted UAV component images are sequentially input into the optimal fault recognition model called to perform UAV component fault recognition and determine the status of each UAV component in the UAV image to be detected;
[0161] 13) Based on the encoding of the UAV components, label the UAV component status recognition results on the UAV image to be detected.
[0162] The UAV components include propellers, arms, fuselage frames, and screws.
[0163] When the UAV component is a propeller, the status types of the UAV component include normal, broken, bent, and notched.
[0164] When the UAV component is an arm, the status types of the UAV component include normal, cracked, and notched;
[0165] When the UAV component is a fuselage frame, the status types of the UAV component include normal, cracked, and notched;
[0166] When the UAV component is a screw, the status types of the UAV component include normal, loose, and missing; loose means that the screw protrudes more than 0.5 mm from the base surface.
[0167] The number of images in the small sample image set ranges from 1000 to 5000;
[0168] The number of images in the optimized image set is greater than 10 6 .
[0169] In the convolutional neural network based on residual connection, the output of the convolutional layer is as follows:
[0170] D' = X' * G'(13)
[0171] In the formula, D' is the output of the convolutional layer; X' is the input; G' is the convolutional kernel;
[0172] In the convolutional neural network based on residual connection, the output of the residual block is as follows:
[0173] E' = w2σ(w1x') + w3x'(14)
[0174] In the formula, E' is the output of the residual block; x' is the output of the previous convolutional layer adjacent to the current residual block; w1, w2 are residual block parameters; σ is the ReLU activation function; w3 is a linear transformation function used to ensure that the dimensions of two adjacent residual blocks are the same.
[0175] When training the convolutional neural network based on residual connection, the convolutional neural network based on compound scaling, and the lightweight convolutional neural network, the loss function Loss is as follows:
[0176]
[0177] Where P' is the number of samples; z is the true label of the sample; z' is the label output by the convolutional neural network; λ is the penalty parameter; ω p' is the connection weight.
[0178] In the generative adversarial network, the loss of the discriminator is Loss D As shown below:
[0179]
[0180] Where R is the output of the fully connected layer in the discriminator; c' is the number of state type labels; z j' is the one-hot encoding form of the state type label; p j' is the output probability distribution of the previous layer of network.
[0181] In the convolutional neural network based on compound scaling, the output of the EfficientNet module is as follows:
[0182] T=Sigmoid(BN(DepthwiseConv(X”)))(17)
[0183] X'=Sigmoid(BN(Conv 1×1 (X)))(18)
[0184] In the formula, Sigmoid is the activation function; BN is batch normalization; Conv 1×1 is a 1×1 convolution operation; DepthwiseConv is a depthwise separable convolution operation; X” is the input.
[0185] In a lightweight convolutional neural network, the steps for a lightweight convolution unit to process an input image include:
[0186] The input image is expanded through point-by-point convolution operation, and then the channels of the input image are grouped through depth-wise convolution operation;
[0187] Normalize the input layer of the lightweight convolutional neural network and the input of each intermediate layer, and then concatenate the input image with the batch-normalized image;
[0188] After splicing, point-by-point convolution is used to shrink the output channel of the lightweight convolutional neural network, and the activation function R_Hard_Swish is used to measure the feature map and output the result;
[0189] Among them, the activation function R_Hard_Swish is as follows:
[0190]
[0191] Among them, X represents the input image, ReLU6 represents the ReLU6 activation function, and β is a constant; the value range of β is (0, 1).
[0192] The update process of the parameters in the generative adversarial network is as follows:
[0193] S1) Initialize the model parameters w, the exponentially weighted average E[g 2 0 = 0, and set the decay coefficient η and the learning rate α;
[0194] S2) Calculate the gradient g t of the loss function J(w t ), that is:
[0195]
[0196] In the formula, is the gradient symbol; t is the number of iterations;
[0197] S3) Update the exponentially weighted average E[g 2 t , that is:
[0198]
[0199] S4) Update the parameter w t+1 , that is:
[0200]
[0201] In the formula, ⊙ represents element-wise multiplication; ε represents a constant term to prevent the denominator from being 0;
[0202] S5) Determine whether the maximum number of iterations is reached or the difference between the parameters of two adjacent iterations is less than a preset threshold. If so, end the iteration; otherwise, set t = t + 1 and return to step S2).
[0203] Example 2:
[0204] A method for batch identification of UAV faults based on image processing includes the following steps:
[0205] 1) Construct a UAV fault retrieval database; the UAV fault retrieval database includes n types of UAV component types and the status types of each UAV component;
[0206] The UAV fault retrieval database is as follows:
[0207]
[0208] Among them, A represents the matrix of UAV component types; a n represents the nth UAV component type; B represents the matrix of UAV component state types; a mapping relationship is formed between matrix A and matrix B; b n represents the state type vector of the nth UAV component; represents the mth n state type of the nth UAV component;
[0209] 2) Collect m ×M images of the ith UAV component under the state type i and label them with the state type label, so as to construct a small sample image set, that is:
[0210]
[0211] In the formula, is the image vector of the ith UAV component under the state type ; is the Mth image of the ith UAV component under the state type ;
[0212] 3) Divide the small sample image set into a training set and a test set; record the average resolution and size χ×φ of the small sample image set; I l is the resolution of the lth image in the small sample image set;
[0213] 4) Use the training set to train the convolutional neural network based on residual connection, the convolutional neural network based on compound scaling, and the lightweight convolutional neural network respectively, and obtain the UAV component fault recognition model F1, the UAV component fault recognition model F2, and the UAV component fault recognition model F3;
[0214] The convolutional neural network based on residual connection includes an input layer, a convolutional layer, at least one residual block, and an output layer;
[0215] The convolutional neural network based on compound scaling includes an input layer, a convolutional layer, an EfficientNet module, a pooling layer, a fully connected layer, a batch normalization layer, a compound scaling layer, and an output layer;
[0216] The lightweight convolutional neural network includes an input layer, a lightweight convolutional unit, and an output layer;
[0217] 5) Use the test set to test the performance of the trained UAV component fault recognition models F1, F2, and F3, and select the UAV component fault recognition model with the best performance as the model to be optimized according to the performance index objective function;
[0218] The performance index objective function is as follows:
[0219] max{ω1f 11 +ω2f 12 ,ω1f 21 +ω2f 22 ,ω1f 31 +ω2f 32 [[ID=17}]}(3)
[0220] Wherein, f 11 , f 21 , f 31 are the average recognition rates of the unmanned aircraft component fault recognition models F1, F2, and F3; f 12 , f 22 , f 32 are the accuracies of the unmanned aircraft component fault recognition models F1, F2, and F3; ω1 and ω2 are weights;
[0221] 6) Use the generative adversarial network to expand the small sample image set to obtain an optimized image set, that is:
[0222]
[0223] Wherein, is the image vector of the i-th unmanned aircraft component in the state type generated by the generative adversarial network; is the M'-th image of the i-th unmanned aircraft component in the state type generated by the generative adversarial network; M' > M;
[0224] The generative adversarial network includes a generator and a discriminator; the generator includes an input layer, a fully connected layer, a convolutional layer, and an output layer; the discriminator includes an input layer, a convolutional layer, a fully connected layer, and an output layer;
[0225] The objective function of the generative adversarial network is as follows:
[0226]
[0227] Among them, G represents the generator, D represents the discriminator, Y represents the state type label, p Z (Z) represents the prior noise distribution; E represents the expectation; Z represents the noise; X is the data of the small sample image set; p data (X) represents the data distribution of the small sample image set;
[0228] 7) Use the optimized image set to train the model to be optimized to obtain the optimal fault recognition model for the i-th unmanned aircraft component;
[0229] 8) Determine whether i > n holds. If so, proceed to step 9); otherwise, set i = i + 1 and return to step 2).
[0230] 9) Write the optimal fault identification models of all UAV components into the UAV fault retrieval database and form a mapping with n types of UAV components, i.e.:
[0231]
[0232] In the formula, C is the optimal fault identification model matrix corresponding to n types of UAV components; c n is the optimal fault identification model of the nth type of UAV component;
[0233] 10) Select the type of UAV component to be detected and call the corresponding optimal fault identification model;
[0234] 11) Obtain the image of the UAV to be detected and perform pixel grid division to obtain the pixel grid matrix K of the UAV image to be detected, i.e.:
[0235]
[0236] In the formula, k eg is the pixel grid serial number of the e-th row and g-th column of the UAV image to be detected; k egx , k egy is the horizontal and vertical coordinates of the lower left vertex of the pixel grid k eg ;
[0237] 12) Perform UAV component identification and segmentation on the UAV image to be detected, obtain several UAV component sub-images, and assign codes to each UAV component sub-image;
[0238] The steps for performing UAV component identification and segmentation include:
[0239] 12.1) Use the Canny algorithm to perform edge extraction on the UAV image to be detected to obtain an edge image;
[0240] Among them, the edge density ρ of the edge image is as follows:
[0241]
[0242] In the formula, N1 is the length of the edge image, N2 represents the width of the edge image, p dj represents the value of the pixel point (d, j) in the edge image, p dj = 1 represents the edge, p dj = 0 represents the background;
[0243] 12.2) Use the Q×Q area centered on each pixel point as a template to calculate the edge density ρ of each pixel pointdj , namely:
[0244]
[0245] 12.3) Based on the edge density ρ of each pixel dj , perform binary image segmentation on the drone image to be detected according to the segmentation principle, segment the background and the drone component area, so as to obtain a binary image P;
[0246] The segmentation principle is as follows:
[0247]
[0248] In the formula, P(d,j) is the pixel value;
[0249] 12.4) Perform an opening operation on the binary image P to eliminate noise and smooth the boundary, and obtain a binary segmentation image p', that is:
[0250]
[0251] In the formula, represents the opening operation; S is the structuring element;
[0252] 12.5) Perform a masking process on the drone image to be detected based on the binary segmentation image p' to obtain a drone component mask image;
[0253] 12.6) Crop the drone components in the drone component mask image and assign codes to construct a drone component sub-image matrix H, that is:
[0254]
[0255] In the formula, is the sth drone component sub-image in the drone component mask image s ; is the code of the sth drone component sub-image; the code is the horizontal and vertical coordinates of the lower left vertex of the central pixel grid of the drone component sub-image in the drone image to be detected; s
[0256] 12) Adjust the resolution and size of each drone component sub-image so that the resolution and size of the drone component sub-image are the same as the average resolution and size of the small sample image set;
[0257] Input the adjusted drone component images into the optimal fault recognition model called in sequence for drone component fault recognition to determine the state of each drone component in the drone image to be detected;
[0258] 13) Based on the encoding of the UAV components, label the UAV component status recognition results on the UAV image to be detected.
[0259] Embodiment 3:
[0260] A method for batch recognition of UAV faults based on image processing, the technical content is the same as that of Embodiment 2. Further, the UAV components include propellers, arms, fuselage frames, and screws.
[0261] Embodiment 4:
[0262] A method for batch recognition of UAV faults based on image processing, the technical content is the same as any one of Embodiments 2-3. Further, when the UAV component is a propeller, the status types of the UAV component include normal, broken, bent, and notched.
[0263] When the UAV component is an arm, the status types of the UAV component include normal, cracked, and notched;
[0264] When the UAV component is a fuselage frame, the status types of the UAV component include normal, cracked, and notched;
[0265] When the UAV component is a screw, the status types of the UAV component include normal, loose, and missing; Loose means that the screw protrudes more than 0.5 mm from the base surface.
[0266] Embodiment 5:
[0267] A method for batch recognition of UAV faults based on image processing, the technical content is the same as any one of Embodiments 2-4. Further, the number of images in the small sample image set ranges from 1000 to 5000;
[0268] The number of images in the optimized image set is greater than 10 6 .
[0269] Embodiment 6:
[0270] A method for batch recognition of UAV faults based on image processing, the technical content is the same as any one of Embodiments 2-5. Further, in the convolutional neural network based on residual connection, the output of the convolutional layer is as follows:
[0271] D' = X' * G'(13)
[0272] In the formula, D' is the output of the convolutional layer; X' is the input; G' is the convolutional kernel;
[0273] In the convolutional neural network based on residual connection, the output of the residual block is as follows:
[0274] E' = w2σ(w1x') + w3x'(14)
[0275] Wherein, E' is the output of the residual block; x' is the output of the previous convolutional layer adjacent to the current residual block; w1 and w2 are the parameters of the residual block; σ is the ReLU activation function; w3 is a linear transformation function for ensuring the consistency of dimensions between two adjacent residual blocks.
[0276] Embodiment 7:
[0277] A method for batch identification of UAV faults based on image processing, the technical content is the same as any one of Embodiments 2-6. Further, when training a convolutional neural network based on residual connection, a convolutional neural network based on compound scaling, and a lightweight convolutional neural network, the loss function Loss is as follows:
[0278]
[0279] Wherein, P' is the number of samples; z is the true label of the sample; z' is the label output by the convolutional neural network; λ is the penalty parameter; ω p' is the connection weight.
[0280] Embodiment 8:
[0281] A method for batch identification of UAV faults based on image processing, the technical content is the same as any one of Embodiments 2-7. Further, in the generative adversarial network, the loss Loss of the discriminator D is as follows:
[0282]
[0283] Wherein, R is the output result of the fully connected layer in the discriminator; c' is the number of state type labels; z j' is the one-hot encoding form of the state type label; p j' is the output probability distribution of the previous layer network.
[0284] Embodiment 9:
[0285] A method for batch identification of UAV faults based on image processing, the technical content is the same as any one of Embodiments 2-8. Further, in the convolutional neural network based on compound scaling, the output of the EfficientNet module is as follows:
[0286] T = Sigmoid(BN(DepthwiseConv(X”)))(17)
[0287] X' = Sigmoid(BN(Conv 1×1 (X)))(18)
[0288] Wherein, Sigmoid is the activation function; BN is batch normalization; Conv 1×11×1 convolution operation; DepthwiseConv is a depthwise separable convolution operation; X" is the input.
[0289] Example 10:
[0290] A method for batch identification of UAV faults based on image processing, the technical content is the same as any one of Examples 2-9. Further, in the lightweight convolutional neural network, the steps for the lightweight convolutional unit to process the input image include:
[0291] Expand the channels of the input image through pointwise convolution operation, and then group the channels of the input image through depth convolution operation;
[0292] Normalize the input of the input layer and each intermediate layer of the lightweight convolutional neural network, and then concatenate the input image with the batch-normalized image;
[0293] After concatenation, use pointwise convolution to contract the output channels of the lightweight convolutional neural network, measure the feature map using the activation function R_Hard_Swish, and output the result;
[0294] Among them, the activation function R_Hard_Swish is as follows:
[0295]
[0296] Among them, X represents the input image, ReLU6 represents the ReLU6 activation function, and β is a constant; the value range of β is (0, 1).
[0297] Example 11:
[0298] A method for batch identification of UAV faults based on image processing, the technical content is the same as any one of Examples 2-10. Further, the update process of the parameters in the generative adversarial network is as follows:
[0299] S1) Initialize the model parameters w, the exponential decay average value E[g 2 0 = 0, and set the decay coefficient η and the learning rate α;
[0300] S2) Calculate the gradient g t of the loss function J(w t with respect to the parameters, that is:
[0301]
[0302] In the formula, is the gradient symbol; t is the iteration number;
[0303] S3) Update the exponential decay average value E[g of the historical gradient square2 t , that is:
[0304]
[0305] S4) Update parameter w t+1 , that is:
[0306]
[0307] In the formula, ⊙ represents element-wise multiplication; ε represents a constant term to prevent the denominator from being 0;
[0308] S5) Determine whether the maximum number of iterations is reached or the difference between the parameters of two adjacent iterations is less than a preset threshold. If so, end the iteration; otherwise, let t = t + 1 and return to step S2).
[0309] Example 12:
[0310] A method for batch identification of UAV faults based on image processing, the technical content is the same as any one of Examples 2-11. The output of the optimal fault identification model is the encoding of the status type label. After obtaining the encoding, it is converted into text and marked in the image.
[0311] Example 13:
[0312] A method for batch identification of UAV faults based on image processing, the technical content is the same as any one of Examples 2-12,
[0313] The convolutional neural network based on residual connection includes an input layer, a convolutional layer, at least one residual block, and an output layer;
[0314] The convolutional neural network based on compound scaling includes an input layer, a convolutional layer, an EfficientNet module, a pooling layer, a fully connected layer, a batch normalization layer, a compound scaling layer, and an output layer;
[0315] The lightweight convolutional neural network includes an input layer, lightweight convolutional units, and an output layer;
[0316] Among them, the convolutional layer and the lightweight convolutional units are used to extract image features.
[0317] The input of the input layer is the image to be detected, and the output of the output layer is the status type label.
[0318] The residual block introduces a skip connection to directly superimpose the input on the output, solving the problem of gradient disappearance / explosion.
[0319] The EfficientNet module balances the computational efficiency and model performance through depthwise separable convolution and channel attention.
[0320] The pooling layer downsamples the feature map to reduce the spatial dimension.
[0321] The fully connected layer maps high-dimensional features to the target output space.
[0322] The batch normalization layer normalizes the data, accelerates training convergence, and stabilizes the gradients.
[0323] The compound scaling layer uniformly adjusts the depth, width, and input resolution of the model.
Claims
1. A method for batch identification of UAV faults based on image processing, characterized in that, It includes the following steps: 1) Construct a UAV fault retrieval database; the UAV fault retrieval database includes n types of UAV component types and the status types of each UAV component. The UAV fault retrieval database is as follows: Among them, A represents the matrix of UAV component types; a n represents the nth UAV component type; B represents the matrix of UAV component state types; there is a mapping relationship between matrix A and matrix B; b n represents the state type vector of the nth UAV component; represents the mth n state type of the nth UAV component; 2) Collect m × M images of the i-th UAV component under the state type i and label them with the state type label, so as to construct a small sample image set, that is: Wherein, is the image vector of the i-th UAV component in the state type ; is the M-th image of the i-th UAV component in the state type ; 3) Divide the small-sample image set into a training set and a test set; record the average resolution of the small-sample image set and the size χ×φ; I l is the resolution of the l-th image in the small-sample image set; 4) Use the training set to train the convolutional neural network based on residual connection, the convolutional neural network based on compound scaling, and the lightweight convolutional neural network respectively to obtain the UAV component fault recognition model F1, the UAV component fault recognition model F2, and the UAV component fault recognition model F3; The convolutional neural network based on residual connection includes an input layer, a convolutional layer, at least one residual block, and an output layer; The convolutional neural network based on compound scaling includes an input layer, a convolutional layer, an EfficientNet module, a pooling layer, a fully connected layer, a batch normalization layer, a compound scaling layer, and an output layer; The lightweight convolutional neural network includes an input layer, a lightweight convolutional unit, and an output layer; 5) Use the test set to test the performance of the trained UAV component fault recognition models F1, F2, and F3, and select the UAV component fault recognition model with the best performance as the model to be optimized according to the performance index objective function; The performance index objective function is as follows: max{ω1f 11 +ω2f 12 ,ω1f 21 +ω2f 22 ,ω1f 31 +ω2f 32} (3) where f 11 , f 21 , f 31 are the average recognition rates of the UAV component fault recognition models F1, F2, and F3; f 12 , f 22 , f 32 are the accuracies of the UAV component fault recognition models F1, F2, and F3; ω1, ω2 are weights; 6) Use the generative adversarial network to expand the small sample image set to obtain an optimized image set, that is: In the formula, is the image vector of the i-th unmanned aircraft component generated by the generative adversarial network under the state type ; is the M'-th image of the i-th unmanned aircraft component generated by the generative adversarial network under the state type ; M' > M; The generative adversarial network includes a generator and a discriminator; the generator includes an input layer, a fully connected layer, a convolutional layer, and an output layer; the discriminator includes an input layer, a convolutional layer, a fully connected layer, and an output layer; The objective function of the generative adversarial network is as follows: Among them, G represents the generator, D represents the discriminator, Y represents the state type label, p Z (Z) represents the prior noise distribution; E represents the expectation; Z represents the noise; X is the data of the few-shot image set; p data (X) represents the few-shot image set data distribution; 7) Use the optimized image set to train the model to be optimized to obtain the optimal fault recognition model for the i-th type of UAV component; 8) Judge whether i > n holds. If so, go to step 9). Otherwise, let i = i + 1 and return to step 2); 9) Write the optimal fault recognition models of all UAV components into the UAV fault retrieval database and form a mapping with n types of UAV component types, that is: Where C is the optimal fault identification model matrix corresponding to n types of unmanned aircraft components; c n is the optimal fault identification model for the nth type of unmanned aircraft component; 10) Select the type of UAV component to be detected and call the corresponding optimal fault recognition model; 11) Obtain the UAV image to be detected and perform pixel grid division to obtain the UAV image pixel grid matrix K to be detected, that is: where k eg is the pixel grid serial number of the g-th column in the e-th row of the UAV image to be detected; k egx , k egy are the horizontal and vertical coordinates of the lower left vertex of the pixel grid k eg respectively. 12) Perform UAV component recognition and segmentation on the UAV image to be detected to obtain several UAV component sub-images, and assign codes to each UAV component sub-image; The steps for UAV component recognition and segmentation include: 12.1) Use the Canny algorithm to extract the edges of the UAV image to be detected to obtain an edge image; Among them, the edge density ρ of the edge image is as follows: Where N1 is the length of the edge image, N2 represents the width of the edge image, and p dj represents the value of the pixel point (d, j) in the edge image, and p dj = 1 represents the edge, and p dj = 0 represents the background; 12.2) Taking the Q×Q region centered on each pixel as a template, calculate the edge density ρ of each pixel dj , that is: 12.3) Based on the edge density ρ of each pixel dj , the image binary segmentation is performed on the drone image to be detected according to the segmentation principle, and the background and the drone component area are segmented to obtain a binary image P; The segmentation principle is as follows: In the formula, P(d,j) is the pixel value; 12.4) Perform an opening operation on the binary image P to eliminate noise and smooth the boundary to obtain a binary segmentation image p', that is: P' = P ο S (11) In the formula, ο represents the opening operation; S is the structuring element; 12.5) Perform masking processing on the UAV image to be detected based on the binary segmentation image p' to obtain a UAV component masking image; 12.6) Crop the UAV components in the masked image of the UAV components and assign codes to construct the sub-image matrix H of the UAV components, that is: In the formula, is the sr-th s unmanned aircraft component sub-image in the unmanned aircraft component mask image; is the code of the sr-th s unmanned aircraft component sub-image; the code is the horizontal and vertical coordinates of the lower left vertex of the central pixel grid of the unmanned aircraft component sub-image in the image of the drone to be detected; 12) Adjust the resolution and size of each UAV component sub-image so that the resolution and size of the UAV component sub-image are the same as the average resolution and size of the small sample image set; Input the adjusted UAV component images into the called optimal fault recognition model in sequence to perform UAV component fault recognition and determine the status of each UAV component in the image to be detected; 13) Based on the codes of the UAV components, label the UAV component status recognition results on the image of the UAV to be detected.
2. The method for batch identification of UAV faults based on image processing according to claim 1, characterized in that: The UAV components include propellers, arms, fuselage frames, and screws.
3. The method for batch identification of UAV faults based on image processing according to claim 2, wherein: When the UAV component is a propeller, the status types of the UAV component include normal, broken, bent, and notched. When the UAV component is an arm, the status types of the UAV component include normal, cracked, and notched; When the UAV component is a fuselage frame, the status types of the UAV component include normal, cracked, and notched; When the UAV component is a screw, the status types of the UAV component include normal, loose, and missing; loose means that the screw protrudes more than 0.5 mm from the base surface.
4. A method for batch identification of UAV faults based on image processing according to claim 1, characterized in that: The number of images in the small sample image set ranges from 1000 to 5000; The number of images in the optimized image set is greater than 10 6 .
5. A method for batch identification of UAV faults based on image processing according to claim 1, characterized in that: In the convolutional neural network based on residual connection, the output of the convolutional layer is as follows: D' = X' * G'(13) In the formula, D' is the output of the convolutional layer; X' is the input; G' is the convolutional kernel; In the convolutional neural network based on residual connection, the output of the residual block is as follows: E' = w2σ(w1x') + w3x'(14) In the formula, E' is the output of the residual block; x' is the output of the previous convolutional layer adjacent to the current residual block; w1 and w2 are residual block parameters; σ is the ReLU activation function; w3 is a linear transformation function used to ensure that the dimensions of two adjacent residual blocks are the same.
6. The method for batch identification of UAV faults based on image processing according to claim 1, characterized in that When training the convolutional neural network based on residual connection, the convolutional neural network based on compound scaling, and the lightweight convolutional neural network, the loss function Loss is as follows: Wherein, P' is the number of samples; z is the true label of the sample; z' is the label output by the convolutional neural network; λ is the penalty parameter; ω p' is the connection weight.
7. A method for batch identification of UAV faults based on image processing according to claim 1, characterized in that In a generative adversarial network, the loss of the discriminator D is as follows: where R is the output result of the fully connected layer in the discriminator; c' is the number of state type labels; z j' is the one-hot encoded form of the state type label; p j' is the output probability distribution of the previous layer network.
8. A method for batch identification of UAV faults based on image processing according to claim 1, characterized in that, In the convolutional neural network based on compound scaling, the output of the EfficientNet module is as follows: T = Sigmoid(BN(DepthwiseConv(X”))) (17) X' = Sigmoid(BN(Conv 1×1 (X))) (18) Wherein, Sigmoid is the activation function; BN is batch normalization; Conv 1×1 is a 1×1 convolution operation; DepthwiseConv is a depthwise separable convolution operation; X” is the input.
9. A method for batch identification of UAV faults based on image processing according to claim 1, characterized in that, In the lightweight convolutional neural network, the steps for the lightweight convolutional unit to process the input image include: Expand the channels of the input image through pointwise convolution operations, and then group the channels of the input image through depth convolution operations; Normalize the inputs of the input layer and each intermediate layer of the lightweight convolutional neural network, and then concatenate the input image with the batch-normalized image; After concatenation, use pointwise convolution to contract the output channels of the lightweight convolutional neural network, and use the activation function R_Hard_Swish to measure the feature map and output the result; Among them, the activation function R_Hard_Swish is as follows: Wherein, X represents the input image, ReLU6 represents the ReLU6 activation function, and β is a constant; the value range of β is (0, 1).
10. A method for batch identification of UAV faults based on image processing according to claim 1, characterized in that, The update process of the parameters in the generative adversarial network is as follows: S1) Initialize the model parameters w, the exponentially weighted average E[g 2 0 = 0, and set the decay coefficient η and the learning rate α; S2) Calculate the gradient g of the loss function J(w t ) with respect to the parameters t , that is: In the formula, is the gradient symbol; t is the number of iterations; S3) Update the exponentially decaying average E[g 2 t , i.e.: S4) Update parameter w t+1 , namely: In the formula, ⊙ represents element-wise multiplication; ε represents a constant term to prevent the denominator from being zero; S5) Determine whether the maximum number of iterations is reached or the difference between the parameters of two adjacent iterations is less than the preset threshold. If so, end the iteration; otherwise, set t = t + 1 and return to step S2).