A method, system, device and medium for identifying defect images of drone inspection

Abstract

The present invention discloses a method, system, device and medium for identifying defect images during drone inspection. The method comprises the following steps: obtaining an image to be identified during drone inspection and preprocessing it; inputting the preprocessed image to be identified into a trained neural network model for defect identification and obtaining an identification result; wherein obtaining the trained neural network model comprises: obtaining a sample image set of drone inspection, annotating defects on each sample image in the sample image set to form an annotated image set; preprocessing each annotated image in the annotated image set and dividing it into a training set and a test set; constructing a neural network model with a multi-scale attention mechanism network structure Swin-Transformer and a feature pyramid network structure SA-FPN as a skeleton; training the constructed neural network model with the training set, and then testing the neural network model with the test set to obtain a trained neural network model; the present invention has better recognition performance than the traditional CNN structure.

Description

Technical Field

[0001] The present invention relates to a method, system, device and medium for identifying defect images inspected by an unmanned aerial vehicle (UAV), and belongs to the technical field of image recognition. Background Art

[0002] Currently, applying deep learning intelligent recognition technology to identify defects in power tower images captured by drones during inspections is a common technique. However, in actual inspections, drone images present complex backgrounds, with diverse power equipment defect types, inconsistent sizes, and varying locations within the images. Currently popular intelligent recognition methods primarily utilize pure convolutional neural networks (CNNs), which have limited adaptability to multi-scale object perception and a receptive field limited by the size of the predefined convolution kernel. Therefore, using traditional CNN architectures to effectively detect defects during transmission line inspections is difficult. Summary of the Invention

[0003] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method, system, device and medium for identifying defect images of drone inspections, so as to solve the technical problem that it is difficult to achieve good performance in detecting transmission line inspection defects using traditional CNN structures.

[0004] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0005] In a first aspect, the present invention provides a method for identifying defect images inspected by a drone, comprising:

[0006] Obtain the images to be identified from drone inspections and perform preprocessing;

[0007] The pre-processed image to be identified is input into the trained neural network model for defect identification and the identification result is obtained;

[0008] The acquisition of the trained neural network model includes:

[0009] Obtain a sample image set of drone inspections, and perform defect annotation on each sample image in the sample image set to form an annotated image set; wherein the defect annotation includes annotating the defect type and the defect bounding box;

[0010] Preprocessing each annotated image in the annotated image set, and dividing the preprocessed annotated images into a training set and a test set;

[0011] Construct a neural network model with a multi-scale attention mechanism network structure Swin-Transformer, a feature pyramid network structure SA-FPN, and a regional feature network structure RPN as the backbone;

[0012] The constructed neural network model is trained using the training set, and then the neural network model is tested using the test set to complete the training of the neural network model and obtain a trained neural network model.

[0013] Optionally, the preprocessing includes cutting the pixel matrix of the input image into a plurality of non-overlapping pixel blocks according to a preset ratio, and synthesizing the pixel blocks into pixel vectors.

[0014] Optionally, the attention mechanism network structure Swin-Transformer includes four cascaded stages, which are respectively recorded as Stage 1, Stage 2, Stage 3, and Stage 4; wherein, the Stage 1 includes two cascaded Swin-TransformerBlocks, the Stage 2 and Stage 4 both include a cascaded Patch Merging module and two Swin-TransformerBlocks, and the Stage 3 includes a cascaded Patch Merging module and six Swin-Transformer Blocks; the Patch Merging module is used to downsample the input image to reduce the resolution of the input image.

[0015] Optionally, the Swin-Transformer Block includes a cascaded first module and a second module;

[0016] The first module includes a cascaded first LN layer, a window multi-head self-attention layer W-MSA, a second LN layer and a first MLP layer, and the input of the first LN layer is connected to the output of the window multi-head self-attention layer W-MSA through a residual connection, and the input of the second LN layer is connected to the output of the first MLP layer through a residual connection;

[0017] The second module includes a cascaded third LN layer, a shifted window multi-head self-attention layer SW-MSA, a fourth LN layer and a second MLP layer, and the input of the third LN layer is connected to the output of the shifted window multi-head self-attention layer SW-MSA through a residual method, and the input of the fourth LN layer is connected to the output of the second MLP layer through a residual method.

[0018] Optionally, the Patch Merging module downsampling the input image includes:

[0019] Extract pixel feature maps of the input image in row and column directions at intervals of 2 to generate patches;

[0020] Connect each patch according to the feature map channel and transform the channel dimension to four times the original one;

[0021] The connected patch is stretched and normalized through a fully connected layer, and the channel dimension is transformed to twice its original size.

[0022] Optionally, the feature pyramid network structure SA-FPN includes three Scale-Attention Blocks, wherein the input of the first Scale-Attention Block is the features output by Stage 1 and Stage 2, the input of the second Scale-Attention Block is the features output by the first Scale-Attention Block and Stage 3, and the input of the third Scale-Attention Block is the features output by the second Scale-Attention Block and Stage 4.

[0023] Optionally, the calculation of the Scale-Attention Block includes:

[0024] Set the three variable matrices Q, K, and V:

[0025] Q=F n ×W

[0026] K=V=[F n ,F n+1 ]

[0027] Where, F n 、F n+1 are the first and second input features of the Scale-Attention Block, respectively, and W is the linear regression matrix weight;

[0028] Calculate the output F of Scale-Attention Block based on Q, K, and V attn :

[0029] F attn =softmax(QK T )V+F n

[0030] In a second aspect, the present invention provides a system for identifying defect images of drone inspections, the system comprising:

[0031] The preprocessing module is used to obtain the images to be identified during drone inspections and perform preprocessing;

[0032] The defect recognition module is used to input the pre-processed image to be recognized into the trained neural network model to perform defect recognition and obtain recognition results;

[0033] The acquisition of the trained neural network model includes:

[0034] A defect annotation module is used to obtain a sample image set of drone inspections and annotate each sample image in the sample image set with defects to form an annotated image set; wherein the defect annotation includes annotating the defect type and the defect bounding box;

[0035] A training preparation module, configured to preprocess each annotated image in the annotated image set and divide the preprocessed annotated images into a training set and a test set;

[0036] The model building module is used to build a neural network model with the multi-scale attention mechanism network structure Swin-Transformer, the feature pyramid network structure SA-FPN and the regional feature network structure RPN as the skeleton;

[0037] The model training module is used to train the constructed neural network model through the training set, and then test the neural network model using the test set to complete the training of the neural network model and obtain a trained neural network model.

[0038] In a third aspect, the present invention provides a device for identifying defect images of drone inspections, including a processor and a storage medium;

[0039] The storage medium is used to store instructions;

[0040] The processor is configured to operate according to the instructions to execute the steps of the above method.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention provides a method, system, device and medium for identifying defect images during drone inspection. By constructing Swin-Transformer as the backbone network and combining it with a neural network model with an SA-FPN structure, it can extract and fuse multi-scale features, and its performance is better than that of the traditional CNN structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of a method for identifying defect images during drone inspection provided by the first embodiment of the present invention;

[0045] Figure 2 This is a flowchart of obtaining a trained neural network model provided in the first embodiment of the present invention;

[0046] Figure 3This is a diagram showing the working principle of the neural network model provided in the first embodiment of the present invention;

[0047] Figure 4 This is a structural diagram of the Swin-Transformer Block provided in Example 1 of the present invention;

[0048] Figure 5 This is a structural diagram of the feature pyramid network structure SA-FPN provided in Example 1 of the present invention;

[0049] Figure 6 This is a diagram of the working principle of the Scale-Attention Block provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0051] Example 1:

[0052] like Figure 1 As shown, the present invention provides a method for identifying defect images inspected by a drone, comprising:

[0053] 1. Obtain the image to be identified from the drone inspection and perform preprocessing;

[0054] Preprocessing includes cutting the pixel matrix of the input image into multiple non-overlapping pixel blocks according to a preset ratio and synthesizing the pixel blocks into pixel vectors.

[0055] 2. Input the preprocessed image to be identified into the trained neural network model for defect identification and obtain the recognition results.

[0056] like Figure 2 As shown, the trained neural network model acquisition includes:

[0057] S1. Obtain a sample image set of drone inspections, and annotate defects on each sample image in the sample image set to form an annotated image set; wherein the defect annotation includes annotating the defect type and the defect bounding box;

[0058] S2. Preprocess each annotated image in the annotated image set, and divide the preprocessed annotated images into a training set and a test set; the preprocessing here is the same as step 1.

[0059] S3. Construct a neural network model with a multi-scale attention mechanism network structure Swin-Transformer, a feature pyramid network structure SA-FPN, and a regional feature network structure RPN as the backbone;

[0060] like Figure 3 As shown in the figure, the attention mechanism network structure Swin-Transformer is used for multi-scale feature extraction, the feature pyramid network structure SA-FPN is used to fuse the extracted multi-scale features to obtain feature maps of each scale, and the regional feature network structure RPN is used to generate candidate box maps based on the feature maps of each scale for predicting defect types and defect bounding boxes.

[0061] (1) The attention mechanism network structure Swin-Transformer includes four cascaded stages, which are respectively denoted as Stage1, Stage2, Stage3, and Stage4; among them, Stage1 includes two cascaded Swin-Transformer blocks, Stage2 and Stage4 both include cascaded Patch Merging modules and two Swin-Transformer blocks, and Stage3 includes cascaded Patch Merging modules and six Swin-Transformer blocks.

[0062] (1.1) Figure 4 As shown, the Swin-Transformer Block includes a cascaded first module and a second module; the first module includes a cascaded first LN layer, a window multi-head self-attention layer W-MSA, a second LN layer and a first MLP layer, and the input of the first LN layer is connected to the output of the window multi-head self-attention layer W-MSA through a residual connection, and the input of the second LN layer is connected to the output of the first MLP layer through a residual connection; the second module includes a cascaded third LN layer, a shifted window multi-head self-attention layer SW-MSA, a fourth LN layer and a second MLP layer, and the input of the third LN layer is connected to the output of the shifted window multi-head self-attention layer SW-MSA through a residual connection, and the input of the fourth LN layer is connected to the output of the second MLP layer through a residual connection.

[0063] In the first module, the input feature z l-1 First, it is normalized by the first LN layer and then the feature learning is performed by W-MSA. The output value is combined with the original feature to perform residual operation to obtain Eigenvalue The value obtained after the second LN layer and the first MLP layer (multi-layer perceptron) is then combined with Perform residual operation to obtain output feature z l , and used as the input of the second module; the second module uses the SW-MSA structure instead of W-MSA, and the remaining layers and operations remain unchanged, and finally the output feature z is obtained l+1 .

[0064] (1.2) The Patch Merging module is used to downsample the input image and reduce the resolution of the input image. The specific process is as follows:

[0065] Extract pixel feature maps of the input image in row and column directions at intervals of 2 to generate patches;

[0066] Connect each patch according to the feature map channel and transform the channel dimension to four times the original one;

[0067] The connected patch is stretched and normalized through a fully connected layer, and the channel dimension is transformed to twice its original size.

[0068] (2) Figure 5 As shown in the figure, the feature pyramid network structure SA-FPN includes three Scale-Attention Blocks, where the input of the first Scale-Attention Block is the features F3 and F4 output by Stage1 and Stage2, the input of the second Scale-Attention Block is the feature F2 output by the first Scale-Attention Block and Stage3, and the input of the third Scale-Attention Block is the feature F1 output by the second Scale-Attention Block and Stage4.

[0069] like Figure 6 As shown, the calculation of Scale-Attention Block includes:

[0070] Set the three variable matrices Q, K, and V:

[0071] Q=F n ×W

[0072] K=V=[F n ,F n+1 ]

[0073] Where, F n 、F n+1 are the first and second input features of the Scale-Attention Block, respectively, and W is the linear regression matrix weight;

[0074] Calculate the output F of Scale-Attention Block based on Q, K, and V attn :

[0075] F attn =softmax(QK T )V+F n

[0076] S4. Train the constructed neural network model using the training set, and then test the neural network model using the test set to complete the training of the neural network model and obtain a trained neural network model;

[0077] The training process involves taking a batch of four sample images per iteration and performing a backpropagation update on the model weights. The initial learning rate is set to 0.0001, the weight decay is set to 0.05, and the AdamW optimizer is used for optimization. A linear learning rate warmup strategy is used every 1000 iterations. The entire training set is trained for a total of 12 epochs, with the learning rate decayed to 1 / 10 of its original value at the 8th and 11th iterations, respectively.

[0078] Experimental verification:

[0079] Three models were trained using a training set of 687 samples and tested using a test set of 229 samples. The three models were: ResNet-50 as the backbone network combined with the traditional FPN structure; Swin-Transformer as the backbone network combined with the traditional FPN structure; and the Swin-Transformer proposed in this invention as the backbone network combined with the SA-FPN structure. The test results are shown in Table 1:

[0080] Table 1:

[0081] Network structure FLOPs #Params mAP <![CDATA[AP 75 ]]> ResNet-50+FPN 206.67G 41.13M 32.5 31.4 Swin-T+FPN 213.42G 41.13M 34.2 34.1 Swin-T+SA-FPN 218.93G 44.95M 35.9 38.1

[0082] The experimental results show that compared with the other two traditional network structures, the Swin-Transformer+SA-FPN structure proposed in this invention can improve the average detection accuracy of drone inspection defect images by 4%. At the same time, the network parameter complexity only increases by 0.42%, and the computational complexity only increases by 2.58%, showing good detection performance.

[0083] Example 2:

[0084] An embodiment of the present invention provides a system for identifying defect images during drone inspection, the system comprising:

[0085] The preprocessing module is used to obtain the images to be identified during drone inspections and perform preprocessing;

[0086] The defect recognition module is used to input the pre-processed image to be recognized into the trained neural network model to perform defect recognition and obtain recognition results;

[0087] Among them, the acquisition of the trained neural network model includes:

[0088] The defect annotation module is used to obtain a sample image set of drone inspections and annotate each sample image in the sample image set with defects to form an annotated image set; wherein the defect annotation includes annotating the defect type and the defect bounding box;

[0089] A training preparation module, configured to preprocess each annotated image in the annotated image set and divide the preprocessed annotated images into a training set and a test set;

[0090] The model building module is used to build a neural network model with the multi-scale attention mechanism network structure Swin-Transformer, the feature pyramid network structure SA-FPN and the regional feature network structure RPN as the skeleton;

[0091] The model training module is used to train the constructed neural network model through the training set, and then test the neural network model using the test set to complete the training of the neural network model and obtain a trained neural network model.

[0092] Example 3:

[0093] Based on the first embodiment, the present invention provides a device for identifying defect images of drone inspections, including a processor and a storage medium;

[0094] The storage medium is used to store instructions;

[0095] The processor is configured to operate according to the instructions to execute the steps of the above method.

[0096] Example 4:

[0097] Based on the first embodiment, the embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the above method are implemented.

[0098] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0099] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0102] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for identifying defect images from drone inspection, characterized in that: include: Obtain the images to be identified from drone inspections and perform preprocessing; The pre-processed image to be identified is input into the trained neural network model for defect identification and the identification result is obtained; The acquisition of the trained neural network model includes: Obtain a sample image set of drone inspections, and perform defect annotation on each sample image in the sample image set to form an annotated image set; wherein the defect annotation includes annotating the defect type and the defect bounding box; Preprocessing each annotated image in the annotated image set, and dividing the preprocessed annotated images into a training set and a test set; Construct a neural network model with a multi-scale attention mechanism network structure Swin-Transformer, a feature pyramid network structure SA-FPN, and a regional feature network structure RPN as the backbone; The constructed neural network model is trained using the training set, and then the neural network model is tested using the test set to complete the training of the neural network model and obtain a trained neural network model; The feature pyramid network structure SA-FPN includes three Scale-Attention Blocks, wherein the input of the first Scale-Attention Block is the features output by Stage 1 and Stage 2, the input of the second Scale-Attention Block is the features output by the first Scale-Attention Block and Stage 3, and the input of the third Scale-Attention Block is the features output by the second Scale-Attention Block and Stage 4. The calculation of the Scale-Attention Block includes: Set the three variable matrices Q, K, and V: ; ; Where, They are the first input feature and the second input feature of Scale-Attention Block respectively. is the linear regression matrix weight; Calculate the output of Scale-Attention Block based on Q, K, and V : 。 2. The method for identifying defect images during drone inspection according to claim 1, characterized in that: The preprocessing includes cutting the pixel matrix of the input image into a plurality of non-overlapping pixel blocks according to a preset ratio, and synthesizing the pixel blocks into pixel vectors.

3. The method for identifying defect images during drone inspection according to claim 1, characterized in that: The attention mechanism network structure Swin-Transformer includes four cascaded stages, which are respectively denoted as Stage 1, Stage 2, Stage 3, and Stage 4; wherein, the Stage 1 includes two cascaded Swin-Transformer Blocks, the Stage 2 and Stage 4 each include a cascaded Patch Merging module and two Swin-Transformer Blocks, and the Stage 3 includes a cascaded Patch Merging module and six Swin-Transformer Blocks; the Patch Merging module is used to downsample the input image to reduce the resolution of the input image.

4. The method for identifying defect images during drone inspection according to claim 3, characterized in that: The Swin-Transformer Block includes a cascaded first module and a second module; The first module includes a cascaded first LN layer, a window multi-head self-attention layer W-MSA, a second LN layer and a first MLP layer, and the input of the first LN layer is connected to the output of the window multi-head self-attention layer W-MSA through a residual connection, and the input of the second LN layer is connected to the output of the first MLP layer through a residual connection; The second module includes a cascaded third LN layer, a shifted window multi-head self-attention layer SW-MSA, a fourth LN layer and a second MLP layer, and the input of the third LN layer is connected to the output of the shifted window multi-head self-attention layer SW-MSA through a residual method, and the input of the fourth LN layer is connected to the output of the second MLP layer through a residual method.

5. The method for identifying defect images during drone inspection according to claim 3, characterized in that: The Patch Merging module downsamples the input image including: Extract pixel feature maps of the input image in row and column directions at intervals of 2 to generate patches; Connect each patch according to the feature map channel and transform the channel dimension to four times the original one; The connected patch is stretched and normalized through a fully connected layer, and the channel dimension is transformed to twice its original size.

6. A drone inspection defect image recognition system, characterized by: The system comprises: The preprocessing module is used to obtain the images to be identified during drone inspections and perform preprocessing; The defect recognition module is used to input the pre-processed image to be recognized into the trained neural network model to perform defect recognition and obtain recognition results; The acquisition of the trained neural network model includes: A defect annotation module is used to obtain a sample image set of drone inspections and annotate each sample image in the sample image set with defects to form an annotated image set; wherein the defect annotation includes annotating the defect type and the defect bounding box; A training preparation module, configured to preprocess each annotated image in the annotated image set and divide the preprocessed annotated images into a training set and a test set; The model building module is used to build a neural network model with the multi-scale attention mechanism network structure Swin-Transformer, the feature pyramid network structure SA-FPN and the regional feature network structure RPN as the skeleton; The model training module is used to train the constructed neural network model through the training set, and then test the neural network model using the test set to complete the training of the neural network model and obtain a trained neural network model; The feature pyramid network structure SA-FPN includes three Scale-Attention Blocks, wherein the input of the first Scale-Attention Block is the features output by Stage 1 and Stage 2, the input of the second Scale-Attention Block is the features output by the first Scale-Attention Block and Stage 3, and the input of the third Scale-Attention Block is the features output by the second Scale-Attention Block and Stage 4. The calculation of the Scale-Attention Block includes: Set the three variable matrices Q, K, and V: ; ; Where, They are the first input feature and the second input feature of Scale-Attention Block respectively. is the linear regression matrix weight; Calculate the output of Scale-Attention Block based on Q, K, and V : 。 7. A device for identifying defect images of drone inspection, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images