An unmanned aerial vehicle target vulnerable part identification method

Abstract

The present application relates to the technical field of unmanned aerial vehicle image capture target detection, and particularly relates to a kind of unmanned aerial vehicle ground target vulnerable part identification method. Through the space transformation module proposed, the input image is adaptively affine transformed in unsupervised learning mode;Through the category-related feature re-labeling module established, the prior category information is integrated into the feature map extracted, and the generated unmanned aerial vehicle capture ground target vulnerable part dataset is input into the vulnerable part target detection deep learning model based on prior category knowledge to train, test and evaluate the model;In the model construction stage, the space transformation module can adaptively crop out interfering objects, which is conducive to improving the detection accuracy of the model;The category-related feature re-labeling module introduces the prior category information into the spatial network, which is conducive to improving the robustness of the model. In the evaluation stage, the width and depth of the model are adjusted to optimize the accuracy and speed of the model to meet the needs of practical applications.

Description

Technical Field

[0001] This invention relates to the field of target detection technology in UAV image capture, and particularly to a method for identifying vulnerable parts of UAV ground targets. Background Technology

[0002] With the explosive development of deep learning, intelligent target recognition, as a key enabling technology for multiple military applications such as detection and early warning, intelligence reconnaissance, situational awareness, and precision guidance, has become a research hotspot in recent years and has yielded a series of positive results. Accurately identifying key parts of a target is of great significance for saving strike costs and improving strike efficiency.

[0003] The paper "Augmented Reality Maintenance Assistant Using YOLOv5" proposes a YOLOv5-based automotive parts detection model. Experimental results show that the model has high detection accuracy and generalization ability, and can be extended to all vehicle models on the market. However, these methods only consider the part detection problem of a single type of object, and research on multi-type objects and the identification of different specific regions is still lacking. Summary of the Invention

[0004] The purpose of this invention is to provide a method for accurately identifying vulnerable parts of ground targets by UAVs, in order to solve the problem of detecting different key parts of different types of targets based on the overall target detection framework for UAV remote sensing images.

[0005] To achieve the aforementioned objective, the present invention employs the following technical solution:

[0006] A method for identifying vulnerable parts of ground targets by unmanned aerial vehicles (UAVs), the method comprising the following steps:

[0007] Step 1: Construct a deep learning model for vulnerable part target detection based on prior category knowledge, specifically as follows:

[0008] Step 1.1: Preprocess the input global object detection image using an improved spatial transformation module. By adjusting the input image size and target pose, extract high-quality global object detection proposal boxes.

[0009] Step 1.2: Use a basic set of "convolution + ReLU + pooling layers" to extract feature maps from the image.

[0010] Step 1.3: Using the classification results output by the overall object detector, the feature map is recalibrated through the category-related feature recalibration module.

[0011] Step 1.4: The recalibrated feature map is used to extract multiple candidate regions through a region generation network.

[0012] Step 1.5: For each candidate region, determine its positive or negative attributes, and extract the feature representation of each positive candidate region.

[0013] Step 1.6: Perform a classification operation on the features of each region, determine the category of the vulnerable part (such as tracks, turret), and return the corresponding confidence level.

[0014] Step 1.7: Simultaneously with Step 1.6, perform a regression operation on the bounding box of each region to fine-tune its position in order to more accurately define the location of vulnerable parts.

[0015] Step 1.8: Based on the results of region classification, candidate boxes with large overlaps can be filtered to remove redundant detection results.

[0016] Step 2: Establish a dataset for detecting vulnerable parts of ground-attack military targets captured by UAVs. Specifically:

[0017] Step 2.1: Place 3 to 6 types of military targets (such as tanks, infantry fighting vehicles, radar antenna vehicles, drones, etc.) and other jamming objects (such as trenches and transport vehicles) on the ground to simulate the ground images captured by drones when performing tasks such as patrol search and dive attack, and establish a drone-captured ground military target detection dataset.

[0018] Step 2.2: Input the UAV-captured ground military target detection dataset into the overall target detector, and extract the overall target detection box based on the detection results. The cropped result will form our UAV-captured ground military target vulnerable part detection dataset.

[0019] Step 2.3: Randomly select 60% of the UAV-captured images of ground military targets as the training set and 40% of the detection data as the test set.

[0020] Step 3: Train the deep learning model for vulnerable part target detection built in Step 1 based on prior category knowledge. Specifically:

[0021] Step 3.1: Set the training parameters and use the Adam stochastic optimization algorithm for training. Set the batch size to Batch = 12, momentum = 0.9, the initial learning rate to Ir = 0.001, and the number of training iterations to Epoch = 100.

[0022] Step 3.2: Input the training set from the dataset of vulnerable parts of ground military targets captured by UAVs obtained in Step 2.3 into Step 1 to construct a deep learning model for vulnerable part target detection based on prior category knowledge;

[0023] Step 3.3: Adaptively scale the input images in the training set according to the input image size set by the network;

[0024] Step 3.4: Train a deep learning model for vulnerable part target detection based on prior category knowledge. Adjust the learning rate and number of iterations according to the average accuracy change and loss change trend of the training set results until the accuracy change and loss change gradually tend to stabilize. Determine the final learning rate and number of iterations.

[0025] Step 3.5: Based on the learning rate and number of iterations determined in Step 3.4, complete the training of the deep learning model for vulnerable part target detection based on prior category knowledge, and obtain a well-converged deep learning model for vulnerable part target detection based on prior category knowledge.

[0026] Step 4: Input the test set from the UAV capture vulnerable part detection dataset for ground military targets obtained in Step 2.2 into the deep learning model for vulnerable part target detection trained in Step 3.5 based on prior category knowledge, and test the performance of the deep learning model for vulnerable part target detection based on prior category knowledge.

[0027] Step 5: Evaluate the model, specifically:

[0028] Based on the test results of step 4, the deep learning model for vulnerable part target detection trained in step 3, based on prior category knowledge, is evaluated in terms of average detection accuracy and detection speed.

[0029] Step 6: Determine whether the detection accuracy and speed of the deep learning model for vulnerable part target detection based on prior category knowledge meet the actual application requirements. If the model meets the actual application requirements, proceed to step 8; otherwise, proceed to step 7.

[0030] Step 7: Correct the width and depth of the deep learning model for vulnerable part target detection based on prior category knowledge built in Step 1, and jump to Step 3.4 to retrain;

[0031] Step 8: Apply the deep learning model for vulnerable part target detection based on prior category knowledge, which meets the practical application requirements in Step 6, to the detection of vulnerable parts in military targets captured by UAVs.

[0032] The improved spatial transformation module described in step 1.1 is a spatial transformation network based on a spatial pyramid structure, consisting of three parts: a positioning network, a mesh generator, and a sampler. Specifically, step 1.1 includes the following steps:

[0033] Step 1.1.1: Locate the network, specifically:

[0034] Convolutional layers are used to adjust the number of feature channels and extract salient features. Then, max pooling layers are used for downsampling to reduce the error in mean shift estimation caused by convolutional layer parameter error noise. Finally, the ReLU activation function is used to perform a non-linear transformation on the extracted features. This "convolutional layer-max pooling layer-ReLU activation function" step is repeated to obtain a small-scale feature map that retains key information. After passing through a spatial pyramid pooling layer, the output is a fixed-dimensional feature vector. Then, after passing through a "fully connected layer-activation function-fully connected layer" step, an affine transformation matrix A with 6-dimensional parameters is obtained. θ .

[0035] Step 1.1.2: Mesh generator, specifically:

[0036] By using parameterized grid sampling, the coordinate positions of each point in the output feature map are generated based on the input feature map and the affine transformation matrix.

[0037]

[0038] Where (x,y) are the coordinates on the input feature map, and (u,v) are the coordinates on the corresponding sampling grid.

[0039] Step 1.1.3: Sampler, specifically:

[0040] The sampler corrects the sampling grid data using bilinear interpolation to ensure that the pixels in the output image are always integer values.

[0041] The category-related feature recalibration module described in step 1.3 can be divided into the following steps:

[0042] Step 1.3.1: Generate high-dimensional feature representations of the input feature map by sequentially using global average pooling layers and convolutional layers;

[0043] Step 1.3.2: Represent the classification result output by the overall object detector as a one-channel feature symbol;

[0044] Step 1.3.3: Using the high-dimensional feature representation obtained in Step 1.3.1 and the channel feature symbols obtained in Step 1.3.2, the feature map obtained in Step 1.2 is recalibrated through operations such as linear regression layers.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] This invention is the first to propose a method for detecting different key parts of different types of targets in UAV remote sensing images, based on an overall target detection framework.

[0047] Because this invention proposes a spatial transformation module based on a spatial pyramid structure, it solves the problem that inaccurate positioning of the overall target detection frame in complex environments will directly affect the detection accuracy of key parts of the target. It can adaptively adjust the target pose, thereby improving the positioning accuracy of key parts of the target.

[0048] Because the present invention designs a category-related feature recalibration module, it can explicitly model the strong correlation between "category and vulnerable part location", thereby improving the robustness and detection accuracy of the vulnerable part detection model.

[0049] The proposed model demonstrates superior localization results under different IoU thresholds. When the IoU threshold is 0.9, the proposed detection framework achieves a 1.91-fold improvement in detection accuracy compared to the traditional Faster R-CNN, proving the effectiveness of the proposed method.

[0050] This invention provides a method for accurate identification of vulnerable parts of ground targets by unmanned aerial vehicles (UAVs). Through a proposed spatial transformation module, the input image can be adaptively transformed using an unsupervised learning approach. Furthermore, a class-related feature recalibration module integrates prior class information into the extracted feature map. Next, training and testing sets are constructed for model training. Finally, the generated dataset of vulnerable parts of ground targets captured by the UAV is fed into a pre-constructed deep learning model for vulnerable part target detection based on prior class knowledge for training, testing, and evaluation. During the model construction phase, the spatial transformation module adaptively removes interfering objects, which helps improve the model's detection accuracy. The class-related feature recalibration module introduces prior class information into the spatial network, which helps improve the model's robustness. In the evaluation phase, the model's accuracy and speed are optimized by adjusting its width and depth to meet the needs of practical applications. Attached Figure Description

[0051] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0052] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] The specific embodiments of the present invention are described below with reference to the accompanying drawings:

[0054] like Figure 1As shown: The present invention provides a method for accurately identifying vulnerable parts of ground targets by unmanned aerial vehicles (UAVs), comprising the following steps:

[0055] Step 1: Construct a deep learning model for vulnerable part target detection based on prior category knowledge. First, to extract high-quality single-target regions from the overall target detection bounding box, a geometric transformation network based on a spatial pyramid structure is designed for image preprocessing. Second, the target category information output from the overall target detection bounding box is modeled as a one-channel descriptor. By learning the strong correlation between prior knowledge and prediction results, the importance of each feature channel is automatically adjusted, and the feature map is recalibrated, thereby automatically generating feature maps related to prior knowledge. Specifically:

[0056] Step 1.1: Preprocess the input global object detection image using an improved spatial transformation module. By adjusting the input image size and target pose, extract high-quality global object detection proposal boxes.

[0057] Step 1.2: Use a basic set of "convolution + ReLU + pooling layers" to extract feature maps from the image. In this embodiment, we selected ResNet50 as our backbone network.

[0058] Step 1.3: Using the classification results output by the overall object detector, the feature map is recalibrated through the category-related feature recalibration module.

[0059] Step 1.4: The recalibrated feature maps are used by a region generation network to extract multiple candidate regions. These candidate boxes have variable sizes and positions. To avoid overcomputing the current detection model, the region generation network typically scales the original image to different sizes and generates a set of anchor-based candidate regions at each scale.

[0060] Step 1.5: For each candidate region, determine its positive or negative attribute, and scale it to a fixed-size feature map using the ROI Align operation. ROI Align performs a more refined interpolation operation on the candidate region to improve detection accuracy.

[0061] Step 1.6: Perform a classification operation on the features of each region, determine the category of the vulnerable part (such as tracks, turret), and return the corresponding confidence level.

[0062] Step 1.7: Simultaneously with Step 1.6, perform a regression operation on the bounding box of each region to fine-tune its position in order to more accurately define the location of vulnerable parts.

[0063] Step 1.8: Based on the results of region classification, non-maximum suppression can be used to filter candidate boxes with large overlaps to remove redundant detection results.

[0064] Step 2: Establish a dataset for detecting vulnerable parts of ground-attack military targets captured by UAVs. Specifically:

[0065] Step 2.1: Place three types of military targets (such as tanks, infantry fighting vehicles, and radar antenna vehicles) and two types of jamming objects (such as trenches and transport vehicles) on the ground to simulate the ground images captured by UAVs when performing tasks such as patrol search and dive attack, and establish a dataset for UAV-captured ground military target detection.

[0066] Step 2.2: Input the UAV-captured ground military target detection dataset into the overall target detector (using YOLOv7 as an example), and extract the overall target detection box based on the detection results of YOLOv7. The cropped result will form our UAV-captured ground military target vulnerable part detection dataset.

[0067] Step 2.3: Randomly select 60% of the UAV-captured images of ground military targets as the training set and 40% of the detection data as the test set.

[0068] Step 3: Train the deep learning model for vulnerable part target detection built in Step 1 based on prior category knowledge. Specifically:

[0069] Step 3.1: Set the training parameters and use the Adam stochastic optimization algorithm for training. Set the batch size to Batch = 12, momentum = 0.9, the initial learning rate to Ir = 0.001, and the number of training iterations to Epoch = 100.

[0070] Step 3.2: Input the training set from the dataset of vulnerable parts of ground military targets captured by UAVs obtained in Step 2.3 into Step 1 to construct a deep learning model for vulnerable part target detection based on prior category knowledge;

[0071] Step 3.3: Adaptively scale the input images in the training set according to the input image size set by the network;

[0072] Step 3.4: Train a deep learning model for vulnerable part target detection based on prior category knowledge. Adjust the learning rate and number of iterations according to the trend of average accuracy and loss changes during training until the accuracy and loss changes gradually stabilize. Determine the final learning rate and number of iterations.

[0073] Step 3.5: Based on the learning rate and number of iterations determined in Step 3.4, complete the training of the deep learning model for vulnerable part target detection based on prior category knowledge, and obtain a well-converged deep learning model for vulnerable part target detection based on prior category knowledge.

[0074] Step 4: Input the test set from the UAV capture vulnerable part detection dataset for ground military targets obtained in Step 2.2 into the deep learning model for vulnerable part target detection trained in Step 3.5 based on prior category knowledge, and test the performance of the deep learning model for vulnerable part target detection based on prior category knowledge.

[0075] Step 5: Evaluate the model, specifically:

[0076] Based on the test results of step 4, the deep learning model for vulnerable part target detection trained in step 3, based on prior category knowledge, is evaluated in terms of average detection accuracy and detection speed.

[0077] Step 6: Determine whether the detection accuracy and speed of the deep learning model for vulnerable part target detection based on prior category knowledge meet the practical application requirements. If the model meets the practical application requirements, proceed to Step 8; otherwise, proceed to Step 7. In this embodiment, when IoU = 0.75, the vulnerable part identification accuracy can reach 87.4%, which meets the application index requirements. In addition, the detection accuracy of this method and the Faster R-CNN method when IoU = 0.9 and IoU = 0.55 are given in Table 1.

[0078] Table 1 Model test results

[0079] 1 Faster RCNN 0.095 0.767 0.902 2 Deep learning model for vulnerable part target detection based on prior category knowledge 0.276 0.874 0.909

[0080] Step 7: Correct the width and depth of the deep learning model for vulnerable part target detection based on prior category knowledge built in Step 1, and jump to Step 3.4 to retrain;

[0081] Step 8: Apply the deep learning model for vulnerable part target detection based on prior category knowledge, which meets the practical application requirements in Step 6, to the detection of vulnerable parts in military targets captured by UAVs.

[0082] The improved spatial transformation module described in step 1.1 is a spatial transformation network based on a spatial pyramid structure, consisting of three parts: a positioning network, a mesh generator, and a sampler. Specifically, step 1.1 includes the following steps:

[0083] Step 1.1.1: Locate the network, specifically:

[0084] Convolutional layers are used to adjust the number of feature channels and extract salient features. Then, max pooling layers are used for downsampling to reduce the error in mean shift estimation caused by convolutional layer parameter error noise. Finally, the ReLU activation function is used to perform a non-linear transformation on the extracted features. This "convolutional layer-max pooling layer-ReLU activation function" step is repeated to obtain a small-scale feature map that retains key information. After passing through a spatial pyramid pooling layer, the output is a fixed-dimensional feature vector. Then, after passing through a "fully connected layer-activation function-fully connected layer" step, an affine transformation matrix A with 6-dimensional parameters is obtained. θ In this embodiment, the specific location network structure is shown in the table below:

[0085] Table 2 Location Network Structure

[0086]

[0087] Step 1.1.2: Mesh generator, specifically:

[0088] By using parameterized grid sampling, the coordinate positions of each point in the output feature map are generated based on the input feature map and the affine transformation matrix.

[0089]

[0090] Where (x,y) are the coordinates on the input feature map I, and (u,v) are the coordinates on the corresponding sampling grid.

[0091] Step 1.1.3: Sampler, specifically:

[0092] The sampler corrects the sampling grid data using bilinear interpolation to ensure that the pixels in the output image are always integer values.

[0093] The category-related feature recalibration module described in step 1.3 can be divided into the following steps:

[0094] Step 1.3.1: Generate high-dimensional feature representations of the input feature map by sequentially using global average pooling layers and convolutional layers;

[0095] Step 1.3.2: Represent the classification result output by the overall object detector as a one-channel feature symbol;

[0096] Step 1.3.3: Using the high-dimensional feature representation obtained in Step 1.3.1 and the channel feature symbols obtained in Step 1.3.2, the feature map obtained in Step 1.2 is recalibrated through operations such as linear regression layers.

[0097] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying vulnerable parts of ground targets by unmanned aerial vehicles (UAVs), characterized in that... The steps of this method include: Step 1: Construct a deep learning model for vulnerable part target detection based on prior category knowledge; Step 2: Establish a dataset for detecting vulnerable parts of ground military targets captured by UAVs, and divide the dataset into a test set and a training set; Step 3: Use the training set established in Step 2 to train the deep learning model for vulnerable part target detection built in Step 1 based on prior category knowledge; Step 4: Use the test set established in Step 2 to test the deep learning model for vulnerable part target detection trained in Step 3 based on prior category knowledge, and obtain the test results. Step 5: Evaluate the test results of Step 4, and determine whether the trained deep learning model for vulnerable part target detection based on prior category knowledge meets the actual application requirements based on the evaluation results. If it does, a qualified deep learning model for vulnerable part target detection based on prior category knowledge is obtained. Otherwise, the deep learning model for vulnerable part target detection based on prior category knowledge constructed in Step 1 is corrected, and then Step 3 is entered to retrain until the actual application requirements are met. Step 6: The obtained deep learning model for vulnerable part target detection based on prior category knowledge, which meets the needs of practical applications, is used for the detection of vulnerable parts in military targets captured by UAVs. In step 1, the constructed deep learning model for vulnerable part target detection based on prior category knowledge includes an improved spatial transformation module, a feature extraction module, a category-related feature recalibration module, a region generation network, and a fully connected layer. The improved spatial transformation module is a spatial transformation network based on a spatial pyramid structure, which is divided into three parts: a positioning network, a mesh generator, and a sampler. The localization network uses convolutional layers to adjust the number of feature channels and extract salient features. It then uses max-pooling layers for downsampling to reduce the error in mean shift estimation caused by convolutional layer parameter error noise. A ReLU activation function is used to perform a non-linear transformation on the extracted features. This process of "convolutional layer-max-pooling layer-ReLU activation function" is repeated to obtain a small-scale feature map that retains key information. After passing through a spatial pyramid pooling layer, a fixed-dimensional feature vector is output. Finally, after passing through a "fully connected layer-activation function-fully connected layer" process, an affine transformation matrix with 6-dimensional parameters is obtained. ; The mesh generator generates the coordinate positions of each point in the output feature map by parametric mesh sampling, based on the input feature map and the affine transformation matrix. Where (x,y) are the coordinates on the input feature map, and (u,v) are the coordinates on the corresponding sampling grid; The sampler corrects the sampling grid data using bilinear interpolation to ensure that the pixels in the output image are always integer values. The category-related feature recalibration module sequentially uses a global average pooling layer and a convolutional layer to generate a high-dimensional feature representation of the input feature map. Then, it represents the classification result output by the overall object detector as a one-channel feature symbol. Using the obtained high-dimensional feature representation and the obtained channel feature symbol, the feature map obtained by the feature extraction network is recalibrated through a linear regression layer.

2. The method for identifying vulnerable parts of ground targets by unmanned aerial vehicles (UAVs) according to claim 1, characterized in that: In step 1, the ground military targets include tanks, infantry fighting vehicles, radar antenna vehicles, and drones.

3. The method for identifying vulnerable parts of ground targets by unmanned aerial vehicles (UAVs) according to claim 1, characterized in that: In step 2, the method for establishing a dataset for detecting vulnerable parts of ground military targets captured by UAVs is as follows: Step 2.1: Place 3-6 types of military targets and jamming objects on the ground to simulate the ground images captured by the UAV during its patrol search and dive attack missions, and establish a dataset for UAV-captured ground military target detection. Step 2.2: Input the UAV-captured ground-attack military target detection dataset into the overall target detector, and extract the overall target detection box based on the detection results. The cropped results will form the UAV-captured ground-attack military target vulnerable part detection dataset. The method for dividing the established detection dataset into a test set and a training set is as follows: 60% of the images of ground military targets captured by drones were randomly selected as the training set, and 40% of the detection data were used as the test set.

4. The method for identifying vulnerable parts of ground targets by unmanned aerial vehicles (UAVs) according to claim 1, characterized in that: In step 3, the method for training a deep learning model for vulnerable part target detection based on prior category knowledge is as follows: Step 3.1: Set the training parameters and use the Adam stochastic optimization algorithm for training. Set the batch size to Batch = 12, Momentum = 0.9, the initial learning rate to Ir = 0.001, and the number of training iterations to Epoch = 100. Step 3.2: Adaptively scale the input images in the training set according to the input image size set by the network; Step 3.3: Train a deep learning model for vulnerable part target detection based on prior category knowledge. Adjust the learning rate and number of iterations according to the average accuracy change and loss change trend during the training process until the accuracy change and loss change gradually tend to stabilize. Determine the final learning rate and number of iterations. Step 3.4: Based on the learning rate and number of iterations determined in Step 3.3, complete the training of the deep learning model for vulnerable part target detection based on prior category knowledge, and obtain the deep learning model for vulnerable part target detection based on prior category knowledge.

Images