Aerial image target detection method based on dual feature relationship distillation
By employing a dual feature relation distillation method, utilizing feature decoupling and local pixel-by-pixel relation distillation modules, the problem of poor detection performance in aerial image detection is solved, achieving more efficient detection performance and a lightweight model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA UNIV OF MINING & TECH
- Filing Date
- 2024-09-09
- Publication Date
- 2026-06-09
AI Technical Summary
Existing relation-based distillation methods are easily affected by interference and noise in aerial image detection, making it difficult to capture local feature relationships and resulting in poor detection performance.
A dual feature relationship distillation method is adopted, which distills the relationship between the target and the background in the aerial image through feature decoupling operation and block learning strategy. Combined with the local pixel-by-pixel relationship distillation module, the model's ability to detect local details and complex scenes is improved.
It significantly improves the detection performance and lightweight nature of aerial image detection models, and enhances the ability to understand complex scenes and perceive details.
Smart Images
Figure CN119131631B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of drone aerial image technology, and more specifically, to a target detection method for aerial images based on dual feature relationship distillation. Background Technology
[0002] Currently, most relationship-based distillation methods utilize the relationships between samples or features in the teacher network for knowledge transfer. By learning these structured relationships, the student network can mimic the teacher network's approach to different things. A commonly used feature relationship distillation method uses the inner product method to calculate the feature relationship matrix of adjacent feature maps of the teacher and student networks respectively, and uses feature relationship loss to bring the structural relationships between the two networks closer.
[0003] This method has achieved good results in image classification or natural image detection tasks. However, in the field of aerial image detection, due to the complex scenes in aerial images, when distilling directly using the relationship between features of adjacent layers, the relationship between features may be affected by interference or noise, resulting in inaccurate relationship features obtained by distillation. Secondly, since the arrangement of targets in aerial images is usually quite dense, if relationship distillation is performed based on the entire feature map, it is difficult to capture the local feature relationships in the image, resulting in information loss and relatively blurred relationships during the distillation process.
[0004] Therefore, when applying feature-based distillation methods to aerial image detection, it is necessary to conduct in-depth research and optimization on these issues in order to improve distillation results and detection performance. Summary of the Invention
[0005] This invention proposes an aerial image target detection method based on dual feature relationship distillation. It comprehensively considers the relationship between intermediate features and utilizes decoupling operations and block learning strategies to enable the student network to better learn the relationship between different regions in the image, enhances the sensitivity to local details, and significantly improves the detection performance of the student network while maintaining a low number of parameters.
[0006] The technical solution for implementing this invention is: a target detection method for aerial images based on dual feature relation distillation, comprising the following steps:
[0007] Step S1: Download the aerial image detection dataset DIOR-R, randomly cut the image into 800×800 blocks, randomly divide the blocks of uniform size into training and test datasets in a 7:3 ratio, perform data augmentation on the training dataset to form a teacher-student network training dataset, and proceed to step S2.
[0008] Step S2: Use the teacher-student network training dataset to pre-train the teacher network to obtain the pre-trained teacher network, then proceed to step S3.
[0009] Step S3: Construct a teacher-student network based on the pre-trained teacher network, student network, feature decoupling relation distillation module, and local pixel-by-pixel relation distillation module, and proceed to step S4.
[0010] Step S4: Input the teacher-student network training dataset into the teacher-student network for training, and extract multi-scale features from the pre-trained teacher network and student network respectively; construct a feature decoupling relation distillation module, use the binarized ground truth map to separate the foreground and background information in the multi-scale features, and then distill the foreground and background relations respectively; construct a local pixel-wise relation distillation module, divide the multi-scale features into N×N block feature maps, and then input them into the graph convolution module GloRe to obtain relation-aware features, and perform local pixel-wise relation distillation; fix the parameters of the pre-trained teacher network, update the parameters of the student network through the overall loss function of the student network, and finally obtain the trained student network, and proceed to step S5.
[0011] Step S5: Input the test dataset into the trained student network for detection, merge the detection results of each block image, output the category and location of all targets in the test dataset in the image, and obtain the detection accuracy of the trained student network.
[0012] Compared with existing technical solutions, the advantages of this invention are:
[0013] (1) Compared with existing methods, this invention proposes an aerial image target detection method based on dual feature relationship distillation, which distills the feature relationship from two perspectives, significantly improves the detection performance of the model, realizes the lightweighting of the aerial image detection model, and provides a new solution for engineering applications.
[0014] (2) The feature decoupling relationship distillation module proposed in this invention decouples intermediate features and distills the relationships between target features and non-target features respectively, which encourages students to learn the correlation between different elements in the network, so as to better understand the structure and relationship of different regions in the image and improve the model's ability to detect complex scenes; the local pixel-by-pixel relationship distillation module divides each layer of features into blocks and obtains the local feature processing method by distilling the feature relationships inside the small blocks of features. This module emphasizes the relationship of the model at the pixel level, improves the model's ability to perceive image details, and effectively solves the problem of insufficient ability of the model to capture the local detail relationship of the entire feature map. Attached Figure Description
[0015] Figure 1This is a network framework diagram of the aerial image target detection method based on dual feature relation distillation of the present invention.
[0016] Figure 2 This is a schematic diagram of the feature decoupling relation distillation module (taking foreground relation distillation as an example) proposed in this invention.
[0017] Figure 3 This is a schematic diagram of the local pixel-by-pixel relationship distillation module proposed in this invention. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0019] Combination Figures 1-3 The aerial image target detection method based on dual feature relation distillation described in this invention includes the following steps:
[0020] Step S1: Download the aerial image detection dataset DIOR-R, randomly cut the image into 800×800 blocks, randomly divide the blocks of uniform size into training and test datasets in a 7:3 ratio, perform data augmentation on the training dataset to form a teacher-student network training dataset, and proceed to step S2.
[0021] Step S2: Use the teacher-student network training dataset to pre-train the teacher network to obtain the pre-trained teacher network, then proceed to step S3.
[0022] Step S3: Construct a teacher-student network based on the pre-trained teacher network, student network, feature decoupling relation distillation module, and local pixel-by-pixel relation distillation module, and proceed to step S4.
[0023] Step S4: Input the teacher-student network training dataset into the teacher-student network for training, and extract multi-scale features from the pre-trained teacher network and student network respectively; construct a feature decoupling relation distillation module, using a binarized ground truth map to separate the foreground and background information in the multi-scale features, and then distill the foreground and background relations respectively; construct a local pixel-wise relation distillation module, dividing the multi-scale features into N×N block feature maps, and then inputting them into the graph convolution module GloRe to obtain relation-aware features, and perform local pixel-wise relation distillation; fix the parameters of the pre-trained teacher network, and update the parameters of the student network through the overall loss function of the student network, finally obtaining the trained student network, as follows:
[0024] First, multi-scale features are extracted from the pre-trained teacher network and student network respectively. The multi-scale features of the pre-trained teacher network are represented as follows: The multi-scale features of student networks are denoted as
[0025] Construct a feature decoupling relationship distillation module:
[0026] For aerial images with complex backgrounds, foreground and background relationships often have different statistical properties. Processing them separately helps the model capture and utilize these structural features. Therefore, it was decided to separate the target features and non-target features from the features and then distill them separately.
[0027] For the multi-scale features of layer l, the foreground and background information in the multi-scale features of this layer are first separated, and a binarized ground truth target image B of the same size as the features of this layer is set. l If a position in the binarized truth target image is within the target truth box, then that position is set to 1; otherwise, it is set to 0. Binarized truth target image B... l The methods for obtaining it are as follows:
[0028]
[0029] Among them, scale l (·) represents the scaling function of the ground truth objective at layer l, which is used to map the rotated ground truth box rbbox to the corresponding feature scale; This represents the binarized value at any position (m, n) in the binarized ground truth target image of the multi-scale feature at layer l.
[0030] Binarized ground truth target image B of the multi-scale features of layer l l With the corresponding multi-scale feature F l Perform a multiplication operation to obtain the foreground and background features of the l-th layer, and separate the foreground features from the multi-scale features of the l-th layer. The calculation formula is:
[0031]
[0032] Background features separated from the multi-scale features of layer l The calculation formula is:
[0033]
[0034] Based on the foreground and background features separated from the multi-scale features of each layer, foreground and background relationships between adjacent feature layers are mined, utilizing the separated features. and The foreground relationship matrix between two foreground features is calculated using the following formula:
[0035]
[0036] in, F represents the foreground relation matrix between the features of layer l and layer l+1. This relation matrix is a simple mapping between the processing of the target context; H represents the height of the multi-scale feature of layer l, and W represents the width of the multi-scale feature of layer l; A(·) represents the adaptive pooling operation, which is used to convert the foreground features of layer l+1 to the same size as the foreground features after masking of layer l; l (m,n) represents the feature value of the multi-scale feature of the l-th layer at any position (m,n).
[0037] Foreground relation matrices between features from all adjacent layers are calculated for both the pre-trained teacher network and the student network. Finally, the set of matrix relations between foreground features obtained by the pre-trained teacher network is... The set of matrix relationships between foreground features obtained by the student network is To reduce the discrepancy in the relationship between foreground features of the student network and the pre-trained teacher network, an L2 loss function is used to optimize the relationship between the two. The sum of the relationship distillation losses for all adjacent foreground features is L. re_1 The calculation formula is:
[0038]
[0039] in, This represents the foreground relation matrix between features of layer l and layer l+1 of the pre-trained teacher network. This represents the foreground relationship matrix between features of layer l and layer l+1 of the student network.
[0040] Using the separated and The formula for calculating the background relationship matrix between two background features from adjacent layers is as follows:
[0041]
[0042] in, This represents the background relationship matrix between the features of layer l and layer l+1.
[0043] The relationship between all adjacent background features, the sum of distillation losses, L re_2 The calculation formula is:
[0044]
[0045] in, This represents the background relationship matrix between features of layer l and layer l+1 of the pre-trained teacher network. This represents the background relationship matrix between the features of layer l and layer l+1 of the student network.
[0046] Distilling foreground relationships between adjacent feature layers can help student networks better understand how target context relationships are processed, thereby improving the student network's ability to understand contextual information; distilling background relationships helps convey the relationships between background regions, helping student networks better distinguish between targets and background.
[0047] Since the foreground relationship matrix and the background relationship matrix between adjacent layers contain information of different importance, different loss weights are added to the two losses. Finally, the total loss function L based on the feature decoupling relationship distillation module is calculated. de_re for:
[0048] L de_re =α1L re_1 +α2L re_2
[0049] Where α1 is the weight of the foreground relation loss function and α2 is the weight of the background relation loss function.
[0050] By decoupling multi-scale features into target features and non-target features, and calculating the relationships between target features and non-target features respectively, the ability of student networks to understand complex scene relationships in aerial images can be improved.
[0051] Constructing a local pixel-by-pixel relationship distillation module:
[0052] While focusing on global target relationships or global non-target relationships, the network may overlook some important local information. Therefore, a local pixel-by-pixel relationship distillation module is introduced to better balance the learning of global and local information.
[0053] Let the multi-scale features of the i-th layer of the pre-trained teacher network be... Multiscale features The feature map is divided into N×N blocks, resulting in k blocks of size C. i A block feature map of ×N×N, denoted as k is calculated as follows:
[0054] k = H i / N×W i / N
[0055] Among them, C i H represents the number of channels for the multi-scale feature of the i-th layer. i W represents the height of the multi-scale features of the i-th layer. i This represents the width of the multi-scale feature in the i-th layer;
[0056] Extract the block feature map of the j-th block in the i-th layer. The feature map is input into a graph convolutional unit GloRe, which includes graph embedding, graph convolution, and reprojection.
[0057] First, the features in the coordinate space are projected onto a low-dimensional feature space. For the input block feature map... The projection is transformed into a block projection feature map through a linear layer. Then to Projection is performed to obtain the graph node features Q. The calculation formula is:
[0058]
[0059] Where D is the learnable projection matrix. The graph node feature Q can aggregate information from multiple regions;
[0060] Then, graph convolution is used to capture the relationships between graph nodes. The calculation formula is as follows:
[0061] Z=((IV)Q)U
[0062] Where I is the identity matrix, V represents the node adjacency matrix. The state is continuously updated during training; U represents the state update matrix. Z represents the relationship characteristics of the graph nodes.
[0063] Finally, the graph node relationship features are reprojected back into the coordinate space to obtain the relationship-aware features, calculated using the following formula:
[0064]
[0065] in, Block feature maps for pre-trained teacher networks Relationship perception characteristics D* is the reprojection matrix, which means we can learn the transpose of the projection matrix D.
[0066] Similarly, the block feature map of the student network is obtained. Relationship perception features The L2 loss is used to shorten the distance between two relation-aware features. The calculation formula is as follows:
[0067]
[0068] Where N×N represents the height and width of the block feature map; Represents the relation-aware features of the block feature map of the j-th block in the i-th layer of the pre-trained teacher network. Perceptual features related to the block feature map of the i-th layer and j-th block of the student network The pixel-by-pixel relationship between the distillation loss function.
[0069] Obtain relational features from all block feature maps of each layer of the pre-trained teacher and student networks, and finally use the local pixel-wise relational distillation loss function L. pprd for:
[0070]
[0071] The Local Pixel-by-Pixel Relation Distillation module focuses on learning and capturing local pixel-by-pixel relationships, improving the model's sensitivity to image details and effectively solving the problem of insufficient model ability to capture image details.
[0072] Construct the overall loss function for the student network:
[0073] When training the student network, all parameters of the pre-trained teacher network remain unchanged, while the parameters of the student network are updated using a loss function. Integrating the loss from the feature decoupling relation distillation module and the local pixel-by-pixel relation distillation loss, the overall loss function L of the student network is obtained as follows:
[0074] L = L task +β1L de_re +β2L pprd
[0075] Among them, L task L represents the original detection task loss of the student network, including classification loss and regression loss. de_re L represents the total loss of the feature decoupling distillation module. pprd This represents the local pixel-by-pixel distillation loss, with β1 and β2 used to control the proportions of different distillation loss terms.
[0076] Proceed to step S5.
[0077] Step S5: Input the test dataset into the trained student network for detection, merge the detection results of each block image, output the category and location of all targets in the test dataset in the image, and obtain the detection accuracy of the trained student network.
[0078] Example 1
[0079] The aerial image target detection method based on dual feature relation distillation described in this invention comprises the following steps:
[0080] Step S1: Download the aerial image detection dataset DIOR-R, randomly cut the image into 800×800 blocks, randomly divide the blocks of uniform size into training and test datasets in a 7:3 ratio, perform data augmentation on the training dataset to form a teacher-student network training dataset, and proceed to step S2.
[0081] Step S2: Use the teacher-student network training dataset to pre-train the teacher network to obtain the pre-trained teacher network, then proceed to step S3.
[0082] Step S3: Construct a teacher-student network based on the pre-trained teacher network, student network, feature decoupling relation distillation module, and local pixel-by-pixel relation distillation module, and proceed to step S4.
[0083] Step S4: Input the teacher-student network training dataset into the teacher-student network for training, and extract multi-scale features from the pre-trained teacher network and student network respectively; construct a feature decoupling relation distillation module, use the binarized ground truth map to separate the foreground and background information in the multi-scale features, and then distill the foreground and background relations respectively; construct a local pixel-wise relation distillation module, divide the multi-scale features into N×N block feature maps, and then input them into the graph convolution module GloRe to obtain relation-aware features, and perform local pixel-wise relation distillation; fix the parameters of the pre-trained teacher network, update the parameters of the student network through the overall loss function of the student network, and finally obtain the trained student network, and proceed to step S5.
[0084] Step S5: Input the test dataset into the trained student network for detection, merge the detection results of each block image, output the category and location of all targets in the test dataset in the image, and obtain the detection accuracy of the trained student network.
[0085] This invention utilizes Python and the PyTorch framework for experiments, with model training and inference performed on an NVIDIA GeForce RTX 3090 graphics card. In the experiments, the model training employed the SGD strategy for optimization, with a momentum decay of 0.9 and a weight decay of 0.0001. The initial learning rate was 0.0025, which was divided by 10 at epochs 24 and 33. The DIOR-R dataset was used for 36 training epochs, with a batch size of 2. For the hyperparameter settings, β1 was set to 0.005, and β2 was set to 0.01.
[0086] To demonstrate the effectiveness of this invention, several popular distillation algorithms from recent years were selected as comparative models. In the experiment, a high-precision remote sensing image target detection algorithm with a ResNet-101 backbone network was used. 2A-Net was used as the teacher network, and MobilenetV2 was used as the backbone extraction network for the student network. All other parameters remained unchanged. The experimental results are shown in Table 1.
[0087] Table 1. Comparison of experimental results with other distillation algorithms on the DIOR-R dataset.
[0088]
[0089] The experimental results in Table 1 demonstrate the practicality and effectiveness of this invention.
Claims
1. A target detection method for aerial images based on dual feature relation distillation, characterized in that, Includes the following steps: Step S1: Download the aerial image detection dataset DIOR-R, randomly cut the image into 800×800 blocks, randomly divide the blocks of uniform size into training dataset and test dataset in a 7:3 ratio, perform data augmentation on the training dataset to form a teacher-student network training dataset, and proceed to step S2. Step S2: Use the teacher-student network training dataset to pre-train the teacher network to obtain the pre-trained teacher network, then proceed to step S3; Step S3: Construct a teacher-student network based on the pre-trained teacher network, student network, feature decoupling relation distillation module, and local pixel-by-pixel relation distillation module, and proceed to step S4; Step S4: Input the teacher-student network training dataset into the teacher-student network for training, and extract the multi-scale features of the pre-trained teacher network and student network respectively; A feature decoupling relation distillation module is constructed to separate foreground and background information from multi-scale features using a binarized ground truth map, and then distill foreground and background relations separately. A local pixel-wise relation distillation module is constructed to divide multi-scale features into N×N block feature maps, and then input them into the graph convolution module GloRe to obtain relation-aware features and perform local pixel-wise relation distillation. The parameters of the pre-trained teacher network are fixed, and the parameters of the student network are updated through the overall loss function of the student network to finally obtain the trained student network. The foreground and background information in multi-scale features are separated using a binarized ground truth target image, as detailed below: Multi-scale features of the pre-trained teacher network and student network are extracted separately. The multi-scale features of the pre-trained teacher network are represented as follows: The multi-scale features of the student network are denoted as ; For the For multi-scale features of a layer, set a binarized ground truth target image of the same size as the features of that layer. If a position in the binarized truth target image is within the target truth box, then that position is set to 1; otherwise, it is set to 0. The methods for obtaining it are as follows: ； in, The true objective is represented in the first... The scaling function of the layer is used to rotate the truth box. Map to the corresponding feature scale; Indicates the first Binarized ground truth target image of multi-scale features at any position Binarization value at; For the Binarized ground truth target map of multi-scale features Corresponding multi-scale features Perform the multiplication operation to obtain the first... Foreground and background features of the first layer, Foreground features separated from multi-scale features The calculation formula is: ； No. Background features separated by multi-scale features The calculation formula is: ； The foreground and background information in each layer of features are separated, and then their respective foreground and background features are obtained; The distillation of foreground and background relationships is as follows: Using the separated and The formula for calculating the foreground relationship matrix between features from adjacent layers is as follows: ； in, Indicates the first Layer and first Foreground relationship matrix between layer features; Indicates the first High multi-scale features across layers Indicates the first The width of multi-scale features across layers; Represents an adaptive pooling operation, used to pool the first... Foreground features of layer 1 are converted to those of layer 2. The same size foreground features after layer masking; Indicates the first Multi-scale features at any location Eigenvalues at; Foreground relation matrices between features from all adjacent layers are calculated for both the pre-trained teacher network and the student network. Finally, the set of matrix relations between foreground features obtained by the pre-trained teacher network is... The set of matrix relationships between foreground features obtained by the student network is as follows: To reduce the discrepancy in the relationship between foreground features of the student network and the pre-trained teacher network, an L2 loss function is used to optimize the relationship between the two. The sum of the relationship distillation losses for all adjacent foreground features is used. The calculation formula is: ； in, Indicates the pre-training teacher network Layer and first Foreground relationship matrix between layer features Indicates the student network Layer and first Foreground relationship matrix between layer features; Using the separated and The formula for calculating the background relationship matrix between two background features from adjacent layers is as follows: ； in, Indicates the first Layer and first Background relationship matrix between layer features; The relationship between all adjacent background features and the sum of distillation losses. The calculation formula is: ； in, Indicates the pre-training teacher network Layer and first Background relationship matrix between layer features Indicates the student network Layer and first Background relationship matrix between layer features; Since the foreground relationship matrix and the background relationship matrix between adjacent layers contain information of different importance, different loss weights are added to the two types of losses. The final total loss function is based on the feature decoupling relationship distillation module. for: ； in, The weights of the foreground relation loss function, The weights of the background relation loss function; Proceed to step S5; Step S5: Input the test dataset into the trained student network for detection, merge the detection results of each block image, output the category and location of all targets in the test dataset in the image, and obtain the detection accuracy of the trained student network.
2. The aerial image target detection method based on dual feature relation distillation according to claim 1, characterized in that, In step S4, a local pixel-by-pixel relationship distillation module is constructed, as follows: Set up a pre-training teacher network. The multi-scale characteristics of the layer are , Multi-scale features Divided into The size of the block feature map is finally obtained. block size is The block feature map, denoted as , The calculation method is as follows: ； in, Indicates the first The number of channels in multi-scale layer features. Indicates the first High multi-scale features across layers Indicates the first The width of multi-scale features across layers; Extract the first Layer Block feature map The feature map is input into the graph convolutional unit GloRe, which includes graph embedding, graph convolution, and reprojection. First, the features in the coordinate space are projected onto a low-dimensional feature space. For the input block feature map... , The linear layer transforms its projection into a block projection feature map. Then to Projection is performed to obtain graph node features , The calculation formula is: ； in, For learnable projection matrices, Graph node features It can aggregate information from multiple regions; Then, graph convolution is used to capture the relationships between graph nodes. The calculation formula is as follows: ； in, It is the identity matrix. ; Represents the node adjacency matrix. It is constantly updated as the training progresses; Represents the state update matrix. ; Represents the relationship characteristics of graph nodes. ; Finally, the graph node relationship features are reprojected back into the coordinate space to obtain the relationship-aware features, calculated using the following formula: ； in, Block feature maps for pre-trained teacher networks Relationship perception characteristics , For the reprojection matrix, the projection matrix can be learned. The transpose of the matrix; Similarly, the block feature map of the student network is obtained. Relationship perception features The L2 loss is used to shorten the distance between two relation-aware features. The calculation formula is as follows: ； in, Indicates the height and width of the block feature map; Indicates the pre-training teacher network Layer Relationship-aware features of block feature maps With student network Layer Relationship-aware features of block feature maps Distillation loss function for pixel-by-pixel relationship; Obtain relational features from all block feature maps of each layer of the pre-trained teacher and student networks, and finally use the local pixel-wise relational distillation loss function. for: 。 3. The aerial image target detection method based on dual feature relation distillation according to claim 2, characterized in that, In step S4, the overall loss function of the student network is constructed as follows: When training the student network, all parameters of the pre-trained teacher network remain unchanged. The parameters of the student network are updated using a loss function. The overall loss function of the student network is obtained by integrating the loss from the feature decoupling relation distillation module and the local pixel-by-pixel relation distillation loss. for: ； in, The original detection task loss of the student network is represented by two parts: classification loss and regression loss. This represents the total loss of the feature decoupling distillation module. This represents the local pixel-by-pixel relationship distillation loss. Used to control the proportion of different distillation loss items.