Aerial image target detection method based on dual feature relationship distillation

Abstract

The invention discloses an aerial image target detection method based on dual feature relationship distillation, which comprises the following steps: firstly, constructing a feature decoupling relationship distillation module, decoupling all multi-scale features into target features and non-target features, and further respectively distilling the relationship between the target features and the relationship between the non-target features; a local pixel-by-pixel relationship distillation module is constructed, a block learning strategy is adopted, and an internal relationship between feature images of each block is calculated by using image convolution, so that the network is more focused on learning and capturing the local pixel-by-pixel relationship, and the learning efficiency is improved. Therefore, the perception and expression capability of the network on local details is obviously improved. According to the method, the relationship of the intermediate features is comprehensively considered, the student network can better understand and learn abundant feature relationship representation in the teacher network through decoupling operation and block learning, the detection performance of the student network is effectively improved, and the lightweight of an aerial image target detection model is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial image technology, and in particular to an aerial image target detection method based on dual feature relationship distillation. Background Art

[0002] At present, most of the relationship-based distillation methods use the relationship between samples or features of the teacher network to transfer knowledge. By learning these structured relationships, the student network can imitate the way the teacher network handles different things. The commonly used feature relationship distillation method is to use the inner product method to calculate the feature relationship matrix of adjacent feature graphs of the teacher network and the student network respectively, and use the feature relationship loss to bring the structural relationship between the two networks closer.

[0003] This method has achieved good results in image classification or natural image detection tasks. However, in the field of aerial image detection, due to the complex scenes of aerial images, when the relationship between adjacent layer features is directly used for distillation, the relationship between features may be affected by interference or noise, resulting in inaccurate relational features obtained by distillation. Secondly, since the objects in aerial images are usually densely arranged, if the relationship distillation is based on the entire feature map, it is difficult to capture the local feature relationship in the image, resulting in information loss and relatively vague relationships during the distillation process.

[0004] Therefore, when applying the feature relationship-based distillation method to the field of aerial image detection, in-depth research and optimization are needed to improve the distillation effect and detection performance. Summary of the invention

[0005] The present invention proposes a method for aerial image target detection based on dual feature relationship distillation, which comprehensively considers the relationship between intermediate features and utilizes the strategy of decoupling operation and block learning to enable the student network to better learn the relationship between different regions in the image and enhance the sensitivity to local details. While ensuring a low parameter amount, the detection performance of the student network is significantly improved.

[0006] The technical solution to realize the present invention is: a method for aerial image target detection based on dual feature relationship distillation, the steps are as follows:

[0007] Step S1, download the aerial image detection dataset DIOR-R, randomly cut the image into 800×800 blocks, randomly divide the unified-size blocks into training and test datasets in a ratio of 7:3, and perform data enhancement on the training dataset to form a teacher-student network training dataset, and then go to step S2.

[0008] Step S2: Use the teacher-student network training data set to pre-train the teacher network to obtain a pre-trained teacher network, and then proceed to step S3.

[0009] Step S3: construct a teacher-student network based on the pre-trained teacher network, student network, feature decoupling relationship distillation module and local pixel-by-pixel relationship distillation module, and proceed to step S4.

[0010] Step S4, input the teacher-student network training data set into the teacher-student network for training, and extract the multi-scale features of the pre-trained teacher network and the student network respectively; construct a feature decoupling relationship distillation module, use the binary true value target map to separate the foreground information and background information in the multi-scale features, and then distill the foreground relationship and background relationship respectively; construct a local pixel-by-pixel relationship distillation module, divide the multi-scale features into N×N block feature maps, and then input them into the graph convolution module GloRe to obtain relationship-aware features and perform local pixel-by-pixel relationship distillation; fix the parameters of the pre-trained teacher network, update the parameters of the student network through the overall loss function of the student network, and finally obtain the trained student network, and go to step S5.

[0011] Step S5: input the test data set into the trained student network for detection, merge the detection results of each block image, output the category and position of all targets in the test data set in the image, and obtain the detection accuracy of the trained student network.

[0012] Compared with the prior art solutions, the advantages of the present invention are:

[0013] (1) Compared with the existing methods, the present invention proposes an aerial image target detection method based on dual feature relationship distillation. The feature relationship is distilled from two perspectives, which significantly improves the detection performance of the model, realizes the lightweight of the aerial image detection model, and provides a new solution for engineering applications.

[0014] (2) The feature decoupling relationship distillation module proposed in the present invention decouples the intermediate features and distills the relationships between target features and non-target features respectively, prompting the student network to learn the correlation between different elements to better understand the structure and relationship of different areas in the image, thereby improving the model's detection ability to cope with complex scenes; the local pixel-by-pixel relationship distillation module divides each layer of features into blocks, and obtains the local feature processing method by distilling the feature relationship within the small block features. This module emphasizes the relationship of the model at the pixel level, improves the model's perception of image details, and effectively solves the problem of the model's insufficient ability to capture the local detail relationship of the entire feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1This is a network framework diagram of the aerial image target detection method based on dual feature relationship distillation of the present invention.

[0016] Figure 2 This is a schematic diagram of the feature decoupling relationship distillation module proposed in the present invention (taking foreground relationship distillation as an example).

[0017] Figure 3 Schematic diagram of the local pixel-by-pixel relationship distillation module proposed in the present invention. DETAILED DESCRIPTION

[0018] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0019] Combination Figure 1 to Figure 3 The method for aerial image target detection based on dual feature relationship distillation of the present invention comprises the following steps:

[0020] Step S1, download the aerial image detection dataset DIOR-R, randomly cut the image into 800×800 blocks, randomly divide the unified-size blocks into training and test datasets in a ratio of 7:3, and perform data enhancement on the training dataset to form a teacher-student network training dataset, and then go to step S2.

[0021] Step S2: Use the teacher-student network training data set to pre-train the teacher network to obtain a pre-trained teacher network, and then proceed to step S3.

[0022] Step S3: construct a teacher-student network based on the pre-trained teacher network, student network, feature decoupling relationship distillation module and local pixel-by-pixel relationship distillation module, and proceed to step S4.

[0023] Step S4: Input the teacher-student network training data set into the teacher-student network for training, and extract the multi-scale features of the pre-trained teacher network and the student network respectively; construct a feature decoupling relationship distillation module, use the binary true value target map to separate the foreground information and background information in the multi-scale features, and then distill the foreground relationship and background relationship respectively; construct a local pixel-by-pixel relationship distillation module, divide the multi-scale features into N×N block feature maps, and then input them into the graph convolution module GloRe to obtain relationship-aware features and perform local pixel-by-pixel relationship distillation; fix the parameters of the pre-trained teacher network, update the parameters of the student network through the overall loss function of the student network, and finally obtain the trained student network, as follows:

[0024] First, the multi-scale features of the pre-trained teacher network and the student network are extracted respectively. The multi-scale features of the pre-trained teacher network are expressed as The multi-scale features of the student network are denoted as

[0025] Construct feature decoupling relationship distillation module:

[0026] For aerial images with complex backgrounds, foreground and background relationships often have different statistical characteristics. Separate processing is more helpful for the model to capture and utilize these structural features. Therefore, it is decided to separate the target features and non-target features in the features and then distill them separately.

[0027] For the multi-scale features of the lth layer, the foreground information and background information in the multi-scale features of this layer are first separated, and a binary true value target image B with the same size as the features of this layer is set l , if a position of the binary true value target image is within the target true value box, then the position is set to 1, otherwise it is set to 0. The binary true value target image B l The way to obtain is as follows:

[0028]

[0029] Among them, scale l (·) represents the scaling function of the ground-truth target at layer l, which is used to map the rotated ground-truth box rbbox to the corresponding feature scale; Represents the binary value at any position (m,n) in the binary true value target image of the multi-scale features of the lth layer.

[0030] Binarized true value target map B of the l-th layer multi-scale features l And the corresponding multi-scale feature F l Perform multiplication operation to obtain the foreground features and background features of the lth layer, and the foreground features separated by the multi-scale features of the lth layer The calculation formula is:

[0031]

[0032] Background features separated by multi-scale features at layer l The calculation formula is:

[0033]

[0034] According to the foreground features and background features separated from each layer of multi-scale features, the foreground relationship and background relationship between adjacent feature layers are mined. and Two foreground features calculate the foreground relationship matrix between adjacent layer features, the formula is:

[0035]

[0036] in, represents the foreground relationship matrix between the features of the lth layer and the l+1th layer. This relationship matrix is ​​a simple mapping of the processing between the target contexts. H represents the height of the multi-scale features of the lth layer, and W represents the width of the multi-scale features of the lth layer. A(·) represents the adaptive pooling operation, which is used to convert the foreground features of the l+1th layer to the same size as the foreground features after masking in the lth layer. F l (m,n) represents the feature value of the l-th layer multi-scale feature at any position (m,n).

[0037] The foreground relationship matrix between all adjacent layer features of the pre-trained teacher network and the student network is calculated respectively. Finally, the matrix relationship set between the foreground features obtained by the pre-trained teacher network is The matrix relationship set between the foreground features obtained by the student network is In order to narrow the relationship difference between the foreground features of the student network and the pre-trained teacher network, the L2 loss function is used to optimize the relationship between the two. The sum of the relationship distillation losses of all adjacent foreground features is L re_1 The calculation formula is:

[0038]

[0039] in, represents the foreground relationship matrix between the features of the lth and l+1th layers of the pre-trained teacher network, Represents the foreground relationship matrix between the features of the lth and l+1th layers of the student network.

[0040] Using the separated and Two background features calculate the background relationship matrix between adjacent layer features, the formula is:

[0041]

[0042] in, Represents the background relationship matrix between the features of the lth layer and the l+1th layer.

[0043] The sum of the relational distillation losses of all adjacent background features L re_2 The calculation formula is:

[0044]

[0045] in, represents the background relationship matrix between the features of the lth and l+1th layers of the pre-trained teacher network, Represents the background relationship matrix between the features of the lth and l+1th layers of the student network.

[0046] Distilling the foreground relations of adjacent feature layers can enable the student network to better understand how to handle the target contextual relations, thereby improving the student network's ability to understand contextual information; distilling background relations helps to convey the relations between background regions, helping the student network to better distinguish between targets and backgrounds.

[0047] Since the foreground relationship matrix between features of adjacent layers and the background relationship matrix between features of adjacent layers contain different information importance, different loss weights are added to the two losses. Finally, the total loss function L based on the feature decoupling relationship distillation module is de_re for:

[0048] L de_re =α1L re_1 +α2L re_2

[0049] Among them, α1 is the weight of the foreground relationship loss function, and α2 is the weight of the background relationship loss function.

[0050] By decoupling multi-scale features into target features and non-target features, the relationships between target features and non-target features are calculated respectively to improve the student network's ability to understand complex scene relationships in aerial images.

[0051] Construct a local pixel-by-pixel relationship distillation module:

[0052] While the network focuses on the global target relationship or the global non-target relationship, it may ignore some important local information. Therefore, a local pixel-by-pixel relationship distillation module is introduced to better balance the learning of global and local information.

[0053] Assume that the multi-scale features of the i-th layer of the pre-trained teacher network are Multi-scale features Divide into N×N block feature maps, and finally get k blocks of size C i ×N×N block feature map, denoted as k is calculated as:

[0054] k=H i / N×W i / N

[0055] Among them, C i H represents the number of channels of the i-th layer multi-scale features. i represents the height of the multi-scale features of the i-th layer, W i Represents the width of the multi-scale features of the i-th layer;

[0056] Extract the block feature map of the jth block of the i-th layer The feature map is input into a graph convolution unit GloRe, wherein the graph convolution unit GloRe includes graph embedding, graph convolution, and reprojection;

[0057] First, the features in the coordinate space are projected into the low-dimensional feature space. For the input block feature map Transform its projection into a block projection feature map through a linear layer Then Projection is performed to obtain the graph node feature Q, The calculation formula is:

[0058]

[0059] Where D is the learnable projection matrix, The graph node feature Q can aggregate information from multiple regions;

[0060] Then graph convolution is used to capture the relationship between graph nodes. The calculation formula is as follows:

[0061] Z=((IV)Q)U

[0062] Where I is the identity matrix, V represents the node adjacency matrix, As training continues to update; U represents the state update matrix, Z represents the graph node relationship feature,

[0063] Finally, the graph node relationship features are reprojected back to the coordinate space to obtain the relationship-aware features. The calculation formula is:

[0064]

[0065] in, Block feature map for the pre-trained teacher network The relationship perception characteristics of D* is the reprojection matrix, which can be used to learn the transposed matrix of the projection matrix D;

[0066] Similarly, we get the block feature map of the student network The relationship perception characteristics are The L2 loss is used to narrow the distance between two relationship-aware features. The calculation formula is:

[0067]

[0068] Among them, N×N represents the height and width of the block feature map; Represents the relation-aware features of the block feature map of the jth block in the i-th layer of the pre-trained teacher network Relationship-aware features with the block feature map of the jth block in the i-th layer of the student network The pixel-by-pixel relationship distillation loss function between .

[0069] Obtain the relational perception features of all block feature maps of each layer of the pre-trained teacher network and the student network, and the final local pixel-by-pixel relation distillation loss function L pprd for:

[0070]

[0071] The local pixel-by-pixel relationship distillation module focuses on learning and capturing local pixel-by-pixel relationships, improving the model's sensitivity to image details and effectively solving the problem of the model's insufficient ability to capture image details.

[0072] Construct the overall loss function of the student network:

[0073] When training the student network, all parameters of the pre-trained teacher network remain unchanged, and the parameters of the student network are updated through the loss function. The loss of the feature decoupling relationship distillation module and the local pixel-by-pixel relationship distillation loss are integrated to obtain the overall loss function L of the student network:

[0074] L=L task +β1L de_re +β2L pprd

[0075] Among them, L task represents the original detection task loss of the student network, including classification loss and regression loss. de_re represents the total loss of the feature decoupling relation distillation module, L pprd represents the local pixel-by-pixel relationship distillation loss, and β1 and β2 are used to control the proportion of different distillation loss terms.

[0076] Go to step S5.

[0077] Step S5: input the test data set into the trained student network for detection, merge the detection results of each block image, output the category and position of all targets in the test data set in the image, and obtain the detection accuracy of the trained student network.

[0078] Example 1

[0079] The method for aerial image target detection based on dual feature relationship distillation of the present invention comprises the following steps:

[0080] Step S1, download the aerial image detection dataset DIOR-R, randomly cut the image into 800×800 blocks, randomly divide the unified-size blocks into training and test datasets in a ratio of 7:3, and perform data enhancement on the training dataset to form a teacher-student network training dataset, and then go to step S2.

[0081] Step S2: Use the teacher-student network training data set to pre-train the teacher network to obtain a pre-trained teacher network, and then proceed to step S3.

[0082] Step S3: construct a teacher-student network based on the pre-trained teacher network, student network, feature decoupling relationship distillation module and local pixel-by-pixel relationship distillation module, and proceed to step S4.

[0083] Step S4, input the teacher-student network training data set into the teacher-student network for training, and extract the multi-scale features of the pre-trained teacher network and the student network respectively; construct a feature decoupling relationship distillation module, use the binary true value target map to separate the foreground information and background information in the multi-scale features, and then distill the foreground relationship and background relationship respectively; construct a local pixel-by-pixel relationship distillation module, divide the multi-scale features into N×N block feature maps, and then input them into the graph convolution module GloRe to obtain relationship-aware features and perform local pixel-by-pixel relationship distillation; fix the parameters of the pre-trained teacher network, update the parameters of the student network through the overall loss function of the student network, and finally obtain the trained student network, and go to step S5.

[0084] Step S5: input the test data set into the trained student network for detection, merge the detection results of each block image, output the category and position of all targets in the test data set in the image, and obtain the detection accuracy of the trained student network.

[0085] The present invention conducts relevant experiments based on Python language and PyTorch framework, and performs model training and reasoning on NVIDIA GeForce RTX3090 graphics card. In the experiment, the model training is optimized using SGD strategy, with momentum decay of 0.9 and weight decay of 0.0001. The initial learning rate is 0.0025, and the learning rate is divided by 10 in the 24th and 33rd rounds. The training rounds of the DIOR-R dataset are 36 rounds, and the batch size of each is 2. For the setting of hyperparameters in the experiment, β1 is set to 0.005 and β2 is set to 0.01.

[0086] In order to demonstrate the effectiveness of the present invention, the present invention selects several distillation algorithms that have been popular in recent years as comparison models. In the experiment, the high-precision remote sensing image target detection algorithm S with the backbone network of ResNet-101 is used. 2A-Net is used as the teacher network, MobilenetV2 is used as the backbone extraction network of the student network, and the other parameters remain unchanged. The comparative experimental results are shown in Table 1:

[0087] Table 1 Comparative experimental results of DIOR-R dataset with other distillation algorithms

[0088]

[0089] The experimental results in Table 1 demonstrate the practicability and effectiveness of the present invention.

Claims

1. A method for aerial image target detection based on dual feature relationship distillation, characterized in that: The following steps are involved: Step S1, download the aerial image detection dataset DIOR-R, randomly cut the image into 800×800 blocks, randomly divide the unified-size blocks into training dataset and test dataset in a ratio of 7:3, and perform data enhancement on the training dataset to form a teacher-student network training dataset, and then go to step S2; Step S2, pre-train the teacher network using the teacher-student network training data set to obtain a pre-trained teacher network, and then proceed to step S3; Step S3, construct a teacher-student network based on the pre-trained teacher network, the student network, the feature decoupling relationship distillation module and the local pixel-by-pixel relationship distillation module, and proceed to step S4; Step S4, inputting the teacher-student network training data set into the teacher-student network for training, and extracting multi-scale features of the pre-trained teacher network and the student network respectively; Construct a feature decoupling relation distillation module, use the binary true value target map to separate the foreground information and background information in the multi-scale features, and then distill the foreground relationship and background relationship respectively; construct a local pixel-by-pixel relationship distillation module, divide the multi-scale features into N×N block feature maps, and then input them into the graph convolution module GloRe to obtain relation-aware features and perform local pixel-by-pixel relationship distillation; fix the parameters of the pre-trained teacher network, update the parameters of the student network through the overall loss function of the student network, and finally obtain the trained student network, and go to step S5; Step S5: input the test data set into the trained student network for detection, merge the detection results of each block image, output the category and position of all targets in the test data set in the image, and obtain the detection accuracy of the trained student network.

2. The method for aerial image target detection based on dual feature relationship distillation according to claim 1, characterized in that: In step S4, the foreground information and background information in the multi-scale features are separated using the binary true value target map, as follows: The multi-scale features of the pre-trained teacher network and the student network are extracted respectively. The multi-scale features of the pre-trained teacher network are expressed as The multi-scale features of the student network are denoted as For the multi-scale features of the lth layer, set a binary true value target map B with the same size as the features of this layer l , if a position of the binary true value target image is within the target true value box, then the position is set to 1, otherwise it is set to 0. The binary true value target image B l The way to obtain is as follows: Among them, scale l (·) represents the scaling function of the ground-truth target at layer l, which is used to map the rotated ground-truth box rbbox to the corresponding feature scale; Represents the binary value at any position (m, n) in the binary true value target image of the multi-scale feature at the lth layer; Binarized true value target map B of the l-th layer multi-scale features l And the corresponding multi-scale feature F l Perform multiplication operation to obtain the foreground features and background features of the lth layer, and the foreground features separated by the multi-scale features of the lth layer The calculation formula is: Background features separated by multi-scale features at layer l The calculation formula is: The foreground information and background information in each layer of features are separated to obtain respective foreground features and background features.

3. The aerial image target detection method based on dual feature relationship distillation according to claim 2 is characterized in that: In step S4, the distillation of foreground relationship and background relationship is as follows: Using the separated and Two foreground features calculate the foreground relationship matrix between adjacent layer features, the formula is: in, represents the foreground relationship matrix between the features of the lth layer and the l+1th layer; H represents the height of the multi-scale features of the lth layer, and W represents the width of the multi-scale features of the lth layer; A(·) represents the adaptive pooling operation, which is used to convert the foreground features of the l+1th layer to the same size as the masked foreground features of the lth layer; F l (m, n) represents the feature value of the l-th layer multi-scale feature at any position (m, n); The foreground relationship matrix between all adjacent layer features of the pre-trained teacher network and the student network is calculated respectively. Finally, the matrix relationship set between the foreground features obtained by the pre-trained teacher network is The matrix relationship set between the foreground features obtained by the student network is In order to narrow the relationship difference between the foreground features of the student network and the pre-trained teacher network, the L2 loss function is used to optimize the relationship between the two. The sum of the relationship distillation losses of all adjacent foreground features is L re_1 The calculation formula is: in, represents the foreground relationship matrix between the features of the lth and l+1th layers of the pre-trained teacher network, Represents the foreground relationship matrix between the features of the lth and l+1th layers of the student network; Using the separated and Two background features calculate the background relationship matrix between adjacent layer features, the formula is: in, Represents the background relationship matrix between the features of the lth layer and the l+1th layer; The sum of the relational distillation losses of all adjacent background features L re_2 The calculation formula is: in, represents the background relationship matrix between the features of the lth and l+1th layers of the pre-trained teacher network, Represents the background relationship matrix between the features of the lth and l+1th layers of the student network; Since the foreground relationship matrix between features of adjacent layers and the background relationship matrix between features of adjacent layers contain different information importance, different loss weights are added to the two losses. Finally, the total loss function L based on the feature decoupling relationship distillation module is de_re for: L de_re =α1L re_1 +α2L re_2 Among them, α1 is the weight of the foreground relationship loss function, and α2 is the weight of the background relationship loss function.

4. The method for aerial image target detection based on dual feature relationship distillation according to claim 3, characterized in that: In step S4, a local pixel-by-pixel relationship distillation module is constructed, as follows: Assume that the multi-scale features of the i-th layer of the pre-trained teacher network are Multi-scale features Divide into N×N block feature maps, and finally get k blocks of size C i ×N×N block feature map, denoted as k is calculated as: k=H i / N×W i / N Among them, C i H represents the number of channels of the i-th layer multi-scale features. i represents the height of the multi-scale features of the i-th layer, W i Represents the width of the multi-scale features of the i-th layer; Extract the block feature map of the jth block of the i-th layer The feature map is input into a graph convolution unit GloRe, wherein the graph convolution unit GloRe includes graph embedding, graph convolution, and reprojection; First, the features in the coordinate space are projected into the low-dimensional feature space. For the input block feature map Transform its projection into a block projection feature map through a linear layer Then Projection is performed to obtain the graph node feature Q, The calculation formula is: Where D is the learnable projection matrix, The graph node feature Q can aggregate information from multiple regions; Then graph convolution is used to capture the relationship between graph nodes. The calculation formula is as follows: Z=((IV)Q)U Where I is the identity matrix, V represents the node adjacency matrix, As training continues to update; U represents the state update matrix, Z represents the graph node relationship feature, Finally, the graph node relationship features are reprojected back to the coordinate space to obtain the relationship-aware features. The calculation formula is: in, Block feature map for the pre-trained teacher network The relationship perception characteristics of D* is the reprojection matrix, which can be used to learn the transposed matrix of the projection matrix D; Similarly, we get the block feature map of the student network The relationship perception characteristics are The L2 loss is used to narrow the distance between two relationship-aware features. The calculation formula is: Among them, N×N represents the height and width of the block feature map; Represents the relation-aware features of the block feature map of the jth block in the i-th layer of the pre-trained teacher network Relationship-aware features with the block feature map of the jth block in the i-th layer of the student network The pixel-by-pixel relationship distillation loss function between them; Obtain the relational perception features of all block feature maps of each layer of the pre-trained teacher network and the student network, and the final local pixel-by-pixel relation distillation loss function L pprd for:

5. The method for aerial image target detection based on dual feature relationship distillation according to claim 4, characterized in that: In step S4, the overall loss function of the student network is constructed as follows: When training the student network, all parameters of the pre-trained teacher network remain unchanged, and the parameters of the student network are updated through the loss function. The loss of the feature decoupling relationship distillation module and the local pixel-by-pixel relationship distillation loss are integrated to obtain the overall loss function L of the student network: L=L task +β1L de_re +β2L pprd Among them, L task represents the original detection task loss of the student network, including classification loss and regression loss. de_re represents the total loss of the feature decoupling relation distillation module, L pprd represents the local pixel-by-pixel relationship distillation loss, and β1 and β2 are used to control the proportion of different distillation loss terms.

Images