A knowledge distillation method for remote sensing target detection based on feature separation attention

Abstract

This invention discloses a knowledge distillation method for remote sensing target detection based on feature-separated attention, comprising: extracting feature attention maps from the feature maps output by the teacher network and the student network respectively; separating the foreground and background regions of the feature maps, and calculating a foreground attention mask using the feature attention map of the teacher network and a background attention mask using the feature attention map of the student network; calculating the L2 distillation loss using the foreground and background attention masks; and transferring the knowledge from the teacher network to the student network based on the L2 distillation loss. This invention can effectively select the region to be distilled, improve distillation efficiency, and improve the detection accuracy of the final lightweight target detection network without changing the student network structure or increasing computational cost.

Description

Technical Field

[0001] This invention relates to the field of knowledge distillation technology, and more specifically to a knowledge distillation method for remote sensing target detection based on feature separation attention. Background Technology

[0002] The emergence of large-scale, high-resolution remote sensing image datasets has enabled the widespread application of deep learning in remote sensing image target detection. However, high-precision algorithms have high time complexity and rely on high-performance graphics processors. In practical engineering applications of remote sensing target detection, embedded systems are a more common platform. Currently, there are some lightweight deep learning target detection algorithms, which, although fast, still cannot meet the detection accuracy requirements of the task.

[0003] Currently, some knowledge distillation methods are used to improve the performance of deep neural networks, such as methods that transfer learning from the network's output, feature maps, and information flow. However, most of these studies focus on image classification. In object detection, the definition of the information to be transferred in knowledge distillation is still unclear. Furthermore, since the proportion of background regions in object detection data is much higher than in classification data, directly performing knowledge distillation using image classification methods can lead to severe interference from background regions, failing to achieve the desired results.

[0004] Therefore, how to provide a knowledge distillation method that can effectively extract the region to be distilled without increasing computational consumption and improving the detection accuracy of lightweight target detection networks is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a knowledge distillation method for remote sensing target detection based on feature separation attention, which can effectively select the region to be distilled, improve distillation efficiency, and improve the detection accuracy of the final lightweight target detection network without changing the student network structure or increasing computational consumption.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A knowledge distillation method for remote sensing target detection based on feature separation attention includes:

[0008] Extract feature attention maps from the feature maps output by the teacher network and the student network respectively;

[0009] The foreground and background regions of the feature map are separated, and the foreground attention mask is calculated using the feature attention map of the teacher network, while the background attention mask is calculated using the feature attention map of the student network.

[0010] Calculate L2 distillation loss using foreground attention mask and background attention mask;

[0011] Based on L2 distillation loss, knowledge from the teacher network is transferred to the student network.

[0012] Preferably, in the above-mentioned knowledge distillation method for remote sensing target detection based on feature separation attention, the feature attention map includes spatial attention and channel attention; wherein, the calculation formula for spatial attention is:

[0013]

[0014] The formula for calculating channel attention is:

[0015]

[0016] Among them, G s G represents spatial attention, used to characterize the importance of each pixel location in each channel. c Channel attention represents the importance of each channel in the feature map; H, W, and C represent the height, width, and dimension of the feature map, respectively; A k,i,j This represents the pixel value of feature map A at coordinates i,j in its k-th channel.

[0017] Preferably, in the above-mentioned knowledge distillation method for remote sensing target detection based on feature separation attention, the step of calculating the foreground attention mask through the feature attention map of the teacher network includes:

[0018] Calculate the feature map A output by the teacher network. t Spatial attention and channel attention;

[0019] Using the Softmax function to process feature map A respectively t Spatial attention and channel attention are probability-normalized;

[0020] The feature map A after probability normalization t Spatial attention and channel attention with labeled foreground mask S f Multiplying them together yields a weighted foreground attention mask J. f .

[0021] Preferably, in the above-mentioned knowledge distillation method for remote sensing target detection based on feature separation attention, the calculation formula for the foreground attention mask is:

[0022]

[0023]

[0024]

[0025] Among them, the probability normalization function The probability of variable z belonging to category i is normalized to the range (0,1), and the sum of all probabilities is 1; H, W, and C represent feature maps A, W, and C, respectively. t Length, width, and dimensions and S represents the spatial attention and channel attention of the teacher network, respectively. f As a foreground mask, set the value to 1 in the foreground region and 0 in the background region.

[0026] Preferably, in the above-mentioned knowledge distillation method for remote sensing target detection based on feature separation attention, it further includes: dividing the foreground mask S according to the size of the target. f Normalization is performed to obtain the normalized foreground mask. Normalized foreground mask Replace the foreground mask S f Computational foreground attention mask; Normalized foreground mask The calculation formula is:

[0027]

[0028] Where T represents the sum of all objectives, t represents each objective, and s t This indicates the area of ​​the target region.

[0029] Preferably, in the above-mentioned knowledge distillation method for remote sensing target detection based on feature separation attention, the step of calculating the background attention mask through the feature attention map of the student network includes:

[0030] Calculate the feature map A of the student network output. s Spatial attention and channel attention;

[0031] The feature map A is processed using softmax. s Spatial attention and channel attention are probability-normalized;

[0032] The feature map A after probability normalization s Spatial attention and channel attention with labeled background mask S b Multiply to obtain the weighted background attention mask J b .

[0033] Preferably, in the aforementioned knowledge distillation method for remote sensing target detection based on feature separation attention, the background attention mask J b The calculation formula is:

[0034]

[0035]

[0036]

[0037] Where H, W, and C represent feature maps A, respectively. s Length, width, and dimensions and S represents the spatial attention and channel attention of the student network, respectively. b Use a background mask, setting the value to 0 for the foreground and 1 for the background.

[0038] Preferably, in the above-mentioned knowledge distillation method for remote sensing target detection based on feature separation attention, it further includes: dividing the background mask S according to the size of the target. b Normalization is performed to obtain a normalized background mask. Normalized background mask Replace background mask S b Computational background attention mask; normalized background mask The calculation formula is:

[0039]

[0040] Where T represents the sum of all objectives, t represents each objective, and s t This represents the area of ​​the target region.

[0041] Preferably, in the above-mentioned knowledge distillation method for remote sensing target detection based on feature separation attention, the formula for calculating the L2 distillation loss is:

[0042] L d =δL2(A s ·J f A t ·J f )+εL2(A s ·J b A t ·J b )

[0043] Where δ and ε are parameters controlling the ratio of foreground attention mask to background attention mask loss calculation, A s and A t J represents the feature graphs of the student network and the teacher network, respectively. f and J b Let X and Y represent the foreground attention mask and background attention mask, respectively; the L2 loss function is a function that calculates the spatial Euclidean distance between the two vectors X and Y, and is calculated as follows: Where x i y i These represent each term of vectors X and Y, for a total of n terms.

[0044] As can be seen from the above technical solution, compared with the prior art, this invention discloses a knowledge distillation method for remote sensing target detection based on feature separation attention. First, the foreground and background regions of the feature map are separated, and attention maps are extracted separately. Feature separation uses a masking approach, extracting the coordinates of the target region from the annotation information and mapping them onto the feature map to be distilled according to the resolution. For the foreground region, a foreground mask is used, i.e., only the target region is considered; for the background region, a background mask is used, i.e., the parts other than the target region are considered. Spatial attention and channel attention are extracted for the foreground and background regions respectively. Since the teacher network has stronger performance, it often obtains a stronger response in the foreground region containing the target compared to the student network's output. Distilling the foreground response of the teacher network can improve the student network's ability to judge foreground targets. Therefore, the foreground attention mask is calculated using the feature map of the teacher network. However, the background may contain some erroneous responses, and the student network's feature extraction ability is relatively weak. Compared to the teacher network, erroneous responses in the background region of the student network's output feature map are more prominent. By distilling the background region of the student network during training, misclassification of the background region can be reduced. Therefore, a background attention mask is calculated using the student network's feature map. L2 loss is calculated separately using the foreground and background attention masks, allowing knowledge from the teacher network to be transferred to the student network. Overall, this invention fuses the information from the feature maps into a single attention map through spatial attention and channel attention, avoiding mutual interference between channels. Furthermore, by distilling false detections of targets in the foreground region and false detections in the background region using foreground and background masks respectively, the detection performance of the lightweight model is significantly improved. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 A flowchart of the remote sensing target detection knowledge distillation method based on feature separation attention provided by the present invention;

[0047] Figure 2 A schematic diagram of the knowledge distillation model provided by this invention;

[0048] Figure 3 A schematic diagram illustrating the process of obtaining the foreground attention mask provided by the present invention;

[0049] Figure 4A schematic diagram illustrating the process of obtaining the background attention mask provided by the present invention;

[0050] Figure 5 This is a schematic diagram of distillation based on feature separation attention provided by the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] like Figure 1 As shown in the figure, this invention discloses a knowledge distillation method for remote sensing target detection based on feature separation attention, including the following steps:

[0053] S1. Extract the feature attention maps from the feature maps output by the teacher network and the student network respectively;

[0054] S2. Separate the foreground and background regions of the feature map, and calculate the foreground attention mask using the feature attention map of the teacher network and the background attention mask using the feature attention map of the student network.

[0055] S3. Calculate the L2 distillation loss using foreground attention mask and background attention mask; based on the L2 distillation loss, transfer the knowledge of the teacher network to the student network.

[0056] The overall structure of the remote sensing target detection knowledge distillation method based on feature separation attention proposed in this invention is as follows: Figure 2 As shown, the model consists of a teacher network and a student network. The teacher network is a high-performance, complex neural network, while the student network is a lightweight, simple neural network. First, the teacher network is pre-trained to achieve convergence and high detection performance. Then, during the training of the student network, the output of the teacher network is used as additional supervisory information to train the student network, transferring knowledge from its feature maps to improve the training effect of the student network.

[0057] This invention addresses knowledge distillation of feature maps in convolutional neural networks. Due to the high dimensionality of feature maps, directly applying L2 loss can easily lead to inter-channel interference, negatively impacting distillation performance. Furthermore, background regions, often overlooked by most object detection knowledge distillation methods, may contain valuable information; effectively learning from these regions can enhance the model's ability to distinguish between positive and negative samples. Distilling the foreground reduces the false positive rate of the student network, thus improving its ability to identify positive samples; conversely, distilling the background reduces the false positive rate, improving its ability to identify negative samples. Therefore, this invention utilizes this foreground and background separation distillation approach, combined with feature map attention, to propose a knowledge distillation method based on feature map separation attention.

[0058] The steps described above are described in further detail below.

[0059] S1. Extract the feature attention maps from the feature maps output by the teacher network and the student network respectively.

[0060] Feature attention maps are divided into spatial attention and channel attention. Spatial attention refers to reducing the dimensionality of features along the channel dimension, representing each pixel with only one value. This attention reflects the importance of a pixel in the feature map; a stronger response indicates a higher probability of the presence of a target. Channel attention refers to reducing the dimensionality of features along the length and width dimensions, with each value reflecting the response of a channel. Since the information contained in each channel is uneven during feature extraction, this attention reflects the importance of each channel in the feature map, allowing the network to focus more on channels rich in relevant information.

[0061] Since there is no labeled information for training the attention module in knowledge distillation, this invention adopts a simple manual design method, namely, taking the absolute value average as the attention map. The feature map A of the teacher network is then used. t Feature graph A of the student network s The spatial attention map and channel attention map are calculated respectively using the following formulas.

[0062] The formula for calculating spatial attention is:

[0063]

[0064] The formula for calculating channel attention is:

[0065]

[0066] Among them, G s G represents spatial attention, used to characterize the importance of each pixel location in each channel. cChannel attention represents the importance of each channel in the feature map; H, W, and C represent the height, width, and dimension of the feature map, respectively; A k,i,j This represents the pixel value of feature map A at coordinates i,j in its k-th channel.

[0067] S2. Separate the foreground and background regions of the feature map, and calculate the foreground attention mask using the feature attention map of the teacher network and the background attention mask using the feature attention map of the student network.

[0068] like Figure 3-4 As shown, knowledge distillation based on feature-separated attention requires first obtaining spatial and channel attention maps through absolute value averaging, then performing feature separation on the feature attention maps using foreground and background masks, and calculating the distillation loss for the foreground and background attention masks respectively. For the foreground attention mask, the spatial and channel attention of the teacher model are multiplied and the foreground portion is extracted using the foreground mask; for the background attention mask, the spatial and channel attention of the student model are multiplied and the background portion is extracted using the background mask. Subsequently, the L2 distillation loss function is calculated on the foreground and background attention masks respectively.

[0069] Because the teacher network has stronger performance, it often obtains a stronger response in foreground regions containing targets compared to the student network's output. Distilling the foreground response of the teacher network can improve the student network's ability to identify foreground targets. Therefore, a foreground attention mask is calculated using the feature map of the teacher network. Specifically,

[0070] 1. The process of obtaining the foreground attention mask is as follows:

[0071] 1) Calculate the feature map A of the teacher network output. t Spatial attention and channel attention;

[0072] 2) Use the Softmax function to process feature map A respectively. t Spatial attention and channel attention are probability-normalized;

[0073] 3) The feature map A after probability normalization t Spatial attention and channel attention with labeled foreground mask S f Multiplying them together yields a weighted foreground attention mask J. f .

[0074] The specific calculation formula is as follows:

[0075]

[0076]

[0077]

[0078] Among them, the probability normalization function The probability of variable z belonging to category i is normalized to the range (0,1), and the sum of all probabilities is 1; H, W, and C represent feature maps A, W, and C, respectively. t Length, width, and dimensions and S represents the spatial attention and channel attention of the teacher network, respectively. f As a foreground mask, set the value to 1 in the foreground region and 0 in the background region.

[0079] To balance the impact of loss functions on targets of different sizes, the foreground mask needs to be normalized according to the target size. This normalized foreground mask... Replace S f Normalized foreground mask The calculation formula is:

[0080]

[0081] T represents the sum of all objectives, t represents each individual objective, and s t This represents the area of ​​the target region, ensuring that targets of different sizes have the same impact on the loss function.

[0082] 2. The background may contain some erroneous responses, and the student network's feature extraction ability is relatively weak. Compared to the teacher network, erroneous responses in the background region of the student network's output feature map are more prominent. Distilling the background region during training of the student network can reduce misclassification of the background region. Similar to the calculation method of the foreground attention mask, the process of obtaining the background attention mask is as follows:

[0083] 1) Calculate the feature map A of the student network output. s Spatial attention and channel attention;

[0084] 2) Apply softmax to feature map A respectively. s Spatial attention and channel attention are probability-normalized;

[0085] 3) The feature map A after probability normalization s Spatial attention and channel attention with labeled background mask S b Multiply to obtain the weighted background attention mask J b .

[0086] The specific calculation formula is as follows:

[0087]

[0088]

[0089]

[0090] Where H, W, and C represent feature maps A, respectively. s Length, width, and dimensions and S represents the spatial attention and channel attention of the student network, respectively. b Use a background mask, setting the value to 0 for the foreground and 1 for the background.

[0091] Similarly, to balance the influence of the background mask, the mask is also weighted by the area of ​​the background region. A weighted background mask is then used. Replace S b Normalized background mask The calculation formula is:

[0092]

[0093] Where T represents the sum of all objectives, t represents each objective, and s t This represents the area of ​​the target region. Using a weighted background mask ensures that the contribution of the background region to the loss function is consistent with that of the foreground region.

[0094] Design of S3 and L2 distillation loss functions.

[0095] like Figure 5 As shown, after obtaining the foreground attention mask and the background attention mask, they are multiplied by the output feature maps of the teacher and the student, respectively, and the distillation loss is calculated. The loss function used is the L2 function:

[0096] L d =δL2(A s ·J f A t ·J f )+εL2(A s ·J b A t ·J b ),

[0097] Where δ and ε are parameters that control the ratio of foreground to background loss calculation, A s and A t J represents the feature maps of the student and teacher models, respectively. f and J b These represent the foreground attention mask and the background attention mask, respectively.

[0098] The L2 loss function is a function that calculates the spatial Euclidean distance between two vectors X and Y, and is calculated as follows:

[0099]

[0100] Where, x i y i These represent each term of vectors X and Y, for a total of n terms.

[0101] This distillation loss function allows knowledge from complex models to be transferred to lightweight models, improving their detection performance. In practical applications, a complex model should first be trained for the target task. Then, the pre-trained complex model is used as additional supervision information, and an extra knowledge distillation loss function based on feature separation attention is added during the training of the lightweight model. After training convergence, the resulting lightweight remote sensing target detection model will show improved detection performance compared to a model without distillation loss.

[0102] This invention, as an extension of lightweight object detection tasks, fuses feature map information into a single attention map using spatial and channel attention methods, avoiding interference between channels. Furthermore, it distills false detections in the foreground region and background region separately using foreground and background masks, significantly improving the detection performance of the lightweight model.

[0103] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0104] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A knowledge distillation method for remote sensing target detection based on feature separation attention, characterized in that, include: Extract feature attention maps from the feature maps output by the teacher network and the student network respectively; The foreground and background regions of the feature map are separated, and the foreground attention mask is calculated using the feature attention map of the teacher network, while the background attention mask is calculated using the feature attention map of the student network. Calculate L2 distillation loss using foreground attention mask and background attention mask; Based on L2 distillation loss, knowledge from the teacher network is transferred to the student network; The foreground mask S of the feature map output by the teacher network is determined based on the size of the target. f Normalization is performed to obtain the normalized foreground mask. Normalized foreground mask Replace the foreground mask S f Computational foreground attention mask; Normalized foreground mask The calculation formula is: Where T represents the sum of all objectives, t represents each objective, and s t Indicates the area of ​​the target region; The background mask S of the feature map output by the student network is determined based on the size of the target. b Normalization is performed to obtain a normalized background mask. Normalized background mask Replace background mask S b Computational background attention mask; normalized background mask The calculation formula is: Where T represents the sum of all objectives, t represents each objective, and s t This represents the area of ​​the target region.

2. The remote sensing target detection knowledge distillation method based on feature separation attention according to claim 1, characterized in that, The feature attention map includes spatial attention and channel attention; wherein, the formula for calculating spatial attention is: The formula for calculating channel attention is: Among them, G s G represents spatial attention, used to characterize the importance of each pixel location in each channel. c Channel attention represents the importance of each channel in the feature map; H, W, and C represent the height, width, and dimension of the feature map, respectively; A k,i,j This represents the pixel value of feature map A at coordinates i,j in its k-th channel.

3. The remote sensing target detection knowledge distillation method based on feature separation attention according to claim 1, characterized in that, The calculation of the foreground attention mask using the feature attention map of the teacher network includes: Calculate the feature map A output by the teacher network. t Spatial attention and channel attention; Using the Softmax function, respectively, the feature map A t Spatial attention and channel attention are probability-normalized; The feature map A after probability normalization t Spatial attention and channel attention are multiplied by the labeled foreground mask Sf to obtain a weighted foreground attention mask J. f .

4. The remote sensing target detection knowledge distillation method based on feature separation attention according to claim 3, characterized in that, The formula for calculating the foreground attention mask is: Among them, the probability normalization function The probability of variable z belonging to category i is normalized to the range (0,1), and the sum of all probabilities is 1; H, W, and C represent feature maps A, W, and C, respectively. t Length, width, and dimensions and S represents the spatial attention and channel attention of the teacher network, respectively. f As a foreground mask, set the value to 1 in the foreground region and 0 in the background region.

5. The remote sensing target detection knowledge distillation method based on feature separation attention according to claim 1, characterized in that, The calculation of the background attention mask using the feature attention map of the student network includes: Calculate the feature map A of the student network output. s Spatial attention and channel attention; The feature map A is processed using softmax. s Spatial attention and channel attention are probability-normalized; The feature map A after probability normalization s Spatial attention and channel attention with labeled background mask S b Multiply to obtain the weighted background attention mask J b .

6. The remote sensing target detection knowledge distillation method based on feature separation attention according to claim 5, characterized in that, Background attention mask J b The calculation formula is: Where H, W, and C represent feature maps A, respectively. s Length, width, and dimensions and S represents the spatial attention and channel attention of the student network, respectively. b Use a background mask, setting the value to 0 for the foreground and 1 for the background.

7. The remote sensing target detection knowledge distillation method based on feature separation attention according to claim 1, characterized in that, The formula for calculating L2 distillation loss is: L d <δL2(A s ·J f ,HAS t ·J f )+εL2(A s ·J b ,HAS t ·J b ) Where δ and ε are parameters controlling the ratio of foreground attention mask to background attention mask loss calculation, A s and A t J represents the feature graphs of the student network and the teacher network, respectively. f and J b Let X and Y represent the foreground attention mask and background attention mask, respectively; the L2 loss function is a function that calculates the spatial Euclidean distance between the two vectors X and Y, and is calculated as follows: Where x i y i These represent each term of vectors X and Y, for a total of n terms.

Images