Cross-resolution distillation low-resolution image target detection method

Abstract

The invention discloses a target detection method for cross-resolution distillation, and relates to the field of computer vision and artificial intelligence. The method comprises the following steps: (1) introducing a double-branch network to construct a cross-resolution distillation framework, in a training process, performing down-sampling on a high-quality image to form a high-low resolution image pair which is respectively used for training a teacher model and a student model, and introducing additional convolution to change the step length of a teacher network so as to enable features in the network to have the same size, so that the features in the network have the same size; then cross-resolution knowledge distillation is carried out; (2) replacing the path aggregation feature pyramid structure in the frame in the step (1) with a backbone enhancement feature pyramid structure; and (3) introducing a space channel attention module in the process of constructing the feature pyramid by the teacher network of the framework in the step (1). According to the method, the low-resolution image feature extraction capability of the detection network is effectively improved, the low-resolution image target detection precision is improved, and the method has a wide application prospect in the fields of image information analysis and the like.

Description

Technical Field

[0001] This invention relates to a low-resolution image target detection method based on cross-resolution knowledge distillation, belonging to the field of computer vision and intelligent information processing. Background Technology

[0002] Object detection, capable of locating objects in images and classifying them, is one of the fundamental tasks in computer vision. In recent years, with the emergence of numerous high-quality datasets, object detection techniques have achieved considerable success. However, in real-world scenarios, due to differences in imaging equipment, object detection is required in images of varying resolutions. When processing low-resolution images, insufficient detail and blurred boundaries lead to a significant decrease in detection accuracy, failing to meet practical application requirements. Furthermore, most current object detection techniques train and optimize models on existing high-quality datasets, and their feature extraction capabilities are not well-suited for low-resolution images, resulting in a significant drop in detection accuracy for low-resolution images.

[0003] Existing methods often use interpolation to fix the image size to the training image size before feeding it into the detection network to mitigate the adverse effects of different resolutions on detection accuracy. However, this offers very limited help in restoring image details. Other research has addressed the issue of excessively small target sizes by introducing image super-resolution algorithms. However, the introduction of super-resolution modules often introduces additional parameters, and the increased image size after super-resolution multiplies computational overhead, severely impacting inference speed. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a target detection method based on cross-resolution distillation. The aim is to design a cross-resolution knowledge distillation framework to improve the detector's ability to extract detailed information from low-resolution images, thereby improving the detection accuracy of low-resolution images.

[0005] This invention adopts the following technical solution: a low-resolution image target detection method based on cross-resolution distillation, the method comprising the following steps:

[0006] (1) Two YOLOv5 networks are introduced to construct a cross-resolution distillation framework. During the training process, the high-quality input images are downsampled to form high-low resolution image pairs. The high-resolution images are used to train the teacher model, and the low-resolution images are used to train the student model.

[0007] (2) In the framework described in (1), the first convolutional layer of the teacher and student networks is replaced with a convolutional layer with a smaller stride, thereby preserving a larger feature size in the feature extraction stage. A convolutional layer with a stride of k (set k=2) is introduced in front of the teacher network in this framework. The stride difference between the teacher and student networks is used to make up for the difference in input size, so that the features in the network have the same size, and then cross-resolution knowledge distillation is performed.

[0008] (3) The path aggregation pyramid structure in the framework described in (2) is replaced with a backbone enhancement feature pyramid structure;

[0009] (4) In the process of constructing a top-down Feature Pyramid (FPN) in the teacher network of the framework described in (2),

[0010] Introduce a spatial channel attention module.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0012] 1. This invention utilizes the proposed cross-resolution knowledge distillation framework to extract high-resolution image features from the teacher network during model training to supervise the student network with low-resolution input, thereby significantly improving the detection accuracy of low-resolution images;

[0013] 2. This invention introduces a backbone-enhanced feature pyramid structure, which makes the enhanced features pay more attention to detailed information. At the same time, by adjusting the channel ratio of the decoded features, the detector pays more attention to small targets, and the model is also made more lightweight.

[0014] 3. In the process of constructing the feature pyramid network in the teacher network, the present invention adds a spatial channel attention module and uses distillation to make the student network pay attention to important channels and pixels. This method of adding attention mechanism can improve the accuracy of the detector and avoid additional computational overhead. Attached Figure Description

[0015] Figure 1 This is the overall network diagram of the target detection method with cross-resolution knowledge distillation of the present invention;

[0016] Figure 2 This is the backbone network structure of the present invention;

[0017] Figure 3 This is the backbone enhancement feature pyramid network of the present invention;

[0018] Figure 4 This is a schematic diagram of the target detection results of the present invention; Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] Figure 1 The diagram illustrates the network structure and principle of a low-resolution image target detection method that utilizes cross-resolution distillation, including the following parts:

[0021] (1) Transresolution distillation framework

[0022] like Figure 1 As shown, in order to improve the detection performance of low-resolution images by obtaining refined information from high-resolution images, the cross-resolution distillation framework mainly made two improvements, including: 1) the feature size of the student network with low-resolution input, so that it can retain more information and keep in line with the feature size of high-resolution images; and 2) the constraint of the magnified low-resolution features and high-resolution features by regression loss, so as to extract more refined information from the teacher to the student during the training process.

[0023] To enable the low-resolution student network to learn more refined features from the high-resolution student network during training, this invention incorporates a teacher branch to extract features from the high-resolution image. During training, high-quality input I... t Downsampling is performed, and high-resolution and low-resolution image pairs are generated through bilinear interpolation, represented as:

[0024] I s =D(I t (1)

[0025] Where D(I) t ) indicates that bilinear interpolation is used to apply I t Downsample by one time. Use I s Train the student model to effectively extract features from low-resolution images, while utilizing I t The teacher network is trained to extract more fine-grained, high-resolution image features to guide student branches.

[0026] In terms of model structure, YOLOv5 was chosen as the main body of the teacher and student networks, consisting of three parts: backbone, neck, and head. The specific structure of the backbone is shown in the instruction manual. Figure 2 As shown, it is composed of stacked CBS and C3 modules. The backbone ultimately uses SPPF (Spatial Pyramid Pooling-Fast). The specific structure of the neck is as follows. Figure 3As shown, the path aggregation feature pyramid network in the original network is replaced by the backbone enhanced feature pyramid network proposed in this invention. Furthermore, the stride of the student model with low-resolution input is adjusted to half that of the teacher model with high-resolution input. This not only helps alleviate the information loss problem of low-resolution images but also ensures spatial dimensionality matching between their features. The spatial dimensions of the two corresponding features of the teacher and student are the same, facilitating mutual learning between corresponding features during training.

[0027] Specifically, the convolution operation C1 in the YOLOv5s backbone is first replaced with a convolutional module with a stride of 1 and a kernel size of 3. Compared to the original network, this method can preserve twice the feature space size, helping to alleviate the problem of loss of small object information due to low input resolution and small feature size. This network is then used as the student network. The teacher network consists of the same model as the student network and an additional convolution, denoted as C. a The convolution has a stride of 2 and a size of 3. Compared to the input, the output feature size of the convolution is halved. The convolution is placed at the beginning of the teacher model to ensure that the feature size is consistent with that of the student branch.

[0028] Therefore, the output features of C1 for teachers and students can be expressed as follows:

[0029]

[0030] in and They have the same dimension. Furthermore, since the feature variations of the remaining modules in the teacher and student networks are consistent, therefore... (output of layer i+1 in the teacher network) and The outputs of the i-th layer in the student network have the same feature size, where i ≥ 1, which solves the feature space mismatch problem when performing knowledge distillation on images of different sizes. Meanwhile, the feature F of the last layer is selected in both the teacher and student networks. l Calculating the distillation loss is optimal because it is not only training-friendly but also utilizes features from the network's final layer, ensuring that the distillation loss effectively updates the parameters of the entire network. Using these improvements, the student network can extract rich high-resolution knowledge from the teacher network, resulting in a high-performance low-resolution image detection model.

[0031] (2) Backbone Enhancement Feature Pyramid Network

[0032] Figure 3The proposed backbone augmented feature pyramid network (FPN) first constructs a top-down FPN to transmit high-level semantic features, gradually introducing low-level texture features from the backbone during this process. Then, within the FPN, pixel shuffling is used to upsample two smaller features, combining the three features (…). and The feature channels are fixed to a uniform size, and the number of channels is adjusted using 1×1 convolutions to ensure they are all the same. This process can be described as follows:

[0033]

[0034] Where Ps n (·) is pixelshuffle, Conv k,s (·) indicates a convolution kernel size of k×k, a stride of s, and n as the upsampling factor, set to 4 or 2 depending on the difference in feature sizes. Then, a fused feature with rich low-level detail texture information and high-level semantic information can be obtained in the following way:

[0035]

[0036] Among them, Concat(·) c This indicates concatenation along the channel dimension. The generated feature X is passed step-by-step through a bottom-up pyramid structure to obtain features suitable for detecting small, medium, and large targets, respectively denoted by F. s F m F l The details are as follows:

[0037]

[0038]

[0039] C3(·) is the same as the C3 module in the backbone network. The corresponding backbone network features are output from each C3 module in the backbone network. From (6), (7), and (8), it can be seen that in the proposed backbone enhancement feature pyramid structure, the features ultimately used for decoding incorporate lower-level backbone network features. This approach can make better use of lower-level characteristics to increase attention to detailed information. Due to the increased number of small targets caused by reduced resolution, this invention also rebalances the channel ratio of the C3 modules in (6), (7), and (8), making the detector more focused on small targets. This not only helps improve the detection accuracy of low-resolution images but also significantly reduces model parameters.

[0040] (3) Backbone Enhancement Feature Pyramid Network

[0041] In the construction of the FPN in the teacher model, we introduce a spatial channel attention module when incorporating backbone features. Through the distillation framework, the student model can also focus on more important pixels and channels. At the same time, since only the student network is used during inference, the attention module introduced in this invention does not introduce additional model complexity, as detailed below:

[0042] In the construction of the FPN in the teacher model, a spatial channel attention module is introduced to process the two-layer backbone features integrated into the FPN. Assuming the backbone feature dimensions are B×C×W×H, the backbone features are first compressed along the spatial dimensions, and then processed through Conv-ReLU-Conv to obtain a channel attention map with dimensions of B×C×1×1. The specific process is as follows:

[0043] Channel_attention=Conv(ReLU(Conv(pool(x) s (9)

[0044] Where pool(·) s This indicates compression along the spatial dimension. The attention map is normalized using the sigmoid function and then multiplied by the input backbone features to obtain channel attention-weighted features:

[0045] F c h annel =x×sigmoid(Channel_attention) (10)

[0046] Next, spatial attention weighting is applied to this feature. First, the maximum and average values ​​of the channel attention weighted feature are calculated along the channel dimension. The two results are then concatenated along the channel dimension and convolved to obtain the spatial attention map, as shown below:

[0047] F spatial =Conv(Concat(max(F) c h annel ) c ,mean(F c h annel ) c (11)

[0048] After normalizing the attention map using the sigmoid function, multiply it by the channel attention-weighted features to obtain the spatial channel attention-weighted features:

[0049] F c h annel =F c h annel ×sigmoid(F spatial (12)

[0050] This feature is fed into the FPN to construct the feature pyramid network.

[0051] (4) Loss Function

[0052] To enable the student network to learn high-resolution feature representations from the teacher network, we introduce a distillation loss Li. dis Utilizing the features F of the last layer in the teacher-student network l The L1 loss is calculated as follows:

[0053]

[0054] in The final characteristic of teachers, This is the final characteristic of the student.

[0055] The detection loss consists of three parts: L box L represents the positional difference between the predicted bounding box and the actual bounding box. obj L represents the confidence loss of the predicted bounding box. cls The category prediction loss can be represented as:

[0056] L det =ω box L box +ω obj L obj +ω cls L cls (14)

[0057] Where L obj and L cls L is calculated using the cross-entropy function. box The loss is calculated using CIOU. ω box ω obj and ω cls As a balancing coefficient, ω is used during the training of the student and teacher networks. box =0.05, ω obj =1, ω cls =0.5.

[0058] This invention uses online distillation to simultaneously train both teacher and student networks and leverages high-resolution image features from the teacher network to guide student network training. This requires simultaneously calculating the detection loss and distillation loss for both teacher and student branches. In summary, the total loss is expressed as:

[0059] L total =α t L t +α s L s +βL dis (15)

[0060] Where α t and α s We test the loss coefficients for the teacher and student branches. $\beta$ represents the distillation loss coefficient. We set α... t =α s =1 and β=0.1.

[0061] In the detection task, since remote sensing data typically requires higher image resolution compared to conventional scenes, this invention conducted relevant experiments on the DOTAv1.0 remote sensing dataset. Images were cropped to 1024*1024 pixels with a 200-pixel overlap. The cropped images were used for training, with YOLOv5s as the baseline. Experiments were conducted on cross-resolution distillation frameworks, backbone augmentation feature pyramid networks, and attention distillation. A GeForce RTX 3090 graphics card was used. During training, stochastic gradient descent (SGD) was used as the optimizer for simultaneous training of the teacher and student models. The initial learning rate was set to 0.01, momentum to 0.937, weight decay to 0.0005, batch size to 8, and epochs to 100. The evaluation image size was set to a lower resolution of 512*512 pixels. The evaluation process was directly performed using the EvaluationTask2 tool from the DOTA dataset website. The results compared with the baseline network on different high-resolution images are shown in Table 1. Furthermore, seven deep learning-based image classification methods were selected as comparison methods:

[0062] Method 1: The method proposed by Ren et al., see reference "Faster r-cnn: Towards real-time object detection with region proposal networks. 2015."

[0063] Method 2: The method proposed by Lin et al., see reference "Focal loss for dense object detection. 2017".

[0064] Method 3: The method proposed by Li et al., see reference "Generalized Focal Loss: Learningqualified and distributed bounding boxes for dense object detection. 2020".

[0065] Method 4: The method proposed by Tian et al., see reference "FCOS: Fully convolutional one-stage object detection. 2019".

[0066] Method 5: The method proposed by Zhang et al., see reference "FCOS: Fully convolutional one-stage object detection. 2019".

[0067] Method 6: The method proposed by Yang et al., see reference "Adaptive knowledge distillation for lightweight remote sensing object detectors optimizing. 2020".

[0068] Method 7: The method proposed by Zhang et al., see reference "Superyolo: Super resolution assisted object detection in multimodal remote sensing imagery. 2023".

[0069] The comparison results are shown in Table 2. This invention significantly improves the detection performance of low-resolution images. Even with low-resolution input, it offers a significant improvement in accuracy compared to other methods. Figure 4 The detection results of the method of the present invention on low-resolution images are shown.

[0070] Table 1 Comparison of detection performance of the baseline network at different resolutions

[0071]

[0072] Table 1 Comparison of SOTA results

[0073]

[0074]

Claims

1. A low-resolution image target detection method based on cross-resolution knowledge distillation, characterized in that, The method comprises the following steps: (1) a teacher-student double-branch cross-resolution distillation framework is constructed by using YOLOv5s, the step length of the student network is reduced, the step length multiplied by the input down-sampling multiple is equal to the step length of the teacher network, high-low resolution image pairs are trained by using the down-sampling, the regression loss is used to constrain the last layer features in the teacher and student networks, and the knowledge of the high-resolution model is distilled into the low-resolution model; (2) in the two branches, the BAFPN proposed in the patent is used to replace the PAFPN; (3) in the teacher network, the CBAM module is introduced into the FPN stage of the backbone feature fusion.

2. The low-resolution image target detection method based on cross-resolution knowledge distillation according to claim 1, characterized in that, The cross-resolution distillation framework established in step (1) has the following specific steps: First, as shown in Fig. 1 of the accompanying drawings, the high-quality input I t is down-sampled by one, generating a high-resolution and a low-resolution image pair by bilinear interpolation, denoted as: I s = D(I t ) (1) Wherein, I s For training student model, make the model can effectively extract low resolution image feature, I t For training teacher network to extract more fine-grained high-resolution image features for guiding student branch; In terms of model structure, firstly, the first convolution module in the backbone of YOLOv5 is replaced by a convolution module with a step size of 1 and a kernel size of 3*3. Compared with the original network, this method can retain twice the feature space size, which helps to alleviate the problem of small target information loss caused by low input resolution and small feature size. The network is used as a student network; the teacher network consists of the same model as the student and an additional convolution, which is represented as C a with a step size of 2 and a kernel size of 3*3. Compared with the input, the output feature size of this convolution is halved. This convolution is located at the starting position of the teacher model to ensure that the feature size is consistent with the student branch. Therefore, the output features of C1 in the teacher and student can be represented as: wherein and have the same dimension; in addition, since the characteristics of the remaining modules in the teacher and student networks change consistently, the characteristics of the output of the i-th layer in the teacher network and the output of the i-th layer in the student network have the same feature size, where i≥1, which solves the feature space mismatch problem when distilling knowledge for images of different sizes; the features F l The distillation loss is calculated as follows: In the inference process, only the student model is used for detecting the target detection of the low-resolution image.

3. The low-resolution image target detection method based on cross-resolution knowledge distillation according to claim 1, characterized in that, The BAFPN proposed in the patent is applied in step (2), and the specific steps are as follows: As shown in Figure 3 of the instruction manual, the FPN is first constructed from top to bottom. Then, in the FPN, two smaller features are upsampled using pixel shuffling, resulting in a combination of three features (…). and The feature channels are fixed to a uniform size, and the number of channels is adjusted by 1×1 convolution to make them the same. This process can be described as follows: where Ps n (·) is pixelshuffe, Conv k,s (·) represents a convolution kernel size of k x k, a step size of s, and n is an up-sampling multiple, which is set to 4 or 2 according to the difference multiple of the feature size; then, a fusion feature with rich low-level detail texture information and high-level semantic information is obtained by the following way, Concat(·) c Concat(·) represents concatenation along the channel dimension, and the generated feature X is transmitted through a bottom-up pyramid structure to obtain features suitable for detecting small, medium and large targets, respectively represented by F s , F m , F l , and the specific process is as follows: wherein C3(·) is the same as the C3 module of the YOLOv5 baseline network, The output backbone network features corresponding to each C3 module in the backbone network; and the application also rebalances the channel number ratio of the C3 modules in (6), (7) and (8), so that the detector is more focused on small size targets, which not only helps to improve the detection accuracy of low resolution images, but also greatly reduces the model parameters.

4. The low-resolution image target detection method based on cross-resolution knowledge distillation according to claim 1, characterized in that, In step (3), the spatial channel attention module is introduced into the FPN stage of the backbone feature fusion in the teacher network, and the specific steps are as follows: In the construction process of the FPN in the teacher model, the spatial channel attention module is introduced to process the two layers of backbone features fused into the FPN, assuming that the dimension of the backbone feature is BxCxWxH, first, the backbone feature is compressed along the spatial dimension, and then the dimension of the channel attention map BxCx1x1 is obtained through the Conv-ReLU-Conv mode, and the specific process is as follows: Channel_attention = Conv(ReLU(Conv(pool(x) s )) (10) where pool(·) s represents the compression along the spatial dimension, and the attention map is normalized by the sigmoid function and multiplied by the input backbone feature to obtain the channel attention weighted feature: F c h annel = x x sigmoid(CΣannel_attention) (11) then, the spatial attention weighting is performed on the feature, the maximum value and the average value are calculated along the channel dimension, respectively, the two results are spliced along the channel dimension, and the spatial attention map is obtained through convolution, and the specific process is as follows: F spatial = Conv(Concat(max(F c h annel ) c , mean(F c h annel ) c )) (12) after the attention map is normalized through the sigmoid function, the spatial channel attention weighted feature is obtained by multiplying the channel attention weighted feature: F c h annel =F c h annel ×sigmoid(F spatial ) (13) the feature is input into the FPN to construct the feature pyramid network; it should be noted that this claim is the way of introducing the attention module, not the attention module itself, through the distillation framework and the attention module of the teacher model, the student model can also pay attention to more important pixels and channels, and at the same time, since only the student network is used in the inference process, no additional model complexity is brought.

Images