A Compressed Image Object Detection Method Based on Degraded Network Feature Learning

Abstract

A method for compressed image object detection based on degraded network feature learning, which relates to the field of computer vision technology. In view of the problem that when the object detection method in the prior art uses thumbnail for detection to reduce the computational amount, the accuracy rate will decrease, even to zero. This application first learns a thumbnail through a supervised method, and then sends the generated thumbnail into the object detector with the network structure unchanged. By this method, the computational amount and memory occupation can be greatly reduced. The downsampling module of this application can make full use of the powerful feature extraction ability of the convolutional neural network to generate a thumbnail from the original large-size image. This thumbnail is generated under the supervision of the image downsampling loss, the knowledge distillation loss, and the object detection loss. Therefore, it has the key information of the original image and can replace the original image to perform the object detection task. And the thumbnail obtained through learning has much better object detection effect than the traditional interpolation method.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and specifically to a method for detecting compressed image targets based on degraded network feature learning. Background Art

[0002] In the past decade or so, convolutional neural networks have achieved a series of remarkable results in many computer vision tasks (such as object detection, semantic segmentation, and human pose estimation, etc.). Thanks to the progress of hardware conditions, people have begun to improve the accuracy of algorithms by designing deeper network structures and using large-size input images. However, the increasingly large network structures also bring high computational costs and memory requirements. At the same time, due to the resource limitations of mobile devices such as smartphones and tablets, it is difficult to deploy such complex algorithms on mobile devices.

[0003] In order to enable the algorithm to run in real time on mobile devices, researchers have proposed many methods for model acceleration and compression. Most methods focus on reducing the number of network layers or designing more efficient structures to reduce model parameters, but they ignore that large-size input images are also an important factor affecting computational costs and memory requirements. In classification tasks, some methods have proven that using thumbnail images (4-fold downsampling) for image classification achieves classification accuracy comparable to that of the original images. However, through experiments, we found that reducing the image resolution performs poorly in object detection tasks. For example, in the recently effective object detection algorithm - Deformable-DETR, after using 2-fold downsampling, the accuracy drops from 43.8% to 35.3%; when the ratio of image resolution reduction is greater than or equal to 4, the algorithm simply cannot detect the target, and the accuracy directly becomes 0. The reason can be found from the image downsampling formula:

[0004] I lr =(I org *k)↓ s +n

[0005] The purpose of image downsampling is to obtain a low-resolution image I org from the original image I lr through a certain downsampling rate s, where k and n represent the downsampling kernel and noise respectively. Now most downsampling methods use a fixed downsampling kernel k, which makes image downsampling an idealized process, resulting in a large difference in the distribution between the low-resolution image and the original image. At the same time, the low-resolution images obtained using this traditional downsampling method often have artifacts, which is not conducive to feature extraction by computers. Therefore, the accuracy of object detection results using low-resolution images obtained by such methods is low. Summary of the Invention

[0006] The object of the present invention is: in view of the problem that when the target detection method in the prior art uses thumbnails for detection to reduce the calculation amount, the accuracy rate will decrease, or even be zero, a compressed image target detection method based on degraded network feature learning is proposed.

[0007] The technical solution adopted by the present invention to solve the above technical problems is:

[0008] A compressed image target detection method based on degraded network feature learning, comprising the following steps:

[0009] The step of obtaining the image to be detected and obtaining the compressed image of the image to be detected, and

[0010] The step of inputting the compressed image of the image to be detected into a trained target detection network for target detection;

[0011] The training process of the target detection network is:

[0012] Step 1: Obtain the original image and obtain the compressed image of the original image;

[0013] Step 2: Respectively extract features from the original image and the compressed image to obtain the original image feature map and the compressed image feature map, and use the detection network to detect the original image to obtain the class prediction value in the original image;

[0014] Step 3: Use Feature map distillation to process the original image feature map and the compressed image feature map to obtain a compressed image feature map with a distribution similar to that of the original image feature map;

[0015] Step 4: Replace the original compressed image feature map with the compressed image feature map with a distribution similar to that of the original image feature map, and use the detection network to obtain the position and class prediction value of the object in the compressed image;

[0016] Step 5: Use Logit distillation to process the class prediction value in the original image and the class prediction value obtained in Step 4 to obtain a class prediction value close to the class prediction value in the original image;

[0017] Step 6: Use the compressed image of the original image as the input, the position of the object in the compressed image obtained in Step 4, and the class prediction value close to the class prediction value in the original image as the output to train the target detection network.

[0018] Further, the compressed image is obtained through a downsampling module, and the downsampling module includes:

[0019] The first layer for mapping a large-size image to 12 channels to learn hidden information, and the first layer is a convolutional layer with a stride of 2 and a convolutional kernel size of 3×3;

[0020] The second layer for mapping the features back to 3 channels to reconstruct the RGB thumbnail, and the second layer is a convolutional layer with a stride of 2 and a convolutional kernel size of 3×3.

[0021] Furthermore, the loss function of the downsampling module is expressed as:

[0022]

[0023] where x and y represent the original large-size image and the generated thumbnail respectively, λ is a hyperparameter used to balance these two parts, μ(·) and σ(·) calculate the first and second moments of each channel in the image respectively, and i represents the serial number of the image channel.

[0024] Furthermore, the compressed image is expressed as:

[0025]

[0026] where represents the downsampling module, x and y represent the original large-size image and the generated thumbnail respectively, and δ represents the parameter of the downsampling module.

[0027] Furthermore, the loss function of the Logit distillation is expressed as:

[0028]

[0029] where represents the softmax function, represents the hyperparameter temperature, p t and p s represent the class prediction values of the teacher and the student respectively.

[0030] Furthermore, the feature extraction is performed through the ResNet50 network.

[0031] Furthermore, the loss function of the Feature map distillation is expressed as:

[0032]

[0033] where F T and F S represent the output feature maps of ResNet50 in the teacher and student networks respectively, L represents the number of convolutional layers of ResNet50, C, H, and W represent the number of channels, height, and width of the feature map respectively, and f represents the feature map alignment operation.

[0034] Furthermore, the overall loss function of the Logit distillation and the Feature map distillation is expressed as:

[0035]

[0036] Among them, α and β are hyperparameters used to balance the losses of the two parts.

[0037] Furthermore, the α and β are 0.5 and 0.1 respectively.

[0038] Furthermore, the loss function of the object detection method is expressed as:

[0039]

[0040] Among them, γ is the weight of the Moment-matching loss.

[0041] The beneficial effects of the present invention are as follows:

[0042] The present application proposes an object detection framework from a new perspective. Different from the traditional method of compressing the model by reducing the model parameters, the present application first learns a thumbnail through a supervised method, and then sends the generated thumbnail into the object detector with the unchanged network structure. By this method, the computational amount and memory occupation can be greatly reduced. In the "compressed image object detection technology based on degraded network feature learning" of the present application, the downsampling module can make full use of the powerful feature extraction ability of the convolutional neural network to generate a thumbnail from the original large-size image. The thumbnail is generated under the supervision of the image downsampling loss, the knowledge distillation loss, and the object detection loss. Therefore, it has the key information of the original image and can replace the original image for the object detection task. And the thumbnail obtained by learning has much better object detection effect than the traditional interpolation method. The distillation supervision strategy can maintain the object detection effect at a result equivalent to that of using the original large-size image for object detection. The "teacher" network is a model trained with the original image, and its weights will be frozen when transferring knowledge to the "student" model. In addition, the Query Filter algorithm eliminates redundant learnable vectors before Logit distillation to reduce the computational amount. The present application reduces the computational amount of the network by reducing the spatial size scale of the input image, and realizes the object detection task on the input compressed image, solving the problem that complex algorithms cannot run in real time on hardware with limited memory resources such as mobile devices. Description of the Drawings

[0043] Figure 1 It is a framework diagram of the compressed image object detection method based on degraded network feature learning;

[0044] Figure 2 It is a structure diagram of the downsampling module;

[0045] Figure 3It is a structural diagram of Logit distillation;

[0046] Figure 4 It is a structural diagram of the feature map;

[0047] Figure 5 It is a comparison diagram of the target detection effect. Specific implementation manners

[0048] It should be specifically noted that, without conflict, the various implementation manners disclosed in this application can be combined with each other.

[0049] Specific implementation manner 1: Refer to Figure 1 This implementation manner will be specifically described. A compressed image target detection method based on degraded network feature learning described in this implementation manner includes the following steps:

[0050] The step of obtaining the image to be detected and obtaining the compressed image of the image to be detected, and

[0051] The step of inputting the compressed image of the image to be detected into the trained target detection network for target detection;

[0052] The training process of the target detection network is as follows:

[0053] Step 1: Obtain the original image and obtain the compressed image of the original image;

[0054] Step 2: Respectively perform feature extraction on the original image and the compressed image to obtain the original image feature map and the compressed image feature map, and use the detection network to detect the original image to obtain the class prediction value in the original image;

[0055] Step 3: Use Feature map distillation to process the original image feature map and the compressed image feature map to obtain a compressed image feature map with a distribution similar to that of the original image feature map;

[0056] Step 4: Use the compressed image feature map with a distribution similar to that of the original image feature map to replace the original compressed image feature map, and use the detection network to obtain the position and class prediction value of the object in the compressed image;

[0057] Step 5: Use Logit distillation to process the class prediction value in the original image and the class prediction value obtained in Step 4 to obtain a class prediction value close to the class prediction value in the original image;

[0058] Step 6: Use the compressed image of the original image as the input, the position of the object in the compressed image obtained in Step 4, and the class prediction value close to the class prediction value in the original image as the output to train the target detection network (student network).

[0059] In view of the deficiencies of existing deep learning-based object detection methods, this application addresses the problem that most existing object detection methods, in order to pursue detection accuracy, usually adopt large deep networks and complex algorithms, resulting in a large number of network parameters and high memory and hardware resource consumption. Some mobile devices in practical applications may not have the ability to run complex large algorithms in real time. This application provides a compressed image object detection technology based on degraded network feature learning. Using the compressed image object detection technology based on degraded network feature learning proposed in this application, the research object is not only limited to low-resolution images obtained by downsampling through a fixed blur kernel, nor is it limited to images obtained by traditional interpolation methods. In particular, it makes it possible to perform object detection on real images. This application is of great significance for the subsequent implementation of real-scene object detection technology and artificial intelligence technology for practical applications.

[0060] To overcome the problem that complex algorithms cannot run in real time on hardware with limited memory resources such as mobile devices. This application proposes a compressed image object detection method based on degraded network feature learning. For the input image of the real scene, first, a degradation model is used to downsample the input image to reduce the scale of the input image in the spatial dimension. Then, the compressed image is sent into any existing object detection method to implement the object detection task. Inspired by the successful application of Transformer in object classification of compressed images, we choose Deformable-DETR as the object detection module to verify the effectiveness of the proposed method. To maintain the object detection accuracy under compressed images, we adopt the traditional idea of knowledge distillation. The original image before compression is input into the same object detection method, and the class labels and location information of its output results will be used as supervision information to further narrow the distance from the class labels and location information output based on the compressed image, achieving the purpose of reducing the computational complexity of the deep model while ensuring high-precision object detection.

[0061] Specifically, this application proposes a compressed image object detection framework based on degraded network feature learning, as Figure 1As shown in the figure, the whole framework consists of four parts. The downsampling module and the distillation supervision strategy are used to improve the inference speed and maintain the object detection effect. The input of the network is the original large-size image, and the low-resolution image generated by the downsampling module is used in the inference process; in order to further improve the detection effect of the low-resolution image, we propose a distillation supervision strategy. (A) The downsampling module is used to learn a thumbnail that is downsampled 4 times from the original large-size image. After the Moment-matching loss optimization, the low-resolution image contains the key information for the object detection task. (B) The "teacher" network is a Deformable-DETR network trained with the original large-size image, and its weights are fixed during the entire training process. In the distillation supervision strategy, the original image and the low-resolution image are respectively fed into two ResNet50 networks to extract features, and the feature map distillation loss is used to narrow the distance between the two feature maps. In addition, unlike the object detection method similar to DETR, we do not directly predict the output of the forward propagation network (FFNs), but use the Query Filter algorithm to remove some redundant learnable vectors (query) and then perform Logit distillation. This can reduce the amount of calculation and has been found through experiments to improve the target detection effect. (C) The "student" model is the final inference model used. Except that the input is a low-resolution image, the other structures are exactly the same as the "teacher" network. (D) The final prediction head contains a series of prediction boxes and category probability values. We use Logit distillation to transfer key information (knowledge) from the "teacher" model to the "student" model. In addition to the loss function mentioned above, the entire network also uses the original Deformable-DETR's loss function for target detection during training. The method proposed in this application is trained end-to-end.

[0062] In the downsampling module, a downsampling module is designed to obtain a thumbnail containing key information for the target detection task. In this application, the first layer of the downsampling module structure is a convolution layer with a step size of 2 and a convolution kernel size of 3×3. It maps the original input image to a 12-channel hidden space to learn hidden information. The second layer has a similar structure to the first layer, mapping the 12-channel hidden space back to 3 channels for reconstructing RGB color images. The convolution layer after each layer of the downsampling structure is connected to a ReLU activation layer for nonlinear transformation. Under the supervision of the Moment-matching loss, the original large-size image eventually becomes a 4-fold downsampled thumbnail.

[0063] In the distillation supervision strategy, in order to make the object detection results using thumbnails equivalent to those of the original image, the distillation supervision strategy is designed. In this application, the decoder structures of both the "teacher" and "student" in the Logit distillation structure are standard Transformer structures, and the feed-forward neural network (FFNs) is a three-layer linear structure with a ReLU activation function. The dimension of the output value of FFNs is R×N×C, representing the class prediction values of the "student" and "teacher" respectively. The yellow box area in the figure is the Query Filter algorithm for removing redundant learnable vectors before Logit distillation, and the final result is that only those learnable vectors with relatively high confidence are used for subsequent Logit distillation.

[0064] In this application, the COCO dataset is used to train the model proposed in this application. The COCO dataset is widely used in object detection tasks and contains images of 80 categories in daily life. We divide the COCO dataset into 80k / 40k / 5k, where 80k images are used for training, 40k images are used for validation, and 5k images are used for testing. The experimental results follow the evaluation metrics given by the COCO official, namely mAP (mean average precision), AP 50 , AP 75 , AP S , AP M , AP L .

[0065] To overcome the above problems, the present invention proposes a compressed image object detection method based on degraded network feature learning, which reduces the computational complexity of the network from a novel perspective. The present invention mainly solves two problems: (1) how to obtain a low-resolution image that removes the redundant information of the original image but contains the key information for object detection; and at the same time, ensure that the distributions of the two images are as similar as possible. (2) how to maintain the detection effect of the low-resolution image so that it can be deployed on mobile devices. For problem (1), the present invention designs an online image downsampling module to obtain a low-resolution image. Different from the traditional method of using ideal downsampling to obtain a low-resolution image, the present invention makes full use of the advantages of convolutional neural networks to obtain a low-resolution image from the original image in a supervised manner. For problem (2), the present invention proposes a knowledge distillation supervision strategy to make the performance of object detection using the low-resolution image equivalent to that using the original image. This strategy can not only obtain the key information when using the original image as input, but also accelerate the speed of using the low-resolution image in the inference stage.

[0066] The present invention breaks through the problem of existing deep learning-based object detection methods that pursue higher accuracy by extracting high-dimensional abstract features of images, resulting in an excessively large number of network model parameters and a large occupation of hardware memory resources. At the same time, it overcomes the problem that existing complex algorithms are not suitable for mobile devices with limited hardware resources. The present invention belongs to the research on object detection technology for real-world scenarios in practical applications, conforms to the development trend of contemporary information industry and intelligent manufacturing, and promotes the implementation and application of artificial intelligence technology to a certain extent.

[0067] Embodiment:

[0068] As Figure 1 shown, the first part is an image downsampling module, which uses convolutional neural networks (CNNs) to generate thumbnails. Due to the powerful learning ability of CNNs, the generated thumbnails contain crucial attributes in the original large-size image. The second part is the distillation supervision strategy. In this module, the "teacher" model is a model trained using the original large-size image, while the "student" model has exactly the same structure as the "teacher" model except that it uses the thumbnail as the input to the network. In addition, the present invention also proposes the Query Filter algorithm to eliminate redundant learnable vectors before Logit distillation, thereby further reducing the computational amount and improving the detection effect. The loss function of the entire invention includes the Moment-matching loss, which can enable the thumbnail to retain crucial information in the original image and make the thumbnail visually more similar to the original image; the Feature map loss and the Logit loss can enable the "student" model to learn the feature maps and class probability values in the "teacher" model. The present invention mainly reduces the computational amount of the network by reducing the spatial size scale of the input image, realizes the object detection task on the compressed input image, and solves the problem that complex algorithms cannot run in real time on hardware with limited memory resources such as mobile devices.

[0069] Prepare training samples. The present invention selects the MS COCO dataset, which is widely used in object detection tasks and contains images of 80 categories in daily life. The COCO dataset is divided into 80k / 40k / 5k, where 80k images are used for training, 40k images are used for validation, and 5k images are used for testing. The experimental results follow the evaluation metrics given by COCO official, namely mAP (mean average precision), AP 50 , AP 75 , AP S , AP M , AP L .

[0070] Design the downsampling module. In the image downsampling module, for a given large-sized image, the goal of the present invention is to generate a thumbnail (4x downsampling) whose distribution is as similar as possible to the original large-sized image. Theoretically, a single-layer convolutional structure can achieve the downsampling function. However, through experiments, we found that the single-layer convolutional structure did not achieve good results. In order to obtain a better downsampled image without introducing too much computational complexity, the downsampling module designed by the present invention uses two convolutional layers (each convolutional layer is followed by a ReLU activation function). In addition, in order to let the thumbnail retain more useful information, a pooling layer is not used in the downsampling module. As Figure 2 shown, the first layer is a 3×3 convolutional operation with a stride of 2, which mainly maps the large-sized image to 12 channels to learn hidden information; the second layer structure is similar to the first layer, and its main function is to map the features back to 3 channels to reconstruct the RGB thumbnail. The entire method for obtaining the thumbnail can be described by the following formula:

[0071]

[0072] where represents the downsampling module, x and y represent the original large-sized image and the generated thumbnail respectively, and δ represents the parameters of the downsampling module.

[0073] As mentioned before, the purpose of designing the downsampling module of the present invention is not only to reduce the resolution of the image, but more importantly, it is hoped that the obtained thumbnail is similar to the original image in terms of distribution, and the thumbnail should be easier for the machine to obtain features. That is to say, the pixel value distribution of the thumbnail should be the same as that of the original image, and the information of each image channel should be kept unchanged. Usually, a simple method is to use the MSE loss function, but the obtained thumbnail looks uncomfortable and generates many artifacts. Therefore, the present invention uses the Moment-matching loss function to optimize the downsampling module, and this loss function is defined as follows:

[0074]

[0075] where x and y represent the original large-sized image and the generated thumbnail respectively, and λ is a hyperparameter used to balance these two parts. μ(·) and σ(·) calculate the first and second moments of each channel in the image respectively. The Moment-matching loss function can make the mean and variance of the thumbnail as close as possible to those of the original image, and i represents the serial number of the image channel.

[0076] Design a distillation supervision strategy. After the downsampling module, a thumbnail that looks very similar to the original image is obtained. Although the thumbnail obtained by learning can replace the thumbnail obtained by the traditional interpolation method and be applied to subsequent target detection tasks, the 4x downsampling still loses too much information used for target detection tasks. In order to further improve the target detection effect of the 4x downsampled image, the present invention proposes a distillation supervision strategy, which can transfer the key information in the original large-size image to the thumbnail through Logit distillation and Feature map distillation.

[0077] (1) Logit Distillation

[0078] Logit Distillation by Introducing “Temperature” To soften the classification prediction results of the "teacher" and "student", in addition, the KL divergence is used to bring the classification prediction results of the "teacher" and "student" closer. In the Logit distillation loss function, the KL divergence of the classification output of the "teacher" network and the classification output of the "student" network is defined as follows:

[0079]

[0080] in represents the softmax function, represents the hyperparameter temperature, p t and p s Represent the predicted values of “teacher” and “student” respectively.

[0081] However, object detection methods like DETR use a series of learnable vectors (query) to directly obtain prediction results. Therefore, the logit distillation of DETR is very different from the logit distillation based on CNNs. Specifically, the dimension of the logit output of CNNs is p∈R B×C , while the dimension of the logical output similar to DETR is p′∈R B×C×N , where B, N, and C represent the batch size, the number of learnable vectors, and the number of categories in the dataset, respectively. Usually, the number of real targets M in an image is much smaller than N, which means that many queries are redundant. If these redundant queries are subsequently used to calculate Logit distillation, it will take up a lot of computation. Therefore, we propose a Query Filter algorithm to remove redundant queries before Logit distillation, which can help reduce the amount of computation and improve detection results.

[0082] like Figure 3As shown, N learnable queries are respectively fed into the Transformer decoders of the "teacher" and "student", and the final outputs are obtained after passing through the feed-forward neural networks (FFNs). In the Query Filter algorithm, the first step is to sort the classification prediction results for all the queries of the "teacher" network using the descending order sorting algorithm. Then, according to the pre-set threshold (0.5), the part of the prediction results greater than the threshold is set to True, thereby obtaining a binary matrix keep. In the second step, the obtained keep matrix is multiplied with the corresponding positions of the prediction matrices of the "teacher" and "student", and finally the prediction matrix with redundant queries removed is obtained.

[0083] (2) Feature map distillation

[0084] In the present invention, we use the method of feature map distillation to improve the object detection effect on thumbnails. As Figure 4 shown, both the "teacher" and "student" networks use ResNet50 as the backbone network. However, since the input of the "student" network is a 4-fold downsampled thumbnail, this results in inconsistent sizes of the intermediate feature maps of the two networks. To calculate the feature map distillation loss, usually deconvolution or interpolation operations are used to transform the feature map of the "student" network into the same size as the feature Figure 1 map of the "teacher" network. In the present invention, we use the linear interpolation method to align the sizes of the feature maps of the two networks. Specifically, in the ResNet50 backbone network, only the feature maps in the third, fourth, and fifth convolutional layers are used for knowledge distillation, and the weights of other layers are frozen. The loss function of feature map distillation is calculated as follows:

[0085]

[0086] where F T and F S represent the output feature maps of ResNet50 in the "teacher" and "student" networks, L represents the number of convolutional layers of ResNet50, C, H, and W respectively represent the number of channels, height, and width of the feature map, and f represents the feature map alignment operation. Finally, the distillation supervision strategy includes two parts: Logit distillation and Feature map distillation. The overall distillation loss is as follows:

[0087]

[0088] where α and β are hyperparameters used to balance the losses of the two parts.

[0089] In summary, the complete loss function of the compressed image object detection method based on degraded network feature learning proposed in this paper is as follows:

[0090]

[0091] where γ is the weight of the Moment-matching loss.

[0092] Train the compressed image object detection network based on degraded network feature learning proposed in the present invention. For simplicity and more stable training, in the present invention, we divide the entire training process into two stages. In the first stage, only the downsampling module is trained, and the Moment-matching loss is used to optimize the parameters of the downsampling module. In the second stage, knowledge distillation is used to further optimize the detection effect of the thumbnail. It should be noted that in the second stage, the parameters of the downsampling module are also continuously optimized. Finally, we obtain the parameters of the trained compressed image object detector based on degraded network feature learning. Specifically, we use the Deformable-DETR algorithm with ResNet50 as the backbone network as the object detector based on the present invention. AdamW is used as the optimizer, and the initial learning rate is 2×10 -4 , and the learning rate is reduced by 10 times every 40 epochs. The size of the original image is (800, 1333), where 800 represents the length of the short side of the image, and 1333 represents the maximum side length of the image. λ is set to 0.1, and the weight of the Moment-matching loss, γ, is set to 1. In the distillation supervision strategy stage, the "temperature" is set to 1, and α and β in the distillation loss are set to 0.5 and 0.1 respectively. The settings of other hyperparameters are the same as those in the Deformable-DETR algorithm. In the training stage, only the downsampling module is trained in the first 5 epochs, and then knowledge distillation is used for the subsequent training process.

[0093] The compressed image object detection network based on degraded network feature learning trained through the above steps breaks through the problem that existing deep learning-based object detection methods pursue higher accuracy by extracting high-dimensional abstract features of images, resulting in too large network model parameters and occupying too much hardware memory resources. At the same time, it overcomes the problem that existing complex algorithms are not suitable for mobile devices with limited hardware resources and can achieve fast and high-accuracy object detection tasks. Experiments prove that the "compressed image object detection technology based on degraded network feature learning" of the present invention can achieve object detection tasks for low-resolution images. Table 1 shows the comparison data of experimental results, where the present invention uses the standard evaluation index mAP in the field of object detection to evaluate the proposed method.

[0094] To verify the object detection effect of the method proposed in the present invention on low-resolution images, we conducted experiments on images downsampled by 2 times and 4 times respectively. We mainly compared with the method of using the original image for object detection and the method of using traditional bicubic interpolation to obtain thumbnails. The results are shown in Table 1. In Table 1, since the "teacher" and "student" networks use the same network structure, the number of network parameters is the same, both are 40M.

[0095] (1) 4x downsampling

[0096] Compared with the bicubic interpolation method, the method proposed in the present invention uses a downsampling module, and the mAP is improved by 3.2% (32.3% mAP vs. 29.1% mAP). Among them, in the detection of large-sized objects, the effect is the most significant, and the mAP is increased from 48.0% to 53.5%, an increase of 5.5%. In addition, in the object detection of medium-sized objects, the mAP is increased by 4.2%; in the detection of small-sized objects, the mAP is increased by 1.7%. The above experimental results also prove the effectiveness of the method proposed in the present invention in low-resolution image object detection. In addition, to verify the efficiency of the algorithm, we calculated the FPS (images / second) in the first 100 images of the COCO validation set during algorithm testing. It can be seen from the last two columns of Table 1 that the FPS of the method proposed in the present invention is 19.91 images per second, which is 1.26 times faster than the method of using the original large-sized image at 15.86 images per second. The main reason is that the method proposed in the present invention only performs convolution calculations on thumbnails containing key information, while the original large-sized image needs to calculate additional redundant information. At the same time, the inference speed of the present invention is comparable to that of using traditional bicubic interpolation to obtain thumbnails, but in terms of object detection results, it is much higher than the traditional method.

[0097] (2) 2x downsampling

[0098] The network structure of the 2x downsampling module is similar to that of the 4x downsampling module, including two convolutional layers. After each convolutional layer, the ReLU activation function is also used. The number of convolutional kernels in the first and second layers is also 12 and 3, respectively. The difference is that the first layer uses a 5×5 convolutional kernel for convolution operations. As can be seen from Table 1, the method proposed in the present invention is 3.0% higher than the traditional method in terms of average accuracy; in terms of inference speed, the method of the present invention processes 19.40 images per second, which is basically the same as 19.57 images per second of the traditional method. In addition, compared with the result of 43.8% for object detection using the original large-size images, although we use 2x downsampled images as input, we can still achieve a relatively good object detection effect (41.6% mAP). In summary, compared with object detection using the original large-size images, the method proposed in the present invention improves the inference speed by 1.22 times with only a 2.2% loss in accuracy, which further verifies the effectiveness of the present invention in object detection on low-resolution images.

[0099] (3) Comparison of object detection effects

[0100] We present some visualization results of object detection on the COCO validation set. As Figure 5 shown, the first column is the 4x downsampled thumbnails generated by the downsampling module. For ease of observation, we magnify them to the same size as the original images, as shown in the second column of the figure. The third column is the results of object detection using the low-resolution images obtained by the bicubic interpolation method, which are also magnified to the same size as the original images. The last column is the visualization results of object detection using Deformable-DETR on the original images.

[0101] From (a) and (d) in the figure, we can find that all objects are correctly detected, which is basically the same as the results of the Deformable-DETR algorithm, while the detection effect of the bicubic interpolation method is not very ideal in some cases. From these visualization results, it can be seen that the thumbnails generated by the downsampling module are easier for machines to understand and contain the key information for object detection.

[0102] Table 1 Experimental results

[0103]

[0104] It should be noted that the specific implementation manners are only explanations and illustrations of the technical solutions of the present invention, and the scope of the rights cannot be limited thereby. Those that are only partial changes made according to the claims and the description of the present invention should still fall within the protection scope of the present invention.

Claims

1. A compressed image object detection method based on degraded network feature learning, characterized in that Including: Steps of obtaining an image to be detected and obtaining a compressed image of the image to be detected, and Steps of inputting the compressed image of the image to be detected into a trained object detection network for object detection; The training process of the object detection network is as follows: Step 1: Obtain an original image and obtain a compressed image of the original image; Step 2: Respectively perform feature extraction on the original image and the compressed image to obtain an original image feature map and a compressed image feature map, and use the detection network to detect the original image to obtain class prediction values in the original image; Step 3: Use Feature map distillation to process the original image feature map and the compressed image feature map to obtain a compressed image feature map with a distribution similar to that of the original image feature map; Step 4: Use the compressed image feature map with a distribution similar to that of the original image feature map to replace the original compressed image feature map, and use the detection network to obtain the position and class prediction values of the objects in the compressed image; Step 5: Use Logit distillation to process the class prediction values in the original image and the class prediction values obtained in Step 4 to obtain class prediction values close to the class prediction values in the original image; Step 6: Use the compressed image of the original image as the input, the position of the objects in the compressed image obtained in Step 4, and the class prediction values close to the class prediction values in the original image as the output to train the object detection network.

2. The compression image target detection method based on degraded network feature learning according to claim 1, characterized in that The compressed image is obtained through a downsampling module, and the downsampling module includes: The first layer for mapping a large-size image to 12 channels to learn hidden information, and the first layer is a convolutional layer with a stride of 2 and a kernel size of 3×3; The second layer for mapping the features back to 3 channels to reconstruct the RGB thumbnail, and the second layer is a convolutional layer with a stride of 2 and a kernel size of 3×3.

3. The compression image target detection method based on degenerated network feature learning according to claim 2, wherein The loss function of the downsampling module is expressed as: where x and y respectively represent the original large-size image and the generated thumbnail, λ is a hyperparameter used to balance these two parts, μ(·) and σ(·) respectively calculate the first and second moments of each channel in the image, and i represents the serial number of the image channel.

4. A method for compressed image object detection based on degraded network feature learning according to claim 3, characterized in that The compressed image is expressed as: Among them, represents the downsampling module, x and y respectively represent the original large-size image and the generated thumbnail, and δ represents the parameter of the downsampling module.

5. A method for compressed image object detection based on degraded network feature learning according to claim 1, characterized in that The loss function of the Logit distillation is expressed as: Among them, represents the softmax function, represents the hyperparameter temperature, p t and p s represent the class prediction values of the teacher and the student respectively.

6. The compression image target detection method based on degraded network feature learning according to claim 1, wherein The feature extraction is performed through a ResNet50 network.

7. A method for detecting compressed image targets based on degenerate network feature learning according to claim 6, characterized in that The loss function of the Feature map distillation is expressed as: Among them, F T and F S represent the output feature maps of ResNet50 in the teacher and student networks, L represents the number of convolutional layers of ResNet50, C, H, and W represent the number of channels, height, and width of the feature maps respectively, and f represents the feature map alignment operation.

8. A method for compressed image object detection based on degraded network feature learning according to claim 1, characterized in that The overall loss function of the Logit distillation and the Feature map distillation is expressed as: where α and β are hyperparameters used to balance the two parts of the loss.

9. A method for detecting compressed image targets based on degraded network feature learning according to claim 8, characterized in that The α and β are 0.5 and 0.1 respectively.

10. A method for compressed image object detection based on degraded network feature learning according to claim 8, characterized in that The loss function of the object detection method is expressed as: Among them, γ is the weight of the Moment-matching loss, representing the loss of the object detection network.

Images