A Deep Learning-Based Low-Resolution Image Object Detection Method

Abstract

This invention discloses a deep learning-based method for low-resolution image target detection. The implementation scheme includes: 1) obtaining a high-resolution image target detection training dataset; 2) creating a low-resolution image target detection training dataset; 3) constructing a low-resolution image target detection model; 4) constructing a loss function; 5) training the model; and 6) detecting low-resolution images. This invention constructs a task decomposition enhancement network to effectively enhance low-resolution images under different degradation conditions. It mainly consists of an image restoration branch and a resolution enhancement branch. These two branches share a common feature extractor, which not only reduces the number of network parameters but also enhances the feature representation ability. By jointly optimizing the task decomposition enhancement network and the detection network, the accuracy of low-resolution image target detection is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a method for low-resolution image target detection based on deep learning. Background Technology

[0002] Convolutional neural network-based object detection methods have achieved superior performance on classic public datasets. These methods can be broadly categorized into two types: two-stage methods, represented by the R-CNN series, and single-stage methods, represented by the YOLO series. These methods are typically trained and optimized on high-resolution images; their performance drops significantly when detecting objects on low-resolution images.

[0003] Existing low-resolution image object detection methods primarily employ deep learning-based image super-resolution reconstruction to enhance the resolution of low-resolution images. The enhanced image is then fed into an existing detection network to improve object detection accuracy. However, these super-resolution reconstruction methods typically train models using only pixel-level loss functions, resulting in models that fail to learn the structural features crucial for object detection, thus hindering the improvement of subsequent detection network accuracy. Furthermore, most super-resolution reconstruction methods are designed for low-resolution images resulting from a single degradation factor. Consequently, these methods struggle to handle low-resolution images in real-world scenarios caused by complex degradation factors, posing a challenge to subsequent object detection stages. Summary of the Invention

[0004] This invention fully considers the shortcomings of existing technologies and aims to provide a deep learning-based method for low-resolution image target detection that can handle low-resolution image target detection under different degradation conditions and achieve high accuracy.

[0005] I. Technical Principles

[0006] This invention constructs a deep learning-based low-resolution image target detection model, which improves the accuracy of low-resolution image target detection by cascading image enhancement networks and target detection networks and jointly optimizing them in an end-to-end learning manner. In this model, a task decomposition enhancement network is designed to break down the overall image enhancement task into an image restoration subtask and a resolution enhancement subtask. These two subtasks are executed by the image restoration branch and the resolution enhancement branch in the task decomposition enhancement network, respectively. Furthermore, a shared feature extractor is introduced into the task decomposition enhancement network to effectively utilize the correlation between these two subtasks to better learn the structured features in the image. By sequentially executing these two subtasks, the quality of the low-resolution image can be progressively enhanced.

[0007] II. Based on the above principles, the present invention is implemented through the following scheme:

[0008] A deep learning-based method for low-resolution image target detection includes the following steps:

[0009] (1) Obtain high-resolution image target detection training dataset: Obtain original high-resolution images and corresponding detection labels from public databases.

[0010] (2) Create a low-resolution image target detection training dataset: Use a degradation model to degrade each high-resolution image in the high-resolution training dataset obtained in step (1) to obtain the corresponding low-resolution image. Use the set of image pairs consisting of high and low resolution images and the detection labels corresponding to these image pairs as the low-resolution image target detection training dataset.

[0011] (3) Constructing a low-resolution image target detection model: This model consists of a task decomposition enhancement network and a detection network. The specific construction process includes the following steps:

[0012] (3-a) Constructing the task decomposition enhancement network: This network consists of an image restoration branch and a resolution enhancement branch; the image restoration branch and the resolution enhancement branch are constructed as follows:

[0013] (3-a-1) Constructing the image restoration branch: This branch consists of shared feature extractor 1, convolutional layer 1, and pixel-wise addition operation; input image I L After passing through shared feature extractor 1 and convolutional layer 1 in sequence, the result obtained is the same as I. L By adding pixels one by one, the intermediate restored image I is obtained. I The construction of the image restoration branch is completed.

[0014] (3-a-2) Constructing the resolution enhancement branch: This branch consists of shared feature extractor 2, bilinear interpolation operation 1, bilinear interpolation operation 2, pixel shuffling layer, and pixel-by-pixel addition operation; the intermediate restored image I obtained in step (3-a-1) is then used to enhance the resolution. I The input is fed into the shared feature extractor 2, and the result is processed by bilinear interpolation operation 1 to obtain the intermediate feature map X. E ; X E The input is fed into the pixel shuffling layer, and the result is the same as I. I The results processed by bilinear interpolation operation 2 are then summed pixel by pixel to obtain the enhanced image I. S The resolution enhancement branch was constructed.

[0015] The shared feature extractor 1 and shared feature extractor 2 have the same structure, both consisting of convolutional layer 2, residual unit 1, and residual unit 2; the input of the shared feature extractor passes through convolutional layer 2, residual unit 1, and residual unit 2 in sequence to obtain the output of the shared feature extractor.

[0016] (3-b) Constructing the detection network: This network consists of a backbone module, a neck module, and a head module; the enhanced image I in step (3-a-2) S The target detection results are obtained by sequentially passing through the backbone module, neck module, and head module.

[0017] (4) Construct the loss function:

[0018] Construct the following joint loss function:

[0019] L = L IR +L RE +L Det

[0020]

[0021]

[0022] Where L IR L represents the image restoration loss function. RE L represents the resolution enhancement loss function. Det Represents the detection loss function; Represents the i-th high-resolution image. Indicates will The corresponding low-resolution image is input into the image restoration branch described in step (3-a-1) to obtain the intermediate restored image; Indicates will The enhanced image is obtained after inputting into the resolution enhancement branch described in step (3-a-2); B(·) is a bicubic interpolation function; N is a positive integer representing the batch size of the training images, and N∈[1,64]; ‖·‖1 represents the L1 norm.

[0023] (5) Training the model: Use the low-resolution image target detection training dataset obtained in step (2) to train the low-resolution image target detection model constructed in step (3). Obtain the loss value according to the loss function constructed in step (4), and use the Adam optimizer to update the model parameters until the loss no longer decreases, and obtain the trained model.

[0024] (6) Low-resolution image target detection: Use the low-resolution image target detection model trained in step (5) to perform target detection on the low-resolution images in the test set to obtain the final detection result.

[0025] The backbone module in step (3-b) uses ResNet18.

[0026] The neck module in step (3-b) uses the original neck module of the CenterNet algorithm.

[0027] The header module in step (3-b) uses the original header module of the CenterNet algorithm.

[0028] The residual unit 1 and residual unit 2 in step (3) have the same structure, both consisting of convolutional layer 3, ReLU layer, convolutional layer 4 and pixel-by-pixel addition operation; the input feature map of the residual unit passes through convolutional layer 3, ReLU layer and convolutional layer 4 in sequence, and the result is then added to the input feature map of the residual unit pixel by pixel to obtain the output of the residual unit.

[0029] In step (4), N is preferably 16.

[0030] The detection loss function in step (4) adopts the original detection loss function of the CenterNet algorithm.

[0031] The present invention has the following advantages:

[0032] First, this invention constructs an effective low-resolution image target detection model. This model improves the accuracy of target detection by jointly optimizing a cascaded image enhancement network and a detection network in an end-to-end learning manner.

[0033] Second, the task decomposition enhancement network constructed in this invention gradually enhances the quality of low-resolution images by splitting the overall enhancement task into an image complex atom task and a resolution enhancement subtask, and executing these two subtasks sequentially.

[0034] Third, this invention introduces a shared feature extractor to effectively utilize the correlation between image complex atom tasks and resolution enhancement subtasks, thereby improving the expressive power of features. Attached Figure Description

[0035] Figure 1 Flowchart of the deep learning-based low-resolution image target detection method according to an embodiment of the present invention;

[0036] Figure 2 Network structure diagram of the low-resolution image target detection model according to an embodiment of the present invention;

[0037] Figure 3 Residual unit structure diagram of an embodiment of the present invention;

[0038] Figure 4 Comparison of target detection results between the embodiments of the present invention and other methods;

[0039] Figure 5 Comparison chart of target detection results between the embodiments of the present invention and other methods. Detailed Implementation

[0040] The specific embodiments of the present invention are described below:

[0041] Example 1

[0042] Figure 1 The flowchart shown is a low-resolution image target detection method based on deep learning according to an embodiment of the present invention. The specific steps are as follows:

[0043] Step 1: Obtain the high-resolution image object detection training dataset

[0044] Obtain the original high-resolution images and corresponding detection labels from a public database.

[0045] Step 2: Create a low-resolution image object detection training dataset

[0046] The high-resolution images in the high-resolution training dataset obtained in step (1) are degraded using a degradation model to obtain the corresponding low-resolution images. The set of image pairs consisting of high-resolution and low-resolution images and the detection labels corresponding to these image pairs are used as the low-resolution image target detection training dataset.

[0047] Step 3: Construct a low-resolution image target detection model

[0048] Figure 2 The diagram shown illustrates the network structure of a low-resolution image target detection model according to an embodiment of the present invention. This model consists of a task decomposition enhancement network and a detection network. The specific construction process includes the following steps:

[0049] (3-a) Constructing the task decomposition enhancement network: This network consists of an image restoration branch and a resolution enhancement branch; the image restoration branch and the resolution enhancement branch are constructed as follows:

[0050] (3-a-1) Constructing the image restoration branch: This branch consists of shared feature extractor 1, convolutional layer 1, and pixel-wise addition operation; the input image I is 256×256×3. L After passing through shared feature extractor 1 and convolutional layer 1 with a 3×3 kernel, the result obtained is the same as I. L By adding pixels one by one, an intermediate restored image I with a size of 256×256×3 is obtained. I The construction of the image restoration branch is completed.

[0051] (3-a-2) Constructing the resolution enhancement branch: This branch consists of shared feature extractor 2, bilinear interpolation operation 1, bilinear interpolation operation 2, pixel shuffling layer, and pixel-by-pixel addition operation; the intermediate restored image I with a size of 256×256×3 obtained in step (3-a-1) is then used. I The input is fed into the shared feature extractor 2, and the result is processed by bilinear interpolation operation 1 to obtain an intermediate feature map X of size 128×128×48. E ; X E The input is fed into the pixel shuffling layer, and the result is the same as I. I The results processed by bilinear interpolation operation 2 are then summed pixel by pixel to obtain an enhanced image I of size 512×512×3. S The resolution enhancement branch was constructed.

[0052] The shared feature extractor 1 and shared feature extractor 2 have the same structure, both consisting of convolutional layer 2, residual unit 1, and residual unit 2. The input of the shared feature extractor with a size of 256×256×3 passes sequentially through convolutional layer 2 with a kernel size of 3×3, residual unit 1, and residual unit 2 to obtain the output of the shared feature extractor with a size of 256×256×48.

[0053] (3-b) Constructing the detection network: This network consists of a backbone module, a neck module, and a head module; the enhanced image I in step (3-a-2) is 512×512×3 in size. S The target detection result is obtained by sequentially passing through the backbone module, neck module, and head module. In this embodiment, the backbone module is ResNet18, and the neck module and head module are the original neck module and head module of the CenterNet algorithm, respectively.

[0054] Figure 3 The diagram shows the structure of the residual unit according to an embodiment of the present invention, which is used to extract depth features. Residual unit 1 and residual unit 2 have the same structure, both consisting of convolutional layer 3, ReLU layer, convolutional layer 4, and pixel-by-pixel addition operation; the input feature map of the residual unit passes through convolutional layer 3 with a kernel of 3×3, ReLU layer, and convolutional layer 4 with a kernel of 3×3 in sequence, and the result is then added to the input feature map of the residual unit pixel by pixel to obtain the output of the residual unit.

[0055] Step 4, construct the loss function

[0056] Construct the following joint loss function:

[0057] L = L IR +L RE +L Det

[0058]

[0059]

[0060] Where L IR L represents the image restoration loss function. RE L represents the resolution enhancement loss function. Det Represents the detection loss function; Represents the i-th high-resolution image. Indicates will The corresponding low-resolution image is input into the image restoration branch described in step (3-a-1) to obtain the intermediate restored image; Indicates will The enhanced image is obtained after inputting into the resolution enhancement branch described in step (3-a-2); B(·) is a bicubic interpolation function; N is a positive integer representing the batch size of the training images, and N∈[1,64]; ‖·‖1 represents the L1 norm; in this example, N is preferably 16, and the detection loss function is the original detection loss function of the CenterNet algorithm.

[0061] Step 5, train the model

[0062] The low-resolution image target detection training dataset obtained in step (2) is used to train the low-resolution image target detection model constructed in step (3). The loss value is obtained according to the loss function constructed in step (4), and the Adam optimizer is used to update the model parameters until the loss no longer decreases, thus obtaining the trained model.

[0063] Step 6, Low-resolution image target detection

[0064] The low-resolution image target detection model trained in step (5) is used to perform target detection on the low-resolution images in the test set to obtain the final detection result.

[0065] Example 2

[0066] The method described in Example 1 was used to perform object detection experiments on low-resolution images from the VOC-D dataset, which was obtained by degrading the public dataset PSCAL VOC val2007. The computer environment for this experiment was as follows: Linux Ubuntu 18.04 operating system, one NVIDIA 1080Ti 11G GPU, and PyTorch software platform.

[0067] In this embodiment, the mean precision (mAP) is used to evaluate the detection performance of different methods. The methods compared include RLFN, DSSR, SCUNet, and AERIS. The mAP values ​​of different methods on the VOC-D dataset are shown in Table 1. As can be seen from Table 1, compared with other methods, the present invention achieved the highest values ​​in all evaluation metrics.

[0068] Figure 4 The figure shown is a comparison of target detection results between the embodiments of the present invention and other methods. Figure 4 The first row, from left to right, shows the high-resolution image, the corresponding low-resolution image, and the object detection result obtained by RLFN; the second row, from left to right, shows the object detection results obtained by DSSR, SCUNet, AERIS, and this invention, respectively. Figure 4 As can be seen, compared to other methods, this invention achieves better detection results. Figure 4 Taking cattle as an example, this invention can detect more cattle. This result shows that the invention can effectively improve the detection accuracy of low-resolution image target detection models by applying a task decomposition enhancement network.

[0069] Table 1

[0070]

[0071] Example 3

[0072] The method described in Example 1 was used to perform object detection experiments on low-resolution images in the COCO-D dataset, which was obtained by degrading the publicly available COCO dataset. The computer environment for this experiment was as follows: Linux Ubuntu 18.04 operating system, one NVIDIA 1080Ti 11G GPU, and PyTorch software platform.

[0073] In this embodiment, the mean accuracy (mAP) is used to evaluate the detection performance of different methods. The methods compared include RLFN, DSSR, SCUNet, and AERIS. s AP m and AP l The values ​​represent the mean accuracy (AP) for small, medium, and large targets, respectively. Table 2 shows the mAP values ​​of different methods on the COCO-D dataset. As can be seen from Table 2, this invention achieves the highest values ​​across all evaluation metrics compared to other methods.

[0074] Figure 5 The figure shown is a comparison of target detection results between the embodiments of the present invention and other methods. Figure 5The first row, from left to right, shows the high-resolution image, the corresponding low-resolution image, and the object detection result obtained by RLFN; the second row, from left to right, shows the object detection results obtained by DSSR, SCUNet, AERIS, and this invention, respectively. Figure 5 As can be seen, compared to other methods, this invention can detect more targets. This result indicates that this invention can effectively improve the accuracy of target detection in low-resolution images.

[0075] Table 2

[0076]

Claims

1. A low-resolution image target detection method based on deep learning, characterized in that... Includes the following steps: (1) Obtain the high-resolution image target detection training dataset: Obtain the original high-resolution images and corresponding detection labels from public databases; (2) Create a low-resolution image target detection training dataset: Use a degradation model to degrade each high-resolution image in the high-resolution training dataset obtained in step (1) to obtain the corresponding low-resolution image. Use the set of image pairs consisting of high and low resolution images and the detection labels corresponding to these image pairs as the low-resolution image target detection training dataset. (3) Constructing a low-resolution image target detection model: This model consists of a task decomposition enhancement network and a detection network. The specific construction process includes the following steps: (3-a) Constructing the task decomposition enhancement network: This network consists of an image restoration branch and a resolution enhancement branch; the image restoration branch and the resolution enhancement branch are constructed as follows: (3-a-1) Constructing the image restoration branch: This branch consists of shared feature extractor 1, convolutional layer 1, and pixel-wise addition operation; input image I L After passing through shared feature extractor 1 and convolutional layer 1 in sequence, the result obtained is the same as I. L By adding pixels one by one, the intermediate restored image I is obtained. I Complete the construction of the image restoration branch; (3-a-2) Constructing the resolution enhancement branch: This branch consists of shared feature extractor 2, bilinear interpolation operation 1, bilinear interpolation operation 2, pixel shuffling layer, and pixel-by-pixel addition operation; the intermediate restored image I obtained in step (3-a-1) is then used to enhance the resolution. I The input is fed into the shared feature extractor 2, and the result is processed by bilinear interpolation operation 1 to obtain the intermediate feature map X. E ; X E The input is fed into the pixel shuffling layer, and the result is the same as I. I The results processed by bilinear interpolation operation 2 are then summed pixel by pixel to obtain the enhanced image I. S Complete the construction of the resolution enhancement branch; The shared feature extractor 1 and shared feature extractor 2 have the same structure, both consisting of convolutional layer 2, residual unit 1, and residual unit 2; the input of the shared feature extractor passes through convolutional layer 2, residual unit 1, and residual unit 2 in sequence to obtain the output of the shared feature extractor; (3-b) Constructing the detection network: This network consists of a backbone module, a neck module, and a head module; the enhanced image I in step (3-a-2) S The target detection results are obtained by sequentially passing through the backbone module, neck module, and head module; (4) Construct the loss function: Construct the following joint loss function: L=L IR +L RE +L Det Where L IR L represents the image restoration loss function. RE L represents the resolution enhancement loss function. Det Represents the detection loss function; Represents the i-th high-resolution image. Indicates will The corresponding low-resolution image is input into the image restoration branch described in step (3-a-1) to obtain the intermediate restored image; Indicates will The enhanced image obtained after inputting into the resolution enhancement branch described in step (3-a-2); B(·) is a bicubic interpolation function; N is a positive integer representing the batch size of the training images, and N∈[1,64]; ‖·‖1 represents the L1 norm; (5) Training the model: Use the low-resolution image target detection training dataset obtained in step (2) to train the low-resolution image target detection model constructed in step (3), obtain the loss value according to the loss function constructed in step (4), and use the Adam optimizer to update the model parameters until the loss no longer decreases, and obtain the trained model. (6) Low-resolution image target detection: Use the low-resolution image target detection model trained in step (5) to perform target detection on the low-resolution images in the test set to obtain the final detection result.

2. The method for low-resolution image target detection based on deep learning as described in claim 1, characterized in that, The backbone module in step (3-b) uses ResNet18.

3. The method for low-resolution image target detection based on deep learning as described in claim 1, characterized in that, The neck module in step (3-b) is the original neck module of the CenterNet algorithm.

4. The method for low-resolution image target detection based on deep learning as described in claim 1, characterized in that, The header module in step (3-b) uses the original header module of the CenterNet algorithm.

5. The method for low-resolution image target detection based on deep learning as described in claim 1, characterized in that, The residual unit 1 and residual unit 2 in step (3-a) have the same structure, both consisting of convolutional layer 3, ReLU layer, convolutional layer 4 and pixel-by-pixel addition operation; the input feature map of the residual unit passes through convolutional layer 3, ReLU layer and convolutional layer 4 in sequence, and the result is then added to the input feature map of the residual unit pixel by pixel to obtain the output of the residual unit.

6. The method for low-resolution image target detection based on deep learning as described in claim 1, characterized in that, The detection loss function L in step (4) Det The original detection loss function of the CenterNet algorithm is used.

Images