A method for object detection based on image super-resolution

Abstract

A target detection method based on image super-resolution includes the following steps: step 1, sending the original image into a target detection network to obtain an image that needs super-resolution; step 2, using a super-resolution network to perform super-resolution on the output feature map to obtain a feature map of a larger size; step 3, re-cropping the feature map of step 2 into several small areas, and then searching for a target frame in each small area; step 4, comparing the feature map with the feature map obtained in step 3 to obtain a loss function of the entire network. The present invention improves the target detection efficiency of the entire network by using a super-resolution network to improve the resolution of the feature map or part of the original image, thereby reducing the complexity of super-resolution and the computational complexity of processing the super-resolution original image, and improving the real-time performance of the entire target detection task based on super-resolution.

Description

Technical Field

[0001] The present invention belongs to the technical field of image target detection, and in particular relates to a target detection method based on image super-resolution. Background Art

[0002] Image super-resolution (SR) is to convert low-resolution (LR) images to high-resolution (HR) images through certain algorithms. High-resolution images have higher pixel density, more detailed information, and more delicate image quality. To obtain high-resolution images, the most direct way is to use a high-resolution camera. However, in actual applications, due to production process and engineering cost considerations, high-resolution and super-resolution cameras are not used to collect image signals in many occasions. Therefore, there is a certain application demand for obtaining HR through super-resolution technology.

[0003] From the definition of this technology, it can be seen that the main function of super-resolution technology is to restore the missing information in the image and construct an image with higher resolution. It is usually used in the fields of photo restoration, film restoration, transmission image compression, etc.

[0004] Super-resolution technology can be implemented by traditional methods such as interpolation, or by deep neural network (DNN) calculations.

[0005] In the field of image processing, there is another important application field, which is target detection. The purpose of the target detection task is to identify the category of an object in a given image and mark the area where the object is located. The currently widely used target detection technology is based on the deep neural network (DNN) algorithm. After the input image is processed by a DNN network, many feature maps are generated. The target to be detected is finally found by searching the frame in the feature map. It should be noted that the size of the feature map is generally much smaller than the size of the original image. Taking YOLOV3 as an example, the size of the input image is 416*416 pixels, and the size of the feature map is only 52*52, 26*26, and 13*13 pixels.

[0006] In the task of target detection, a relatively difficult problem is that small targets are difficult to extract using DNN networks because their features are difficult to extract. The target detection algorithm based on DNN usually has a low detection success rate for small targets. Therefore, a very intuitive idea is to enlarge the original image using image super-resolution technology, and use the enlarged image for target detection. Since the size of small targets is enlarged, the probability of recognition can be improved.

[0007] The best object detection algorithm based on super-resolution is the TDSR algorithm, which is a super-resolution algorithm driven by object detection. In this algorithm, a super-resolution network is used to perform super-resolution processing on the original image to be detected to obtain an enlarged image. Then, the object detection DNN network is used to detect the enlarged image, and finally the smaller objects on the original image can be detected.

[0008] In the neural network system, an important concept is neural network training. Training refers to using the loss function tool to calculate the parameter values ​​in the neural network. The loss function measures the contribution of each parameter in the neural network to the output of the neural network. First, the loss function is used to calculate the loss of the output using the current neural network parameters, and then the neural network parameters are incrementally adjusted according to the loss value, and the output is calculated again. The above operation is repeated many times, and finally the network parameters converge to a stable value. The above process is called training.

[0009] During the training process of the TDSR network, due to the introduction of the super-resolution network, it is necessary to modify the loss function based on the target detection results to revise the parameters of the super-resolution network. In other words, the super-resolution task is designed for the accuracy of target detection.

[0010] There are two disadvantages of the prior art.

[0011] One is: for the target detection network, the computing speed of the network is related to the size of the input image. If the side length of the input image increases by 1 time, the computing amount of the entire subsequent target detection DNN network may increase by many times. In addition, the computing amount of super-resolution itself is also very large. Taking YOLOV3 as an example, the input image increases from 320*320 to 608*608, and the image size increases by less than 1 time, but the computing amount of the DNN network increases by nearly 4 times (38.97vs140.69TFLOPS). Therefore, the computing cost of improving the resolution of the original image is very large, which has a great impact on the real-time performance of the system.

[0012] The second disadvantage of this technology is that since the DNN network uses the feature map after computation for target recognition, there is a certain gap between the original image and the feature map, so increasing the resolution of the original image cannot significantly improve the system's ability to recognize objects. Therefore, performing super-resolution operations on the original image has limited effect on improving the DNN network's ability to extract target features and detect targets. Summary of the invention

[0013] The purpose of the present invention is to provide a target detection method based on image super-resolution to solve the above problems.

[0014] To achieve the above object, the present invention adopts the following technical solutions:

[0015] A target detection method based on image super-resolution comprises the following steps:

[0016] Step 1: Send the low-resolution original image to the target detection network to obtain the feature map that needs super-resolution;

[0017] Step 2: Use a super-resolution network to perform super-resolution on the output feature map to obtain a feature map of a larger size;

[0018] Step 3: Re-crop the feature map in step 2 into several small areas, ensure that each small area is consistent with the size of the target frame extraction network of the original network, and then search for the target frame in each small area;

[0019] In step 4, the high-resolution original image corresponding to the original low-resolution image is sent to the current network, and the original target detection network without super-resolution work is used to obtain a feature map of the same size after steps 1-3; then the feature map is compared with the feature map obtained in step 3 to obtain the loss function of the entire network.

[0020] Furthermore, step 1 includes case 1: downsampling the high-definition original image to obtain the original image size, sending it to the target detection network, and obtaining the feature map through the target detection network.

[0021] Furthermore, a super-resolution feature map is obtained by using a gradient map or a residual network.

[0022] Furthermore, step 1 includes situation 2: the original image is sent to a two-stage target detection network, and a candidate frame area of ​​the image is obtained after passing through the first-stage network; based on the obtained candidate frame, the image area corresponding to the candidate frame area in the original image is calculated as the image that needs super-resolution.

[0023] Furthermore, if multiple candidate boxes are found, the image areas corresponding to the multiple candidate boxes are extracted respectively, and the union part is taken; certain threshold conditions can be used to screen valid candidate boxes, such as candidate box size, position, overlapping parts between candidate boxes, etc.

[0024] Furthermore, in step 4, the first-order or second-order loss of the feature map and the feature map obtained in step 3 is used as the super-resolution loss, and the loss after target detection and the super-resolution loss are combined as the loss function of the entire network.

[0025] Furthermore, in step 4, the resolution multiple of the low-resolution image and the high-resolution original image is consistent with the super-resolution network magnification in step 2.

[0026] Compared with the prior art, the present invention has the following technical effects:

[0027] The present invention improves the resolution of feature maps or part of the original maps using a super-resolution network, thereby improving the target detection efficiency of the entire network. This reduces the complexity of super-resolution and the computational complexity of processing the super-resolution original map, and improves the real-time performance of the entire super-resolution-based target detection task.

[0028] The present invention performs super-resolution on the feature map, so that the features with improved resolution are closer to the features of target detection, and can improve the target detection capability, rather than simply improving the resolution of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the existing target detection network;

[0030] Figure 2 Part of the original image with super resolution in the improved object detection network. DETAILED DESCRIPTION

[0031] The present invention is further described below in conjunction with the accompanying drawings:

[0032] See also Figure 1 and Figure 2 This solution uses a super-resolution network to increase the resolution of the feature map instead of using the resolution of the original feature map, and sends the feature map with the increased resolution to the subsequent target frame extraction algorithm for extraction. Because super-resolution processing is used on the feature map, the target detection network needs to be processed accordingly so that it can respond to the target frame extraction on the larger feature map after super-resolution.

[0033] Step 1: Downsample the high-resolution original image (HR) to get the original image size, send it to the target detection network, and get the feature map through the target detection network.

[0034] Taking the YOLOV3 network as an example, the 1684*1684 resolution image is downsampled by 4 times to get the original 416*416 image, which is sent to the target detection network. The size of the feature map output by the network is 13*13.

[0035] Step 2: Use a super-resolution network to perform super-resolution on the output feature map to obtain a feature map of a larger size. It is necessary to use a gradient map or other methods to obtain a super-resolution feature map.

[0036] Step 2a: Optionally, in step 2, a residual network can be used to obtain a super-resolved feature map.

[0037] Taking the YOLOV3 network as an example, the size of the final output feature map is 13*13. After 4 times super-resolution, a 52*52 feature map is obtained.

[0038] Step 3: Re-crop the feature map into several small areas, ensure that each small area is consistent with the size of the target box extraction network of the original network, and then search for the target box in each small area, so that the subsequent target detection network does not need to be modified.

[0039] Taking the YOLOV3 network as an example, after the super-resolution obtains a 52*52 feature map, it is cropped into 16 13*13 areas, and then each area is sent to the original 13*13 target box extraction network for recognition.

[0040] By analogy, if super-resolution is performed directly on the original image, the entire target detection network needs to perform target detection on a 1684*1684 image, and the amount of computation is much greater than the current difficulty of detection on a 52*52 feature map.

[0041] Step 4: Send the high-definition original image (HR) to the current network, and use the same network to obtain a feature map of the same size after steps 1-3. Then the difference between the feature map and the feature map obtained in step 3 is used as the super-resolution loss, and the loss after target detection (such as common classification loss and bbox loss) and the super-resolution loss are processed together (such as weighted average, etc.) as the loss function of the entire network.

[0042] In a two-stage target detection network (a typical Faster RCNN network), since a two-stage feature extraction method is used, candidate boxes where targets may exist are extracted in the first stage, and only target detection is performed on the candidate box area in the second stage. Therefore, a super-resolution network can be used to perform super-resolution on the candidate box area. In this way, there is no need to perform super-resolution on the complete feature map, and the overall network speed will be faster.

[0043] Step 1: Send the original image to the two-stage object detection network. After passing through the first stage network, the candidate box area of ​​the image is obtained.

[0044] Taking the Fasterrcn network as an example, after the original image passes through the backbone network and the RPN network, several candidate box areas will be output.

[0045] Step 2: Based on the obtained candidate box, calculate the candidate box and crop the image at the corresponding position in the original image as the image that needs super-resolution.

[0046] Step 2a: If multiple candidate boxes are found, extract the part that is the union of the multiple candidate boxes.

[0047] Step 2b: Certain threshold conditions can be used to filter valid candidate boxes, such as candidate box size, position, overlap between candidate boxes, etc.

[0048] Step 3-5: Same as Step 2-4 of Solution 1, use the cropped original image obtained in Step 2 as the feature map for subsequent input. The target detection DNN network used can be the two-stage target detection network used in Step 1, or any other target detection network.

[0049] In most cases, step 2 will not find a candidate box, and the entire super-resolution network will not be executed, which will not cause excessive burden on the entire system.

Claims

1. A target detection method based on image super-resolution, characterized in that: The following steps are involved: Step 1: Send the low-resolution image to the target detection network for processing to obtain the first feature map; Step 2, using a super-resolution network to perform super-resolution on the first feature map to obtain a second feature map of a larger size; Step 3, re-cropping the second feature map in step 2 into two or more small areas, ensuring that the size of each small area is consistent with the output feature map of the target frame extraction network of the target detection network, and then performing subsequent target processing on each small area; Step 4, sending the high-definition image to the target detection network in step 1 to obtain a third feature map of the same size as the second feature map; then comparing the third feature map with the second feature map as a loss function of the target detection network, and using the loss function for training the target detection network; In step 1, the method includes: reducing the high-definition original image to the original image size by downsampling, sending the image to the target detection network, and obtaining the first feature map through the target detection network; Use gradient maps or residual networks to obtain super-resolution feature maps; In step 4, the difference between the feature map and the feature map obtained in step 3 is used as the super-resolution loss, and the target detection loss after target detection and the super-resolution loss are combined as the loss function of the entire network; In step 1, the target detection network is a two-stage target detection network, and the target detection network obtains one or more candidate frame areas after passing through the first stage network; the corresponding image area in the original image is calculated according to the candidate frame area, and the image area is used as the feature map required for super-resolution in step 2.

2. The target detection method based on image super-resolution according to claim 1, characterized in that: If more than one candidate box area is found, the union of the image areas calculated for the multiple candidate boxes is used as the feature map.

3. The target detection method based on image super-resolution according to claim 1, characterized in that: Use certain threshold conditions to filter valid candidate boxes, such as candidate box size, position, and overlap between candidate boxes.

4. The target detection method based on image super-resolution according to claim 1, characterized in that: After obtaining the image area corresponding to the cropped candidate box, the original object detection network is used, or any other object detection network is used for subsequent object detection.

Images