Small target detection method and system based on region-of-interest super-division and storage medium

Through super-resolution reconstruction of the region of interest, the edge and detail blurring caused by insufficient resolution in small object detection is solved, the detection performance is improved, and it is suitable for infrared image processing, and the recognition ability of small objects is improved.

CN119027650APending Publication Date: 2024-11-26WU HAN XUAN YUAN ZHI JIA KE JI YOU XIAN GONG SI
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411121423.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing small object detection algorithms have blurred edges and details in object detection with small resolution, resulting in weak features and even disappearance, insufficient detection performance, especially in the field of infrared image processing, it is difficult to effectively identify small objects.

Method used

By performing super-resolution reconstruction of the region of interest (ROI), low-resolution region of interest is extracted using the Faster R-CNN algorithm, and super-resolution reconstruction algorithm such as EDSR network is applied to perform super-resolution reconstruction of the image to recover Detailed information to improve small object detection performance.

Benefits of technology

It improves the accuracy and robustness of small object detection, so that the algorithm can effectively identify small objects in more application scenarios, provide higher quality input images, and enhances the performance of target detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027650A_ABST
    Figure CN119027650A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image processing, in particular to a small target detection method and system based on region-of-interest super-division and a storage medium, and the method comprises the steps: taking an input image as a first image, and carrying out the target detection of the first image through a Faster R-CNN algorithm, extracting a target-containing region of interest in the first image as a low-resolution region of interest; performing image super-resolution reconstruction on the low-resolution region of interest by using a super-resolution reconstruction algorithm to obtain a high-resolution second image; and performing small target detection on the second image to obtain the position of the small target, and mapping the position of the small target back to the coordinate space of the first image. The small target detection performance of the algorithm can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically to a small target detection method, system and storage medium based on region of interest super-resolution. Background Art

[0002] With the popularity of deep learning, the field of computer vision has developed rapidly. Existing small target detection algorithms are emerging in an endless stream. The detection performance of small targets is improved by optimizing data, network, and loss function. However, targets with smaller resolutions have problems such as blurred edges and details. After multiple layers of convolution, their features may become weaker or even disappear. At present, there are many solutions for small target detection tasks. Although there has been a significant improvement on the original basis, the performance of small target detection is still insufficient. Small target detection is still a challenging research field.

[0003] A similar prior art includes a Chinese patent application with publication number CN116503641A, which discloses a small target detection method based on multi-scale feature fusion, comprising the following steps: collecting training images, marking the object positions in the images and labeling the object classifications, and obtaining a data set of object classifications and positions; establishing a detection network model; inputting the training images into the detection network model for training, and obtaining a trained detection network model; and inputting the image to be identified into the detection network model to obtain a detection result.

[0004] There is also a Chinese patent application with announcement number CN113222867B, which provides an image data enhancement method and system based on multiple template images, including: classifying the image data to be enhanced, and selecting N images from each type of image data as template images; performing pixel-level fusion on the template images to obtain a fused image; inputting the fused image into a segmentation network and an evaluation network respectively to obtain an initial probability distribution matrix and N initial confusion matrices; wherein the segmentation network and the evaluation network are coupled; multiplying the N initial confusion matrices element-by-element with the initial probability distribution matrix to obtain N new probability distribution matrices; calculating the cross entropy of the N new probability distribution matrices and the corresponding template images respectively, and forming a loss function with the traces of the N confusion matrices; based on the goal of minimizing the loss function, iteratively updating the parameters of the segmentation network and the evaluation network, and after reaching the preset optimization conditions, using the segmentation network to output the expanded image.

[0005] Although the above two patent applications have greatly improved the processing effect based on the original image, the small target detection performance is still insufficient. Therefore, the present invention provides a small target detection method, system and storage medium based on region of interest super-resolution to improve the algorithm's detection performance for small targets. Summary of the invention

[0006] By super-resolution reconstruction of the region of interest (ROI), the details of the ROI are restored, the detection performance of the algorithm for small targets is improved, the application scenarios of the target detection algorithm are enriched, and the robustness of the algorithm is improved. The present invention is applicable to the field of infrared image processing, especially in infrared image reconstruction and target detection applications.

[0007] To achieve the above-mentioned object of the invention, the present invention provides a small target detection method based on region of interest super-resolution, which is implemented by the following steps:

[0008] Step S1: taking an input image as a first image, performing target detection on the first image using the Faster R-CNN algorithm, and extracting a region of interest containing the target in the first image as a low-resolution region of interest, wherein the target refers to an object to be identified or detected in the first image;

[0009] Step S2: using a super-resolution reconstruction algorithm to perform image super-resolution reconstruction on the low-resolution region of interest to obtain a high-resolution second image;

[0010] Step S3: perform small target detection on the second image, obtain the position of the small target, and map the position of the small target back to the coordinate space of the first image to obtain a third image, wherein the small target refers to a target that is smaller in size than the target in the first image.

[0011] As a preferred technical solution of the present invention, in step S1, the Faster R-CNN algorithm extracts features of the first image and outputs a feature image by using a shared convolutional feature layer of a convolutional neural network.

[0012] As a preferred technical solution of the present invention, the region extraction network takes the feature image as input, moves on the feature image in a sliding window manner, and predicts multiple candidate region bounding boxes and confidence scores according to each sliding window position on the feature image.

[0013] As a preferred technical solution of the present invention, when predicting multiple candidate region bounding boxes and confidence scores respectively according to each sliding window position on the feature image, the confidence scores of each candidate region are sorted from large to small, and the three candidate regions with the highest confidence scores are obtained as target candidate regions, wherein the scales and aspect ratios of each candidate region are different.

[0014] As a preferred technical solution of the present invention, after obtaining the three candidate regions with the highest confidence scores as target candidate regions, the target candidate regions are also input into a region of interest pooling layer.

[0015] As a preferred technical solution of the present invention, the region of interest pooling layer maps the coordinates of the target candidate regions of different sizes to the first image, and captures the candidate regions on the first image and maps them to a fixed image of a fixed size, wherein loss calculation is required when predicting the coordinates and width and height of the bounding boxes of multiple target candidate regions when mapped to the size of the first image.

[0016] As a preferred technical solution of the present invention, the super-resolution reconstruction algorithm performs super-resolution reconstruction on the fixed image of a fixed size, and sends the ROI after super-resolution reconstruction to the classification and regression branch for the target classification and the bounding box regression, wherein the super-resolution reconstruction algorithm is an EDSR network.

[0017] As a preferred technical solution of the present invention, the loss calculation includes:

[0018] The reconstructed small target detection network loss function mainly includes three parts: classification loss L C , bounding box regression loss L B and super-resolution loss L S , expressed as follows:

[0019]

[0020]

[0021]

[0022] L total =L C +αL B +L S

[0023] L cls (p i )=-(1-p i ) γ log(p i )

[0024]

[0025]

[0026] Among them, L total represents the sum of all losses, N cls Indicates the number of anchors, N reg Represents the number of regression boxes, N sr represents the number of pixels, α represents the balance factor, i represents the anchor index, and p i represents the probability of a positive prediction, represents the corresponding GT prediction probability, t i represents the predicted bounding box, Indicates the GT box corresponding to the anchor predicted to be positive, L cls represents the focal loss, L reg represents the smooth L1 loss, L sr represents L1 loss, otherwise represents other situations, γ represents the focusing parameter, and n represents the number of pixels.

[0027] The present invention also provides a small target detection system based on region of interest super-resolution as described above, comprising the following modules:

[0028] an extraction unit, configured to use an input image as a first image, perform target detection on the first image using a Faster R-CNN algorithm, and extract a region of interest containing the target in the first image as a low-resolution region of interest, wherein the target refers to an object to be identified or detected in the first image;

[0029] A reconstruction unit, configured to perform image super-resolution reconstruction on the low-resolution region of interest using a super-resolution reconstruction algorithm to obtain a high-resolution second image;

[0030] A mapping unit performs small target detection on the second image, obtains the position of the small target, and maps the position of the small target back to the coordinate space of the first image to obtain a third image, wherein the small target refers to a target that is smaller in size than the target in the first image.

[0031] The present invention also provides a storage medium, wherein the storage medium stores computer executable instructions, and when the computer executable instructions are executed by a processor, the above-mentioned small target detection method based on super-resolution of a region of interest is implemented.

[0032] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0033] The technical solution of the present invention uses the Faster R-CNN algorithm to perform target detection on the first image, extracts the region of interest containing the target in the first image as a low-resolution region of interest; identifies the region in the image that may contain the target, namely the region of interest (ROI), and provides accurate positioning for subsequent target recognition. Use a super-resolution reconstruction algorithm to perform image super-resolution reconstruction on the low-resolution region of interest to obtain a high-resolution second image; by utilizing the local details and global context information of the image, more details and clarity are restored, making the target visually clearer and easier to identify, providing a higher quality input image for subsequent target detection, thereby improving the accuracy of target detection. Perform small target detection on the second image, obtain the position of the small target, and map the position of the small target back to the coordinate space of the first image, so that the final small target detection result can be seamlessly connected and integrated with the first image.

[0034] Through the coordination of the above steps, the target detection network structure is redesigned, and the detection performance of the algorithm for small targets is improved by super-resolution reconstruction of the region of interest (ROI), which improves the robustness of the algorithm and makes the method applicable to more application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0036] Figure 1 It is a flowchart of the steps of the small target detection method based on super-resolution of the region of interest in the present invention;

[0037] Figure 2 This is a structural diagram of the small target detection system based on region of interest super-resolution in the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0040] With the popularity of deep learning, the field of computer vision has developed rapidly. Existing small target detection algorithms are emerging in an endless stream. The detection performance of small targets is improved by optimizing data, network, and loss function. However, targets with smaller resolutions have problems such as blurred edges and details. After multiple layers of convolution, their features may become weaker or even disappear. At present, there are many solutions for small target detection tasks. Although there has been a significant improvement on the original basis, the performance of small target detection is still insufficient.

[0041] In view of the above technical problems, the present invention proposes Figure 1 The small target detection method based on region of interest super-resolution shown in the figure includes the following steps:

[0042] Step S1: Take the input image as the first image, use the Faster R-CNN algorithm to perform target detection on the first image, and extract the region of interest containing the target in the first image as the low-resolution region of interest, wherein the target refers to the object that needs to be identified or detected in the first image.

[0043] Specifically, in complex images, targets may appear at any position and may overlap or be confused with other objects. The Faster R-CNN algorithm can effectively identify areas in the image that may contain targets, namely, regions of interest (ROIs), providing accurate positioning for subsequent target recognition. Moreover, Faster R-CNN only performs subsequent processing (such as super-resolution reconstruction, target classification, etc.) on the extracted regions of interest, rather than processing the entire image, which can greatly reduce the amount of calculation and improve the processing speed. In addition, in real-world applications, the scale of the target may also change greatly. Faster R-CNN can process targets of different scales through the region extraction network (RPN) and ROI pooling layer, and extract regions of interest of corresponding scales. The scales can be 32×32, 64×64, 128×128, 256×256, 512×512, etc., which will be described in detail below. At the same time, when extracting the region of interest, the interference of background information can also be reduced, improving the accuracy of target detection. For example, a surveillance image containing multiple pedestrians is being processed. The background of this image is complex, and the scales, postures and positions of the pedestrians are different. If the traditional target detection algorithm is used directly, it may be difficult to accurately identify all the pedestrians. Through the above method, the input surveillance image is used as the first image, and the Faster R-CNN algorithm is used to detect the target. The RPN network of Faster R-CNN will generate a large number of candidate regions (i.e., regions of interest) on the image. Each candidate region contains a possible target. By screening and classifying these candidate regions, the regions of interest containing pedestrians are extracted. These regions of interest usually contain most of the information about the pedestrians and remove most of the background information, so as to facilitate subsequent processing of the extracted regions of interest, such as super-resolution reconstruction, target classification, etc.

[0044] Step S2: Use a super-resolution reconstruction algorithm to perform image super-resolution reconstruction on the low-resolution region of interest to obtain a high-resolution second image.

[0045] Specifically, the region of interest containing the target in the first image is extracted and belongs to a low-resolution image. In the low-resolution image, the target loses a lot of details due to pixel limitations, which makes subsequent recognition or detection difficult. In order to solve the problem of target detail loss or difficulty in recognition caused by insufficient image resolution, these low-resolution regions of interest are reconstructed by super-resolution through a super-resolution reconstruction algorithm. The local details and global context information of the image can be used to restore more details and clarity, making the target visually clearer and easier to identify, providing a higher quality input image for subsequent target detection, thereby improving the algorithm's detection performance for small targets.

[0046] Step S3: perform small target detection on the second image, obtain the position of the small target, and map the position of the small target back to the coordinate space of the first image to obtain a third image, wherein the small target refers to a target that is smaller than the target in the first image.

[0047] Specifically, a small target refers to a target that is smaller than the target in the first image. The small target is a small part of the target and is smaller in size. For example, the pixel area of ​​the small target can range from 32*32 to 256*256, and the feature information contained is also less. It is more difficult for traditional target detection algorithms to accurately identify and locate it. Through the super-resolution reconstruction of step S2 above, the small target is enlarged in size, the detail information is richer, and the position of the small target can be accurately located, which solves the problem of detection difficulty caused by the small size of the target in the original image (first image) and significantly improves the accuracy of small target detection. Through coordinate space mapping, the position of the small target in the second image is mapped back to the first image, so that the final target detection result can be seamlessly connected and fused with the first image to obtain a third image after the reconstruction of the first image.

[0048] For example, when processing a macro photography image containing multiple vehicles, due to the relatively small size of the vehicles, the traditional target detection algorithm may not be able to accurately identify and locate these vehicles. Use the Faster R-CNN algorithm to perform target detection on the original image (first image) and extract the region of interest containing the vehicle as a low-resolution region of interest; apply super-resolution reconstruction algorithms such as SRCNN or ESRGAN to these low-resolution regions of interest to reconstruct them into a high-resolution second image. In the reconstructed image, the size of the vehicle is magnified and the detail information is richer; perform small target detection on the second image and use a detection algorithm specifically for small targets (such as YOLOv4-tiny or SSD, etc.) to accurately identify and locate the position of the vehicle. Since the vehicle is clearer and easier to identify in the high-resolution image, the detection algorithm can complete this task more accurately; map the detected vehicle position back to the coordinate space of the original image, which can be achieved by recording the position and size of the region of interest in the first image and converting the coordinates in the second image accordingly. Finally, mark the detected vehicle position on the original image.

[0049] Regarding mapping the position of the small target back to the coordinate space of the first image, first determine that the small target and the first image are in a reduced relationship, calculate the coordinate transformation parameters from the small target to the first image, and apply the inverse transformation to map the position of the small target in the second image back to the coordinate space of the first image. After the coordinate transformation, it may be necessary to use remapping technology to ensure that the mapped coordinates are integers and correspond to valid pixel positions in the first image. Interpolation algorithms such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation may also be involved to calculate pixel values ​​at non-integer coordinates. Finally, the position of the small target is mapped back to the coordinate space of the first image to achieve reconstruction of the first image.

[0050] Through the coordination of the above steps, the target detection network structure is redesigned, and the detection performance of the algorithm for small targets is improved by super-resolution reconstruction of the region of interest (ROI), which improves the robustness of the algorithm and makes the method applicable to more application scenarios.

[0051] Furthermore, in the above step S1, the Faster R-CNN algorithm extracts features of the first image and outputs a feature image by using a shared convolutional feature layer of a convolutional neural network.

[0052] Specifically, traditional target detection methods are difficult to adapt to complex image changes. Through convolutional neural networks (CNN), Faster R-CNN can automatically learn and extract deep features in images. Faster R-CNN uses a shared convolutional neural network to first extract features from the entire input image to obtain a feature image (FeatureMap). This feature image is then shared by the candidate region network (RPN) and the Fast R-CNN detector, avoiding repeated feature extraction processes, thereby greatly improving the detection speed and providing a solid foundation for subsequent target detection tasks. Faster R-CNN is a deep neural network architecture for object detection, which is further developed on the basis of R-CNN and Fast R-CNN. The core innovation of Faster R-CNN is the introduction of Region Proposal Network (RPN), a fully convolutional network that can simultaneously predict the location of the object's bounding box and the score of whether each location is an object, thereby greatly reducing the time overhead of candidate box calculation.

[0053] Furthermore, the region extraction network takes the feature image as input, moves on the feature image in a sliding window manner, and predicts multiple candidate region bounding boxes and confidence scores according to each sliding window position on the feature image.

[0054] Specifically, the Region Proposal Network (RPN) is a key component in the Faster R-CNN algorithm. The region proposal network takes the feature image as input and predicts the candidate region bounding box directly on the feature image by using a shared convolutional feature layer through a sliding window, which greatly improves the efficiency of candidate region generation. RPN predicts multiple candidate region bounding boxes of different scales and ratios for each sliding window position and gives a confidence score for each bounding box. It effectively solves the problems of low candidate region generation efficiency, many redundant candidate regions, separation from subsequent tasks, and lack of multi-scale and multi-aspect ratio adaptability in traditional target detection methods. The solution of these problems enables Faster R-CNN to achieve faster detection speed while maintaining high accuracy. When the target may present different scales and aspect ratios in the image, RPN presets multiple anchors of different scales and ratios, so that the algorithm can predict candidate region bounding boxes with different scales and aspect ratios, which helps to improve the robustness of target detection, thereby improving the algorithm's detection performance for small targets.

[0055] Furthermore, when predicting multiple candidate region bounding boxes and confidence scores respectively according to each sliding window position on the feature image, the confidence scores of each candidate region are sorted from large to small, and the three candidate regions with the highest confidence scores are obtained as the target candidate regions, wherein the scales and aspect ratios of each candidate region are different.

[0056] Specifically, since the position and size of the target in the image may vary greatly, RPN uses anchors with different scales and aspect ratios. For example, common anchor scales may include 32×32, 64×64, 128×128, 256×256, 512×512, etc. (Note: different scales may be used in the actual algorithm). These scales are designed to cover different target sizes from small to large. Candidate regions with different scales and aspect ratios can be predicted. During the prediction process, RPN generates a large number of candidate region bounding boxes for each sliding window position. However, not all candidate regions contain targets or have high confidence. Therefore, by selecting only the three candidate regions with the highest confidence scores, it can be ensured that these regions are more likely to be the locations of real targets, and the false detection and missed detection caused by low-quality candidate regions are reduced, while reducing the computational complexity of subsequent classification and regression tasks, thereby improving the overall target detection efficiency. This makes Faster R-CNN more practical and robust while maintaining high accuracy.

[0057] Furthermore, after obtaining the three candidate regions with the highest confidence scores as target candidate regions, the target candidate regions are also input into the region of interest pooling layer.

[0058] Specifically, the ROI pooling layer is performed after the shared convolution feature layer. It uses the feature image that has been calculated and only performs pooling operations on the target candidate area, avoiding repeated calculations. The ROI pooling layer maintains the spatial position information of the target candidate area on the feature image, so that subsequent classification and regression tasks can accurately locate the position of the target in the original image. At the same time, the ROI pooling layer divides the corresponding area of ​​each target candidate area on the feature image, performs the maximum pooling operation on each sub-area, and converts target candidate areas of different sizes into fixed-size feature vectors, so that they can be compatible with subsequent fully connected layers, classifiers, and regressors, thereby completing small target detection tasks.

[0059] Furthermore, the region of interest pooling layer maps the coordinates of target candidate regions of different sizes to the first image, and captures the candidate regions on the first image and maps them into a fixed image of a fixed size, wherein loss calculation is required when predicting the coordinates and width and height of the bounding boxes of multiple target candidate regions when mapped to the size of the first image.

[0060] Specifically, due to the different sizes and shapes of the candidate regions, the feature sizes extracted directly from these regions may be inconsistent and cannot be directly input into the subsequent fully connected layer or classifier. Therefore, the ROI pooling layer maps candidate regions of different sizes to a fixed map of fixed size to achieve standardization of feature sizes, so that the subsequent network layers can process these fixed-size features. At the same time, in target detection, it is necessary not only to identify the category of the target, but also to determine the exact position of the target in the image. If the candidate region is directly scaled or cropped, its original spatial position information may be lost. The ROI pooling layer can maintain the spatial position information of the candidate region in the original image through coordinate mapping and interception operations, making subsequent classification and positioning more accurate. By calculating the loss between the predicted bounding box and the true bounding box (such as smooth L1 loss), this difference can be quantified and used to guide the training process of the model. Loss calculation can not only help us understand the performance of the model, but also optimize the model parameters through the back propagation algorithm to improve the performance of small target detection.

[0061] Furthermore, the super-resolution reconstruction algorithm performs super-resolution reconstruction on a fixed image of a fixed size, and sends the super-resolution reconstructed ROI to the classification and regression branches for target classification and bounding box regression, wherein the super-resolution reconstruction algorithm is an EDSR (Efficient Sub-Pixel Convolutional Neural Network) network.

[0062] Specifically, low-resolution fixed images may lose some important details and affect detection performance. The EDSR network can improve the resolution of fixed images by super-resolution reconstruction of fixed images of fixed size, which helps to retain more details. When processing fixed images, traditional convolutional neural networks may cause the resolution of fixed images to decrease due to pooling operations or convolution step sizes, thereby reducing the feature representation capability. The EDSR network uses sub-pixel convolution technology to directly generate high-resolution fixed images through convolution operations without introducing additional parameters. This technology not only improves the resolution of fixed images, but also enhances the feature representation capability, allowing the model to better capture the details in the image. At the same time, when performing target classification and bounding box regression on low-resolution fixed images, classification errors or inaccurate positioning may occur due to insufficient information. Super-resolution reconstruction of fixed images by the EDSR network can improve the resolution and feature representation capability of fixed images, so that subsequent classification and regression tasks can be performed on higher-resolution fixed images, thereby improving the accuracy of classification and the accuracy of bounding box regression.

[0063] Furthermore, the above loss calculation includes:

[0064] The reconstructed small target detection network loss function mainly includes three parts: classification loss L C , bounding box regression loss L B and super-resolution loss L S , expressed as follows:

[0065]

[0066]

[0067]

[0068] L total =L C +αL B +L S

[0069] L cls (p i )=-(1-p i ) γ log(p i )

[0070]

[0071]

[0072] Among them, L totalrepresents the sum of all losses, N cls Indicates the number of anchors, N reg Represents the number of regression boxes, N sr represents the number of pixels, α represents the balance factor, i represents the anchor index, and p i represents the probability of a positive prediction, represents the corresponding GT prediction probability, t i represents the predicted bounding box, Indicates the GT box corresponding to the anchor predicted to be positive, L cls represents the focal loss, L reg represents the smooth L1 loss, L sr represents L1 loss, otherwise represents other situations, γ represents the focusing parameter, and n represents the number of pixels.

[0073] Specifically, in object detection, the classifier needs to accurately identify the category of the object in the image. The classification loss is used to measure the difference between the target category predicted by the model and the true category. By minimizing the classification loss, the model can learn how to more accurately identify the target category in the image. In addition to identifying the category of the target, object detection also needs to determine the precise location of the target in the image. The bounding box regression loss is used to measure the difference between the bounding box predicted by the model and the true bounding box. By minimizing the bounding box regression loss, the model can learn how to more accurately predict the bounding box of the target, thereby improving the positioning accuracy. In object detection, the use of super-resolution techniques (such as EDSR) can improve the resolution of the fixed map, thereby improving the detection performance of small objects. However, the super-resolution reconstruction process may introduce errors. The super-resolution loss is used to measure the difference between the reconstructed high-resolution fixed map and the original high-resolution fixed map (or some reference standard), and by minimizing the super-resolution loss, it is ensured that the error generated by the super-resolution reconstruction process is as small as possible to improve the performance of small object detection.

[0074] The loss function is designed to simultaneously optimize the performance of classification, localization and super-resolution reconstruction in target detection, thereby improving the overall target detection accuracy.

[0075] The present invention also provides a Figure 2 The small target detection system based on region of interest super-resolution shown in the figure includes the following modules:

[0076] An extraction unit, configured to use the input image as a first image, perform target detection on the first image using a Faster R-CNN algorithm, and extract a region of interest containing a target in the first image as a low-resolution region of interest, wherein the target refers to an object to be identified or detected in the first image;

[0077] A reconstruction unit, configured to perform image super-resolution reconstruction on a low-resolution region of interest using a super-resolution reconstruction algorithm to obtain a high-resolution second image;

[0078] The mapping unit detects small targets in the second image, obtains the positions of the small targets, and maps the positions of the small targets back to the coordinate space of the first image to obtain a third image, wherein the small targets refer to targets that are smaller in size than the targets in the first image.

[0079] The present invention also provides a computer storage medium, which stores program instructions. When the program instructions are executed, the device where the computer storage medium is located is controlled to execute the above-mentioned small target detection method based on super-resolution of the region of interest.

[0080] In summary, the present invention uses the Faster R-CNN algorithm to perform target detection on the first image, extracts the region of interest containing the target in the first image as a low-resolution region of interest; identifies the region in the image that may contain the target, namely the region of interest (ROI), and provides accurate positioning for subsequent target recognition. Use a super-resolution reconstruction algorithm to perform image super-resolution reconstruction on the low-resolution region of interest to obtain a high-resolution second image; by utilizing the local details and global context information of the image, more details and clarity are restored, making the target visually clearer and easier to identify, providing a higher quality input image for subsequent target detection, thereby improving the accuracy of target detection. Perform small target detection on the second image, obtain the position of the small target, and map the position of the small target back to the coordinate space of the first image, so that the final small target detection result can be seamlessly connected and integrated with the first image.

[0081] Through the coordination of the above steps, the target detection network structure is redesigned, and the detection performance of the algorithm for small targets is improved by super-resolution reconstruction of the region of interest (ROI), which improves the robustness of the algorithm and makes the method applicable to more application scenarios.

[0082] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0083] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The above-mentioned program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0084] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0086] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A small target detection method based on region of interest super-resolution, characterized in that: The method comprises the following steps: Step S1: taking an input image as a first image, performing target detection on the first image using the Faster R-CNN algorithm, and extracting a region of interest containing the target in the first image as a low-resolution region of interest, wherein the target refers to an object to be identified or detected in the first image; Step S2: using a super-resolution reconstruction algorithm to perform image super-resolution reconstruction on the low-resolution region of interest to obtain a high-resolution second image; Step S3: perform small target detection on the second image, obtain the position of the small target, and map the position of the small target back to the coordinate space of the first image to obtain a third image, wherein the small target refers to a target that is smaller in size than the target in the first image.

2. The method according to claim 1, characterized in that In step S1, the Faster R-CNN algorithm extracts features of the first image by using a shared convolutional feature layer of a convolutional neural network and outputs a feature image.

3. The method according to claim 2, characterized in that The region extraction network takes the feature image as input, moves on the feature image in a sliding window manner, and predicts multiple candidate region bounding boxes and confidence scores according to each sliding window position on the feature image.

4. The method according to claim 3, characterized in that When predicting multiple candidate region bounding boxes and confidence scores respectively according to each sliding window position on the feature image, the confidence scores of the candidate regions are sorted from large to small, and the three candidate regions with the highest confidence scores are obtained as target candidate regions, wherein the scales and aspect ratios of the candidate regions are different.

5. The method according to claim 4, characterized in that After obtaining the three candidate regions with the highest confidence scores as target candidate regions, the target candidate regions are also input into a region of interest pooling layer.

6. The method according to claim 5, characterized in that The region of interest pooling layer maps the coordinates of the target candidate regions of different sizes to the first image, and captures the candidate regions on the first image and maps them into a fixed image of a fixed size, wherein loss calculation is required when predicting the coordinates and width and height of the bounding boxes of multiple target candidate regions when mapped to the size of the first image.

7. The method according to claim 6, characterized in that The super-resolution reconstruction algorithm performs super-resolution reconstruction on the fixed image of a fixed size, and sends the super-resolution reconstructed ROI to the classification and regression branch for the target classification and the bounding box regression, wherein the super-resolution reconstruction algorithm is an EDSR network.

8. The method according to claim 6, characterized in that The loss calculation includes: The reconstructed small target detection network loss function mainly includes three parts: classification loss L C , bounding box regression loss L B and super-resolution loss L S , expressed as follows: L total =L C +αL B +L S L cls (p i )=-(1-p i ) γ log(p i ) Among them, L total represents the sum of all losses, N cls Indicates the number of anchors, N reg Represents the number of regression boxes, N sr represents the number of pixels, α represents the balance factor, i represents the anchor index, and p i represents the probability of a positive prediction, represents the corresponding GT prediction probability, t i represents the predicted bounding box, Indicates the GT box corresponding to the anchor predicted to be positive, L cls represents the focal loss, L reg represents the smooth L1 loss, L sr represents L1 loss, otherwise represents other situations, γ represents the focusing parameter, and n represents the number of pixels.

9. A small target detection system based on region of interest super-resolution, used to implement the method according to any one of claims 1 to 8, characterized in that: The system includes the following modules: an extraction unit, configured to use an input image as a first image, perform target detection on the first image using a Faster R-CNN algorithm, and extract a region of interest containing the target in the first image as a low-resolution region of interest, wherein the target refers to an object to be identified or detected in the first image; A reconstruction unit, configured to perform image super-resolution reconstruction on the low-resolution region of interest using a super-resolution reconstruction algorithm to obtain a high-resolution second image; A mapping unit performs small target detection on the second image, obtains the position of the small target, and maps the position of the small target back to the coordinate space of the first image to obtain a third image, wherein the small target refers to a target that is smaller in size than the target in the first image.

10. A computer storage medium, characterized in that: The storage medium stores program instructions, wherein when the program instructions are executed, the device where the storage medium is located is controlled to execute the small target detection method based on region of interest super-resolution according to any one of claims 1 to 8.

Citation Information

Cited By

  • Remote sensing image fine-grained target identification method based on local multi-scale super-division

    CN120808177A

  • A remote sensing image fine-grained target recognition method based on local multi-scale super-resolution

    CN120808177B