Accurate and efficient ultra-large pixel scale image detection method
By adopting a hierarchical detection framework from coarse to thin in super-large pixel-scale images, combining reinforcement learning to screen areas and perform fine detection, the balance between detection accuracy and efficiency is solved, and efficient object detection is achieved.
Patent Information
- Application Number
- CN202311840677.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art is difficult to achieve a good balance between detection accuracy and efficiency when processing super-large pixel-scale images, especially in the case of large changes in the target scale, the calculation complexity and time overhead are too large, and the method based on sliding windows is used. It will waste computing resources.
Using a hierarchical detection framework from coarse to fine and detection methods with feature enhancement, coarse detection is performed using a high-power downsampling whole graph to perform coarse detection. Through the neural network agent trained by reinforcement learning, the areas that need further detection are selected and in these areas Fine detection is carried out on the top, and combined with non-maximum suppression fusion to obtain the target coordinate information of the whole image.
It effectively improves the detection accuracy of super-large pixel-scale images, reduces the waste of computing resources, and achieves the balance of detection speed and accuracy, adapts to the needs of target scale changes.
Smart Images

Figure CN119027635A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular to the field of detection of images with ultra-large pixel scale. Background Art
[0002] Object detection is a classic and challenging task in the field of computer vision, which aims to provide the location and category information of objects in images or videos. This field has many application scenarios in reality, such as security monitoring, drone inspection, autonomous driving, etc. With the development of technologies such as computational photography and semiconductor chips, the resolution of images has increased rapidly and has recently reached hundreds of millions of pixels. These images are taken by ultra-high pixel cameras, which can retain high-resolution detail information of local areas while maintaining a large field of view (covering an area of more than 1 square kilometer).
[0003] Although object detection algorithms have achieved some success on images with normal resolution, there are still two major challenges in directly applying them to process images with ultra-large pixel scales. First, facing such a large pixel scale, directly using existing methods for detection will bring great computational complexity and unacceptable time overhead; second, ultra-high pixel scale images with large fields of view have significant changes in target scale (target scale changes up to 100 times), which poses a severe challenge to achieving high-precision detection.
[0004] Initially, sliding window-based strategies were used to detect objects on images with very large pixel scales. The images were decomposed into blocks of the same size and the detection algorithm was run on each block separately to ensure high detection accuracy. However, such methods waste expensive detector computing resources on a large number of blocks containing only background, so it takes a lot of time to detect a single frame. Some recent methods for object detection on high-resolution images usually start with coarse detection on downsampled low-resolution images, and then cluster and pack the coarse detection results to generate blocks, which are then detected again. In addition, there are also methods that directly generate local blocks that may contain objects based on downsampled low-resolution images, and then use parallel computing to improve detection efficiency. However, the accuracy of these methods depends largely on the detection performance of low-resolution images, which cannot be well guaranteed. In short, it is difficult for these methods to achieve a good balance between detection accuracy and efficiency. Summary of the invention
[0005] In view of the above analysis of the prior art and the existing problems, the purpose of the present invention is to propose an accurate and efficient ultra-large pixel scale image detection method.
[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:
[0007] An accurate and efficient method for detecting images at a very large pixel scale, comprising the following steps:
[0008] Step 1: Use the detection algorithm to obtain a rough detection result on the whole image with high downsampling.
[0009] Step 2: Use a neural network agent trained with reinforcement learning to select areas that require further inspection;
[0010] Step 3: The areas that need further detection selected by the agent are enlarged to a low-multiple downsampling size, and then after local and global feature enhancement, fine detection results are obtained on these areas.
[0011] Step 4: The rough detection results With detailed detection results Perform non-maximum suppression fusion to obtain the target coordinate information of the entire image.
[0012] Compared with the existing sliding window-based detection method, this invention effectively solves the problem of low detection accuracy caused by the large-scale changes of targets in ultra-large pixel-scale images by combining a coarse-to-fine hierarchical detection framework and a detection method with feature enhancement. At the same time, combined with reinforcement learning, by adjusting the reward factor, the intelligent agent dynamically selects the area to enter the fine detection stage, while maintaining accuracy, accelerating the detection of ultra-large pixel-scale images. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 The present invention is a flow chart of the method.
[0014] Figure 2 Schematic diagram of the specific structure of the neural network agent in the embodiment of the present invention.
[0015] Figure 3 Schematic diagram of the specific structure of a detector with feature enhancement in an embodiment of the present invention. DETAILED DESCRIPTION
[0016] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0017] Reference Figure 1 , an accurate and efficient ultra-large pixel scale image detection method of this embodiment specifically includes the following steps:
[0018] Step 1: Use the detection algorithm to obtain a rough detection result on the whole image with high downsampling. The specific downsampling multiple can be adjusted according to computing resources and application scenarios. The present invention takes 8-fold downsampling as an example. The main purpose of the coarse detection here is to quickly and completely identify targets with lower detection difficulty. The obtained coarse detection results will also be used for enhanced training of neural network intelligent agents, and there is no restriction on the specific detection algorithm.
[0019] Step 2: Use a neural network agent trained by reinforcement learning to filter out areas that need further inspection. The training method of reinforcement learning is: divide the whole image with 8 times downsampling evenly into multiple sub-regions. There is no limit on the number of divided sub-regions. The present invention takes the division into 16 sub-regions as an example, and models the process of the agent selecting whether further inspection is needed according to the input sub-region map into a Markov decision process. The likelihood function of the agent's action a is as follows:
[0020] π θ (a|x)=p(1-p) (1-a)
[0021] Among them, θ represents the parameters of the neural network agent, x is the low-resolution local area map seen by the agent, and p is the probability value of the area output by the agent that needs further detection. In order to achieve a balance between detection accuracy and speed, the reward function for training the agent is designed as follows:
[0022]
[0023]
[0024] in, is the rough detection result, is the fine detection result, Y is the annotated true bounding box, Recall() represents the recall function, is the reward factor, which ranges from [0,1]. The larger the reward factor, the more likely the trained agent is to select sub-regions that can significantly improve the detection accuracy for further detection. By adjusting the reward factor, a trade-off between detection speed and accuracy can be achieved. The policy gradient algorithm is used to update the parameters of the neural network agent, and the optimization function is as follows:
[0025]
[0026] Among them, J is the gradient used to update the neural network agent, and b is the average reward value during the training process, which is used to reduce the variance of the back-propagation gradient to achieve more stable training. In order to make the agent make more reasonable decisions, the specific structure of the agent in this embodiment is as follows: the input is a local area image of H*W*3 downsampled 8 times, and the decision probability value of p∈[0,1] is output. ResNet-50 is used as the basic feature extractor, and its second stage outputs a feature map of (H / 4)*(W / 4)*256, the third stage outputs a feature map of (H / 8)*(W / 8)*512, the fourth stage outputs a feature map of (H / 16)*(W / 16)*1024, and the fifth stage outputs a feature map of (H / 32)*(W / 32)*2048. The feature maps of the four stages are respectively subjected to global average pooling, and then spliced together to obtain a vector containing multi-scale information. The vector is mapped to a single output value using a fully connected layer, and the output value is limited to between 0 and 1 using a sigmoid function. During the reasoning process, if the value output by the agent is greater than 0.5, it is considered that the sub-region needs further inspection.
[0027] Step 3: Enlarge the area that needs further detection selected by the agent. The size of the enlarged area can be adjusted according to the computing resources and application scenarios. The present invention takes the enlargement to 2 times the downsampling size as an example, and uses a detector with feature enhancement to obtain fine detection results on these areas. The specific structure of the detector with feature enhancement in this embodiment is as follows: the input is a local area image that is downsampled by 2 times of H*W*3, and the upper left corner coordinates and length and width (x, y, w, h) of the target are output. The first part uses ResNet-50 as the backbone network; the second part consists of a local enhancement module, a global long-distance dependency extraction module, and a global enhancement module. First, the local enhancement module enhances the features of the third, fourth, and fifth stages output by the backbone network to obtain the corresponding enhanced features P3, P4, and P5. The local enhancement module The structure is as follows: the first layer is a 7*7 depth convolution with a step size of 1, a pad of 3, and a dilation rate of 1; the second layer is a 7*7 depth convolution with a step size of 1, a pad of 12, and a dilation rate of 4; the third layer is a layer normalization layer; the fourth layer is a point-by-point convolution; the fifth layer is an activation layer with ReLU as the activation function; the sixth layer is a linear mapping layer with a fully connected layer for mapping; the structure of the global long-distance dependency extraction module is as follows: a multi-head self-attention layer, and the channel dimensions of Q, K, and V are 16, 16, and 32 respectively , and 1 feedforward network layer, consisting of 1 fully connected layer, 1 ReLU activation layer and 1 fully connected layer. The global long-distance dependency extraction module acts on P5 to obtain the global long-distance dependency G; the global enhancement module consists of three stages, each stage contains two sub-modules: the feature aggregation module and the feature fusion module. The feature aggregation module scales and splices the features of different stages. The feature fusion module consists of 1*1 convolution and ConvNeXt modules. In the first stage, the feature aggregation module splices the enhanced feature P5 and the global long-distance dependency G along the channel dimension, and obtains the new feature N5 through the feature fusion module. In the second stage, the feature aggregation module splices N5, G, and P4 along the channel dimension, and obtains the new feature N4 through the feature fusion module. In the third stage, the feature aggregation module splices N5, G, N4, and P3 along the channel dimension, and obtains the new feature N3 through the feature fusion module; the third part is the detection head, which consists of a classification branch and a regression branch. Each branch is stacked by 4 convolution modules, and the enhanced features are used to output the coordinate information of the target.
[0028] Step 4: The rough detection results With detailed detection results Perform non-maximum suppression fusion to obtain the target coordinate information of the entire image.
Claims
1. An accurate and efficient method for detecting images at a very large pixel scale, characterized in that: The following steps are involved: Step 1: Use the detection algorithm to obtain a rough detection result on the whole image with high downsampling. Step 2: Use a neural network agent trained with reinforcement learning to select areas that require further inspection; Step 3: The areas that need further detection selected by the agent are enlarged to a low-multiple downsampling size, and then after local and global feature enhancement, fine detection results are obtained on these areas. Step 4: The rough detection results With detailed detection results Perform non-maximum suppression fusion to obtain the target coordinate information of the entire image.
2. The accurate and efficient ultra-large pixel scale image detection method according to claim 1, characterized in that: In step 2, the specific structure of the neural network agent is: The input is a local area image downsampled 8 times of H*W*3, and the decision probability value of p∈[0,1] is output. It passes through a feature extractor in the middle. The second stage of the feature extractor outputs a feature map of (H / 4)*(W / 4)*256, the third stage outputs a feature map of (H / 8)*(W / 8)*512, the fourth stage outputs a feature map of (H / 16)*(W / 16)*1024, and the fifth stage outputs a feature map of (H / 32)*(W / 32)*2048. Global average pooling is used for the feature maps of the four stages, and then they are spliced together to obtain a vector containing multi-scale information. A fully connected layer is used to map the vector to a single output value, and the sigmoid function is used to limit the output value to between 0 and 1.
3. The accurate and efficient ultra-large pixel scale image detection method according to claim 1, characterized in that: In step 2, when the output value of the neural network agent is greater than 0.5, the current area is considered to be an area that needs further detection.
4. The accurate and efficient ultra-large pixel scale image detection method according to claim 1, characterized in that: In step 2, when the neural network agent is trained by reinforcement learning, the reward function of the training agent is designed as follows: Among them, a is the action of the agent, is the rough detection result, is the fine detection result, Y is the annotated true bounding box, Recall() represents the recall function, is the reward factor.
5. The accurate and efficient ultra-large pixel scale image detection method according to claim 4, characterized in that: The reward factor The range is [0,1], the reward factor The larger it is, the more the trained agent tends to select areas that can greatly improve the detection accuracy for further detection.
6. The accurate and efficient ultra-large pixel scale image detection method according to claim 1, characterized in that: In step 3, a detector with feature enhancement is used to achieve local and global feature enhancement. The specific structure of the detector is: the input is a downsampled local area image of H*W*3, and the output is the coordinates of the upper left corner and the length and width (x, y, w, h) of the target, including a backbone network, a local enhancement module, a global long-distance dependency extraction module, a global enhancement module and a detection head module; First, the local enhancement module enhances the feature maps of the third, fourth and fifth stages output by the backbone network to obtain the corresponding enhanced features P3, P4 and P5; the global long-distance dependency extraction module acts on the enhanced feature P5 to obtain the global long-distance dependency G; The global enhancement module includes three stages: in the first stage, the enhanced feature P5 and the global long-distance dependency G are spliced along the channel dimension and feature fusion is performed to obtain a new feature N5; in the second stage, the new feature N5, the global long-distance dependency G and the enhanced feature P4 are spliced along the channel dimension and feature fusion is performed to obtain a new feature N4; in the third stage, the new feature N5, the global long-distance dependency G, the new feature N4 and the enhanced feature P3 are spliced along the channel dimension and feature fusion is performed to obtain a new feature N3; finally, the detection head module uses the enhanced features to output the coordinate information of the target.
7. The accurate and efficient ultra-large pixel scale image detection method according to claim 6, characterized in that: The structure of the local enhancement module is as follows: the first layer is a 7*7 depth convolution with a step size of 1, a pad of 3, and a dilation rate of 1; the second layer is a 7*7 depth convolution with a step size of 1, a pad of 12, and a dilation rate of 4; the third layer is a layer normalization layer; the fourth layer is a point-by-point convolution layer; the fifth layer is an activation layer; and the sixth layer is a linear mapping layer.
8. The accurate and efficient ultra-large pixel scale image detection method according to claim 6, characterized in that: The structure of the global long-distance dependency extraction module includes: a multi-head self-attention layer and a feedforward network layer.
9. The accurate and efficient ultra-large pixel scale image detection method according to claim 6, characterized in that: Each stage of the global enhancement module includes a feature aggregation module and a feature fusion module. The feature aggregation module scales and splices the features of different stages, and the feature fusion module consists of a 1*1 convolution and a ConvNeXt module.
