High-resolution image target detection acceleration method and system based on image block combination

Abstract

A high-resolution image target detection acceleration method based on image block combination, after extracting the image block sequence from the high-resolution image in the offline stage, ① the image block sequence is used as the state, ② the image block arrangement is used as the action, and ③ the target detection accuracy of the synthesized sub-image is used as the reward, and the policy network is trained using the reinforcement learning method; in the online stage, the trained policy network is used to obtain the corresponding combination strategy according to the input image block sequence, and the sub-image is synthesized based on the combination strategy, and the target detection result is obtained by mapping back to the original image to achieve accelerated detection. The present invention reduces the number of times the target detection algorithm is run, and ensures that the accuracy of target detection remains at a high level while speeding up the algorithm operation speed.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, in particular to a high-resolution image target detection acceleration method and system based on image block combination. Background Art

[0002] As the resolution of image acquisition devices continues to increase, efficient analysis of high-resolution images becomes increasingly important.

[0003] In 2019, Fan Yang et al. proposed a method to accelerate the detection of targets by only performing target detection on some image blocks instead of the entire image in the paper “Clustered object detection in aerial images” published in Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 8311 to 8320. This method requires the introduction of a large number of image blocks to improve the accuracy of target detection in the process of processing high-resolution images, and the presence of a large number of image blocks will slow down the speed of target detection. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a high-resolution image target detection acceleration method and system based on image block combination, which combines image blocks of high-resolution images into sub-images for target detection. Compared with directly performing target detection on image blocks, the present invention reduces the number of times the target detection algorithm is run, thereby speeding up the algorithm operation speed while ensuring that the accuracy of target detection remains at a high level.

[0005] The present invention is achieved through the following technical solutions:

[0006] The present invention relates to a high-resolution image target detection acceleration method based on image block combination. After extracting an image block sequence from a high-resolution image in an offline stage, a policy network is trained using a reinforcement learning method, with ① the image block sequence as a state, ② the image block arrangement as an action, and ③ the target detection accuracy of a synthesized sub-image as a reward. In an online stage, the trained policy network is used to obtain a corresponding combination strategy according to an input image block sequence, and a sub-image is synthesized based on the combination strategy. After the target detection result is obtained, the accelerated detection is achieved by mapping it back to the original image.

[0007] The image block sequence refers to: after downsampling the high-resolution image, a number of image blocks are extracted using a target detection algorithm, and the number of targets contained in the image blocks is estimated. Several image blocks belonging to the same image are arranged in descending order according to the number of targets contained therein to form an image block sequence.

[0008] The target detection algorithm refers to an algorithm that outputs the category information and location information of the target in a given image, and is implemented by the method described in, but not limited to, Ren S, He K, Girshick R, et al. Faster r-cnn: Towards real-time object detection with region proposal networks [J]. Advances in neural information processing systems, 2015, 28.

[0009] The image block arrangement refers to different image block arrangements obtained by performing arbitrary exchange operations on the image blocks in a given image block sequence. This arrangement determines the order in which each image block is considered when synthesizing sub-images, that is, it corresponds to different sub-image synthesis methods.

[0010] The synthesis means: according to the arrangement order of the image blocks, one image block is taken out each time and placed in a blank sub-image template. While ensuring that there is no overlap between the image blocks in the sub-image, each image block preferentially occupies the space in the lower left corner of the sub-image. When the current image block exceeds the boundary of the sub-image, a new blank sub-image is created to place the current image block. This process is performed until all image blocks are placed in the sub-image.

[0011] The accuracy of target detection is a common objective quality evaluation indicator for target detection: mean average precision (mAP).

[0012] The strategy network includes: an encoder and a decoder, wherein: the encoder extracts image block size and content features from an input image block sequence, and the decoder outputs an image block combination strategy according to the image block features.

[0013] The training is the process of optimizing the parameters of the policy network using reinforcement learning methods based on the state, action, reward and preset network structure, which specifically includes:

[0014] Step 1) Initialization: Randomly initialize the policy network parameters θ; initialize the trajectory set T to an empty set; initialize the number of sampling times N, the learning rate α, and the number of iterations L.

[0015] Step 2) Collect training data: Take the image block sequence S i is the state, the arrangement of image blocks is the action, and the target detection accuracy of the synthesized sub-image is the reward R i , with the current policy network with fixed parameters as θ as the policy, execute the Markov decision process to obtain the combined image block sequence τ i =(v 0 , v 1 , v 2 , ...), where v j represents a single image block, and τ i Add the trajectory set T.

[0016] Step 3) Update network parameters: When the size of the trajectory set T |T| = N, update θ according to the policy gradient algorithm: Calculating average reward Clear the trajectory set T and jump to step 4. When the size of the trajectory set T |T|<N, jump to step 2.

[0017] Step 4) Determine whether the policy network converges: When calculating the average reward If there is no increase during L consecutive iterations, terminate the training, otherwise jump to step 2.

[0018] The mapping refers to the process of mapping the target detection result in the sub-image coordinate system to the original image coordinate system according to the position of the image block in the sub-image in the original image.

[0019] The present invention relates to an image target detection system for implementing the above method, comprising: an image block extraction module, a policy network training module and an image block combination module, wherein: the image block extraction module is connected to the policy network training module and transmits the image blocks extracted by it, and the policy network module is connected to the image block combination module and transmits the trained policy network.

[0020] Technical Effects

[0021] The present invention models image block synthesis as a Markov decision process with an image block sequence as a state, an image block arrangement as an action, and a target detection accuracy of a synthesized sub-image as a reward. The synthesis of image blocks is completed by executing the process, using a strategy network to model the strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of the present invention;

[0023] Figure 2 It is the structural diagram of the strategy network;

[0024] Figure 3 Schematic diagram of image block combination;

[0025] Figure 4 Detection schematic diagram. DETAILED DESCRIPTION

[0026] like Figure 1 As shown, this embodiment relates to a high-resolution image target detection acceleration method based on image block combination, including:

[0027] Step 1) extracting image blocks using a target detection algorithm, specifically including:

[0028] 1-1) The high-resolution image is downsampled by 4 times, and the rough detection box is extracted using the Faster R-CNN algorithm. Then, the detection box is clustered using the Mean Shift algorithm. The clustering bandwidth is set to 0.025, and the minimum bounding rectangle of the detection box of each category is used as the image block.

[0029] 1-2) According to each image block v i The number and area of ​​the detection boxes in the image are used to estimate the number of targets in each image block. i and the average area n of the object in the image block i . Using the coordinates of the center point of the image block (x i ,y i ), width w i , long h i , aspect ratio r i , image block area a i , the average area of ​​the object in the image block o i , the target number of image blocks n i , image patch v i It can be expressed as (x i ,y i , w i ,h i , r i , a i , o i , n i ).

[0030] 1-3) Several image blocks belonging to the same image are arranged in descending order according to the number of detection frames they contain to obtain an image block sequence S = (v 1 , v 2 , v 3 , ..., v n ).

[0031] Step 2) Training Figure 2 The strategy network shown includes an encoder and a decoder, specifically including:

[0032] 2-1) Initialization: Randomly initialize the strategy network parameters θ; initialize the trajectory set T to an empty set; initialize the subgraph set V to an empty set; initialize a blank subgraph C with a length of H and a width of W 1 ; Initialize trajectory τ 1 Initialize sampling times N, learning rate α, and iteration times L. In this embodiment, H is set to 1000, W is set to 1000, N is set to 64, α is set to 0.001, and L is set to 5.

[0033] The encoder includes: a fully connected layer for extracting image block features, and a long short-term memory (LSTM) network for encoding the sequential relationship of image blocks; the decoder includes: a long short-term memory network for sequential decoding and an attention head for selecting image blocks, and the attention head is implemented in the manner described in Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. arXiv preprint arXiv:1506.03134, 2015.

[0034] 2-2) Collect training data: For the i-th image in the training set, generate the image block sequence S as described in step 1) i , the policy network takes the image block sequence S i As input, output an image block number j.

[0035] 2-3) According to sequence S i The number in gets the image block v corresponding to the current sequence number j j , the image block v j Insert into current subgraph C k In order to ensure v j With C k When there are already image blocks that do not overlap, v j Priority to occupy C k The space in the lower left corner, when v j Beyond C k The boundary of v j Take it out, and then put subgraph C k Add subgraph set V and create a new blank subgraph C k+1 , with C k+1 As the current subgraph, jump to 2-3. Otherwise, v j Put in C k Inside, v j Add trajectory τ i Inside, skip to 2-4.

[0036] 2-4) When the trajectory length |τ i| and sequence length |S i | are equal, that is, the generated sorting method has taken all image blocks into consideration, then the current sub-image C k Add sub-graph set V, use but not limited to FasterR-CNN algorithm to detect objects in all sub-graphs in V, and calculate the accuracy as reward R i , clear the subgraph set V, and set the trajectory τ i Add to the trajectory set T and jump to 2-5. Otherwise, add the image block v j Input the decoder of the policy network, decode to get the next image block number j+1, and use j+1 as the current image block number, jumping to 2-3.

[0037] 2-5) Update network parameters: When the size of the trajectory set T |T| = N, the policy gradient algorithm is used to update θ: Calculating average reward And clear the trajectory set T, jump to 2-6. When the size of the trajectory set T |T|<N, jump to 2-2.

[0038] 2-6) Determine whether the strategy network converges: When calculating the average reward If there is no growth during L consecutive iterations, terminate; otherwise, jump to 2-2.

[0039] Step 3) combining image blocks, specifically including:

[0040] 3-1) Initialization: Load the policy network parameters θ trained in step 2), initialize the sub-image set V to an empty set; for the test image in the offline stage, initialize the image block sequence S according to the method described in 1). Initialize a blank sub-image C with a length of H and a width of W 1 In this embodiment, H is set to 1000 and W is set to 1000.

[0041] 3-2) Execute action: The policy network takes the image block sequence S as input and outputs an image block number j.

[0042] 3-3) Get the current image block v according to the number in the sequence S j , the image block v j Insert into current subgraph C k In order to ensure v j With C k When there are already image blocks that do not overlap, v j Priority to occupy C k The space in the lower left corner, when v j Beyond C k The boundary of v j Take it out, and then put subgraph C k Add subgraph set V and create a new blank subgraph Ck+1 , with C k+1 As the current subgraph, jump to 3-3. Otherwise, v j Put in C k Inside, skip to 3-4.

[0043] 3-4) When the current image block v j If the subscript j is equal to the sequence length |S|, that is, the generated sorting method has taken all image blocks into consideration, then the current sub-image C k Add the sub-image set V, use the FasterR-CNN algorithm to detect objects in all sub-images in V, and jump to 3-5. Otherwise, add the image block v j Input the decoder of the policy network, decode to get the next image block number j+1, use j+1 as the current image block number, and jump to 3-3.

[0044] 3-5) Coordinate mapping: According to the position of the image block in the sub-image in the original image, the position information of the target in the sub-image is mapped to the original image to obtain the final detection result.

[0045] After specific practical experiments, in the Ubuntu16.04 operating system, a single GTX 1080Ti graphics card environment, a whole system was built based on the open source framework Pytorch, and training and testing were carried out on the billion-pixel object detection dataset PANDA. The evaluation index mAP was calculated when IOU=0.5. The experimental data that can be obtained are shown in Table 1.

[0046] Table 1 Experimental results

[0047] Down Sample+Sliding Window This embodiment mAP 0.715 0.715 Number of target detection runs 15019 3671 Speed ​​(pictures / second) 0.07 0.23

[0048] Compared with the existing technology of downsampling high-resolution images and then performing sliding window block detection (Down Sample + Sliding Window), the performance indicators of this method are improved in that: under similar target detection accuracy (mAP), the target detection speed of this method is 3.2 times that of the original one. This method combines a large number of image blocks into a small number of sub-images and only performs target detection on a small number of sub-images, which significantly reduces the number of target detection runs, thereby reducing the average detection time of each high-resolution image.

[0049] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A high-resolution image target detection acceleration method based on image block combination, characterized in that: After extracting the image block sequence from the high-resolution image in the offline stage, the policy network is trained using the reinforcement learning method, with ① the image block sequence as the state, ② the image block arrangement as the action, and ③ the target detection accuracy of the synthesized sub-image as the reward. In the online stage, the trained policy network is used to obtain the corresponding combination strategy according to the input image block sequence, and the sub-image is synthesized based on the combination strategy. After the target detection result is obtained, it is mapped back to the original image to achieve accelerated detection.

2. The high-resolution image target detection acceleration method based on image block combination according to claim 1 is characterized in that: The image block sequence is constructed by downsampling a high-resolution image, extracting a number of image blocks using a target detection algorithm, and estimating the number of targets contained in the image blocks. The image blocks belonging to the same image are arranged in descending order according to the number of targets contained therein.

3. The high-resolution image target detection acceleration method based on image block combination according to claim 1 is characterized in that: The image block arrangement refers to: for a given image block sequence, different image block arrangements are obtained by performing any exchange operation on the image blocks therein. This arrangement determines the order in which each image block is considered when synthesizing sub-images, that is, it corresponds to different sub-image synthesis methods.

4. The high-resolution image target detection acceleration method based on image block combination according to claim 1 is characterized in that: The synthesis means: according to the arrangement order of the image blocks, one image block is taken out each time and placed in a blank sub-image template. While ensuring that there is no overlap between the image blocks in the sub-image, each image block preferentially occupies the space in the lower left corner of the sub-image. When the current image block exceeds the boundary of the sub-image, a new blank sub-image is created to place the current image block. This process is performed until all image blocks are placed in the sub-image.

5. The high-resolution image target detection acceleration method based on image block combination according to claim 1 is characterized in that: The accuracy of target detection is a common objective quality evaluation indicator for target detection: the average accuracy of the category.

6. The high-resolution image target detection acceleration method based on image block combination according to claim 1 is characterized in that: The strategy network includes: an encoder and a decoder, wherein: the encoder extracts image block size and content features from an input image block sequence, and the decoder outputs an image block combination strategy according to the image block features.

7. The high-resolution image target detection acceleration method based on image block combination according to claim 6 is characterized in that: The encoder includes: a fully connected layer for extracting image block features, and a long short-term memory network for encoding the sequential relationship of image blocks; the decoder includes: a long short-term memory network for sequential decoding and an attention head for selecting image blocks.

8. The high-resolution image target detection acceleration method based on image block combination according to claim 1 is characterized in that: The training is the process of optimizing the parameters of the policy network using reinforcement learning methods based on the state, action, reward and preset network structure, which specifically includes: Step 1) Initialization: Randomly initialize the policy network parameters θ; initialize the trajectory set T to an empty set; initialize the number of sampling times N, the learning rate α, and the number of iterations L; Step 2) Collect training data: Take the image block sequence S i is the state, the arrangement of image blocks is the action, and the target detection accuracy of the synthesized sub-image is the reward R i , with the current policy network with fixed parameters as θ as the policy, execute the Markov decision process to obtain the combined image block sequence τ i =(v0, v1, v2, ...), where v j represents a single image block, and τ i Add trajectory set T; Step 3) Update network parameters: When the size of the trajectory set T |T| = N, update θ according to the policy gradient algorithm: Calculating average reward And clear the trajectory set T and jump to step 4; when the size of the trajectory set T |T|<N, jump to step 2; Step 4) Determine whether the policy network converges: When calculating the average reward If there is no increase during L consecutive iterations, terminate the training, otherwise jump to step 2.

9. The high-resolution image target detection acceleration method based on image block combination according to claim 1 is characterized in that: The mapping refers to the process of mapping the target detection result in the sub-image coordinate system to the original image coordinate system according to the position of the image block in the sub-image in the original image.

10. An image object detection system implementing the high-resolution image object detection acceleration method based on image block combination as described in any one of claims 1 to 9, characterized in that: include: An image block extraction module, a policy network training module and an image block combination module, wherein: the image block extraction module is connected to the policy network training module and transmits the image blocks extracted by it, and the policy network module is connected to the image block combination module and transmits its trained policy network.

Images