Post-processing method, device and system for small target detection in large images

Abstract

The present invention provides a post-processing method, device, and system for small target detection in large images. The method includes: looping and executing the following steps in parallel until all feature map arrays are traversed: S1, reading the feature map array obtained by NPU inference; S2, sorting the feature maps in the feature map array according to confidence and category; wherein, during sorting, a non-recursive merge sorting dedicated circuit structure is adopted, and sorting is performed in parallel for each merge step; S3, calculating the intersection-over-union ratio using a dedicated circuit structure, and filtering the sorted feature map array based on the intersection-over-union ratio to obtain candidate targets that meet the requirements; S4, transmitting the filtered candidate targets to the NPU. This embodiment uses a multi-stage pipeline method to implement the parallel expansion of the candidate box sorting algorithm in the feature map, thereby optimizing the timing relationship of the loop, minimizing the loop time, and improving the post-processing speed.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a post-processing method, device and system for detecting small objects in large images. Background Art

[0002] Object detection, also known as object extraction, is a type of image segmentation based on the geometric and statistical characteristics of objects. Object detection not only determines the classification of objects in an image but also identifies their location and encloses them with bounding boxes. This combines object segmentation and recognition, and its accuracy and real-time performance are crucial capabilities of AI systems.

[0003] The larger the receptive field size for object detection, the wider the original image it can access, and the more global the features it detects. Conversely, the smaller the receptive field size, the more local and detailed the features it captures. Objects in large-format images, such as those from remote sensing, are relatively small. A large receptive field size can lose local features and details, leading to a sharp decline in detection performance. Reducing the receptive field size significantly improves object detection, but the number of candidate boxes increases dramatically, creating a significant computational load.

[0004] A neural network processor is a dedicated processing chip that integrates large-scale matrix neural computing. It has the characteristics of low cost, low power consumption, and high computing power. It is particularly good at performing neural network inference calculations. However, the post-processing calculations of neural network inference have strong timing dependence and low spatial parallelism. The use of neural network processors for calculations takes a long time and consumes high power. For example, Atlas200 DK is an embedded device used for deep learning model inference. It is often used for the implementation of deep learning models in practical applications, such as target detection. The main process of Atlas200 DK inference deep learning models is: preprocessing, inference, data decoding and post-processing (see Figure 1 ), the operations are executed sequentially, and the next step can be executed only after the previous step is completed. The inference process will be executed on the inference hardware unit of Atlas200 Dk. Because of hardware optimization and support, the speed of the inference stage is relatively fast, with an average speed of about tens of milliseconds. Preprocessing mainly includes operations such as image scaling, and data decoding mainly obtains information such as coordinates and confidence from the feature map after inference. Post-processing is mainly non-maximum suppression operation. Preprocessing, data decoding, and post-processing are all performed on the ARM CPUcortex A53 of Atlas200 DK without any hardware optimization, and the speed is slow, especially the post-processing operation. For large images that need to detect small targets, the number of candidate boxes is large, so post-processing becomes the fundamental factor that increases the delay of the entire inference system. After calculation, for a 1920X1080 image, the post-processing operation takes about 200ms, which leads to Figure 1Under the inference process, the overall inference speed is about 5fps. For larger images, such as remote sensing images, if small target detection is performed on the Atlas200 DK, the post-processing delay will be greater and the inference speed will be further reduced. Summary of the Invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a post-processing method, device and system for small target detection in large images.

[0006] The present invention is achieved in that:

[0007] A post-processing method for small target detection in large images, comprising:

[0008] The following steps are executed in a loop in parallel until all feature map arrays are traversed:

[0009] S1, reads the feature map array obtained by NPU inference;

[0010] S2, sorting the feature maps in the feature map array according to confidence and category; wherein, during the sorting, a non-recursive merge sort dedicated circuit structure is adopted, and the sorting is performed in parallel for each merge step;

[0011] S3, using a dedicated circuit structure to calculate the intersection-over-union ratio, and filtering the sorted feature map array according to the intersection-over-union ratio to obtain candidate targets that meet the requirements;

[0012] S4, transmit the screened candidate targets to the NPU.

[0013] Preferably, S2 is specifically:

[0014] A dedicated circuit structure reads the feature maps in the memory into multiple BRAMs in batches, and uses multiple sets of parallel sorting units to merge and sort the data in the BRAMs before writing them back to the memory. For any two elements A and B in the feature map array, they are first sorted by category. If the category priority of element A is greater than that of element B, then A is sorted first, otherwise B is sorted first. If elements A and B have the same category, they are sorted according to their confidence levels, with the element with higher confidence levels being placed before the element with lower confidence levels.

[0015] Preferably, S3 specifically includes:

[0016] S31, initialize the first pointer and tail pointer registers to point to the first memory address of the sorted feature map array;

[0017] S32, determining whether the tail pointer is less than the length of the feature map array; if so, executing S33, if not, executing S35;

[0018] S33, initialize the pointer i register to point to the first address of the feature map array memory, and determine whether the pointer i is less than the length of the feature map array; if the pointer i is greater than or equal to the length of the feature map array, the process ends, otherwise execute S34;

[0019] S34, determine whether the flag bit of the feature map pointed to by pointer i is true. If it is true, first point pointer j to the first address of the result array memory, and assign the feature map pointed to by pointer i to the location pointed to by pointer j. Then, pointer i and pointer j are incremented by 1 at the same time, and return to S33; if the flag bit is false, directly return to S33;

[0020] S35, determining whether the two conditions are simultaneously met: the tail pointer is less than the length of the feature map array and the category of the feature map pointed to by the first pointer is the same as the category of the feature map pointed to by the tail pointer; if so, executing S36, otherwise executing S37;

[0021] S36, increment the tail pointer and return to S35;

[0022] S37, decrement the tail pointer, and then determine whether the first pointer is smaller than the tail pointer; if not, add 1 to the value of the tail pointer and assign it to the first pointer, then increment the tail pointer and return to S32. If so, execute S38;

[0023] S38, determine whether the flag bit of the feature map pointed to by the first pointer is false; if so, increment the first pointer and execute S37; if not, define a pointer k and initialize it to the value of the first pointer plus 1, and execute S39;

[0024] S39, determine whether the pointer k is less than or equal to the tail pointer; if not, after incrementing the head pointer, execute S37; if so, execute S310;

[0025] S310, determine whether the flag bit of the feature map pointed to by pointer k is true; if so, increment pointer k and return to S39; if not, calculate the intersection-over-union ratio based on the data pointed to by pointer k and the feature map pointed to by the first pointer and execute S311;

[0026] S311, compare the size of the intersection-union ratio and the preset threshold. If the intersection-union ratio is greater than the threshold, then set the flag bit of the pointer k pointing to the feature map to false, increment the pointer k, and return to S38; otherwise, increment the pointer k and return to S38.

[0027] Preferably, for each element in the feature map array, there are two sets of coordinates (x1, y1) and (x2, y2), which respectively indicate the upper left corner and lower right corner of the element;

[0028] Then for elements A and B, the calculation steps of the intersection and union ratio are as follows:

[0029] Use combinational logic circuits to calculate the intersection rectangle width and height;

[0030] The width of the intersection rectangle is calculated according to the formula width = min(A.x2 - B.x2) - max(A.x1 - B.x1); the height of the intersection rectangle is calculated according to the formula height = min(A.y2 - B.y2) - max(A.y1 - B.y1);

[0031] Use the hardware multiplier to calculate the intersection rectangle area s1, s1 is calculated according to the formula s1 = height * width;

[0032] Use the hardware multiply-accumulator to calculate the area s2 of the union rectangle. s2 is calculated according to the formula s2 = (A.x2 - A.x1) * (A.y2 - A.y1) + (B.x2 - B.x1) * (B.y2 - B.y1) - s1. Where MN represents the value of the N coordinate of element M.

[0033] A hardware divider is used to calculate the intersection-over-union ratio (IOU) = s1 / s2 based on the intersection rectangle area s1 and the union rectangle area s2.

[0034] The embodiment of the present invention further provides a post-processing device for small target detection in large images, which includes an FPGA;

[0035] The FPGA includes an NMS module, which is used to implement the post-processing method for small target detection in large images as described above.

[0036] Preferably, the FPGA further includes:

[0037] Hard core processor module, Processor System Reset module, AXI BRAM Controller module, AXIInterconnect module, Block Memory Generator module; among them:

[0038] The working clock pin of the hard core processor module is connected to the low-speed synchronous clock pin of the Processor System Reset module;

[0039] The working clock pin of the hard core processor module is connected to the working clock pins of each master and slave interface of the AXI Interconnect module;

[0040] The working clock pin of the hard core processor module is connected to the working clock pin of the NMS module;

[0041] The working clock pin of the hard core processor module is connected to the working clock pin of the slave interface of the AXI BRAM Controller module;

[0042] The FPGA working clock pin of the hard core processor module is connected to the working clock pins of each master and slave interface of the AXI Interconnect module;

[0043] The FPGA working clock pin of the hard core processor module is connected to the working clock pin of the NMS module;

[0044] The FPGA reset pin of the hard-core processor module is connected to the external reset input pin of the Processor System Reset module;

[0045] The AXI fast master interface pins of the hard core processor module are connected to the AXI slave interface pins of the AXI Interconnect module;

[0046] The AXI slow master interface pins of the hard core processor module are connected to the AXI slave interface pins of the AXI Interconnect module;

[0047] The external device reset pin of the Processor System Reset module is connected to the master-slave interface reset pin of the AXI Interconnect module;

[0048] The external device reset pin of the Processor System Reset module is connected to the reset pin of the NMS module;

[0049] The external device reset pin of the Processor System Reset module is connected to the slave interface reset pin of the AXI BRAM Controller module;

[0050] Connect the AXI slave interface pins of the Processor System Reset module to the AXI master interface pins of the NMS module;

[0051] The AXI master interface pins of the Processor System Reset module are connected to the AXI slave interface pins of the AXI BRAM Controller module;

[0052] The AXI master interface pins of the Processor System Reset module are connected to the AXI slave interface control pins of the NMS module;

[0053] The write pin of the AXI BRAM Controller module is connected to the read pin of the Block Memory Generator module.

[0054] Preferably, the hard-core processor module is configured to receive a feature map of the neural network inference sent by the NPU and send the feature map to the AXI BRAM Controller module via the AXI fast master interface pin;

[0055] AXI BRAM Controller module, which is used to write the feature map to the Block Memory Generator module through the write pin after receiving the feature map through the AXI slave interface pin;

[0056] Multiple Block Memory Generator modules for caching feature maps;

[0057] The NMS module is used to communicate with the AXI Interconnect module through the AXI master interface pin, and the AXI Interconnect module communicates with the AXI BRAM Controller module through the AXI master interface pin to obtain the feature map in the BlockMemory Generator module. After obtaining the feature map, the NMS module executes the post-processing acceleration method for small target detection in large images as described above. After execution, the candidate target is copied; the NMS module is also used to communicate with the AXI slave interface pin of the AXI Interconnect module through the AXI master interface pin, and the AXI Interconnect module communicates with the AXI slow interface pin of the hard-core processor module through the AXI slave interface pin. The hard-core processor module copies the candidate target from the NMS module through the AXI slow interface pin and transmits it to the NPU.

[0058] An embodiment of the present invention further provides a post-processing system for small target detection in large images, which includes an NPU and the post-processing device for small target detection in large images as described above; wherein:

[0059] On the NPU side, the input raw image data is pre-processed, inference is performed to generate feature map data, and then sent to the FPGA;

[0060] On the FPGA side, the inferred feature map array is sorted and filtered in parallel, and then the candidate targets are transmitted to the NPU.

[0061] On the NPU side, the filtered candidate targets are output.

[0062] Preferably, on the NPU side, it includes:

[0063] S201, after reading the original image data, execute S202;

[0064] S202, label each read original image data with a serial number i1, and put the original image data into the buffer pool, and execute S203;

[0065] S203, preprocessing the original image data. After the preprocessing is completed, the preprocessed image data is used to perform inference on the hardware inference unit. After the inference is completed, the data is decoded to obtain the inferred feature map, and S204 is executed.

[0066] S204, slicing the feature map, marking each feature map data slice with a sequence number j, and then sending the feature map data slice to the FPGA end using the UDP protocol;

[0067] On the FPGA side:

[0068] S205, initialize the receiving flag flag to true;

[0069] S206, receiving the inferred feature map data slice j;

[0070] S207, determine whether the inferred feature map data slice number j is equal to 0, if it is equal to 0, then set the receiving flag flag to true and execute S208, otherwise directly execute S208;

[0071] S208, determine whether the sequence number j is continuous. If not, set the receiving flag flag to false and execute S209. Otherwise, execute S209 directly.

[0072] S209, determine whether the receiving flag flag is true. If not true, discard the current inferred feature map data slice j and execute S206. Otherwise, execute S206 directly.

[0073] S210, determining whether the currently inferred feature map data slice is the last slice, if not, executing S206, otherwise executing S211;

[0074] S211, reorganize the feature map data slices to obtain the original sequence number i1 and the sequence number j of each inferred feature map data slice, reorganize according to the sequence number j, put the reorganized feature map into the buffer pool, and execute S212;

[0075] S212, determine whether the buffer pool is empty, if not, then execute S213, otherwise execute S212;

[0076] S213. Read the feature map from the buffer pool, and send the feature map to the NMS module through the data interaction API compiled in HLS. After generating candidate targets, use the original sequence number i1 as i2 and put it into the message header. Recombine the data packets according to the candidate targets, and use the UDP protocol to transmit the data packets to the NPU.

[0077] On the NPU side:

[0078] S214. After receiving the candidate targets, parse the candidate targets to obtain the coordinate information and the sequence number i2, and execute S215.

[0079] S215. Compare the sequence number i2 with the sequence number i1 of the original image data in the local buffer. If the sequence number i1 is equal to i2, it means a match. Directly retrieve the current original image data from the buffer pool, combine the coordinate information with the original image data and output them together, and execute S201. If i1 > i2, it means that data is lost when the FPGA side transmits candidate targets to the NPU side. Then discard the candidate targets and execute S214. If i1 < i2, it means that data is lost when the NPU transmits the inferred feature map to the FPGA side. Then directly discard the original image data from the buffer pool and execute S215 again.

[0080] In this embodiment, a multi-stage pipeline method is adopted for expansion, so as to achieve the parallel expansion of the candidate box sorting algorithm in the feature map, thereby optimizing the timing relationship of the loop, minimizing the loop time consumption, and improving the post-processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. Obviously, the following described drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0082] Figure 1 It is a flowchart of the existing Atlas200 DK inference deep learning model.

[0083] Figure 2 It is a schematic flowchart of the post-processing method for small target detection in large-scale images provided by the first embodiment of the present invention.

[0084] Figure 3 It is a result display diagram of the pipeline and parallel expansion.

[0085] Figure 4 It is a schematic structural diagram of the post-processing device for small target detection in large-scale images provided by the second embodiment of the present invention.

[0086] Figure 5 This is a workflow diagram of the post-processing system for small target detection in large images provided by the third embodiment of the present invention. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0088] See also Figure 2 A first embodiment of the present invention provides a post-processing method for detecting small objects in large images, which can be performed by a post-processing device for detecting small objects in large images (hereinafter referred to as the post-processing device) to implement the following steps:

[0089] The following steps are executed in a loop in parallel until all feature map arrays are traversed:

[0090] S1, reads the feature map array obtained by NPU inference.

[0091] In this embodiment, the post-processing device may include an FPGA, and the steps of this embodiment are implemented by the FPGA.

[0092] The feature map array is obtained by inference by an embedded neural network processor (NPU). Specifically, in this embodiment:

[0093] First, after the NPU reads the raw image data, it marks each piece of raw image data read with a serial number i1 and puts the raw image data into the buffer pool.

[0094] Then, the original image data is preprocessed. After the preprocessing is completed, the preprocessed image data is used to perform inference work on the hardware inference unit. After the inference is completed, the data is decoded to obtain the inferred feature map.

[0095] Next, the feature map is sliced, each feature map data slice is marked with a sequence number j, and then the feature map data slice is sent to the FPGA end using the UDP protocol.

[0096] Finally, the FPGA reads the inferred feature map array.

[0097] S2, sorting the feature maps in the feature map array according to confidence and category; wherein, during the sorting, a non-recursive merge sort dedicated circuit structure is adopted, and the sorting is performed in parallel for each merge step.

[0098] In this embodiment, for example:

[0099] For any two elements A and B in the feature map array, first sort them by category. If the category priority of element A is greater than that of element B, then A is sorted first, otherwise B is sorted first.

[0100] If elements A and B have the same category, they are sorted according to their confidence levels, with elements with higher confidence levels placed before elements with lower confidence levels.

[0101] Particularly, in this embodiment, during sorting, a non-recursive merge sort dedicated circuit structure is used to execute the merge sort algorithm, and sorting can be executed in parallel for each merge step.

[0102] S3 uses a dedicated circuit structure to calculate the intersection-over-union ratio, and filters the sorted feature map array according to the intersection-over-union ratio to obtain candidate targets that meet the requirements.

[0103] S3 specifically includes:

[0104] S31, initialize the first pointer and tail pointer registers to point to the first memory address of the sorted feature map array;

[0105] S32, determining whether the tail pointer is less than the array length of the feature map array; if so, executing S33, if not, executing S35;

[0106] S33, initialize the pointer i register to point to the first address of the feature map array memory, and determine whether the pointer i is less than the length of the feature map array; if the pointer i is greater than or equal to the length of the feature map array, the process ends, otherwise execute S34;

[0107] S34, determine whether the flag bit of the feature map pointed to by pointer i is true. If it is true, first point pointer j to the first address of the result array memory, and assign the feature map pointed to by pointer i to the location pointed to by pointer j. Then, pointer i and pointer j are incremented by 1 at the same time, and return to S33; if the flag bit is false, directly return to S33;

[0108] S35, determining whether the two conditions are simultaneously met: the tail pointer is less than the length of the feature map array and the category of the feature map pointed to by the first pointer is the same as the category of the feature map pointed to by the tail pointer; if so, executing S36, otherwise executing S37;

[0109] S36, increment the tail pointer and return to S35;

[0110] S37, decrement the tail pointer, and then determine whether the first pointer is smaller than the tail pointer; if not, add 1 to the value of the tail pointer and assign it to the first pointer, then increment the tail pointer and return to S32. If so, execute step S38;

[0111] S38, determine whether the flag bit of the feature map pointed to by the first pointer is false; if so, increment the first pointer and execute S37; if not, define a pointer k and initialize it to the value of the first pointer plus 1, and execute S39;

[0112] S39, determine whether the pointer k is less than or equal to the tail pointer; if not, after incrementing the head pointer, execute S37; if so, execute S310;

[0113] S310, determine whether the flag bit of the feature map pointed to by pointer k is true; if so, increment pointer k and return to S39; if not, calculate the intersection-over-union ratio based on the data pointed to by pointer k and the feature map pointed to by the first pointer, and execute S311;

[0114] S311, compare the size of the intersection-union ratio and the preset threshold. If the intersection-union ratio is greater than the threshold, then set the flag bit of the pointer k pointing to the feature map to false, increment the pointer k, and return to S38; otherwise, increment the pointer k and return to S38.

[0115] Among them, for each element in the feature map array, there are two sets of coordinates (x1, y1) and (x2, y2), which indicate the upper left corner and lower right corner of the element respectively;

[0116] Then for elements A and B, the calculation steps of the intersection and union ratio are as follows:

[0117] Use combinational logic circuits to calculate the intersection rectangle width width and height height.

[0118] The width of the intersection rectangle is calculated according to the formula width = min(A.x2-B.x2)-max(A.x1-B.x1);

[0119] The height of the intersection rectangle is calculated according to the formula height = min(A.y2-B.y2)-max(A.y1-B.y1);

[0120] Use the hardware multiplier to calculate the intersection rectangle area s1, s1 is calculated according to the formula s1 = height * width;

[0121] Use the hardware multiplier-accumulator to calculate the area s2 of the union rectangle. s2 is calculated according to the formula s2 = (A.x2-A.x1)*(A.y2-A.y1)+(B.x2-B.x1)*(B.y2-B.y1)-s1. Where MN represents the value of the N coordinate of element M. For example, A.x2 represents the x2 coordinate of element A, B.x2 represents the x2 coordinate of element B, and so on.

[0122] A hardware divider is used to calculate the intersection-over-union ratio (IOU) = s1 / s2 based on the intersection rectangle area s1 and the union rectangle area s2.

[0123] S4, transmit the screened candidate targets to the NPU.

[0124] It should be noted that, in this embodiment, the above steps S1-S4 are carried out using a multi-stage pipeline method, thereby realizing the parallel development of the candidate box sorting algorithm in the feature map.

[0125] like Figure 3 As shown, regarding pipelines: for example, during sorting, the subsequent feature maps to be sorted can be read, and there is a pipeline here; when filtering the current feature map array, the feature map array of subsequent data can be sorted, and there is also a pipeline here. Regarding parallel expansion: As mentioned above, this embodiment uses a non-recursive merge sort algorithm. For each merge step, sorting can be performed in parallel; for S1 reading data, it can be read in parallel; for S4 output results, it can also be copied in parallel.

[0126] In summary, this embodiment optimizes the timing relationship of the loop and minimizes the loop time by adopting a pipeline and parallel design for the above steps S1-S4, thereby improving the processing speed.

[0127] The second embodiment of the present invention further provides a post-processing device for small target detection in a large image, which includes an FPGA;

[0128] The FPGA is used to implement the post-processing method for small target detection in large images as described in any of the above embodiments.

[0129] Among them, such as Figure 4 As shown, in particular, the FPGA includes:

[0130] Hard core processor module 210, Processor System Reset module 220, AXI BRAM Controller module 230, AXI Interconnect module 240, Block Memory Generator module 250, and NMS module 260; wherein:

[0131] The working clock pin of the hard core processor module 210 is connected to the low-speed synchronous clock pin of the Processor System Reset module 220;

[0132] The working clock pin of the hard core processor module 210 is connected to the working clock pins of each master and slave interface of the AXI Interconnect module 240;

[0133] The working clock pin of the hard core processor module 210 is connected to the working clock pin of the NMS module 260;

[0134] The working clock pin of the hard core processor module 210 is connected to the working clock pin of the slave interface of the AXI BRAM Controller module 230;

[0135] The FPGA working clock pin of the hard core processor module 210 is connected to the FPGA working clock pin of the AXI Interconnect module 240;

[0136] The FPGA working clock pin of the hard core processor module 210 is connected to the working clock pin of the NMS module 260;

[0137] The FPGA reset pin of the hard core processor module 210 is connected to the external reset input pin of the Processor System Reset module 220;

[0138] The AXI fast master interface pins of the hard core processor module 210 are connected to the AXI slave interface pins of the AXI Interconnect module 240;

[0139] The AXI slow master interface pins of the hard core processor module 210 are connected to the AXI slave interface pins of the AXI Interconnect module 240 ;

[0140] The external device reset pin of the Processor System Reset module 220 is connected to the master-slave interface reset pin of the AXI Interconnect module 240;

[0141] The external device reset pin of the Processor System Reset module 220 is connected to the reset pin of the NMS module 260;

[0142] The external device reset pin of the Processor System Reset module 220 is connected to the slave interface reset pin of the AXI BRAMController module 230;

[0143] The AXI slave interface pin of the Processor System Reset module 220 is connected to the AXI master interface pin of the NMS module 260;

[0144] The AXI master interface pin of the Processor System Reset module 220 is connected to the AXI slave interface pin of the AXI BRAM Controller module 230;

[0145] The AXI master interface pin of the Processor System Reset module 220 is connected to the AXI slave interface control pin of the NMS module 260;

[0146] The write pin of the AXI BRAM Controller module 230 is connected to the read pin of the Block Memory Generator module 250;

[0147] In this embodiment, in particular:

[0148] The hard core processor module 210 is used to receive the feature map of the neural network inference sent by the NPU and send the data to the AXI BRAM Controller module 230 through the AXI fast master interface pin;

[0149] AXI BRAM Controller module 230 is used to write data to the Block Memory Generator module 250 through the write pin after receiving the feature map through the AXI slave interface pin;

[0150] a plurality of Block Memory Generator modules 250 for caching feature maps;

[0151] The NMS module 260 is configured to communicate with the AXI Interconnect module 240 via the AXI master interface pins, and the AXI Interconnect module 240 communicates with the AXI BRAM Controller module 230 via the AXI master interface pins to obtain the feature map from the Block Memory Generator module 250. After obtaining the feature map, the NMS module 260 executes the above-mentioned post-processing acceleration method for small object detection in large images, generates candidate objects, and copies the candidate objects.

[0152] The NMS module 260 is further configured to communicate with the AXI slave interface pins of the AXI Interconnect module 240 via the AXI master interface pins. The AXI Interconnect module 240 communicates with the AXI slow interface pins of the hard-core processor module 210 via the AXI slave interface pins. The hard-core processor module 210 copies the candidate target from the NMS module 260 via the AXI slow interface pins and transmits it to the NPU.

[0153] See also Figure 5 The third embodiment of the present invention further provides a post-processing system for detecting small targets in large images, which includes an NPU and the post-processing device for detecting small targets in large images as described above; wherein:

[0154] On the NPU side, the input raw image data is pre-processed, inference is performed to generate feature map data, and then sent to the FPGA;

[0155] On the FPGA side, the inferred feature map array is sorted and filtered in parallel, and then the candidate targets are transmitted to the NPU.

[0156] On the NPU side, the filtered candidate targets are output.

[0157] Preferably, on the NPU side, it includes:

[0158] S201, after reading the original image data, execute S202;

[0159] S202, label each read original image data with a serial number i1, and put the original image data into the buffer pool, and execute S203;

[0160] S203, preprocessing the original image data. After the preprocessing is completed, the preprocessed image data is used to perform inference on the hardware inference unit. After the inference is completed, the data is decoded to obtain the inferred feature map, and S204 is executed.

[0161] S204, slicing the feature map, marking each feature map data slice with a sequence number j, and then sending the feature map data slice to the FPGA end using the UDP protocol;

[0162] On the FPGA side:

[0163] S205, initialize the receiving flag flag to true;

[0164] S206, receiving the inferred feature map data slice j;

[0165] S207, determine whether the inferred feature map data slice number j is equal to 0, if it is equal to 0, then set the receiving flag flag to true and execute S208, otherwise directly execute S208;

[0166] S208, determine whether the sequence number j is continuous. If not, set the receiving flag flag to false and execute S209. Otherwise, execute S209 directly.

[0167] S209, determine whether the receiving flag flag is true. If not true, discard the current inferred feature map data slice j and execute S206. Otherwise, execute S206 directly.

[0168] S210, determining whether the currently inferred feature map data slice is the last slice, if not, executing S206, otherwise executing S211;

[0169] S211. Recombine the fragmented feature map data to obtain the original sequence number i1 and the sequence number j of each fragmented feature map data after inference. Recombine according to the sequence number j, and put the recombined feature map into the buffer pool, then execute S212;

[0170] S212. Determine whether the buffer pool is empty. If it is not empty, then execute S213; otherwise, execute S212;

[0171] S213. Read the feature map data from the buffer pool, and send this feature map to the NMS module through the data interaction API compiled in HLS. After generating candidate targets, take the original sequence number i1 as i2 and put it into the message header. Recombine the data packets according to the candidate targets, and use the UDP protocol to transmit the data packets to the NPU;

[0172] On the NPU side:

[0173] S214. After receiving the candidate targets, parse the candidate targets to obtain the coordinate information and the sequence number i2, then execute S215;

[0174] S215. Compare the sequence number i2 with the sequence number i1 of the original image data in the local buffer. If the sequence number i1 is equal to i2, it means they match. Directly take out the current original image data from the buffer pool, and output it together with the combined coordinate information and the original image data, then execute S201. If i1 > i2, it means data loss occurred when the FPGA side transmitted the candidate targets to the NPU side. Then discard the candidate targets and execute S214. If i1 < i2, it means data loss occurred when the NPU transmitted the feature map data after inference to the FPGA side. Then directly discard the original image data from the buffer pool and execute S215 again.

[0175] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of a code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0176] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0177] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. It should be noted that, in this article, the terms "include", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0178] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A post-processing method for small target detection in large images, characterized in that: include: The following steps are executed in a loop in parallel until all feature map arrays are traversed: S1, reads the feature map array obtained by NPU inference; S2, sorting the feature maps in the feature map array according to confidence and category; wherein, during the sorting, a non-recursive merge sort dedicated circuit structure is adopted, and the sorting is performed in parallel for each merge step; S3, using a dedicated circuit structure to calculate the intersection-over-union ratio, and filtering the sorted feature map array according to the intersection-over-union ratio to obtain candidate targets that meet the requirements; S4, transmit the screened candidate targets to the NPU.

2. The post-processing method for small target detection in large images according to claim 1 is characterized in that: S2 is specifically: A dedicated circuit structure reads the feature maps in the memory into multiple BRAMs in batches, and uses multiple sets of parallel sorting units to merge and sort the data in the BRAMs before writing them back to the memory. For any two elements A and B in the feature map array, they are first sorted by category. If the category priority of element A is greater than that of element B, then A is sorted first, otherwise B is sorted first. If elements A and B have the same category, they are sorted according to their confidence levels, with the element with higher confidence levels being placed before the element with lower confidence levels.

3. The post-processing method for small target detection in large images according to claim 1, characterized in that: S3 specifically includes: S31, initialize the first pointer and tail pointer registers to point to the first memory address of the sorted feature map array; S32, determining whether the tail pointer is less than the length of the feature map array; if so, executing S33, if not, executing S35; S33, initialize the pointer i register to point to the first address of the feature map array memory, and determine whether the pointer i is less than the length of the feature map array; if the pointer i is greater than or equal to the length of the feature map array, the process ends, otherwise execute S34; S34, determine whether the flag bit of the feature map pointed to by pointer i is true. If it is true, first point pointer j to the first address of the result array memory, and assign the feature map pointed to by pointer i to the location pointed to by pointer j. Then, pointer i and pointer j are incremented by 1 at the same time, and return to S33; if the flag bit is false, directly return to S33; S35, determining whether the two conditions are simultaneously met: the tail pointer is less than the length of the feature map array and the category of the feature map pointed to by the first pointer is the same as the category of the feature map pointed to by the tail pointer; if so, executing S36, otherwise executing S37; S36, increment the tail pointer and return to S35; S37, decrement the tail pointer, and then determine whether the first pointer is smaller than the tail pointer; if not, add 1 to the value of the tail pointer and assign it to the first pointer, then increment the tail pointer and return to S32. If so, execute S38; S38, determine whether the flag bit of the feature map pointed to by the first pointer is false; if so, increment the first pointer and execute S37; if not, define a pointer k and initialize it to the value of the first pointer plus 1, and execute S39; S39, determine whether the pointer k is less than or equal to the tail pointer; if not, after incrementing the head pointer, execute S37; if so, execute S310; S310, determine whether the flag bit of the feature map pointed to by pointer k is true; if so, increment pointer k and return to S39; if not, calculate the intersection-over-union ratio based on the data pointed to by pointer k and the feature map pointed to by the first pointer, and execute S311; S311, compare the size of the intersection-union ratio and the preset threshold. If the intersection-union ratio is greater than the threshold, then set the flag bit of the pointer k pointing to the feature map to false, increment the pointer k, and return to S38; otherwise, increment the pointer k and return to S38.

4. The post-processing method for small target detection in large images according to claim 3 is characterized in that: For each element in the feature map array, there are two sets of coordinates (x1, y1) and (x2, y2), which indicate the upper left corner and lower right corner of the element respectively. Then for elements A and B, the calculation steps of the intersection and union ratio are as follows: Use combinational logic circuits to calculate the intersection rectangle width and height; The width of the intersection rectangle is calculated according to the formula width = min(A.x2 - B.x2) - max(A.x1 - B.x1); the height of the intersection rectangle is calculated according to the formula height = min(A.y2 - B.y2) - max(A.y1 - B.y1); Use the hardware multiplier to calculate the intersection rectangle area s1, s1 is calculated according to the formula s1 = height * width; Use the hardware multiply-accumulator to calculate the area s2 of the union rectangle. s2 is calculated according to the formula s2 = (A.x2 - A.x1) * (A.y2 - A.y1) + (B.x2 - B.x1) * (B.y2 - B.y1) - s1. Where MN represents the value of the N coordinate of element M. A hardware divider is used to calculate the intersection-over-union ratio (IOU) = s1 / s2 based on the intersection rectangle area s1 and the union rectangle area s2.

5. A post-processing device for small target detection in large images, characterized in that: Including FPGA; The FPGA includes an NMS module, which is used to implement the post-processing method for small target detection in a large image as described in any one of claims 1 to 4.

6. The post-processing device for small target detection in large images according to claim 5, characterized in that: The FPGA further includes: Hard core processor module, Processor System Reset module, AXI BRAM Controller module, AXIInterconnect module, Block Memory Generator module; among them: The working clock pin of the hard core processor module is connected to the low-speed synchronous clock pin of the Processor System Reset module; The working clock pin of the hard core processor module is connected to the working clock pins of each master and slave interface of the AXI Interconnect module; The working clock pin of the hard core processor module is connected to the working clock pin of the NMS module; The working clock pin of the hard core processor module is connected to the working clock pin of the slave interface of the AXI BRAM Controller module; The FPGA working clock pin of the hard core processor module is connected to the working clock pins of each master and slave interface of the AXI Interconnect module; The FPGA working clock pin of the hard core processor module is connected to the working clock pin of the NMS module; The FPGA reset pin of the hard-core processor module is connected to the external reset input pin of the Processor System Reset module; The AXI fast master interface pins of the hard core processor module are connected to the AXI slave interface pins of the AXI Interconnect module; The AXI slow master interface pins of the hard core processor module are connected to the AXI slave interface pins of the AXI Interconnect module; The external device reset pin of the Processor System Reset module is connected to the master-slave interface reset pin of the AXI Interconnect module; The external device reset pin of the Processor System Reset module is connected to the reset pin of the NMS module; The external device reset pin of the Processor System Reset module is connected to the slave interface reset pin of the AXI BRAM Controller module; Connect the AXI slave interface pins of the Processor System Reset module to the AXI master interface pins of the NMS module; The AXI master interface pins of the Processor System Reset module are connected to the AXI slave interface pins of the AXI BRAM Controller module; The AXI master interface pins of the Processor System Reset module are connected to the AXI slave interface control pins of the NMS module; The write pin of the AXI BRAM Controller module is connected to the read pin of the Block Memory Generator module.

7. The post-processing device for small target detection in large images according to claim 6, characterized in that: The hard-core processor module is used to receive the feature map of the neural network inference sent by the NPU and send the feature map to the AXI BRAM Controller module through the AXI fast master interface pin; AXI BRAM Controller module, which is used to write the feature map to the Block Memory Generator module through the write pin after receiving the feature map through the AXI slave interface pin; Multiple Block Memory Generator modules for caching feature maps; The NMS module is used to communicate with the AXI Interconnect module through the AXI master interface pins, and the AXI Interconnect module communicates with the AXI BRAM Controller module through the AXI master interface pins to obtain the feature map in the BlockMemory Generator module. After obtaining the feature map, the NMS module performs acceleration processing to generate candidate targets and copy the candidate targets. The NMS module is also used to communicate with the AXI slave interface pins of the AXI Interconnect module through the AXI master interface pins. The AXI Interconnect module communicates with the AXI slow interface pins of the hard-core processor module through the AXI slave interface pins. The hard-core processor module copies the candidate target from the NMS module through the AXI slow interface pins and transmits it to the NPU.

8. A post-processing system for small target detection in large images, characterized by: The apparatus comprises an NPU and a post-processing device for detecting small targets in large images according to any one of claims 5 to 7; wherein: On the NPU side, the input raw image data is pre-processed, inference is performed to generate feature map data, and then sent to the FPGA; On the FPGA side, the inferred feature map array is sorted and filtered in parallel, and then the candidate targets are transmitted to the NPU. On the NPU side, the filtered candidate targets are output.

9. The post-processing system for small target detection in large images according to claim 8, characterized in that: On the NPU side, it includes: S201, after reading the original image data, execute S202; S202, label each read original image data with a serial number i1, and put the original image data into the buffer pool, and execute S203; S203, preprocessing the original image data. After the preprocessing is completed, the preprocessed image data is used to perform inference on the hardware inference unit. After the inference is completed, the data is decoded to obtain the inferred feature map, and S204 is executed. S204, slicing the feature map, marking each feature map data slice with a sequence number j, and then sending the feature map data slice to the FPGA end using the UDP protocol; On the FPGA side: S205, initialize the receiving flag flag to true; S206, receiving the inferred feature map data slice j; S207, determine whether the inferred feature map data slice number j is equal to 0, if it is equal to 0, then set the receiving flag flag to true and execute S208, otherwise directly execute S208; S208, determine whether the sequence number j is continuous. If not, set the receiving flag flag to false and execute S209. Otherwise, execute S209 directly. S209, determine whether the receiving flag flag is true. If not true, discard the current inferred feature map data slice j and execute S206. Otherwise, execute S206 directly. S210, determining whether the currently inferred feature map data slice is the last slice, if not, executing S206, otherwise executing S211; S211, reorganize the feature map data slices to obtain the original sequence number i1 and the sequence number j of each inferred feature map data slice, reorganize according to the sequence number j, put the reorganized feature map into the buffer pool, and execute S212; S212, determine whether the buffer pool is empty, if not, then execute S213, otherwise execute S212; S213. Read the feature map from the buffer pool, send the feature map to the NMS module through the data interaction API compiled in HLS. After generating the candidate targets, use the original sequence number i1 as i2 and put it into the message header. Recombine the data packets according to the candidate targets, and transmit the data packets to the NPU using the UDP protocol. On the NPU side: S214. After receiving the candidate targets, parse the candidate targets to obtain the coordinate information and the sequence number i2, and execute S215. S215. Compare the sequence number i2 with the sequence number i1 of the original image data in the local buffer. If the sequence number i1 is equal to i2, it means a match. Directly retrieve the current original image data from the buffer pool, combine the coordinate information with the original image data and output them together, and execute S201. If i1 > i2, it means that data is lost when the FPGA side transmits the candidate targets to the NPU side. Then discard the candidate target and execute S214. If i1 < i2, it means that data is lost when the NPU transmits the inferred feature map to the FPGA side. Then directly discard the original image data from the buffer pool and execute S(215) again.

Images