An FPGA hardware implementation method of an anchor-free decoupling structure detection head

Abstract

The application provides an FPGA hardware implementation method of an anchor-free decoupling structure detection head, relates to the technical field of target detection deep learning algorithms, and mainly comprises the following steps: defining the size of a block feature map processed by an FPGA hardware calculation unit each time, and then dividing the feature map into a plurality of block feature maps; firstly, calculating the maximum confidence value of the first block feature map in each channel and saving the maximum confidence value; judging whether there are pixel points greater than a confidence threshold value; if there are, only the bounding box of the pixel point with the confidence value greater than the threshold value needs to be calculated; if not, the next block feature map is calculated until the last block feature map; after all the pixel points with the confidence value greater than the confidence threshold value are obtained, non-maximum suppression calculation is performed. The application only calculates the bounding box of the pixel point with the confidence value greater than the threshold value, can reduce the calculation amount and improve the real-time performance, only uses programmable logic resources to realize the non-maximum suppression algorithm, and can improve the universality and real-time performance.

Description

Technical Field

[0001] This invention relates to the field of deep learning algorithm technology for target detection, and in particular to an FPGA hardware implementation method for a detection head with an anchor-free decoupled structure. Background Technology

[0002] In the field of computer vision, deep learning algorithms for object detection are constantly being updated and iterated. For example, YOLO evolved from version v1 in 2015 to version v8 in 2023. The programmability of FPGAs is well-suited for these continuously evolving deep learning algorithms. Object detection algorithms consist of a backbone network, a neck network, and a detection head. Currently, mainstream detection heads employ an anchor-free decoupled structure, separating classification and regression. The detection head generates a large number of candidate boxes for the same object, and these candidate boxes often overlap. Therefore, a non-maximum suppression algorithm is needed to find the optimal object bounding box and remove redundant bounding boxes.

[0003] When implementing a detection head using an FPGA, firstly, the detection head involves a large number of bounding box calculations, and calculating each one individually will reduce real-time performance. Secondly, the detection head includes non-maximum suppression algorithms, which are often implemented using the hard or soft cores of the FPGA's on-chip processor, resulting in long computation times and poor real-time performance. Summary of the Invention

[0004] Based on the above problems, the purpose of this invention is to adopt a new FPGA hardware implementation method that calculates bounding boxes only for pixels with confidence scores greater than a threshold, thereby reducing computational load and improving real-time performance. It also uses only programmable logic resources to implement the non-maximum suppression algorithm, thereby improving versatility and real-time performance.

[0005] The technical solution adopted by this invention to achieve its objective is an FPGA hardware implementation method for an anchor-free decoupling structure detection head, comprising the following steps:

[0006] S1. Define the anchorless decoupled structure detection head as including classification and regression branches;

[0007] S2. The detection head processes feature maps of different sizes, where the value of the classification branch feature map represents the confidence of the bounding box, and the value of the regression branch feature map represents the distance of the bounding box.

[0008] S3. Define the size of the block feature map processed by the FPGA hardware computing unit each time, and divide the classification branch feature map and regression branch feature map into several block feature maps;

[0009] S4. First, calculate the maximum confidence of the first block feature map in the classification branch in each channel direction, and save the corresponding channel, i.e. the corresponding category.

[0010] S5. Determine whether there are any pixels in the above block feature maps that have a confidence threshold. If there are, calculate the bounding box of the pixel with a confidence threshold in the corresponding block feature map of the regression branch. If not, calculate the next block feature map until the last block feature map. The coordinate information of the bounding box is [(x1,y1),(x2,y2)], where (x1,y1) is the coordinate of the upper left corner and (x2,y2) is the coordinate of the lower right corner.

[0011] S6. After obtaining information on all pixels with a confidence level greater than the confidence threshold, perform non-maximum suppression calculation.

[0012] Furthermore, the feature map in step S2 has three sizes: 20×20, 40×40, or 80×80.

[0013] Furthermore, the feature map size mentioned in step S3 is 8×8. The feature maps of 20×20, 40×40, and 80×80 are divided into 8×8 blocks and their sizes are 3×3, 5×5, and 10×10, respectively. If there are blocks smaller than 8×8 after the division, "0" is padded to fill the size to 8×8.

[0014] Specifically, for a 20×20 classification branch feature map or regression branch feature map, it is divided into 8×8 blocks. In addition to generating 4 standard 8×8 block feature maps, it will also generate 2 8×4, 2 4×8 and 1 4×4 blocks. These 5 blocks smaller than 8×8 are padded with "0"s to make them 8×8 in size.

[0015] Furthermore, the number of channels in step S4 is 80.

[0016] Furthermore, the specific steps of step S6 are as follows:

[0017] S61. Write the information of all pixels with confidence scores greater than the confidence threshold obtained in step S5 into the "ping" buffer of the "ping-pong" buffer. The pixel information is: {[coordinates], confidence score, type}. At this time, the control signal pingpong of the "ping-pong" buffer is 0, which means "ping" read and "pong" write.

[0018] S62. Read all pixels written to the "ping-pong" buffer, find the maximum confidence value for different types, and output it.

[0019] S63. Read the pixels written to the "ping-pong" buffer one by one, calculate the IOU between the current candidate box and the candidate box with the maximum value of the same type. If the IOU is less than the threshold, write the information of the pixel into the "ping-pong" buffer. If the IOU is greater than the threshold, discard the pixel. The IOU (Intersection of Union) is the intersection of two boxes divided by the union of the two boxes.

[0020] S64. After reading all the pixels written to the "ping-pong" buffer, if there is at least one pixel with an IOU less than the threshold, the "ping-pong" buffer control signal pingpong is flipped, that is, the control signal pingpong = 1, which means "ping" write and "pong" read, and the process returns to step S62; if there is no pixel with an IOU less than the threshold, the non-maximum suppression calculation is completed.

[0021] The beneficial effects of this invention are as follows:

[0022] (i) Only the bounding boxes are calculated for pixels with a confidence level greater than the threshold, which greatly reduces the amount of computation and improves real-time performance;

[0023] (ii) The nonmaximum suppression algorithm is implemented using general-purpose programmable resources, without the need for hard or soft cores of the FPGA on-chip processor, thus improving real-time performance and versatility. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of two branches of the detection head decoupling structure of the present invention;

[0025] Figure 2 This is a schematic diagram of the block calculation process for the detection head of the present invention;

[0026] Figure 3 This is a schematic diagram of the nonmaximum suppression calculation process of the present invention;

[0027] Figure 4 This is a schematic diagram illustrating four possible scenarios for the IOU of this invention;

[0028] Figure 5 This is a flowchart of the detection head block calculation process according to an embodiment of the present invention;

[0029] Figure 6 This is a flowchart of the nonmaximum suppression calculation in an embodiment of the present invention. Detailed Implementation

[0030] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Example

[0032] Figures 1-6This invention illustrates a specific embodiment of the FPGA hardware implementation method for the anchorless decoupling structure detection head of the present invention, comprising the following steps:

[0033] S1. Define the anchorless decoupled structure detection head as including classification and regression branches;

[0034] S2. The detection head processes feature maps of different sizes, where the value of the classification branch feature map represents the confidence of the bounding box, and the value of the regression branch feature map represents the distance of the calculated bounding box; the feature map has three sizes, namely 20×20, 40×40, or 80×80; each classification branch feature map contains 80 channels, i.e., 80 categories, and each regression feature map contains 4 channels, i.e., 4 distances;

[0035] S3. Because FPGA hardware cannot generate large feature maps at once, it is necessary to perform block-based computation on the feature maps. The size of each block feature map processed by the FPGA hardware computation unit is defined as 8×8. The classification branch feature map and regression branch feature map are divided into several block feature maps. When dividing the classification branch feature map and regression branch feature map into 8×8 blocks, if any of the resulting blocks are smaller than 8×8, zeros are padded to bring them up to 8×8. That is, the sizes of 20×20, 40×40, and 80×80 feature maps after being divided into 8×8 blocks are 3×3, 5×5, and 10×10, respectively. r8×8 is the row coordinate after the division, and c8×8 is the division value. The column coordinates are as follows; This embodiment takes a 20×20 feature map as an example for specific explanation. The 20×20 feature map is divided into 8×8 blocks. In addition to generating 4 standard 8×8 block feature maps, 2 8×4, 2 4×8 and 1 4×4 blocks will also be generated. These 5 blocks smaller than 8×8 are padded with "0" to fill to the size of 8×8. Finally, the 20×20 feature map is decomposed into 3×3 8×8 block feature maps. The coordinates of the 8×8 block feature map are (r8x8, c8x8). The processing order is (0,0), (0,1), (0,2), (1,0), (1,1), (1,2), (2,0), (2,1), (2,2).

[0036] S4. First, calculate the maximum confidence of the first 8×8 block feature map in the classification branch, i.e., the 8×8 block feature map with coordinates (0,0), in each channel direction, and save the corresponding channel, i.e. the corresponding category.

[0037] S5. Determine whether there are any pixels in the above block feature maps that have a confidence threshold. If there are, calculate the bounding box of the pixel with a confidence threshold in the corresponding block feature map of the regression branch. If not, calculate the next block feature map until the last block feature map. The coordinate information of the bounding box is [(x1,y1),(x2,y2)], where (x1,y1) is the coordinate of the upper left corner and (x2,y2) is the coordinate of the lower right corner.

[0038] like Figure 5 As shown, only the 8×8 block feature map with coordinates (0,1) has 9 pixels (R1C6, R1C7, R2C7, R3C6, R3C7, R4C7, R6C6, R6C7, R7C7) whose maximum confidence score is greater than the confidence threshold. Therefore, we only need to calculate the bounding boxes [(x1,y1),(x2,y2)] of these 9 pixels: R1C6, R1C7, R2C7, R3C6, R3C7, R4C7, R6C6, R6C7, R7C7, and R7C7, which greatly reduces the computational cost of the bounding boxes.

[0039] S6. After obtaining information on all pixels with a confidence level greater than the confidence threshold, perform non-maximum suppression calculation.

[0040] In this embodiment, the specific steps of step S6 are as follows:

[0041] S61. Write the information of all 9 pixels with confidence scores greater than the confidence threshold obtained in step S5 into the "ping" buffer of the "ping-pong" buffer. The pixel information is: {[coordinates], confidence score, type}. At this time, the control signal pingpong of the "ping-pong" buffer is 0, which means "ping" read and "pong" write.

[0042] S62. Read all pixels written to the "ping-pong" buffer, find the maximum confidence value for different types, and output it.

[0043] like Figure 6 As shown, in the first loop, two pixels, R2C7 and R3C7, of different types were found. Their confidence scores were the highest in their respective types, and the information of these two pixels, {[coordinates], confidence scores, type}, was output. R2C7 belongs to type 1 and R3C7 belongs to type 2.

[0044] S63. Read the pixels written to the "ping-pong" buffer one by one, calculate the IOU between the current candidate box and the candidate box with the maximum value of the same type. If the IOU is less than the threshold, write the information of the pixel into the "ping-pong" buffer. If the IOU is greater than the threshold, discard the pixel. The IOU (Intersection of Union) is the intersection of two boxes divided by the union of the two boxes.

[0045] Generally speaking, the IOU may have four situations, as shown in the appendix. Figure 4 As shown;

[0046] like Figure 6 As shown, in the first loop, pixels R6C6, R6C7, and R7C7 are of the same type as the maximum value pixel R3C7, all belonging to type 2. Furthermore, the IOU of these three pixels and the maximum value pixel R3C7 is less than the IOU threshold, so their information is written to the "ping-pong" buffer. Pixels R1C6 and R1C7 are of the same type as the maximum value pixel R2C7, all belonging to type 1. However, the IOU of R1C6, R1C7, and the maximum value pixel R2C7 is greater than the IOU threshold, so they are discarded. Pixels R3C6 and R4C7 are of the same type as the maximum value pixel R3C7, all belonging to type 2. However, the IOU of R3C6, R4C7, and the maximum value pixel R3C7 is greater than the IOU threshold, so they are discarded.

[0047] S64. After reading all pixels written to the "ping-pong" buffer, if there is at least one pixel with an IOU less than the threshold, the "ping-pong" buffer control signal pingpong is toggled, i.e., control signal pingpong = 1, representing "ping" write and "pong" read, and the process returns to step S62; Figure 6 As shown, in the first loop of this embodiment, the IOU of the three pixels R6C6, R6C7, and R7C7 is less than the threshold. Therefore, the "pingpong" buffer control signal is flipped, and the process returns to step S62 to continue running. At this time, the "pingpong" buffer contains the three pixels R6C6, R6C7, and R7C7.

[0048] Run step S62 again, read all the pixels written to the "ping-pong" buffer again, namely the three pixels R6C6, R6C7, and R7C7, find the maximum confidence value for different types, and output it.

[0049] like Figure 6 As shown, in the second loop, pixel R6C7 was found out from the three pixels. It has the highest confidence and the information of this pixel {[coordinates], confidence, category} is output. R6C7 belongs to category 2.

[0050] Then, read each pixel written to the "ping-pong" buffer and calculate the IOU between the current candidate box and the candidate box of R6C7, such as... Figure 6 As shown, in the second loop of this embodiment, there are no pixels with IOU less than the threshold, thus completing the non-maximum suppression calculation.

[0051] The above embodiments of the present invention are merely illustrative examples and are not intended to limit the implementation of the invention. Those skilled in the art can make other variations and modifications based on the above description. It is impossible to exhaustively list all possible implementations here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. An FPGA hardware implementation method for an anchor-free decoupled structure detection head, characterized in that, Includes the following steps: S1. Define the anchorless decoupled structure detection head as including classification and regression branches; S2. The detection head processes feature maps of different sizes, where the value of the classification branch feature map represents the confidence of the bounding box, and the value of the regression branch feature map represents the distance of the bounding box. S3. Define the size of the block feature map processed by the FPGA hardware computing unit each time, and divide the classification branch feature map and regression branch feature map into several block feature maps; S4. First, calculate the maximum confidence of the first block feature map in the classification branch in each channel direction, and save the corresponding channel, i.e. the corresponding category. S5. Determine whether there are any pixels in the above block feature maps that have a confidence threshold. If there are, calculate the bounding box of the pixel with a confidence threshold in the corresponding block feature map of the regression branch. If not, calculate the next block feature map until the last block feature map. The coordinate information of the bounding box is [(x1,y1),(x2,y2)], where (x1,y1) is the coordinate of the upper left corner and (x2,y2) is the coordinate of the lower right corner. S6. After obtaining information on all pixels with a confidence level greater than the confidence threshold, perform non-maximum suppression calculation.

2. The FPGA hardware implementation method for an anchorless decoupled structure detection head according to claim 1, characterized in that: The feature map in step S2 has three sizes: 20×20, 40×40, or 80×80.

3. The FPGA hardware implementation method for an anchorless decoupled structure detection head according to claim 2, characterized in that: The feature map size mentioned in step S3 is 8×8. The feature maps of 20×20, 40×40 and 80×80 are divided into 8×8 blocks and their sizes are 3×3, 5×5 and 10×10 respectively. If there are blocks smaller than 8×8 after division, "0" is added to fill the size to 8×8.

4. The FPGA hardware implementation method for an anchorless decoupled structure detection head according to claim 1, characterized in that: In step S4, the number of channels is 80.

5. The FPGA hardware implementation method for an anchorless decoupled structure detection head according to claim 1, characterized in that: The specific steps of step S6 are as follows: S61. Write all pixel information with confidence scores greater than the confidence threshold obtained in step S5 into the "ping" buffer of the "ping-pong" buffer. The pixel information is: {[coordinates], confidence score, type}. At this time, the control signal pingpong of the "ping-pong" buffer is 0, which means "ping" read and "pong" write. S62. Read all pixels written to the "ping-pong" buffer, find the maximum confidence value for different types, and output it. S63. Read the pixels written to the "ping-pong" buffer one by one, calculate the IOU between the current candidate box and the candidate box with the maximum value of the same type. If the IOU is less than the threshold, write the information of the pixel into the "ping-pong" buffer. If the IOU is greater than the threshold, discard the pixel. The IOU is the intersection of the two boxes divided by the union of the two boxes. S64. After reading all the pixels written to the "ping-pong" buffer, if there is at least one pixel with an IOU less than the threshold, the "ping-pong" buffer control signal pingpong is flipped, that is, the control signal pingpong = 1, which means "ping" write and "pong" read, and the process returns to step S62; if there is no pixel with an IOU less than the threshold, the non-maximum suppression calculation is completed.

Images