Intelligent processing unit executing non-maximum suppression algorithm

Abstract

The invention discloses an intelligent processing unit for executing a non-maximum suppression algorithm, and relates to the technical field of data processing. The smart processing unit executes a non-maximum suppression algorithm to filter a plurality of frames, each frame having a confidence. The intelligent processing unit comprises a register circuit and a calculation circuit. The computing circuit performs the steps of: storing a plurality of frames to the register circuit; selecting a plurality of candidate frames from the plurality of frames, the confidence of the plurality of candidate frames being greater than a first threshold; calculating the area of each candidate frame; selecting the candidate frame with the maximum confidence from the plurality of candidate frames; calculating an intersection area of the candidate frame with the maximum confidence coefficient and a target candidate frame, wherein the target candidate frame is a frame except the candidate frame with the maximum confidence coefficient in the plurality of candidate frames; calculating the union set area of the candidate box with the maximum confidence coefficient and the target candidate box; and when the ratio of the intersection area to the union area is greater than a second threshold value, shielding the target candidate frame.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to an intelligent processing unit for performing a non-maximum suppression algorithm. BACKGROUND

[0002] In the field of image processing, a non-maximum suppression algorithm is often used to filter candidate boxes so as to avoid repeated processing of overlapping boxes. A general purpose computer is usually configured with a central processing unit (CPU) and a dynamic random access memory (DRAM). However, because the non-maximum suppression algorithm involves a large amount of computation, and the CPU is not a hardware specifically designed to perform the algorithm, the general purpose computer cannot efficiently perform the algorithm. Furthermore, because the CPU needs to frequently access the DRAM when performing the non-maximum suppression algorithm, the execution speed is also limited by the speed at which the CPU accesses the DRAM. SUMMARY

[0003] In view of the deficiencies of the prior art, an object of the present application is to provide an intelligent processing unit for performing a non-maximum suppression algorithm to improve the deficiencies of the prior art.

[0004] An embodiment of the present application provides an intelligent processing unit. The intelligent processing unit is used to perform a non-maximum suppression algorithm to filter a plurality of boxes, each box having a confidence. The intelligent processing unit comprises a register circuit and a calculation circuit. The calculation circuit is coupled to the register circuit and is used to perform the following steps: storing the plurality of boxes to the register circuit; selecting a plurality of candidate boxes from the plurality of boxes, wherein the confidence of the plurality of candidate boxes is greater than a first threshold; calculating an area of each candidate box; selecting a candidate box with the greatest confidence from the plurality of candidate boxes; calculating an intersection area of the candidate box with the greatest confidence and a target candidate box, the target candidate box being one of the plurality of candidate boxes other than the candidate box with the greatest confidence; calculating a union area of the candidate box with the greatest confidence and the target candidate box; and masking the target candidate box when a ratio of the intersection area to the union area is greater than a second threshold.

[0005] The technical means embodied by the embodiments of the present application can improve at least one of the shortcomings of the prior art, and thus the present application can improve the execution efficiency of the non-maximum suppression algorithm compared to the prior art.

[0006] The features, implementations, and effects of the present application will be described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0008] Figure 1 is a functional block diagram of an embodiment of the electronic device of the present application;

[0009] Figures 2A-2B is a flow chart of an embodiment of the image processing method of the present application;

[0010] Figure 3A is a schematic diagram showing two frames overlapping;

[0011] Figure 3B is a schematic diagram showing two frames not overlapping; and

[0012] Figure 4 is a schematic diagram of an embodiment of the vector processor 122 and the system frequency CLK of the present application.

[0013] Reference signs:

[0014] 100: electronic device;

[0015] 101: intelligent processing unit;

[0016] 102: memory;

[0017] 110: register circuit;

[0018] 112: mask register;

[0019] 114: scalar register;

[0020] 116: virtual register;

[0021] 120: calculation circuit;

[0022] 122: vector processor;

[0023] 124: scalar processor;

[0024] 130: instruction control circuit;

[0025] 140: DMA circuit;

[0026] INST: instruction;

[0027] VR0: first block;

[0028] VR1: second block;

[0029] VR2: The third block;

[0030] VR3: The fourth block;

[0031] 200: Image processing methods;

[0032] S205, S210, S212, S215, S220, S225, S230, S235, S240, S245, S250, S255, S260: steps;

[0033] 310: target candidate box;

[0034] 315: intersection area;

[0035] 320: The candidate box with the highest confidence;

[0036] 410_1, 410_2, 410_3, 410_M: multiplication circuit;

[0037] CLK: system frequency;

[0038] T1: cycle. DETAILED DESCRIPTION

[0039] The technical terms used in the following descriptions refer to the customary terms in this technical field. If some terms are explained or defined in this specification, the interpretation of those terms shall be based on the explanation or definition in this specification.

[0040] The disclosure of this application includes an intelligent processing unit that performs a non-maximum suppression algorithm. Because some components of the intelligent processing unit of this application may be known components individually, the following description will omit details of known components without affecting the full disclosure and feasibility of the device invention. Furthermore, part or all of the image processing method of this application may be implemented in software and / or firmware and may be executed by the intelligent processing unit of this application or its equivalent. Without affecting the full disclosure and feasibility of the method invention, the following description of the method invention will focus on the steps rather than the hardware.

[0041] Figure 1is a functional block diagram of an embodiment of an electronic device of the present application. The electronic device 100 includes an intelligence processing unit (IPU) 101 and a memory 102 (e.g., a Dynamic Random Access Memory (DRAM)) coupled to each other. The intelligence processing unit 101 includes a register circuit 110, a computation circuit 120, an instruction control circuit 130, and a direct memory access (DMA) circuit 140 coupled to each other.

[0042] The register circuit 110 includes a mask register 112, a scalar register 114, and a virtual register 116. The virtual register 116 includes a first block VR0, a second block VR1, a third block VR2, and a fourth block VR3.

[0043] The computation circuit 120 includes a vector processor 122 and a scalar processor 124.

[0044] The instruction control circuit 130 controls the computation circuit 120 and the DMA circuit 140 according to an instruction INST. In some embodiments, the instruction control circuit 130 includes a storage circuit (e.g., a volatile memory) to buffer the instruction INST.

[0045] Figures 2A-2B is a flowchart of an embodiment of an image processing method of the present application. The image processing method 200 can be performed by the intelligence processing unit 101 and includes the following steps.

[0046] Step S205: The DMA circuit 140 moves or copies data (including but not limited to vertex coordinates and confidence levels) of a plurality of boxes from the memory 102 to the first block VR0 of the virtual register 116. In the virtual register 116, each box corresponds to an index value. A box has four vertices, and a coordinate of a vertex corresponds to a position of the vertex in a frame.

[0047] Step S210: The computation circuit 120 (more specifically, the vector processor 122) selects, from the plurality of boxes in the virtual register 116, boxes with confidence levels greater than a first threshold value as candidate boxes, and stores index values of the plurality of candidate boxes to the second block VR1 of the virtual register 116. In some embodiments, the first threshold value is stored in the scalar register 114.

[0048] Step S212: The computing circuit 120 (more specifically, the vector processor 122) stores the data of the plurality of candidate boxes into the third block VR2 of the virtual register 116 based on the index values of the plurality of candidate boxes.

[0049] Step S215: The computing circuit 120 (more specifically, the vector processor 122) reads the data of the candidate boxes from the third block VR2 of the virtual register 116 and calculates the area of each candidate box. The vector processor 122 can calculate the area of a candidate box based on the coordinates of the vertices of the candidate box. In some embodiments, the area of a candidate box is stored in the third block VR2 of the virtual register 116.

[0050] Step S220: The computing circuit 120 (more specifically, the vector processor 122) finds out the candidate box with the highest confidence among the plurality of candidate boxes (stored in the third block VR2 of the virtual register 116) based on the mask data of the mask register 112 and stores the index value of the candidate box with the highest confidence (i.e., the reference index value) into the scalar register 114. More specifically, if there are N candidate boxes (N is a positive integer) in the third block VR2 of the virtual register 116, the mask data in the mask register 112 includes N corresponding bits indicating whether the N candidate boxes are masked. When a candidate box is masked (e.g., the corresponding bit in the mask register 112 is modified from a first logic value (e.g., 0) to a second logic value (e.g., 1)), the computing circuit 120 excludes the candidate box (i.e., the candidate box does not participate in the comparison) in step S220. The first time step S220 is executed, no candidate box is masked. If the computing circuit 120 cannot find a candidate box with the highest confidence in this step (e.g., when all candidate boxes are masked), the reference index value is an invalid value (e.g., a certain preset value).

[0051] Step S225: The computing circuit 120 (more specifically, the vector processor 122) determines whether the reference index value is valid. If yes, step S235 is executed; otherwise, step S230 is executed.

[0052] Step S230: The DMA circuit 140 moves or copies all the boxes in the fourth block VR3 of the virtual register 116 to the memory 102. The fourth block VR3 is used to store output boxes (to be described in detail below).

[0053] Step S235: The computing circuit 120 (more specifically, the vector processor 122) masks the most confident candidate box by modifying the bit in the mask register 112 corresponding to the most confident candidate box or its reference index value. In this way, the most confident candidate box selected in the current round of operation will not participate in the next round of step S220.

[0054] Step S240: The computing circuit 120 (more specifically, the vector processor 122) reads the data of the most confident candidate box from the third block VR2 of the virtual register 116 based on the reference index value, and stores the data of the most confident candidate box to the scalar register 114.

[0055] Step S245: The computing circuit 120 (more specifically, the scalar processor 124) moves or copies the data of the most confident candidate box from the scalar register 114 to the fourth block VR3 of the virtual register 116. As mentioned in step S230, the fourth block VR3 stores the output box, i.e. the box to be found by the non-maximum suppression algorithm.

[0056] Step S250: The computing circuit 120 (more specifically, the vector processor 122) calculates the intersection area of the most confident candidate box and other candidate boxes. More specifically, the computing circuit 120 takes the unmasked candidate boxes as target candidate boxes in turn, and then calculates the intersection area of the most confident candidate box and the target candidate box.

[0057] Step S255: The computing circuit 120 (more specifically, the vector processor 122) calculates the union area of the most confident candidate box and other candidate boxes. More specifically, the computing circuit 120 takes the unmasked candidate boxes as target candidate boxes in turn, and then calculates the union area of the most confident candidate box and the target candidate box.

[0058] Figure 3A is a schematic diagram of two boxes overlapping, and Figure 3B is a schematic diagram of two boxes not overlapping. In Figure 3A , the target candidate box 310 and the most confident candidate box 320 have an intersection region 315 (i.e. the intersection area of the two boxes is equal to the area of the intersection region 315), while in Figure 3B , the two boxes have no intersection region (i.e. the intersection area of the two boxes is equal to 0). Whether two boxes overlap and how to calculate the intersection area and the union area of two boxes based on the vertex coordinates of the two boxes are well known to those skilled in the art, and thus will not be described in detail.

[0059] Step S260: The computing circuit 120 (more specifically, the vector processor 122) masks the bounding box whose intersection over union (IoU) is greater than the second threshold by modifying the mask register 112. When the target bounding box has an intersection over union with the bounding box of the highest confidence that is greater than the second threshold, the target bounding box is determined to be duplicated with the bounding box of the highest confidence, and is masked (e.g., by setting the target bit in the mask register 112 corresponding to the target bounding box to logic 1). After step S260, the flow goes back to step S220 to find the next bounding box of the highest confidence from the remaining bounding boxes.

[0060] In some embodiments, the computing circuit 120 (more specifically, the vector processor 122) performs steps S250-S260 based on inequality (1) (where A is the area of the target bounding box, B is the area of the bounding box of the highest confidence, I is the intersection area, U is the union area (i.e., U = A + B - I), and t is the second threshold). That is, step S255 can be omitted. When inequality (1) is true, the target bounding box is masked.

[0061] I / U > t;

[0062] I / (A+B-I) > t;

[0063] I > t*(A+B-I);

[0064] I > t*(A+B) - t*I;

[0065] I + t*I > t*(A+B);

[0066] I*(1+t) > t*(A+B);

[0067] I > (t / (1+t))*(A+B) …… (1);

[0068] Figure 4 is a schematic diagram of an embodiment of the vector processor 122 and the system frequency CLK of the present application. The vector processor 122 operates according to the system frequency CLK, and includes M multiplication circuits (410_1, 410_2, 410_3, …, 410_M) (M is an integer greater than 1). The period of the system frequency CLK is T1. Since the vector processor 122 includes M multiplication circuits, the vector processor 122 can substantially simultaneously calculate the areas of M bounding boxes (step S215) in one period T1 of the system frequency CLK. The greater M is, the higher the efficiency of the vector processor 122 is. In some embodiments, M is 16.

[0069] In summary, the intelligent processing unit 101 of the present application is more suitable for performing the non-maximum suppression algorithm than the conventional CPU, greatly speeding up the image processing speed (e.g., faster completion of the image processing method 200). Figures 2A-2B

[0070] Although the embodiments of the present application are described above, these embodiments are not intended to limit the present application, and those skilled in the art can make changes to the technical features of the present application according to the explicit or implicit contents of the present application. Any such changes may fall within the scope of the patent protection sought by the present application, and the scope of the patent protection of the present application is defined by the patent application range of the specification.​

Claims

1. An intelligent processing unit, characterized in that: The intelligent processing unit is configured to execute a non-maximum suppression algorithm to filter a plurality of boxes, each box having a confidence level, and includes: a register circuit; and A calculation circuit, coupled to the register circuit, is configured to perform the following steps: storing the plurality of frames to the register circuit; Selecting a plurality of candidate frames from the plurality of frames, wherein the confidence levels of the plurality of candidate frames are greater than a first threshold; Calculate an area of ​​each candidate box; Selecting a candidate frame with the highest confidence from the multiple candidate frames; Calculating an intersection area between the candidate frame with the highest confidence and a target candidate frame, where the target candidate frame is a frame among the multiple candidate frames except the candidate frame with the highest confidence; Calculating a union area of ​​the candidate box with the maximum confidence and the target candidate box; and The target candidate frame is masked when a ratio of the intersection area to the union area is greater than a second threshold.

2. The intelligent processing unit according to claim 1, wherein: The intelligent processing unit operates according to a system frequency, the calculation circuit includes a vector processor, the vector processor includes M multiplication circuits, and the vector processor uses the M multiplication circuits to calculate the areas of M candidate boxes in one cycle of the system frequency, where M is an integer greater than 1.

3. The intelligent processing unit according to claim 2, wherein: The register circuit includes a mask register, the mask register includes a plurality of bits corresponding to the plurality of candidate boxes, and the vector processor further performs the following steps: A target bit corresponding to the target candidate frame in the mask register is modified to mask the target candidate frame.

4. The intelligent processing unit according to claim 3, wherein: The vector processor further performs the following steps: A bit corresponding to the candidate box with the highest confidence in the mask register is modified to mask the candidate box with the highest confidence.

5. The intelligent processing unit according to claim 2, wherein: The register circuit includes a virtual register and a scalar register, the candidate box with the highest confidence corresponds to a reference index value in the virtual register, and the vector processor further performs the following steps: The reference index value is stored in the scalar register.

6. The intelligent processing unit according to claim 5, wherein: The vector processor further performs the following steps: Data of the candidate box with the highest confidence is read from the virtual register based on the reference index value, and the data is stored in the scalar register.

7. The intelligent processing unit according to claim 6, wherein: The virtual register includes a first block and a second block. The data is stored in the first block before being stored in the scalar register. The computing circuit includes a scalar processor, and the scalar processor moves or copies the data from the scalar register to the second block.

8. The intelligent processing unit according to claim 7, wherein: The intelligent processing unit further includes a direct memory access circuit coupled to a memory, wherein the direct memory access circuit is coupled to the virtual register and the memory, and the direct memory access circuit moves or copies the data in the second block to the memory.

9. The intelligent processing unit according to claim 5, wherein: The virtual register includes a first block, a second block, and a third block. The plurality of frames are stored in the first block. Each of the plurality of frames has an index value. The vector processor further performs the following steps: storing the plurality of index values ​​of the plurality of candidate frames in the second block; and Data of the candidate frames are stored in the third block based on the index values ​​of the candidate frames.

10. The intelligent processing unit according to claim 1, wherein: The target candidate frame has a first area, the candidate frame with the highest confidence has a second area, and the step of shielding the target candidate frame includes: When the inequality I>(t / (1+t))*(A+B) is true, the target candidate box is masked; Wherein, I is the intersection area, t is the second threshold, A is the first area, and B is the second area.

Images