A YOLO V3 and YOLO V3 Tiny network switching method based on FPGA

Abstract

The invention discloses a YOLO V3 and YOLO V3 Tiny network switching method based on FPGA. The invention proposes that, without adding additional control logic, the method can control data to pass through or skip these functional modules by configuring parameters, so as to achieve parameter control to run or standby according to a preset YOLO_V3 network structure. Similarly, the parameters can be configured into a YOLO V3 Tiny3 network structure. Such a design can switch the role of the network, and can also dynamically adjust the selection of the network as needed, so as to achieve the requirements of higher accuracy, faster speed and lower power consumption.

Description

Technical Field

[0001] The present invention relates to the technical field of network switching methods, and in particular to a YOLO V3 and YOLO V3Tiny network switching method based on FPGA. Background Art

[0002] With the increasing popularity of artificial intelligence technology, its application in all walks of life is becoming more and more extensive. However, although target detection based on FPGA benefits from the high-speed computing and low power consumption of FPGA, it is also limited by its hardware characteristics. Limited resources can only accommodate a limited number of network models. This has caused an embarrassing problem. The accuracy of the YOLO V3 network under FPGA is about 10% to 15% higher than that of the YOLO V3 Tiny. However, the frame rate of the YOLO V3 network based on FPGA is only 1.836FPS, while the YOLO V3 Tiny has 26.141FPS under the same environment. However, under the common requirements of accuracy and speed, one FPGA development board cannot accommodate the two network structures.

[0003] Schematic diagram of the YOLO model structure under traditional FPGA Figure 1 As shown in the figure, the main convolution module consists of N_PE computing units. The weights are cached to the FPGA through the AXI bus, where these convolution units perform calculations and other operations at the same time, and then write the results into the cache. This design structure is simple and direct, but when the logic is designed, the shape of the network has been determined, and subsequent data can only flow according to the current design, and the network loses its variability.

[0004] The above-mentioned FPGA (Field Programmable Gate Array) is a field programmable gate array. It is a product further developed on the basis of programmable devices such as PAL, GAL, CPLD, etc. It appears as a semi-custom circuit in the field of application-specific integrated circuits (ASICs), which not only solves the shortcomings of customized circuits, but also overcomes the shortcomings of the limited number of gate circuits of the original programmable devices. FPGA adopts the concept of logic cell array LCA, which includes three parts: configurable logic module CLB, input and output module IOB and internal wiring. Embedded programmable logic array has the following advantages: (1) It can realize soft and hard IP cores; (2) It is independent of the process; (3) It is easy to integrate into the system-on-chip (SoC); (4) The compact embedded programmable logic array can achieve higher logic density at low power consumption; (5) The target application is highly customizable.

[0005] According to the above description, the present invention proposes a switching method between YOLO V3 Tiny and YOLO V3 networks based on an embedded programmable logic array. This method solves the conflict problem of requiring both accuracy and speed under limited FPGA resources, further simplifies the integrated control logic, and reduces redundant design, thereby achieving optimized configuration of the system in terms of power consumption, area and speed to achieve higher efficiency. Summary of the invention

[0006] The object of the present invention is to provide a YOLO V3 and YOLO V3 Tiny network switching method based on FPGA to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A YOLO V3 and YOLO V3 Tiny network switching method based on FPGA. The operation of the calculation program is initialized by a register instruction driver. The control parameters, weight parameters, quantization parameters and input data are input into DataBuffer. The control parameters will automatically select the calculation logic and calculation sequence of the entire project for calculation. After the calculation is completed, the data is written back to DDR to complete the memory to memory operation.

[0009] As a further solution of the present invention: the operation of the calculation program is initialized by the register instruction driver, and the control parameters, weight parameters, quantization parameters and input data are input into the DataBuffer. The control parameters will automatically select the calculation logic and calculation order of the entire project for calculation. After the calculation is completed, the data is written back to the DDR to complete the memory to memory operation.

[0010] As a further solution of the present invention: the control parameters generated by the parameter table are first entered into the controller for parsing, and then the control parameters are distributed to various interfaces in the program, and the FM data read from the DDR is controlled to be sent to the functional modules that need to pass through or skip, and finally the control data is written back to the DDR or directly output, and the write completion signal will drive the controller to continue to distribute the control parameters of the next layer of network structure, and thus the cycle is repeated to achieve the preset network structure.

[0011] As a further solution of the present invention: when the parameters are sent to the controller for analysis, at this time, the control signal can be manually input into the judgment logic as needed, or the network structure can be selected by presetting in the parameter table. After the network structure is selected, the controller will distribute the control parameters of the corresponding network structure to each module, and the control program will work.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] 1. The network switching method proposed in the present invention can realize the network switching according to the preset judgment conditions or user needs by sending the configured control parameters into the FPGA without using the valuable additional resources of the FPGA.

[0014] 2. The switching method of the present invention can completely pre-configure all parameters outside the program according to the needs to generate a binary file. It only needs to input the generated binary file into the program and distribute the parameters to each interface. This can reduce the design pressure of the FPGA on the one hand, and on the other hand, the FPGA has a very rare flexibility without adding too much logic. Because the configuration of the parameters is preset outside the program, the FPGA resources can be further reduced, and the saved resources can be further optimized to improve the efficiency.

[0015] 3. Another advantage of the present invention is that if the YOLO model is optimized, pruned, etc., there is no need to adjust the program. It is only necessary to modify the parameters in the parameter table and adjust them to the optimized structure to complete the adjustment, which greatly reduces the workload of secondary development. This is an effect that cannot be achieved by traditional neural network development based on FPGA. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A traditional schematic diagram of an FPGA-based YOLO V3 and YOLO V3 Tiny network switching method using FPGA to implement YOLO.

[0017] Figure 2 A schematic diagram of a network switching method in a YOLO V3 and YOLO V3 Tiny network switching method based on FPGA.

[0018] Figure 3 A schematic diagram of the internal interaction relationship in an FPGA-based YOLO V3 and YOLO V3 Tiny network switching method.

[0019] Figure 4 A schematic diagram of the internal interaction relationship in an FPGA-based YOLO V3 and YOLO V3 Tiny network switching method.

[0020] Figure 5 A schematic diagram of the internal judgment conditions in an FPGA-based YOLO V3 and YOLO V3 Tiny network switching method.

[0021] Figure 6Schematic diagram of the YOLO_V3_Tiny control parameter table in an FPGA-based YOLO V3 and YOLO V3 Tiny network switching method.

[0022] Figure 7 Schematic diagram of the YOLO_V3 control parameter table in an FPGA-based YOLO V3 and YOLO V3 Tiny network switching method.

[0023] Figure 8 A schematic diagram of the external interaction relationship in an FPGA-based YOLO V3 and YOLO V3 Tiny network switching method. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] See also Figures 1 to 4 In an embodiment of the present invention, a YOLO V3 and YOLO V3 Tiny network switching method based on FPGA is provided. The YOLO V3 and YOLO V3 Tiny network switching method based on FPGA is superior to the traditional neural network design architecture under FPGA in the following aspects:

[0026] By optimizing the architecture using the method of the present invention, YOLO_V3 and YOLO_V3_Tiny can be switched arbitrarily according to needs, achieving the effect of both accuracy and speed under the limited resources of FPGA.

[0027] like Figure 2 As shown in the figure, the operation of the calculation program is initialized by the register instruction (command) driver, and the control parameters (General Para Control), weight parameters (Weight Control), quantization parameters (Quantize Control) and input data (Feature Map) are input into the DataBuffer. The control parameters will automatically select the calculation logic (Convolution, BN, Scale, Relu, Eltwise, Interp, Quantize) and calculation order of the entire project for calculation. After the calculation is completed, the data is written back to DDR to complete the memory to memory operation.

[0028] A unique FPGA design architecture Figure 2As shown in the figure, the functions required by the two networks are separated into separate modules, and the controller outputs preset control signals to control the direction of the data flow, so as to achieve the effect of switching different networks as needed. In this way, the switching between YOLO V3 Tiny and YOLO V3 networks can be achieved without adding redundant designs based on limited resources.

[0029] The structural design of the network switching method of the present invention is as follows: Figure 3 As shown, the control parameters generated by the parameter table are first parsed in the controller, and then the control parameters are distributed to various interfaces in the program, and the FM data in the DDR is read and sent to the functional modules that need to be passed or skipped. Finally, the control data is written back to the DDR or output directly. The write completion signal will drive the controller to continue to distribute the control parameters of the next layer of network structure, and the cycle will be repeated to achieve the preset network structure.

[0030] The structural design of the present invention is as follows: Figure 4 When the parameters are sent to the controller for analysis, you can manually input the control signal into the judgment logic according to your needs, or preset it in the parameter table to select the network structure. After the network structure is selected, the controller will distribute the control parameters of the corresponding network structure to each module, and the control program will work. In the judgment module.

[0031] like Figure 5 On the one hand, it is to generate start-stop signals to control the program operation, and on the other hand, there will be some preset considerations for special cases. For example, a certain layer in YOLO needs to be Eltwise, and when the layer counting logic is met, the preset situation under this condition will be triggered. Or when the entire network operation is completed, it is decided whether to switch the model structure or continue to use the structure to continue running according to the preset or manual control. It can also be manually operated or preset to control the program to work or standby, further reducing power consumption. Therefore, the present invention can be flexibly changed in the judgment logic according to needs.

[0032] like Figure 6 , Figure 7 As shown, Figure 6 For Tiny Network, Figure 7 For V3 network, the "Type" field describes the calculation method of this layer of network. Based on this field, the functions that need to be implemented in this layer can be clearly determined, thereby determining the control parameters of this layer. The "Calculation Module" field indicates the calculation module that needs to be started to complete the calculation of this layer, and fill in the corresponding control parameters according to the calculation module. The "OutPut Size" field describes the size of the output of this layer, which can determine the data size of the output of this layer, and this field also determines the data size written to DDR, write shape and other parameters.

[0033] According to the above instructions, the user only needs to fill in the parameters and shape of each layer in the network, the read and write addresses of the data and other parameters, generate a binary file and send it to the program, and the program will run according to the control parameters. Specific embodiment 1

[0035] Suppose in a certain experimental scenario, a drone identifies ground personnel targets at high altitude. In the process of fast movement of the drone, if it uses traditional methods, either the tiny network, then the frame rate is fast enough, but the accuracy is insufficient, and many targets will be filtered out; and although the V3 network has high enough accuracy, the frame rate is very low due to the problem of computational complexity. When the drone moves at high speed, it is very likely to miss the target completely due to frame loss.

[0036] Therefore, under this method, Figure 8 As shown, when the drone is moving at high speed, the sensor transmits the image data to the FPGA. By default, the tiny network is used to meet the high frame rate requirements. After the target is found, the preset logic in the judgment module, such as the case of a small number of categories with low confidence, immediately switches to the V3 network, or the network can be replaced through manual intervention and identification is performed again. At this time, although the frame rate is low, the key target can be completely captured, so that it can achieve the frame rate and meet the high enough accuracy in some scenarios. After completing the target identification, preset conditions can be added to the judgment logic as needed, and the default Tiny network can be automatically or manually switched. Similarly, on the drone, when it does not need to work, the program can be put on standby to further reduce power consumption. Therefore, compared with the traditional design architecture, the proposed method has a great improvement in speed, accuracy, power consumption, design complexity, flexibility, etc.

[0037]

[0038] Table 1 Specific embodiment 2

[0040] Assuming that based on the design architecture of this method, in the V3 network, such as Figure 7 , 75 layers of Conv are pruned to 41 layers of Conv. At this time, you only need to modify the number of layers and shape of the parameter table, the address of the read and write data, the modules that each layer of the network needs to pass through, and other parameters according to the pruned network structure. There is no need to adjust the FPGA code at all, which greatly reduces the workload of secondary development to achieve the requirement of quickly changing the network structure.

[0041] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A YOLOV3 and YOLOV3Tiny network switching method based on FPGA, characterized in that: The operation of the calculation program is initialized by the register instruction driver, and the control parameters, weight parameters, quantization parameters and input data are input into the DataBuffer. The control parameters will automatically select the calculation logic and calculation order of the entire project for calculation. After the calculation is completed, the data is written back to the DDR to complete the memory operation. When the parameters are sent to the controller for analysis, the control signal is manually input into the judgment logic, or the network structure is selected by preset in the parameter table. After the network structure is selected, the controller will distribute the control parameters of the corresponding network structure to each module, and the control program will work. Among them, the control parameters generated by the parameter table first enter the controller for parsing, and then the control parameters are distributed to various interfaces in the program, and the FM data in the DDR is controlled to be sent to the functional modules that need to pass through or skip. Finally, the control data is written back to the DDR or directly output. The write completion signal will drive the controller to continue to distribute the control parameters of the next layer of network structure, and thus cycle to achieve the preset network structure.

Images