A real-time analysis method for FPGA neural network accelerators

Abstract

The present invention relates to a real-time performance analysis method for FPGA neural network accelerators, which belongs to the field of real-time system technology and includes the following steps: pre-analyzing the FPGA system-on-chip to obtain DPU deployment information; extracting hardware factors that will affect the CNN reasoning of the FPGA system-on-chip based on the principles of different hardware components, and establishing a system hardware model and a DPU execution model through experimental verification; establishing a generalized formula based on the model to construct a universal shared bus arbitration and memory access contention model for the FPGA system-on-chip. The worst-case values ​​of the parameters in the universal shared bus arbitration and memory access contention model are measured experimentally; the parameter values ​​are substituted into the model formula, and the worst-case execution time limit of each CNN reasoning is statically analyzed based on the FPGA system-on-chip hardware model. This application considers the FPGA system-on-chip in a multi-DPU deployment environment and is a further attempt at static real-time performance analysis of multi-core heterogeneous systems.

Description

Technical Field

[0001] The present invention relates to a real-time analysis method for an FPGA neural network accelerator, and belongs to the technical field of real-time systems. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] With the continuous development of artificial intelligence, convolutional neural networks (CNNs) are widely used in real-time embedded systems. For example, the open-source autonomous driving system Apollo uses multiple CNN network models to implement tasks such as image classification and obstacle detection. Real-time embedded systems have high performance requirements. To improve the performance of neural network inference on resource-constrained embedded devices, the industry often uses heterogeneous embedded computing platforms instead of traditional embedded platforms. Typical heterogeneous processors used for CNN inference include general-purpose graphics processing units (GPUs), field-programmable gate arrays (FPGAs), and neural network processing units (NPUs).

[0004] Among different types of heterogeneous processors, FPGAs offer the best reconfigurability and energy efficiency, leading to a significant amount of research on FPGA-based CNN accelerators. Through customized hardware design and hardware-software collaboration, the performance and energy efficiency of FPGA-based CNN accelerators have significantly improved over the past decade. In 2019, Xilinx released the Vitis AI framework and Deep Learning Processing Unit (DPU). The Xilinx DPU is a programmable accelerator specifically designed for convolutional neural network inference. Built on a specialized instruction set, it facilitates the rapid and efficient deployment and acceleration of CNNs on a variety of Xilinx FPGA devices. Due to Xilinx FPGA's leading position in the industry, deploying the Xilinx DPU based on the Vitis AI framework has now become a more common development process for accelerating neural networks using FPGAs.

[0005] Most FPGA-based CNN inference acceleration research, like Xilinx DPUs, focuses on improving the performance and energy efficiency of accelerators, while ignoring the real-time nature of the design. In real-time embedded systems, worst-case execution time (WCET) analysis is crucial for system design safety and resource utilization.

[0006] However, the inventors of this disclosure discovered that the only current static real-time analysis framework for CNN inference based on Xilinx DPUs analyzes only real-time systems deployed on a single DPU. Furthermore, the framework's estimated WCET bound is 2.16 times higher than the actual average measured value, potentially leading to significant resource over-provisioning in real-time system designs. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this paper provides a method for real-time analysis of FPGA systems deploying multiple DPUs. By establishing a model based on shared bus arbitration and memory access contention, this paper proposes the first static real-time analysis framework for CNN inference in a multi-DPU environment. This framework significantly reduces the worst-case execution time (WCET) bound for a single DPU analyzed in previous papers and helps system designers estimate the design space before design, thereby better utilizing system resources during the design process.

[0008] The technical solutions of the present invention are as follows:

[0009] A real-time performance analysis method for FPGA neural network accelerators includes the following steps:

[0010] The worst-case values ​​of parameters in the general shared bus arbitration and memory access contention models are measured experimentally;

[0011] Substituting the parameter values ​​into the model formula, the worst-case execution time (WCET) bound of each CNN inference is statically analyzed based on the FPGA system-on-chip hardware model.

[0012] Preferably, the method for constructing a universal shared bus arbitration and memory access contention model for an FPGA system-on-chip comprises the following steps:

[0013] Perform preliminary analysis on the FPGA system-on-chip to obtain DPU deployment information;

[0014] Based on the principles of different hardware components, we extracted the hardware factors that affect CNN reasoning in the FPGA system-on-chip. Through experimental verification, we established the system's hardware model and the DPU's execution model.

[0015] Based on the model, a generalized formula is established to construct a universal shared bus arbitration and memory access contention model for FPGA on-chip systems.

[0016] Further preferably, the pre-analysis of the FPGA system on chip is performed using Vivado software when deploying the DPU, or using VitisAnalyzer software for analysis after deployment.

[0017] Further preferably, the DPU deployment information refers to information such as the number and model of DPUs deployed on the FPGA, the deployment method of the DPU, and the connection method of the PS system of the PGA.

[0018] Further preferably, the effects of different hardware components on CNN reasoning are verified through experiments, specifically:

[0019] Using the control variable method, each time the DPU is redeployed, only the information of one hardware component is changed. CNN inference is performed multiple times, and the average inference time is recorded.

[0020] Based on the average inference time of CNN under different hardware configurations, we observe the impact of different hardware components on the inference time of CNN.

[0021] Further preferably, the hardware model of the system is established as follows: the instruction interfaces of all DPUs share the LPD PL-PS interface through the same AXI interconnection, and at the same time, the data transactions sent from the data interfaces of all DPUs will share the AXI interconnection in the PS part, and finally accept the transaction arbitration of the DDR port arbiter together with the instruction transactions.

[0022] Furthermore, it is preferred that a DPU execution model be established for each DPU. Specifically, the model includes input, execution process, and output. The execution process divides a DPU's CNN inference into four phases: read instruction phase (INS), read data phase (R), write data phase (W), and build phase (ELAB). The read instruction phase refers to the process in which the DPU reads DPU instructions stored in the DDR through the instruction interface; the read data phase refers to the process in which the DPU reads DPU data stored in the DDR through its two data interfaces; the write data phase refers to the process in which the DPU writes the calculated data to the DDR through its two data interfaces; and the build phase refers to the time when the DPU performs only calculations without any memory access. Past research has shown that the DPU's read data phase is performed simultaneously with the read instruction phase and the write data phase.

[0023] More preferably, the generalized formula is established as follows:

[0024] Step 1: Based on the DPU execution model, we can obtain the general DPU inference time calculation formula:

[0025] T total =max{T R ,T INS +T W}+T ELAB

[0026] Since there is no memory access during the DPU establishment phase, T ELAB It can be considered as a constant value calculation, which is related to the computational workload of the CNN network and the computing power of the DPU (that is, the computational workload of the CNN network divided by the computing power of the DPU, minus the DPU memory access time. Generally speaking, the value of this parameter is very small and can be ignored.). Therefore, in the following part, we only need to analyze the other three stages. We have the following calculation method:

[0027]

[0028] Among them, T trsf Refers to the sum of the transmission time of all transactions. The transmission time here does not take into account conflicts with other DPUs. wait Refers to the transaction waiting time caused by conflicts with other DPUs; the calculation methods of these two times will be introduced in detail later.

[0029] Step 2: Obtain the calculation method of transmission time based on AXI read and write specifications and DPU principles;

[0030] The reading and writing of data and instructions are completed through AXI transactions. An AXI transaction includes multiple AXI transfers. A complete read transaction includes a read address request and multiple data returns. A complete write transaction includes a write address request, multiple data writes, and a write return. For the convenience of public representation, we record the transmission time of an address request as For write transactions, the transmission time here includes the write return; the read and write time of a data is recorded as The read and write time required for different PL-PS interfaces is These times as parameters can be measured experimentally;

[0031] Because the Xilinx DPU has two data interfaces, both of which can read and write data to and from the DDR, and the DPU has only one instruction interface responsible for reading instructions from the DDR, the transmission time calculation method for the instruction read phase is slightly different from that for the data read and write phase:

[0032] For the DPU's read instruction phase, the transmission time is calculated as follows:

[0033]

[0034] Among them, N INS Indicates the total number of read instruction transactions for a CNN inference, Δ INS Indicates the total number of instructions for a CNN inference; (Since the LPD interface bit width is 32 bits, one instruction is recorded as 32 bits).

[0035] For the DPU's read / write data phase, the transfer time is calculated as follows:

[0036]

[0037] in, and Respectively represent the total number of read / write data transactions issued by the two data interfaces during a CNN inference process, and They represent the total amount of data passing through the two data interfaces during a CNN inference process (since the HP and HPC interface widths are 128 bits, one data point is recorded as 128 bits);

[0038] Step 3: Obtain a calculation method for the waiting time based on the FPGA SoC hardware model and the arbitration principles of AXI interconnection and DDR;

[0039] Since both the AXI interconnect and the DDR controller use polling arbitration by default, the DDR controller can be considered a special AXI interconnect and a general arbitration model formula can be established. An AXI interconnect in the Xilinx DPU deployment process usually includes multiple master interfaces and one slave interface. Therefore, we can establish a general n:1 interconnect model. We assume that there are n master interfaces and one slave interface on the AXI interconnect. For the DPU D a For a read / write transaction issued, assuming that the transaction is transmitted through the i-th port of the AXI interconnect, the maximum number of transactions it needs to wait on the AXI interconnect can be calculated as follows:

[0040]

[0041]

[0042] Here use Indicates the total number of transactions in the jth port; because transactions passing through the same port may come from different DPUs, Indicates the jth port from DPUD a The number of transactions; i, j refer to ports.

[0043] Furthermore, the official DPU image of the ZCU102 board was analyzed using the VitisAnalyzer tool. Three B4096 Xilinx DPUs were deployed on the PL side. In this configuration, the instruction interfaces of the three DPUs shared the LPD PL-PS interface through the same AXI interconnect. At the same time, data transactions sent from the six data interfaces of the three DPUs shared the AXI interconnect on the PS side and ultimately underwent transaction arbitration by the DDR port arbiter together with the instruction transactions. The hardware factors affecting DPU execution time were divided into three levels: the AXI interconnect (IC1) on the PL side, the AXI interconnect (IC2) on the PS side, and the DDR port arbiter (PA).

[0044] Based on the arbitration model formula, the three levels of transaction arbitration are analyzed in a similar but not identical way. First, in a complete CNN execution, the total waiting time of each stage is It can be obtained by adding the waiting time of all transactions at the three levels. The calculation method is as follows:

[0045]

[0046] in, That is, they represent three levels of waiting time, and INS / R / W represent three different DPU stages;

[0047] Since the Xilinx DPU has two data interfaces but only one instruction interface, the analysis of the instruction read phase is different from that of the data read and write phase.

[0048] During the instruction read phase, the waiting time at each level is equal to the number of transactions the DPU needs to wait for multiplied by the maximum transmission time of each transaction, calculated as follows:

[0049]

[0050]

[0051]

[0052] in, XPI ins They represent the interfaces on the three levels of interconnection directly connected to the DPU instruction interface and the corresponding DDR ports; express Depends on which PL-PS interface the command interface is connected to; here use The reason why the transmission time from PA layer to DDR must be less than

[0053] During the data read / write phase, the interaction between the two data ports needs to be considered. First, for the IC1 and IC2 layers, the interaction between the two ports is relatively small, so a more pessimistic analysis method is adopted here, taking the worst-case scenario analysis, that is, for each transaction issued by the other DPU, there is a possibility that the two ports will wait. The calculation method is as follows:

[0054]

[0055]

[0056] Among them, D_0 and D_1 refer to the two data ports of the DPU respectively, IC D_0 and IC D_1 Refers to the AXI interconnect port that transactions issued by the two data ports of the DPU pass through at this layer; and Refers to the number of waiting transactions on each AXI interconnect, which are calculated by the previous formula; and Refers to the PL-PS interface through which the transaction passes, corresponding to

[0057] As the last layer before DDR, the PA layer involves a large amount of traffic. Therefore, to minimize the worst-case execution time (WCET) bound, a more detailed analysis method is required. When the two data ports of the DPU are ultimately connected to the same DDR port, the latency calculation is relatively simple, requiring only one port to be calculated:

[0058]

[0059]

[0060] in, Refers to the number of transactions that the DDR port needs to wait for, calculated by the previous formula;

[0061] When the two data ports of the DPU are connected to different DDR ports, the worst-case execution time is considered here, that is, only one DDR port can access the data in the DDR at the same time, so the transactions on the two DDR ports will not wait for each other at the same time, so this part of the time needs to be subtracted. In this case, the calculation method is as follows:

[0062]

[0063]

[0064] in, Refers to the number of read transactions and write transactions on the DDR port, respectively. and Refers to the number of read (write) transactions on the DDR port that transactions issued by the DPU data ports data0 and data1 pass through;

[0065] After obtaining the general shared bus arbitration and memory access contention model, we can perform static real-time analysis of CNN inference on FPGA in a multi-DPU environment.

[0066] The beneficial effects of the present invention are:

[0067] 1. The method described in this disclosure performs static WCET analysis on CNN reasoning on FPGA in a multi-DPU environment, which is of great significance to the field of real-time systems and avoids the waste of system resources.

[0068] 2. The method described in this disclosure analyzes shared bus arbitration and memory access contention in a fine-grained manner, greatly reducing the worst-case execution time (WCET) bound of CNN inference and improving resource utilization.

[0069] 3. The method described in this disclosure considers FPGA-based systems on a chip in a multi-DPU deployment environment and represents a further attempt at static real-time analysis of multi-core heterogeneous systems. Using static real-time analysis during system deployment helps developers understand whether task execution meets performance targets and facilitates the design of system scheduling methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation to the present disclosure.

[0071] Figure 1 A flowchart of a static real-time analysis method for CNN reasoning on FPGAs in a multi-DPU environment provided by an embodiment of the present disclosure.

[0072] Figure 2 A schematic diagram of a multi-DPU FPGA system-on-chip architecture provided in an embodiment of the present disclosure.

[0073] Figure 3 A schematic diagram of the parallel execution model of the DPU provided in an embodiment of the present disclosure.

[0074] Figure 4 This is a comparison chart of experimental results provided by the embodiments of the present disclosure. The vertical axis represents the execution time of a CNN inference in milliseconds. DETAILED DESCRIPTION

[0075] The present invention will be further described below through examples with reference to the accompanying drawings.

[0076] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0077] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0078] Example 1:

[0079] This paper takes the Xilinx Zynq UltraScale+ series ZCU102 board as an example to provide a static real-time analysis method. The static real-time analysis needs to follow the following Figure 1 The steps shown are analyzed below:

[0080] First, for FPGA SoCs deploying multiple DPUs, a general shared bus arbitration and memory access contention model is constructed.

[0081] We used the VitisAnalyzer tool to analyze the official DPU image of the ZCU102 board, as shown in the following example: Figure 2 As shown, three Xilinx B4096 DPUs were deployed on the PL side. At a 300MHz clock frequency, to measure the impact of multiple DPUs running simultaneously on the CNN inference time on each DPU and the specific influencing factors, a controlled experiment was designed, repeating CNN inference tens of thousands of times. (In principle, a higher number of executions reflects the average behavior of CNN inference, so the higher the number, the better, while also taking experimental time into consideration.) For example, a single DPU was used to execute Mobilenetv2 network inference independently, recording the inference time for each iteration. A control group was then established, running that DPU simultaneously with another DPU executing YOLOv4, recording the inference time for each iteration.

[0082] Experimental results show that the inference time of a single MobileNetV2 network running on a single DPU remains stable at around 3.5ms, with the longest time of 3.59ms and the shortest time of 3.34ms differing by only 7%. However, when running MobileNetV2 simultaneously with another DPU, the inference time ranges from 4ms to 12ms, a difference of approximately 300%. These results demonstrate that when multiple DPUs run simultaneously, memory access contention caused by the shared bus impacts CNN inference time.

[0083] The next step is to build a system hardware model based on the hardware principles of the FPGA SoC. In this configuration, the instruction interfaces of the three DPUs share the LPD PL-PS interface via the same AXI interconnect. Data transactions sent from the six data interfaces of the three DPUs also share the AXI interconnect on the PS side and ultimately undergo transaction arbitration by the DDR port arbiter along with the instruction transactions. The figure shows the hardware factors affecting DPU execution time at three levels: the PL-side AXI interconnect (IC1), the PS-side AXI interconnect (IC2), and the DDR port arbiter (PA).

[0084] For each DPU, a DPU execution model also needs to be established, such as Figure 3As shown. This model divides a CNN inference of the DPU into four stages: the read instruction stage (INS), the read data stage (R), the write data stage (W), and the establishment stage (ELAB). Among them, the read instruction stage refers to the process of the DPU reading the DPU instructions stored in the DDR through the instruction interface; the read data stage refers to the process of the DPU reading the DPU data stored in the DDR through its two data interfaces; the write data stage refers to the process of the DPU writing the calculated data to the DDR through its two data interfaces; the establishment stage refers to the time when the DPU only performs calculations without any memory access behavior. Past research has shown that the DPU's read data stage is carried out simultaneously with the read instruction stage and the write data stage.

[0085] Based on hardware principles and models, worst-case execution time (WCET) analysis can be performed to establish a generalized calculation formula from top to bottom. In this example, the formula establishment process is as follows:

[0086] Step 1: According to Figure 3 Based on the DPU parallel execution model, we can get the general DPU inference time calculation formula:

[0087] T total =max{T R , T INS +T W}+T ELAB

[0088] Since there is no memory access during the DPU establishment phase, T ELAB It can be regarded as a constant value calculation, which is related to the computational load of the CNN network and the computing power of the DPU (that is, the computational load of the CNN network divided by the computing power of the DPU, minus the memory access time of the DPU. Generally speaking, the value of this parameter is very small and can be ignored). Therefore, in the following part, we only need to analyze the other three stages. We have the following calculation method:

[0089]

[0090] Among them, T trsf Refers to the sum of the transmission time of all transactions. The transmission time here does not take into account conflicts with other DPUs. wait This refers to the transaction waiting time caused by conflicts with other DPUs. The calculation methods of these two times will be described in detail later.

[0091] Step 2: Obtain the calculation method of transmission time based on the AXI read and write specifications and DPU principles.

[0092] The reading and writing of data and instructions are completed through AXI transactions. An AXI transaction includes multiple AXI transfers. A complete read transaction includes a read address request and multiple data returns; a complete write transaction includes a write address request, multiple data writes, and a write return. For the convenience of public representation, we record the transmission time of an address request as (For write transactions, the transmission time here includes the write return time), and the read and write time of a data is recorded as The read and write time required for different PL-PS interfaces is These time parameters can be measured through experiments.

[0093] Because the Xilinx DPU has two data interfaces, both of which can read and write data to and from the DDR, and the DPU has only one instruction interface responsible for reading instructions from the DDR, the transmission time calculation method for the instruction read phase is slightly different from that for the data read and write phase:

[0094] For the DPU's read instruction phase, the transmission time is calculated as follows:

[0095]

[0096] Among them, N INS Indicates the total number of read instruction transactions for a CNN inference, Δ INS Indicates the total number of instructions for a CNN inference (since the LPD interface width is 32 bits, one instruction is recorded as 32 bits).

[0097] For the DPU's read / write data phase, the transfer time is calculated as follows:

[0098]

[0099] in, and Respectively represent the total number of read / write data transactions issued by the two data interfaces during a CNN inference process, and They respectively represent the total amount of data passing through the two data interfaces during a CNN inference process (since the bit width of the HP and HPC interfaces is 128 bits, one data is recorded as 128 bits).

[0100] Step 3: Obtain a calculation method for the waiting time based on the FPGA SoC hardware model and the arbitration principles of AXI interconnection and DDR.

[0101] Since both the AXI interconnect and the DDR controller use polling arbitration by default, the DDR controller can be considered a special AXI interconnect and a general arbitration model formula can be established. An AXI interconnect in the Xilinx DPU deployment process usually includes multiple master interfaces and one slave interface. Therefore, we can establish a general n:1 interconnect model. We assume that there are n master interfaces and one slave interface on the AXI interconnect. For the DPU D a For a read / write transaction issued, assuming that the transaction is transmitted through the i-th port of the AXI interconnect, the maximum number of transactions it needs to wait on the AXI interconnect can be calculated as follows:

[0102]

[0103]

[0104] Here use Indicates the total number of transactions in the jth port. Because transactions passing through the same port may come from different DPUs, we use Indicates the number of DPUs in the jth port a The number of transactions. i and j refer to ports.

[0105] Based on the arbitration model formula, Figure 2 The three levels of transaction arbitration in

[15] are analyzed in a similar but not identical way. First, the total waiting time of each stage in a complete CNN execution is It can be obtained by adding the waiting time of all transactions at the three levels. The calculation method is as follows:

[0106]

[0107] in, That is, they represent three levels of waiting time, and INS / R / W represent three different DPU stages.

[0108] Since the Xilinx DPU has two data interfaces but only one instruction interface, the analysis of the instruction read phase is different from that of the data read and write phase.

[0109] During the instruction read phase, the waiting time at each level is equal to the number of transactions the DPU needs to wait for multiplied by the maximum transmission time of each transaction, calculated as follows:

[0110]

[0111]

[0112]

[0113] in, XPI ins They represent the interfaces on the three levels of interconnection directly connected to the DPU instruction interface and the corresponding DDR ports; express Depends on which PL-PS interface the command interface is connected to; here use The reason why the transmission time from PA layer to DDR must be less than

[0114] During the data read / write phase, the interaction between the two data ports needs to be considered. First, for the IC1 and IC2 layers, the interaction between the two ports is relatively small, so a more pessimistic analysis method is adopted here, taking the worst-case analysis (i.e., for each transaction issued by the other DPU, it is possible that the two ports will wait). The calculation method is as follows:

[0115]

[0116]

[0117] Among them, D_0 and D_1 refer to the two data ports of the DPU respectively, IC D_0 and IC D_1 Refers to the AXI interconnect port that transactions issued by the two data ports of the DPU pass through at this layer; and Refers to the number of waiting transactions on each AXI interconnect, which are calculated by the previous formula; and Refers to the PL-PS interface through which the transaction passes, corresponding to

[0118] As the last layer before DDR, the PA layer involves a large amount of traffic. Therefore, to minimize the worst-case execution time (WCET) bound, a more detailed analysis method is required. When the two data ports of the DPU are ultimately connected to the same DDR port, the latency calculation is relatively simple, requiring only one port to be calculated:

[0119]

[0120]

[0121] in, Refers to the number of transactions that the DDR port needs to wait for, calculated by the previous formula.

[0122] When the two data ports of the DPU are connected to different DDR ports, the worst-case execution time is considered here, that is, only one DDR port can access the data in the DDR at the same time, so the transactions on the two DDR ports will not wait for each other at the same time, so this part of the time needs to be subtracted. In this case, the calculation method is as follows:

[0123]

[0124]

[0125] in, Refers to the number of read transactions and write transactions on the DDR port, respectively. and Refers to the number of read (write) transactions on the DDR port that transactions issued by the DPU's data ports data0 and data1 pass through.

[0126] After obtaining the general shared bus arbitration and memory access contention model, we can perform static real-time analysis of CNN inference on FPGA in a multi-DPU environment.

[0127] To calculate the current values ​​of the parameters in the formula, it is necessary to execute multiple CNN inferences on the FPGA where the image is deployed and measure the relevant data. In this example, we measured the parameter values ​​during one million CNN inferences using the AXI Performance Monitor (APM) and Xilinx ILA. These measurements revealed that the bus activity on the FPGA was stable for each inference of the same CNN. The resulting bus activity parameter data for each CNN inference is shown in Table 1.

[0128] Table 1 CNN inference bus activity parameters (number of transactions and amount of data)

[0129]

[0130] We consider the worst case to measure the transaction transmission time, that is, take the maximum value of all experimental data as the boundary of each transmission time. The transmission time parameter values ​​measured in the experiment are shown in Table 2.

[0131] Table 2 Transmission time parameter values ​​(unit: cycle)

[0132]

[0133] Finally, by substituting the various parameter values ​​into the general calculation formula, the CNN inference WCET boundary corresponding to each DPU can be calculated.

[0134] In this embodiment, we set up four groups of experiments on the ZCU102 board with the official image deployed. They are: two DPUs simultaneously execute the OD_SSD model and the PD_SSD model, two DPUs simultaneously execute the Mobilenetv2 model and the YOLOv3 model, three DPUs simultaneously execute the OD_SSD model, the PD_SSD model, and the YOLOv3 model, and three DPUs simultaneously execute the Mobilenetv2 model, the OD_SSD model, and the PD_SSD model. In each group of experiments, each DPU performs tens of thousands of CNN inferences, records the actual worst-case execution time, and compares it with the worst-case execution time obtained by static analysis, as shown in the figure. Figure 4 The experimental results show that the WCET bound obtained by using this method for static real-time analysis is no more than 1.45 times the actual value.

[0135] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A real-time analysis method for FPGA neural network accelerator, characterized in that: The steps are as follows: The worst-case values ​​of parameters in the general shared bus arbitration and memory access contention models are measured experimentally; Substitute the parameter values ​​into the model formula and statically analyze the worst-case execution time bound for each CNN inference based on the FPGA system-on-chip hardware model. A method for constructing a universal shared bus arbitration and memory access contention model for an FPGA system-on-chip includes the following steps: Perform preliminary analysis on the FPGA system-on-chip to obtain DPU deployment information; Based on the principles of different hardware components, we extracted the hardware factors that affect CNN reasoning in the FPGA system-on-chip. Through experimental verification, we established the system's hardware model and the DPU's execution model. Based on the model, a generalized formula is established to build a general shared bus arbitration and memory access contention model for FPGA system-on-chip. The generalized formula is as follows: Step 1: Based on the DPU execution model, obtain the general DPU inference time calculation formula: T total =max{T R ,T INS +T W }+T ELAB T ELAB It is regarded as a constant value calculation, and the other three stages are analyzed. The calculation method is as follows: Among them, T trsf Refers to the sum of the transmission time of all transactions; and T wait Refers to the transaction waiting time caused by conflicts with other DPUs; Step 2: Obtain the calculation method of transmission time based on AXI read and write specifications and DPU principles; The reading and writing of data and instructions are completed through AXI transactions. An AXI transaction includes multiple AXI transfers. A complete read transaction includes a read address request and multiple data returns. A complete write transaction includes a write address request, multiple data writes, and a write return. The transmission time of an address request is recorded as For write transactions, the transmission time here includes the write return; the read and write time of a data is recorded as The read and write time required for different PL-PS interfaces is These times as parameters can be measured experimentally; For the DPU's read instruction phase, the transmission time is calculated as follows: Among them, N INS Indicates the total number of read instruction transactions for a CNN inference, Δ INS Indicates the total number of instructions for a CNN inference; For the DPU's read / write data phase, the transfer time is calculated as follows: in, and Respectively represent the total number of read / write data transactions issued by the two data interfaces during a CNN inference process, and They represent the total amount of data passing through the two data interfaces during a CNN inference process; Step 3: Obtain a calculation method for waiting time based on the FPGA SoC hardware model and the arbitration principle of AXI interconnection and DDR; Establish a general n:1 interconnect model. Assume that there are n master interfaces and one slave interface on the AXI interconnect. a For a read / write transaction issued, assuming that the transaction is transmitted through the i-th port of the AXI interconnect, the maximum number of transactions it needs to wait on the AXI interconnect is calculated as follows: Here use Indicates the total number of transactions in the jth port; here we use Indicates the number of DPUs in the jth port a The number of transactions.

2. The real-time analysis method for FPGA neural network accelerator according to claim 1, characterized in that: Pre-analysis of the FPGA system-on-chip involves using Vivado software when deploying the DPU, or using Vitis Analyzer software after deployment.

3. The real-time analysis method for FPGA neural network accelerator according to claim 1, characterized in that: DPU deployment information refers to the number and model of DPUs deployed on the FPGA, the DPU deployment method, and the connection method information of the PGA PS system.

4. The real-time analysis method for FPGA neural network accelerator according to claim 1, characterized in that: The effects of different hardware components on CNN reasoning were verified experimentally, specifically: Using the control variable method, each time the DPU is redeployed, only the information of one hardware component is changed. CNN inference is performed multiple times, and the average inference time is recorded. Based on the average inference time of CNN under different hardware configurations, we observe the impact of different hardware components on the inference time of CNN.

5. The real-time analysis method for FPGA neural network accelerator according to claim 1, characterized in that: The hardware model of the system is established as follows: the instruction interfaces of all DPUs share the LPD PL-PS interface through the same AXI interconnect. At the same time, data transactions sent from the data interfaces of all DPUs will share the AXI interconnect in the PS part and ultimately be arbitrated by the DDR port arbiter together with the instruction transactions.

6. The real-time analysis method for FPGA neural network accelerator according to claim 1, characterized in that: For each DPU, a DPU execution model also needs to be established for it. Specifically, the model includes input, execution process, and output. The execution process divides a CNN inference of the DPU into four stages: read instruction stage INS, read data stage R, write data stage W, and establishment stage ELAB; among them, the read instruction stage refers to the process of the DPU reading the DPU instructions stored in the DDR through the instruction interface; the read data stage refers to the process of the DPU reading the DPU data stored in the DDR through its two data interfaces; the write data stage refers to the process of the DPU writing the calculated data to the DDR through its two data interfaces; the establishment stage refers to the time when the DPU only performs calculations without any memory access behavior. The DPU's read data stage is carried out simultaneously with the read instruction stage and the write data stage.

7. The real-time analysis method for FPGA neural network accelerator according to claim 1, characterized in that: We used Vitis Analyzer to analyze the official DPU image of the ZCU102 board. Three B4096 Xilinx DPUs were deployed on the PL side. The instruction interfaces of the three DPUs shared the LPD PL-PS interface through the same AXI interconnect. Data transactions sent from the three DPUs' six data interfaces also shared the AXI interconnect on the PS side and ultimately, along with instruction transactions, were arbitrated by the DDR port arbiter. The hardware factors affecting DPU execution time were categorized into three levels: AXI interconnect IC1 on the PL side, AXI interconnect IC2 on the PS side, and the DDR port arbiter PA. Based on the arbitration model formula, the three levels of transaction arbitration are analyzed. First, in a complete CNN execution, the total waiting time of each stage is It can be obtained by adding the waiting time of all transactions at the three levels. The calculation method is as follows: in, That is, they represent three levels of waiting time, and INS / R / W represent three different DPU stages; During the instruction read phase, the waiting time at each level is equal to the number of transactions the DPU needs to wait for multiplied by the maximum transmission time of each transaction, calculated as follows: in, XPI ins They represent the port numbers to which the DPU command interface is connected at three levels respectively; express Depends on which PL-PS interface the command interface is connected to; here use The reason why the transmission time from PA layer to DDR must be less than In the read / write data phase, first, for the IC1 and IC2 layers, the impact between the two ports is small. We take the worst-case analysis, that is, for each transaction issued by other DPUs, it is possible that the two ports will wait. The calculation method is as follows: Among them, D_0 and D_1 refer to the two data ports of the DPU respectively, IC D_0 and IC D_1 Refers to the AXI interconnect port that transactions issued by the two data ports of the DPU pass through at this layer; and Refers to the number of waiting transactions on each AXI interconnect; and Refers to the PL-PS interface through which the transaction passes, corresponding to The PA layer is the last layer before DDR. When the two data ports of the DPU are ultimately connected to the same DDR port, the latency calculation is relatively simple, as only one port needs to be calculated: in, Refers to the number of transactions that the DDR port needs to wait for; When the two data ports of the DPU are connected to different DDR ports, the calculation method is as follows: in, Refers to the number of read transactions and write transactions on the DDR port, respectively. and Refers to the number of read / write transactions on the DDR port that transactions issued by the DPU data ports data0 and data1 pass through; After obtaining the general shared bus arbitration and memory access contention model, we can perform static real-time analysis of CNN inference on FPGA in a multi-DPU environment.

Images