FPGA hardware acceleration method and device based on DPU

Abstract

The invention provides a DPU-based FPGA hardware acceleration method and device, relates to the technical field of data processing, is applied to a deep learning processing unit DPU, the DPU is connected with a field programmable gate array FPGA chip, and the method comprises the steps that initial data are acquired, the initial data at least comprise input data of a target convolutional neural network, and the input data of the target convolutional neural network are acquired; the target convolutional neural network is used for performing data processing on the initial data; extracting an equivalent instruction after format conversion of the target convolutional neural network; and based on the initial data and the equivalent instruction, controlling a field programmable gate array (FPGA) chip to accelerate reasoning of the target convolutional neural network. The problem that an existing accelerator limits neural network acceleration is solved, and based on high-level parallelism and energy efficiency of the DPU, neural network reasoning acceleration can be flexibly achieved, the reasoning speed is increased, and the calculation efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a DPU-based FPGA hardware acceleration method and device. Background Art

[0002] In the field of image and video processing, more and more hardware accelerators such as application-specific integrated circuits (ASICs), graphics processing units (GPUs), and field-programmable gate arrays (FPGAs) are being used to improve the throughput of neural network model inference.

[0003] Among them, FPGA (Field Programmable Gate Array) has many significant advantages, such as programmability, high performance, customizability and scalability, low power consumption, rapid prototyping and development, easy maintenance and upgrade, and high reliability and stability, which make it a powerful tool in fields such as deep learning and artificial intelligence.

[0004] However, globally deploying the model to the FPGA chip in the form of a data stream requires a long hardware design time, and the hardware needs to be redesigned when the model is updated, which cannot adapt to the complex and changeable related application requirements. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, the first purpose of this application is to propose a DPU-based FPGA hardware acceleration method to flexibly accelerate neural network reasoning and improve the reasoning speed.

[0007] The second objective of this application is to propose a DPU-based FPGA hardware acceleration device.

[0008] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a DPU-based FPGA hardware acceleration method, which is applied to a deep learning processing unit DPU, wherein the DPU is connected to a field programmable gate array FPGA chip, including:

[0009] Acquire initial data, wherein the initial data at least includes input data of a target convolutional neural network, and the target convolutional neural network is used to perform data processing on the initial data;

[0010] Extracting equivalent instructions obtained by format conversion of the target convolutional neural network;

[0011] Based on the initial data and the equivalent instructions, the FPGA chip is controlled to accelerate the reasoning of the target convolutional neural network.

[0012] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a DPU-based FPGA hardware acceleration device, which is applied to a deep learning processing unit DPU, wherein the DPU is connected to a field programmable gate array FPGA chip, including:

[0013] An acquisition module, used to acquire initial data, wherein the initial data at least includes input data of a target convolutional neural network, and the target convolutional neural network is used to perform data processing on the initial data;

[0014] An extraction module, used to extract equivalent instructions obtained by format conversion of the target convolutional neural network;

[0015] An acceleration module is used to control the FPGA chip based on the initial data and the equivalent instructions to accelerate the reasoning of the target convolutional neural network.

[0016] The DPU-based FPGA hardware acceleration method and device provided in this application are applied to the deep learning processing unit DPU, and the DPU is connected to the field programmable gate array FPGA chip, by obtaining initial data, wherein the initial data at least includes the input data of the target convolutional neural network, and the target convolutional neural network is used to process the initial data; extracting equivalent instructions after format conversion of the target convolutional neural network; based on the initial data and the equivalent instructions, controlling the field programmable gate array FPGA chip to accelerate the reasoning of the target convolutional neural network. The limitation problem of existing accelerators on neural network acceleration is solved, and based on the high level of parallelism and energy efficiency of DPU, it can flexibly accelerate neural network reasoning, improve reasoning speed, and improve computing efficiency.

[0017] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0019] Figure 1 A schematic diagram of a flow chart of a DPU-based FPGA hardware acceleration method provided in an embodiment of the present application;

[0020] Figure 2 A flowchart of another DPU-based FPGA hardware acceleration method provided in an embodiment of the present application;

[0021] Figure 3 A diagram of the internal architecture of a DPU provided in an embodiment of the present application;

[0022] Figure 4 A DPU-based FPGA hardware acceleration structure architecture diagram provided in an embodiment of the present application;

[0023] Figure 5 An operating power consumption diagram of a DPU design provided in an embodiment of the present application;

[0024] Figure 6 An FPGA floor plan diagram of a DPU accelerator provided in an embodiment of the present application;

[0025] Figure 7 A schematic diagram of the structure of a DPU-based FPGA hardware acceleration device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0027] The following describes the DPU-based FPGA hardware acceleration method and device according to an embodiment of the present application with reference to the accompanying drawings.

[0028] Figure 1 A flowchart of a DPU-based FPGA hardware acceleration method provided in an embodiment of the present application.

[0029] Given that many environments have put forward more stringent requirements and restrictions on the operating power, memory, and energy consumption of electrical equipment, FPGA can achieve higher performance at lower power consumption than GPU. In the face of evolving deep learning technology and new computing, FPGA's programmability makes it more flexible than ASIC. Therefore, FPGA has gradually become an important hardware platform for accelerating reasoning in related applications.

[0030] However, there are still many deficiencies and challenges in realizing the flexibility and inference speed of neural networks on FPGAs, such as:

[0031] The column-based fine-grained pipeline architecture and padding jump technology are used to shorten the computation time of convolutional neural networks, and the reasoning acceleration of the YOLOv2-Tiny deep neural network model is realized on FPGA. However, it takes a long time to globally deploy the model on the FPGA chip in the form of data flow, and the hardware needs to be redesigned when the model is updated, which cannot adapt to the complex and changeable related application requirements.

[0032] In order to realize an efficient computing system for real-time object detection in SSDLite, a neural processing unit (NPU) is designed, which consists of heterogeneous units such as processing, scaling and accumulation, as well as data acquisition and formatting units. It can effectively accelerate deep neural networks, but the types of computing operators supported are obviously limited.

[0033] In view of the above problems, the embodiment of the present application provides a DPU-based FPGA hardware acceleration method to flexibly accelerate neural network reasoning and improve the reasoning speed. The DPU-based FPGA hardware acceleration method is executed by a deep learning processing unit (DPU). The DPU is a configurable computing engine used to optimize convolutional neural networks and is widely used to accelerate deep learning reasoning algorithms such as VGG, Resnet, OLO, SSD, FPN, etc. The parallel utilization in the DPU is configurable and can be selected according to the target hardware platform and application, and the DPU is connected to a field programmable gate array FPGA chip.

[0034] like Figure 1 As shown, the method comprises the following steps:

[0035] Step 101, obtaining initial data, wherein the initial data at least includes input data of a target convolutional neural network, and the target convolutional neural network is used to perform data processing on the initial data.

[0036] Optionally, the target convolutional neural network is a convolutional neural network that achieves a preset purpose, for example, it may be a convolutional neural network that achieves target detection or target recognition in coal mines.

[0037] Among them, the type of input data is determined according to the type of the target convolutional neural network. For example, the target convolutional neural network is a convolutional neural network that realizes underground target detection in coal mines, and the input data can be coal mine images; the target convolutional neural network is a convolutional neural network that realizes coal mine parameter prediction, and the input data can be coal mine environmental parameters.

[0038] Furthermore, the initial data may also include model weights of the target convolutional neural network.

[0039] Optionally, the DPU is connected to an off-chip memory storing the initial data, and obtains the initial data by receiving the initial data transmitted by the off-chip memory.

[0040] Step 102, extracting equivalent instructions obtained by format conversion of the target convolutional neural network.

[0041] Optionally, equivalent instructions supported by the DPU are obtained by converting the format of the target convolutional neural network.

[0042] In an embodiment of the present application, the format conversion is implemented through the Vitis AI framework.

[0043] Through format conversion, the DPU can identify the target convolutional neural network and process it.

[0044] Step 103, based on the initial data and the equivalent instructions, the field programmable gate array FPGA chip is controlled to accelerate the reasoning of the target convolutional neural network.

[0045] Optionally, the DPU includes a hybrid computing array, the DPU is connected to the FPGA, and manages task scheduling. When the initial data is loaded into the DPU, the DPU schedules and allocates different computing tasks to each PE in the hybrid computing array according to equivalent instructions, thereby realizing control of the FPGA chip.

[0046] In this embodiment, by obtaining initial data, wherein the initial data at least includes the input data of the target convolutional neural network, the target convolutional neural network is used to process the initial data; extracting equivalent instructions after format conversion of the target convolutional neural network; based on the initial data and the equivalent instructions, controlling the field programmable gate array FPGA chip to accelerate the reasoning of the target convolutional neural network. The limitation problem of existing accelerators on neural network acceleration is solved, and based on the high level of parallelism and energy efficiency of DPU, the neural network reasoning can be flexibly accelerated, the reasoning speed can be improved, and the computing efficiency can be improved.

[0047] This embodiment provides another DPU-based FPGA hardware acceleration method, which is applied to a deep learning processing unit DPU. The DPU is connected to a field programmable gate array FPGA chip. Figure 2 A flowchart of another DPU-based FPGA hardware acceleration method provided in an embodiment of the present application.

[0048] like Figure 2 As shown, the DPU-based FPGA hardware acceleration method may include the following steps:

[0049] Step 201, obtaining initial data, wherein the initial data at least includes input data of a target convolutional neural network, and the target convolutional neural network is used to perform data processing on the initial data.

[0050] It should be noted that the DPU includes: a high-performance scheduler, a hybrid computing array, an instruction fetch unit and a global memory pool.

[0051] Among them, the high-performance scheduler is used to obtain task information and execution requirements; the hybrid computing array is used to execute tasks; the instruction fetch unit is used to extract equivalent instructions from the off-chip memory; and the global memory pool is used as a buffer for initial data and equivalent instructions.

[0052] See also Figure 3, the DPU consists of a high-performance scheduler, a hybrid computing array, an instruction fetch unit, and a global memory pool. Among them, the high-performance scheduler is responsible for managing and scheduling task execution. The DPU obtains task information and execution requirements through the high-performance scheduler, and assigns tasks to different processing units (PEs) in the hybrid computing array for execution. The main function of the instruction fetch unit is to obtain model-related DPU instructions, that is, equivalent instructions, from the memory. The global memory pool acts as a buffer for input and output data and intermediate data from the DPU, resulting in high throughput.

[0053] It should be noted that the hybrid computing array combines multiple different types of processing elements (PE), such as general processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), machine learning accelerators, etc. Each type of PE has specific computing capabilities and advantages. This structure is designed to achieve efficient processing and optimization of complex computing tasks by integrating multiple computing capabilities.

[0054] It should be noted that the FPGA chip includes an ARM processor.

[0055] In an embodiment of the present application, the initial data is offloaded to the DPU through the processor, wherein the DPU is connected to the processor through an advanced extensible interface AXI.

[0056] Optionally, the target convolutional neural network is a convolutional neural network that achieves a preset purpose, for example, it may be a convolutional neural network that achieves target detection or target recognition in coal mines.

[0057] Among them, the type of input data is determined according to the type of the target convolutional neural network. For example, the target convolutional neural network is a convolutional neural network that realizes underground target detection in coal mines, and the input data can be coal mine images; the target convolutional neural network is a convolutional neural network that realizes coal mine parameter prediction, and the input data can be coal mine environmental parameters.

[0058] Furthermore, the initial data may also include model weights of the target convolutional neural network.

[0059] In the embodiment of the present application, the initial data is stored in an off-chip memory.

[0060] Among them, off-chip memory refers to the main memory or external storage device located outside the DPU. The global memory pool and off-chip memory jointly support the needs of the DPU to perform complex computing tasks through data loading, storage, exchange and management, ensuring computing efficiency and data security.

[0061] Optionally, the DPU is connected to an off-chip memory storing the initial data, and obtains the initial data by receiving the initial data transmitted by the off-chip memory.

[0062] In an embodiment of the present application, the DPU runs at a clock frequency generated by a mixed mode clock manager (MMCM).

[0063] Step 202: Convert the format of the target convolutional neural network into a compatible mode supported by the DPU, and compile the target convolutional neural network in the compatible mode into equivalent instructions.

[0064] Optionally, convert the target convolutional neural network from DNN framework formats (such as TensorFlow and PyTorch) to a compatible model supported by the DPU engine.

[0065] It should be noted that this conversion relies on the unified software stack Vitis AI framework, which is responsible for quantizing and pruning the model. The API provided by the Vitis AI library function manages the communication between the ARM processor and the DPU deep learning processing accelerator, where Vitis AI includes hardware driver support for the DPU.

[0066] Furthermore, the compatible model is compiled into equivalent instructions supported by the DPU, and performance analysis is performed.

[0067] Optionally, equivalent instructions are stored in off-chip memory.

[0068] Step 203, controlling the hybrid computing array on the field programmable gate array FPGA chip based on the initial data and the equivalent instructions to accelerate the reasoning of the target convolutional neural network.

[0069] Optionally, the DPU includes a hybrid computing array, the DPU is connected to the FPGA, and manages task scheduling. When the initial data is loaded into the DPU, the DPU schedules and allocates different computing tasks to each PE in the hybrid computing array according to equivalent instructions, thereby realizing control of the FPGA chip.

[0070] Furthermore, after the DPU execution is completed, the initial data and equivalent instructions are stored back in the off-chip memory.

[0071] As an example, using the coal mine object recognition model as the target convolutional neural network, see Figure 4 The coal mine target recognition model is built based on the deep learning framework. The API provided by the Vitis AI library function is used to manage the communication between the ARM processor and the DPU deep learning processing accelerator. The DPU is connected to the ARM processor core on the FPGA chip through the AXI bus. The data movement inside and outside the chip is controlled by the customized PetaLinux operating system running on the ARM processor. The initial data and equivalent instructions are stored in the off-chip memory, which is controlled by the DDR controller through the AXI bus.

[0072] As part of configuring the DPU for acceleration, initial data and equivalent instructions are initially loaded into off-chip memory before being moved to the on-chip DPU buffers. Once the DPU completes a process, the initial data and equivalent instructions are retrieved from the output on-chip buffers back into memory for any required post-processing. The initial data includes the model input image of size 640×640, and the model weights.

[0073] At startup, the DPU retrieves equivalent instructions from off-chip memory, and then schedules and allocates different computing tasks to each PE in the hybrid computing array based on the equivalent instructions. Further, the initial data and equivalent instructions are loaded into the DPU, and the preparation work is completed. The DPU can use its integrated dedicated hardware accelerator to speed up the calculation steps in the reasoning process. In order to reduce the overall memory bandwidth, the data is kept in use as much as possible, which enhances the overall performance of the system in terms of latency and energy consumption.

[0074] Furthermore, the entire hardware-software coordinated acceleration system is deployed on the Xilinx ZynqUltraScale+MPSoC ZCU102 FPGA evaluation kit, and a 3-core B4096 architecture DPU is implemented in the hardware programmable logic, 300MHz, and hardware synthesis is performed using Vivado2022.1. The HA-YOLOv5 model is quantized and compiled based on Vitis AI V2.5 to obtain the DPU executable model. That is, a 3-core B4096 architecture DPU accelerator is implemented on the target Xilinx ZynqUltraScale+MPSoC ZCU102 FPGA evaluation kit, and the DPU runs at a clock frequency of 300MHz. Please refer to Figure 5 , shows the operating power consumption of the DPU design. The total power consumption on the chip is 22.375W, of which the dynamic power consumption generated by clocks, internal signals and programmable logic resources is 21.445W. The floor plan of the DPU in the FPGA programmable logic is shown in Figure 6 As shown, the lower left corner is the ARM core of the processing system, and the rest is mainly FPGA programmable logic resources and dedicated hardware logic blocks. It can be seen that the DPU accelerator utilizes most of the on-chip programmable logic resources. In other words, the DPU resource utilization rate deployed on ZCU102 is very high and the resources are fully used.

[0075] The Petalinux operating system runs on the ARM processor of ZCU102. The operating system is built with Xilinx Petalinx 2022.1 tools and includes Vitis AI programmable libraries and DPU drivers. An embedded control software application for the HA-YOLOv5 model is built on top of the operating system to manage the communication between the ARM and DPU accelerators. Before the DPU performs model inference acceleration, the model file, image test data, and software application are first loaded into the system-on-chip, and 100 test images are iteratively inferred to evaluate the hardware inference performance after FPGA deployment.

[0076] In an embodiment of the present application, after the quantized model is compiled into equivalent instructions supported by the DPU, the corresponding embedded test program is written, the test set and the model are loaded, and after the hardware is deployed, 100 test images are iterated in a continuous loop for 3 rounds to count the hardware inference throughput (i.e., the number of inference frames per second).

[0077] The results show that the DPU hardware inference throughput reaches 32.51 frames per second, which meets the real-time inference requirements and is 2.41 times the inference throughput of the CPU platform, with a significant acceleration effect. Furthermore, the inference energy consumption of different platforms is evaluated. The DPU hardware inference energy consumption of this application is reduced by 94.46% compared with the CPU and 80.18% compared with the GPU, and the inference energy consumption is significantly reduced.

[0078] The embodiments of the present application can reduce the energy consumption of hardware inference, increase the throughput of hardware inference, achieve high-speed and efficient data processing, improve the real-time performance and responsiveness of the system, and shorten the development cycle of the hardware accelerator.

[0079] In this embodiment, by obtaining initial data, wherein the initial data at least includes the input data of the target convolutional neural network, the target convolutional neural network is used to process the initial data; extracting equivalent instructions after format conversion of the target convolutional neural network; based on the initial data and the equivalent instructions, controlling the field programmable gate array FPGA chip to accelerate the reasoning of the target convolutional neural network. The limitation problem of existing accelerators on neural network acceleration is solved, and based on the high level of parallelism and energy efficiency of DPU, the neural network reasoning can be flexibly accelerated, the reasoning speed can be improved, and the computing efficiency can be improved.

[0080] In order to implement the above embodiments, the present application also proposes a DPU-based FPGA hardware acceleration device.

[0081] Figure 7 A schematic diagram of the structure of a DPU-based FPGA hardware acceleration device provided in an embodiment of the present application.

[0082] like Figure 7As shown, the DPU-based FPGA hardware acceleration device 700 includes: an acquisition module 701, an extraction module 702 and an acceleration module 703.

[0083] An acquisition module 701 is used to acquire initial data, wherein the initial data at least includes input data of a target convolutional neural network, and the target convolutional neural network is used to perform data processing on the initial data;

[0084] An extraction module 702 is used to extract equivalent instructions obtained by format conversion of the target convolutional neural network;

[0085] The acceleration module 703 is used to control the FPGA chip based on the initial data and equivalent instructions to accelerate the reasoning of the target convolutional neural network.

[0086] Furthermore, in a possible implementation of the embodiment of the present application, the extraction module 702 is specifically configured to:

[0087] The format of the target convolutional neural network is converted into a compatible mode supported by the DPU, and the target convolutional neural network in the compatible mode is compiled into equivalent instructions.

[0088] Furthermore, in a possible implementation of the embodiment of the present application, the FPGA chip includes a processor, and the acquisition module 701 is specifically used to:

[0089] The initial data is offloaded to the DPU by the processor, wherein the DPU is connected to the processor via the Advanced eXtensible Interface (AXI).

[0090] Furthermore, in a possible implementation of the embodiment of the present application, initial data and equivalent instructions are stored in an off-chip memory.

[0091] Furthermore, in a possible implementation of the embodiment of the present application, the device is also used for:

[0092] After the DPU execution is completed, the original data and equivalent instructions are stored back to the off-chip memory.

[0093] Furthermore, in a possible implementation of the embodiment of the present application, the DPU includes:

[0094] A high-performance scheduler for obtaining task information and execution requirements;

[0095] Hybrid computing arrays for task execution;

[0096] An instruction fetch unit, used to fetch equivalent instructions from an off-chip memory;

[0097] A global memory pool used as a buffer for initial data and equivalent instructions.

[0098] Furthermore, in a possible implementation of the embodiment of the present application, the DPU runs at a clock frequency generated by a mixed-mode clock manager.

[0099] It should be noted that the aforementioned explanation of the DPU-based FPGA hardware acceleration method embodiment is also applicable to the DPU-based FPGA hardware acceleration device of this embodiment, and will not be repeated here.

[0100] In the embodiment of the present application, by obtaining initial data, wherein the initial data at least includes the input data of the target convolutional neural network, the target convolutional neural network is used to process the initial data; extracting equivalent instructions after format conversion of the target convolutional neural network; based on the initial data and the equivalent instructions, controlling the field programmable gate array FPGA chip to accelerate the reasoning of the target convolutional neural network. The limitation problem of existing accelerators on neural network acceleration is solved, and based on the high level of parallelism and energy efficiency of DPU, it can flexibly accelerate neural network reasoning, improve reasoning speed, and improve computing efficiency.

[0101] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the DPU-based FPGA hardware acceleration method provided in the above embodiments.

[0102] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the DPU-based FPGA hardware acceleration method provided in the above embodiments.

[0103] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which, when executed by a processor, implements the DPU-based FPGA hardware acceleration method provided in the above embodiments.

[0104] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0105] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0106] This application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, this application is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, risks can be minimized by limiting data collection and deleting data. In addition, when applicable, personal identifiers are removed from such personal information to protect the privacy of users.

[0107] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0108] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0109] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0110] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0111] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0112] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0113] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0114] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A DPU-based FPGA hardware acceleration method, characterized in that: The method is applied to a deep learning processing unit DPU, wherein the DPU is connected to a field programmable gate array FPGA chip, and includes the following steps: Acquire initial data, wherein the initial data at least includes input data of a target convolutional neural network, and the target convolutional neural network is used to perform data processing on the initial data; Extracting equivalent instructions obtained by format conversion of the target convolutional neural network; Based on the initial data and the equivalent instructions, the FPGA chip is controlled to accelerate the reasoning of the target convolutional neural network.

2. The DPU-based FPGA hardware acceleration method according to claim 1, characterized in that: The extracting equivalent instructions obtained by format conversion of the target convolutional neural network includes: The format of the target convolutional neural network is converted into a compatible mode supported by the DPU, and the target convolutional neural network in the compatible mode is compiled into the equivalent instructions.

3. The DPU-based FPGA hardware acceleration method according to claim 1, characterized in that: The FPGA chip includes a processor, and obtaining initial data includes: The initial data is unloaded to the DPU through the processor, wherein the DPU is connected to the processor through an advanced extensible interface AXI.

4. The DPU-based FPGA hardware acceleration method according to claim 1, characterized in that: The initial data and the equivalent instructions are stored in an off-chip memory.

5. The DPU-based FPGA hardware acceleration method according to claim 4, characterized in that: The method further comprises: After the DPU is executed, the initial data and the equivalent instructions are stored back in an off-chip memory.

6. The DPU-based FPGA hardware acceleration method according to claim 1, characterized in that: The DPU includes: A high-performance scheduler for obtaining task information and execution requirements; Hybrid computing arrays for task execution; An instruction fetching unit, used for fetching the equivalent instruction from an off-chip memory; A global memory pool is used as a buffer for the initial data and the equivalent instructions.

7. The DPU-based FPGA hardware acceleration method according to claim 1, characterized in that: The DPU runs at a clock frequency generated by a mixed-mode clock manager.

8. A DPU-based FPGA hardware acceleration device, characterized in that: Applied to a deep learning processing unit DPU, the DPU is connected to a field programmable gate array FPGA chip, and includes: An acquisition module, used to acquire initial data, wherein the initial data at least includes input data of a target convolutional neural network, and the target convolutional neural network is used to perform data processing on the initial data; An extraction module, used to extract equivalent instructions obtained by format conversion of the target convolutional neural network; An acceleration module is used to control the FPGA chip based on the initial data and the equivalent instructions to accelerate the reasoning of the target convolutional neural network.

9. The DPU-based FPGA hardware acceleration device according to claim 8, characterized in that: The extraction module is specifically used for: The format of the target convolutional neural network is converted into a compatible mode supported by the DPU, and the target convolutional neural network in the compatible mode is compiled into the equivalent instructions.

10. The DPU-based FPGA hardware acceleration device according to claim 8, characterized in that: The FPGA chip includes a processor, and the acquisition module is specifically used for: The initial data is unloaded to the DPU through the processor, wherein the DPU is connected to the processor through an advanced extensible interface AXI.

Images