Multi-FPGA (Field Programmable Gate Array) cooperative multi-deep neural network pipeline acceleration method

Abstract

The invention discloses a multi-FPGA (Field Programmable Gate Array) collaborative multi-deep neural network pipeline acceleration method, which comprises the following steps of: according to network structures and calculation requirements of different to-be-deployed deep neural network models, obtaining operators required by reasoning of each deep neural network model; dividing the reasoning process of each to-be-deployed deep neural network model into a plurality of task segments; according to operator and task segment division results, sending corresponding scheduling instructions to a plurality of FPGA calculation units, deploying operators required by reasoning of each deep neural network model to the corresponding FPGA calculation units according to categories, and controlling the corresponding FPGA calculation units to execute pipeline operation. Calculating each task segment of the to-be-deployed deep neural network model, and completing the reasoning of each to-be-deployed deep neural network model. According to the method, different calculation tasks in the reasoning process of multiple models can be subjected to pipelined processing, and the total time of multi-model reasoning is effectively shortened.

Description

Technical Field

[0001] The present invention relates to the field of neural network accelerators, and in particular to a multi-FPGA collaborative multi-depth neural network pipeline acceleration method. Background Art

[0002] At present, deep neural networks have been widely used in communication signal recognition, radio frequency fingerprint recognition, radar interference recognition and other application fields. In order to improve performance, the number of parameters and computational complexity of deep neural networks have rapidly expanded during their application, and the computing power requirements of hardware platforms during their deployment have also increased accordingly.

[0003] In the prior art, GPU (Graphics Processing Unit) is usually used to train and deploy deep neural networks. This technology has problems such as low energy efficiency and limited flexibility. Compared with GPU, FPGA (Field Programmable Gate Array) has higher energy efficiency and superior flexibility. FPGA can use the programmable logic units such as LUT (Lookup Table) and DSP (Digital Signal Processor) for parallel computing, and through customized hardware design, it can obtain higher energy efficiency, computing efficiency and resource efficiency. Therefore, the method of using FPGA to deploy deep neural networks is favored by many researchers.

[0004] However, existing solutions for accelerating deep neural network inference based on FPGA chips are mostly concentrated on a single chip. When processing large-scale deep neural networks, the single-chip FPGA will limit the overall computing power due to its limited computing and storage resources. Therefore, related technologies that use multiple FPGA chips to collaboratively complete the deployment of deep neural networks have emerged.

[0005] Deploying deep neural networks on multiple FPGAs can make full use of the parallel computing capabilities of FPGAs to improve computing efficiency. In existing literature, the Chinese invention patent "A method for collaborative training of neural networks with multiple FPGAs based on distributed optimization" (application number: CN202310598533) divides the neural network into multiple FPGAs connected in a star or ring shape after serial division, and the host computer is used as the core scheduling process to accelerate network training, but multiple FPGAs are more in line with the current situation of limited FPGA chip resources for inference acceleration. The Chinese invention patent "A large-scale convolutional neural network acceleration and deployment implementation method based on multiple FPGAs" (application number: CN202410025778) uses multiple FPGAs for cascading, and through task allocation, all FPGAs can balance the workload while reducing idle time to improve computing efficiency, but its operation is limited to supporting a single deep neural network model.

[0006] The shortcomings of existing solutions mainly include two aspects: on the one hand, when using multiple FPGAs for inference acceleration, there is a lack of appropriate model splitting methods and task allocation strategies, which leads to the need to redeploy the accelerator after changing the network model; on the other hand, existing methods are limited to deploying a single network model and fail to fully utilize the high parallelism and pipeline design advantages of multiple FPGAs.

[0007] Therefore, the present invention proposes a multi-FPGA collaborative multi-deep neural network pipeline acceleration method and system, which can execute multiple deep neural network models in parallel, determine the task division according to the hierarchical structure and computational amount of different network models, and accurately allocate the computing tasks to the computing operators inside each FPGA through the efficient scheduling strategy of the upper computer. It also uses a multi-FPGA pipeline design to reduce the data waiting time between operators and improve the computing parallelism and the overall computing efficiency of the system. Summary of the invention

[0008] In view of this, the present invention provides a multi-FPGA collaborative multi-deep neural network pipeline acceleration method and system, which is used to at least solve some of the technical problems in the background technology.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] On one hand, the present invention discloses a multi-FPGA collaborative multi-deep neural network pipeline acceleration method, comprising the following steps:

[0011] Model operator identification:

[0012] According to the network structure and computing requirements of different deep neural network models to be deployed, obtain the operators required for reasoning of each deep neural network model;

[0013] Model task division:

[0014] Divide the tasks according to the network hierarchical structure and computing requirements of each deep neural network model to be deployed, and divide the reasoning process of each deep neural network model to be deployed into multiple task segments;

[0015] Pipeline operation control:

[0016] According to the division results of operators and task segments required for reasoning of each deep neural network model to be deployed, corresponding scheduling instructions are sent to multiple FPGA computing units, and the operators required for reasoning of each deep neural network model are deployed to the corresponding FPGA computing units according to categories, and the corresponding FPGA computing units are controlled to perform pipeline operations to perform calculations on each task segment of the deep neural network model to be deployed and complete the reasoning of each deep neural network model to be deployed.

[0017] Furthermore, in the above model operator identification step, the operators required for reasoning of each deep neural network model are obtained, including:

[0018] Extract the hierarchical structure of each deep neural network model to be deployed, and identify and obtain various computing operators involved in the model reasoning process.

[0019] Furthermore, the above-mentioned computing operators include convolution operators, pooling operators and fully connected operators.

[0020] Furthermore, the above model task division step specifically includes:

[0021] For each deep neural network model, task segments are divided from top to bottom according to the hierarchical structure categories of the deep neural network model. During the division process, adjacent levels of the same hierarchical structure category are divided into the same task segment.

[0022] Furthermore, in the above pipeline operation control step, corresponding scheduling instructions are sent to multiple FPGA computing units, and obtained through the following steps:

[0023] According to the operators and task segment division results required for inference of each deep neural network model, the weight parameter segment allocation order, task segment execution order and data transmission path of each deep neural network model are generated as scheduling instructions for each FPGA.

[0024] Furthermore, in the above pipeline operation control step, controlling the corresponding FPGA computing unit to perform the pipeline operation specifically includes:

[0025] According to the execution order of the task segments of different deep neural network models to be deployed, the different task segments of the corresponding deep neural network models to be deployed are controlled to be executed alternately in different FPGA computing units until the calculation of the task segments of all models is completed.

[0026] On the other hand, the present invention also discloses a multi-FPGA collaborative multi-depth neural network pipeline acceleration system, including several FPGA computing units, a main control unit and a controller;

[0027] Among them, each FPGA computing unit is used to calculate different network structures of the deep neural network model and perform reasoning tasks corresponding to different network structures of the deep neural network model;

[0028] The main control unit is used to exchange data with each FPGA computing unit;

[0029] The controller can send corresponding scheduling instructions to multiple FPGA computing units according to the division results of operators and task segments required for reasoning of each deep neural network model to be deployed, deploy the operators required for reasoning of each deep neural network model to the corresponding FPGA computing units according to categories, and control the corresponding FPGA computing units to perform pipeline operations to support each FPGA computing unit to perform calculations on each task segment of the deep neural network model to be deployed, and complete the reasoning task of each deep neural network model to be deployed.

[0030] Preferably, in the above system, each operator required for inference of the deep neural network model to be deployed is obtained by the following steps:

[0031] The controller extracts the hierarchical structure of each deep neural network model to be deployed, identifies and obtains various computing operators involved in the model reasoning process.

[0032] Preferably, in the above system, the controller sends corresponding scheduling instructions to multiple FPGA computing units, which are obtained by the following steps:

[0033] According to the division results of operators and task segments required for reasoning of each deep neural network model, the execution order and data transmission path of each deep neural network model task segment are generated as scheduling instructions for each FPGA.

[0034] Preferably, in the above system, controlling the corresponding FPGA computing unit to perform pipeline operation specifically includes:

[0035] The controller controls the different task segments of the corresponding deep neural network models to be deployed to be executed alternately in different FPGA computing units according to the execution order of the task segments of different deep neural network models to be deployed, until the task segments of all models are calculated.

[0036] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a multi-FPGA collaborative multi-depth neural network pipeline acceleration method and system, which has the following beneficial effects:

[0037] The present invention can realize the coordinated acceleration of multiple deep neural network models on multiple FPGAs at the same time. The technical solution of the present invention makes full use of the high parallelism and flexibility of multi-FPGA collaboration, can deploy specific operators on the computing unit according to the characteristics of the model network structure, is applicable to a variety of deep neural network models, and can pipeline different computing tasks in the reasoning process of multiple models, effectively reducing the overall time of multi-model reasoning, so that multiple deep neural network models can be coordinated and accelerated on multiple FPGAs at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 It is an overall flow chart of the multi-FPGA collaborative multi-deep neural network pipeline acceleration method provided by an embodiment of the present invention.

[0040] Figure 2 It is a schematic diagram of the overall system structure of a method for implementing multi-FPGA collaborative multi-deep neural network pipeline acceleration provided by an embodiment of the present invention.

[0041] Figure 3 It is a flow chart of the inter-board communication connection operation in the method provided in an embodiment of the present invention.

[0042] Figure 4 It is a flowchart of the operation of the task allocation strategy in the method provided by an embodiment of the present invention.

[0043] Figure 5 It is a schematic diagram of the task division of the AlexNet model in the method provided in an embodiment of the present invention.

[0044] Figure 6 It is a schematic diagram of the task division of the VGG16 model in the method provided in an embodiment of the present invention.

[0045] Figure 7 It is a flow chart of the operation of setting the computing unit in the method provided by an embodiment of the present invention.

[0046] Figure 8 It is a flow chart of pipeline operation control in the method provided by an embodiment of the present invention.

[0047] Fig. 9 It is a timing diagram of the execution of pipeline operation in the method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The embodiment of the present invention firstly discloses a multi-FPGA collaborative multi-deep neural network pipeline acceleration method, which mainly includes three steps: model operator identification, model task division, and pipeline operation control.

[0050] In the specific implementation process, Figure 1 As shown, the acceleration method may include inter-board communication connection, task allocation strategy, computing unit setting, flow line operation control and other processes. The above method can be based on Figure 2 The system structure shown is implemented. The steps of the present invention in the specific implementation process are described in detail below in conjunction with the accompanying drawings. The multi-FPGA collaborative multi-deep neural network pipeline acceleration method includes the following specific implementation steps:

[0051] S101, inter-board communication connection: In this step, the host computer is used as the main control unit, responsible for the overall control and scheduling tasks, and each FPGA is used as a computing unit to perform specific computing tasks. The FPGAs are connected through high-speed interfaces to achieve fast inter-board data transmission; each FPGA is connected to the host computer equipped with a controller through a high-speed bus.

[0052] S102, task allocation strategy: In this step, the host computer divides tasks according to the network hierarchy structure and computing requirements of each model to be deployed, and divides the reasoning process of each model into multiple task segments.

[0053] S103, computing unit setting: In this step, the operators required for model reasoning are identified according to the network structure and computing requirements of each network model, and different operators are deployed on each FPGA computing unit; after each network model is divided into task segments, the parameters required for the computing task segments are stored in the off-chip memory of the main control unit.

[0054] S104, pipeline operation control: In this step, the controller in the host computer sends corresponding scheduling instructions through the high-speed bus according to the settings of the computing units and the task division of each model to be deployed, and controls each computing unit FPGA to perform pipeline operations to perform the calculation of the task segment and the reasoning of each model.

[0055] In step S101 of the above embodiment, the main control unit refers to the host computer responsible for the overall system control and scheduling tasks, which can send data to other FPGAs and exchange data; the computing unit refers to an FPGA equipped with computing operators to perform calculations on different network layer structures; the high-speed interface refers to a communication interface such as PCIE, MiniSAS, QSFP+, etc. that can connect FPGAs to each other to achieve high-speed data transmission; the controller refers to a program in the host computer that can generate scheduling instructions for controlling each computing unit according to the division of computing tasks, and generate code or configuration files that can be executed by the FPGA.

[0056] Furthermore, if Figure 3 As shown, the inter-board communication connection in step S101 includes the following detailed steps:

[0057] S1011: Hardware connection: The host computer is used as the main control unit, and each FPGA is used as a computing unit. The FPGAs are connected to the host computer through a high-speed bus to achieve fast data transmission between boards.

[0058] For example, the host computer can be an ARM-based processor or CPU, with the host computer as the main control unit and the FPGA as the computing unit. The FPGA as the computing unit is connected to the host computer as the main control unit through the PCIe high-speed communication bus, and the computing units are also connected to each other to achieve data communication between boards.

[0059] S1012: Data transmission module control: Design driver and interface application programs on the host computer side, design data transmission modules adapted to high-speed interfaces on the main control unit side and each computing unit side, and realize data transmission and reception between each FPGA.

[0060] For example, the PCIe hardware module IP core provided by the manufacturer Xilinx is used to implement data communication between FPGAs.

[0061] S1013: Data Scheduling: The controller of the host computer is responsible for scheduling the data transmission of the entire system, reading parameters from the off-chip memory, sending the parameters required for the calculation of each task segment to the computing unit, and controlling the computing unit to pass the calculation results of each task segment to the next computing unit to ensure the orderly transmission of the data flow.

[0062] In the above step S102, the network hierarchical structure of the model refers to the type and connection method of each network layer of the model; the computing requirements refer to the specific requirements of different network layer types for computing operations and algorithm implementation; the task segment refers to dividing the same type of network layers into a segment of computing tasks in turn according to the model reasoning process, and dividing the model reasoning process into multiple independent computing tasks.

[0063] Furthermore, if Figure 4 As shown, the task allocation strategy in step S102 includes the following detailed steps:

[0064] S1021: Model structure analysis: The host computer extracts the hierarchical structure of each neural network model and analyzes the various types of computing operators involved in the model reasoning process.

[0065] For example, the hierarchical structure of a convolutional neural network model includes convolutional layers, pooling layers, and fully connected layers, and the operators involved include convolution operators, pooling operators, and fully connected operators.

[0066] S1022: Task division strategy: The upper computer determines whether task division is required based on the network hierarchical structure of each model to be deployed. During the process of model reasoning layer by layer, if the hierarchical structure remains unchanged, it is determined that no division is required and the reasoning continues to the next layer. If the hierarchical structure changes, it is determined that the division operation is performed and the model to which each divided task segment belongs is marked until the compiler divides each model to be deployed into multiple task segments.

[0067] For example, if the first network layer is a convolutional layer, the second network layer is a pooling layer, and the third network layer is also a convolutional layer, the first layer is divided into one task segment, the second layer is divided into one task segment, and the third layer is divided into one task segment. Figure 5 A schematic diagram of the task division results of the AlexNet network model is given. Figure 6 A schematic diagram of the task division results of the VGG16 network model is given.

[0068] S1023: Weight parameter division: The weight parameters corresponding to the task segments obtained by dividing each model to be deployed are also divided into parameter segments accordingly, and the model to which each divided parameter segment belongs is marked.

[0069] In the above step S103, the operator refers to a basic computing unit used to calculate different network layers in the network model; the off-chip memory refers to a device such as SDRAM, DDR3, etc. outside the chip used to expand the storage capacity of the system.

[0070] Furthermore, if Figure 7 As shown, the calculation unit setting in step S103 includes the following detailed steps:

[0071] S1031: Operator hardware deployment: Determine all computing operators involved in the network model to be deployed and classify them according to their functional characteristics, and deploy operators of each functional type on specific computing units. Each computing unit undertakes computing tasks of different operators.

[0072] For example, a convolution operator is deployed on computing unit 1 for convolution layer calculation, a pooling operator is deployed on computing unit 2 for pooling layer calculation, a fully connected operator is deployed on computing unit 3 for fully connected layer calculation, and so on.

[0073] S1032: Operator resource configuration: Evaluate the hardware resources of the FPGA, and based on the characteristics of the operators and the size of the computing load they bear, map low-load operators to computing units composed of a single FPGA, and map high-load operators to computing units composed of multiple FPGAs.

[0074] For example, the pooling operator is deployed on a computing unit consisting of a single FPGA, and the convolution operator is deployed on a computing unit consisting of two or more FPGAs to improve the computing speed of the convolution operation.

[0075] S1033: Weight parameter configuration: configure an off-chip memory for the main control unit, mark and initialize the weight parameters of all models to be deployed, and store them on the off-chip memory according to the results of task division.

[0076] In the above step S104, the high-speed bus refers to a bus such as PCIe used for communication between the host computer and each FPGA; the scheduling instruction refers to an instruction that controls the collaborative calculation of the computing units to perform model reasoning; the pipeline operation refers to the operation in which the computing unit completes a specific computing task at each stage and passes the result to the computing unit of the next stage to realize parallel computing in the form of a pipeline.

[0077] Furthermore, if Figure 8 As shown, the pipeline operation control in step S104 includes the following detailed steps:

[0078] S1041: Generate scheduling instructions: The controller in the host computer generates the allocation order of each model weight parameter segment, the execution order of the task segment and the data transmission path as scheduling instructions for each FPGA according to the configuration of the computing unit and the task division result.

[0079] S1042: Pipeline processing: The upper computer allocates weight parameter segments to corresponding computing units through scheduling instructions, controls the execution process and data flow of each computing unit, and monitors the execution progress of the task segments of each computing unit. The entire reasoning process is carried out in a pipeline form, and the task segment calculations of each model are executed alternately until the task segment calculations of all models are completed.

[0080] For example, at a certain stage in the computational reasoning process, the convolutional computing unit is responsible for executing task segment x of model 1. When entering the next stage, the calculation result of task segment x of model 1 is passed to the pooling computing unit for the next step of calculation. At the same time, task segment y of model 2 is sent to the convolutional computing unit for calculation. When entering the next stage, the result of task segment y of model 2 is passed to the pooling computing unit, and task segment x+1 of model 1 is sent to the convolutional computing unit for calculation. This is repeated alternately until all the task segments of the models are calculated. For a specific timing diagram of the execution of the system pipeline operation, please refer to Fig. 9 shown.

[0081] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0082] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-FPGA collaborative multi-deep neural network pipeline acceleration method, characterized in that: The following steps are involved: Model operator identification: According to the network structure and computing requirements of different deep neural network models to be deployed, obtain the operators required for reasoning of each deep neural network model; Model task division: Divide the tasks according to the network hierarchical structure and computing requirements of each deep neural network model to be deployed, and divide the reasoning process of each deep neural network model to be deployed into multiple task segments; Pipeline operation control: According to the division results of operators and task segments required for reasoning of each deep neural network model to be deployed, corresponding scheduling instructions are sent to multiple FPGA computing units, and the operators required for reasoning of each deep neural network model are deployed to the corresponding FPGA computing units according to categories, and the corresponding FPGA computing units are controlled to perform pipeline operations to perform calculations on each task segment of the deep neural network model to be deployed and complete the reasoning of each deep neural network model to be deployed.

2. The multi-FPGA collaborative multi-deep neural network pipeline acceleration method according to claim 1 is characterized in that: In the model operator identification step, obtaining the operators required for reasoning of each deep neural network model specifically includes: Extract the hierarchical structure of each deep neural network model to be deployed, and identify and obtain various computing operators involved in the model reasoning process.

3. The multi-FPGA collaborative multi-deep neural network pipeline acceleration method according to claim 2 is characterized in that: The computing operators include a convolution operator, a pooling operator and a fully connected operator.

4. The multi-FPGA collaborative multi-deep neural network pipeline acceleration method according to claim 1 is characterized in that: The model task division step specifically includes: For each deep neural network model, task segments are divided from top to bottom according to the hierarchical structure categories of the deep neural network model. During the division process, adjacent levels of the same hierarchical structure category are divided into the same task segment.

5. The method for accelerating a multi-FPGA collaborative multi-deep neural network pipeline according to claim 1, characterized in that: In the pipeline operation control step, corresponding scheduling instructions are sent to multiple FPGA computing units, and obtained through the following steps: According to the operators and task segment division results required for inference of each deep neural network model, the weight parameter segment allocation order, task segment execution order and data transmission path of each deep neural network model are generated as scheduling instructions for each FPGA.

6. The multi-FPGA collaborative multi-deep neural network pipeline acceleration method according to claim 1 is characterized in that: In the pipeline operation control step, controlling the corresponding FPGA computing unit to perform the pipeline operation specifically includes: According to the execution order of the task segments of different deep neural network models to be deployed, the different task segments of the corresponding deep neural network models to be deployed are controlled to be executed alternately in different FPGA computing units until the calculation of the task segments of all models is completed.

7. A multi-FPGA collaborative multi-deep neural network pipeline acceleration system, characterized in that: It includes several FPGA computing units, a main control unit and a controller; Among them, each FPGA computing unit is used to calculate different network structures of the deep neural network model and perform reasoning tasks corresponding to different network structures of the deep neural network model; The main control unit is used to exchange data with each FPGA computing unit; The controller can send corresponding scheduling instructions to multiple FPGA computing units according to the division results of operators and task segments required for reasoning of each deep neural network model to be deployed, deploy the operators required for reasoning of each deep neural network model to the corresponding FPGA computing units according to categories, and control the corresponding FPGA computing units to perform pipeline operations to support each FPGA computing unit to perform calculations on each task segment of the deep neural network model to be deployed, and complete the reasoning task of each deep neural network model to be deployed.

8. The multi-FPGA collaborative multi-deep neural network pipeline acceleration system according to claim 7, characterized in that: The operators required for inference of each deep neural network model to be deployed are obtained through the following steps: The controller extracts the hierarchical structure of each deep neural network model to be deployed, identifies and obtains various computing operators involved in the model reasoning process.

9. The multi-FPGA collaborative multi-deep neural network pipeline acceleration system according to claim 7, characterized in that: In the controller, corresponding scheduling instructions are sent to multiple FPGA computing units, and obtained by the following steps: According to the division results of operators and task segments required for reasoning of each deep neural network model, the execution order and data transmission path of each deep neural network model task segment are generated as scheduling instructions for each FPGA.

10. The multi-FPGA collaborative multi-deep neural network pipeline acceleration system according to claim 7, characterized in that: Control the corresponding FPGA computing unit to perform pipeline operations, including: The controller controls the different task segments of the corresponding deep neural network models to be deployed to be executed alternately in different FPGA computing units according to the execution order of the task segments of different deep neural network models to be deployed until the calculation of the task segments of all models is completed.

Images