A multi-fpga collaborative training neural network method based on distributed optimization

Abstract

The application provides a kind of multi-FPGA collaborative training neural network method based on distributed optimization, with host computer as core control module, uses Ethernet and FPGA communication, distributes training data, finally recycles training parameters;By building the training unit of convolutional neural network on a single FPGA, it includes the calculation module of neural network basic unit, the instruction module of control operation, the DDR-RAM storage module with RAM as cache area and the neural network training module with gradient descent method to update training parameters by BP algorithm using cross-entropy function as loss function, to establish the global objective function of target collaborative optimization;Through UART serial port, communication connection is established between different FPGAs, and necessary handshake signals for starting and ending training are transmitted. Through collaborative control of multiple FPGAs and training of convolutional neural network based on distributed optimization algorithm, the advantages of FPGA underlying parallel computing and low power consumption are fully utilized, and the compatibility is good, which is conducive to the application and progress of CNN accelerator technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of neural network accelerator, in particular to a multi-FPGA collaborative training neural network method based on distributed optimization. BACKGROUND

[0002] Deep learning algorithm is widely used due to its superior performance, but as human beings enter the era of big data, the explosive growth and increasing complexity of data make centralized algorithm inherently insufficient and unable to handle such large data. At the same time, as the network layer of deep learning becomes deeper and the model size becomes larger, centralized algorithm also cannot store and calculate the huge model parameters. From the perspective of computing power and data storage capacity, a single machine cannot handle such large-scale data, so parallelization is urgently needed. Distributed deep learning algorithm, aiming at high computational complexity, large training data size, and large model size, adopts data parallel or model parallel method, i.e. multi-process and multi-thread parallel computing, data is distributed to multiple nodes, multiple nodes train the model, and finally the training parameters are optimized.

[0003] With the development of multi-agent system theory and coordination technology, distributed optimization can be realized by means of multi-agent network. Distributed optimization algorithm has gradually become a hot research direction. System resources are shared by all hosts, and each host not only trains the model by using its own computing power and resources, but also uses the resources and data of other hosts. In the distributed algorithm, each mobile terminal is connected in a certain network structure, and when they work collaboratively, they do not need to know the information of all "participants" in the global network. In the process of information exchange with neighboring nodes, the global problem can be optimized and solved.

[0004] With the continuous development of electronic information technology, more and more large convolutional neural networks have realized training and deployment on hardware edge devices. FPGA (Field Programmable Gate Array) is a semi-custom circuit in the field of ASIC (Application Specific Integrated Circuit), which balances flexibility and efficiency. With its relatively low price, low power consumption and parallel computing advantages compared with GPU, FPGA has become an ideal hardware device for training and deploying convolutional neural networks. However, in the existing technology, there is no scheme for multiple FPGAs to run collaboratively, and multiple FPGAs can fully exert the advantages of FPGA in parallel computing and low power consumption. SUMMARY

[0005] The application aims to provide a multi-FPGA cooperative training neural network method for cooperative control of multiple FPGAs, convolutional neural network training based on a distributed optimization algorithm, and full play of the advantages of FPGA bottom layer parallel computing and low power consumption.

[0006] To achieve the above-mentioned purpose, the application provides a multi-FPGA cooperative training neural network method based on distributed optimization, comprising the following steps:

[0007] Step 1: building a training unit of a convolutional neural network on a single FPGA, i.e. building a convolution, pooling, full connection, activation function layer and data buffer module;

[0008] Step 1.1: using an instruction module to encode the required convolution, pooling, full connection, activation function and other calculation requirements and the corresponding initial storage address, sending the model information after encoding to the FPGA end for decoding operation, and then obtaining the calculation type and storage address information;

[0009] Step 1.2: developing a multiplier and an adder on the FPGA chip to form a basic calculation module;

[0010] Step 1.3: integrating the RAM and DDR resources on the FPGA chip to build a data buffer for the huge data calculation and storage requirement in the neural network accelerator;

[0011] Step 1.4: optimizing the network by using a distributed stochastic gradient descent algorithm;

[0012] Step 2: using Ethernet to communicate with the PC, using the host computer as the core scheduling process, distributing data, and finally recycling the training parameters;

[0013] Step 3: establishing a communication topology relationship among multiple FPGAs.

[0014] Further, in step 1.2, the shift register is used to read the required information from the input information stream, and the remaining information is combined for calculation, a PE calculation unit is built, and based on this, convolution and full connection calculation are performed; pooling is realized by using shift register and size comparison and mean calculation; a simple calculation platform is built by using a lookup table for different types of activation functions.

[0015] Further, in step 1.3, based on the characteristics of RAM that the read-write is flexible but the storage capacity is limited, and DDR that the storage capacity is large but limited by bandwidth, a finite state machine is built to allocate addresses for the intermediate calculation results and the weights obtained by training and other large-scale intermediate information that need to be stored, and the information is transmitted to the DDR for storage. If the data bit width generated by the calculation module is greater than the DDR bit width, the data outside the DDR bit width is first stored in the RAM, and then transmitted after the bit width is idle.

[0016] Further, in step 1.4, the method for optimizing the network is as follows:

[0017] The loss function is measured by using the cross-entropy function, and the calculation formula is as follows:

[0018]

[0019] The final goal is to optimize the global objective function as follows, where f(x) is the local function to be optimized:

[0020]

[0021] The output value of each neuron, the error term, the gradient of each neuron connection weight and bias, the convolution weight and bias term are read from the data buffer, and the weight and bias are updated by using the gradient descent method through the BP algorithm, and the specific function is as follows:

[0022]

[0023]

[0024] In the formula, η is the learning rate, ω is the weight value, b is the bias value, ω l and b l represent the weight and bias of the l-1 layer to the l layer, O k l The kth neuron of the lth layer has used the activation function; in order to consider the training accuracy requirement, all weight parameters, intermediate activation and calculation weights, loss gradients are stored with 16-bit fixed-point precision, and multiplication and division are replaced by digital shift as much as possible; considering the large data processing requirement, a pipeline can be built to use the on-chip resources of FPGA as much as possible to improve the parallelism; finally, the weight value and bias term when the loss function reaches the minimum value are obtained through the gradient descent algorithm.

[0025] Further, in step 2, the PS end of the ZYNQ development board has an Ethernet interface, the Ethernet address is modified, so that multiple FPGAs can communicate with the host computer through the Ethernet; the host computer randomly allocates training data and sends it to the PS end of the ZYNQ development board, and then transmits it to the PL end for training through the AXI-DMA; the obtained intermediate value of the calculation weight is transmitted to the buffer end; finally, after receiving the signal of the end of training of each development board, the final weight value is received for unified processing.

[0026] Further, in step 3, the following steps are included:

[0027] Step 3.1: Establish a communication connection between different FPGAs through the UART serial port to transmit necessary training start and end handshake signals, and ensure that each stage of training is synchronized.

[0028] Step 3.2: Set the weight value of the convolution training parameter according to the communication topology relationship.

[0029] Further, in step 3.2, the center node of the star network is connected to each node, so the weight is set to 1 / num_user, and the remaining connected nodes are only connected to the center node, and the weight is set to 1-1 / num_user; the default of the ring network communication is bidirectional, and the positions P m,n of the corresponding nodes connected to each other are set to 1 / 3; the complete network is connected to other nodes, so in the n*n matrix, P m,n of the corresponding nodes connected to each other are set to 1 / num_user.

[0030] Compared with the prior art, the advantages of the present application are:

[0031] The present application realizes the cooperative control of multiple FPGAs, and trains the convolutional neural network based on a distributed optimization algorithm, fully utilizes the advantages of FPGA bottom parallel computing and low power consumption, and has good compatibility, which is conducive to promoting the application and progress of CNN accelerator technology. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a schematic diagram of the PE calculation module in the embodiment of the present application;

[0033] Figure 2 is a schematic block diagram of the data buffer in the embodiment of the present application;

[0034] Figure 3 is a schematic diagram of the hardware accelerator structure under a single FPGA in the embodiment of the present application;

[0035] Figure 4 is a schematic block diagram of the overall structure of the multi-FPGA cooperative training neural network in the embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be further described below.

[0037] The present application provides a multi-FPGA collaborative training neural network method based on distributed optimization, comprising the following steps:

[0038] Step 1: as shown in the figure, the training of the convolutional neural network is built on a single FPGA, and the convolution, pooling, full connection, activation function layer and data buffer module are built. Figure 3

[0039] Step 1.1: design an instruction module, encode the required convolution, pooling, full connection, activation function and other calculation requirements and the corresponding initial storage address, and send the model information code in the form of hexadecimal code from the total console to the FPGA end for operation. The FPGA end decodes and further obtains the effective information required for calculation, such as the type of calculation and the address of storage.

[0040] Step 1.2: develop a multiplier and an adder on the FPGA chip to form a basic calculation module. As shown in the figure, the shift register is used to read the required information from the input information stream, and the calculation is combined with other information to build a PE calculation unit. Based on this, the convolution and full connection calculation are carried out; the pooling is realized by using shift register and size comparison, and the mean value calculation; a simple calculation platform is built for different types of activation functions by using lookup table. Figure 1

[0041] Step 1.3: in view of the huge data calculation and storage demand of the neural network accelerator, integrate the RAM and DDR resources on the FPGA chip to build a data buffer. Considering the characteristics of flexible RAM reading and writing but limited storage capacity, and large DDR storage capacity but limited by bandwidth. Build a finite state machine, allocate addresses for large-scale intermediate information such as intermediate calculation results and training weights, and store them in DDR. If the data bit width generated by the calculation module is greater than the DDR bit width, the data outside the DDR bit width will be stored in the RAM first, and then transmitted after the bit width is idle.

[0042] Figure 2 ​​The data flow diagram of the application (N > M) is shown, N is the bit width of the data generated by the calculation module, and M is the bandwidth of the DDR read and write data. The overall core of the system is a state machine control, and the intermediate results are temporarily stored in the RAM. When the DDR read and write lines are idle, the data is read out and written in to ensure that the system is not limited by the DDR bandwidth and waits for data reading and writing. When N > M, the RAM is responsible for storing and transmitting data from M to N-1 bits, and the DDR is responsible for storing and transmitting data from 0 to M-1 bits.

[0043] Step 1.4: Optimizing the network using a distributed stochastic gradient descent algorithm. The loss function uses the Cross Entropy function to measure, and the calculation formula is as follows:

[0044]

[0045] The final goal is to optimize the global objective function as follows, where f(x) is the local function to be optimized:

[0046]

[0047] The output value of each neuron, the error term, the gradient of each neuron connection weight and bias, the convolution weight and bias term read from the data buffer, and the weight and bias updated by the BP algorithm using the gradient descent method are calculated. The specific function is as follows:

[0048]

[0049]

[0050] η is the learning rate, ω is the weight value, b is the bias value, ω l and b l represent the weight and bias of the l-1 layer to the l layer, O k l The kth neuron of the lth layer has used the activation function. Here, considering the accuracy of the training, all weight parameters, intermediate activations and loss gradients are stored with 16-bit fixed-point precision, and multiplication and division are replaced by digital shift as much as possible. Considering the large data processing requirements, a pipeline can be built to maximize the use of on-chip resources and improve parallelism. Finally, the weight value and bias term when the loss function reaches the minimum value are obtained by the gradient descent algorithm.

[0051] Figure 4 The overall system architecture after the single computer neural network training system is built is shown, including the connection between FPGA and host computer, the communication between PL and PS in ZYNQ development board and the establishment of communication topology relationship between multiple FPGAs:

[0052] Step 2: communicate with PC by Ethernet, PC as the core scheduling process, distribute data, and finally recycle training parameters.

[0053] The PS end of the ZYNQ development board has an Ethernet interface, and the Ethernet address is modified so that multiple FPGAs can communicate with the host computer through Ethernet. The host computer randomly distributes training data and sends it to the PS end of the ZYNQ development board, and then transmits it to the PL end through AXI-DMA for training. The intermediate value of the calculated weight is transmitted to the buffer end. Finally, after receiving the signal that the training of each development board is completed, the final weight value is received for unified processing.

[0054] Step 3: establish communication topology relationship among multiple FPGAs.

[0055] Step 3.1: establish communication connection between different FPGAs through UART serial port to transmit necessary training start and end handshake signals to ensure synchronization of each stage of training.

[0056] Step 3.2: set the weight value of the convolution training parameter according to the communication topology relationship, taking the common star type, ring type and complete type optimization as examples. In the star network, the center node is connected to each node, so the weight is set to 1 / num_user, and the remaining connected nodes are only connected to the center node, and the weight is set to 1-1 / num_user; in the ring network communication, the corresponding nodes with mutual connection are set to 1 / 3; in the complete network, each node is connected to other nodes, so in the n*n matrix, the corresponding nodes with mutual connection are set to 1 / num_user. m,n m,n

[0057] The above is only a preferred embodiment of the present application, and does not limit the present application in any way. Any person skilled in the art can make any form of equivalent replacement, modification or change to the technical solutions and technical content disclosed in the present application without departing from the scope of the technical solutions of the present application, and such changes still fall within the protection scope of the present application.​​

Claims

1. A method for collaborative training of neural networks using multi-FPGA based on distributed optimization, characterized in that, Includes the following steps: Step 1: Build the training unit of the convolutional neural network on a single FPGA, namely, the construction of convolution, pooling, fully connected, activation function layers and data caching modules; Step 1.1: The instruction module performs encoding operations based on the required convolution, pooling, fully connected, and activation function calculations and the corresponding initial storage addresses. After the model information is encoded, it is sent to the FPGA for decoding to obtain the types of calculations and storage address information. Step 1.2: Develop an on-chip multiplier and adder for the FPGA to form a basic computational module; Step 1.3: To address the massive data computation and storage requirements of neural network accelerators, integrate the on-chip RAM and DDR resources of the FPGA to build a data cache; Step 1.4: Optimize the network using the distributed stochastic gradient descent algorithm; Step 2: Communicate with the PC via Ethernet. The host computer acts as the core scheduling and processing unit, allocates data, and finally collects the training parameters. Step 3: Establish communication topology between multiple FPGAs; Step 3 includes the following steps: Step 3.1: Establish a communication connection between different FPGAs via UART serial port to transmit the necessary handshake signals for the start and end of training, ensuring that each stage of training is carried out synchronously; Step 3.2: Set the weight values ​​of the convolution training parameters according to the communication topology; In step 3.2, the central node of the star network is connected to every other node, so its weight is set to 1 / num_user. The remaining connected nodes are only connected to the central node, and their weights are set to 1 - 1 / num_user. The ring network communication is bidirectional by default, so the corresponding interconnected node positions P are... m,n The weight is set to 1 / 3; in a complete network, because every node is connected to every other node, P in an n*n matrix... m,n The weights are all set to 1 / num user.

2. The multi-FPGA collaborative training neural network method based on distributed optimization according to claim 1, characterized in that, In step 1.2, the shift register is used to read the information required for calculation from the input information stream, and the calculation is performed in combination with the remaining information to build a PE calculation unit. Based on this, convolution and fully connected calculations are performed. Pooling is achieved by shift registers, size comparison, and mean calculation. A simple calculation platform is built using lookup tables for different types of activation functions.

3. The multi-FPGA collaborative training neural network method based on distributed optimization according to claim 1, characterized in that, In step 1.3, based on the characteristics of RAM being more flexible in reading and writing but having limited storage capacity, and DDR having large storage capacity but being limited by bandwidth, a finite state machine is built to allocate addresses for intermediate information that needs to be stored on a large scale, such as intermediate calculation results and training weight values, and store them in DDR. If the data bit width generated by the calculation module is greater than the DDR bit width, the data beyond the DDR bit width is first stored in RAM and then transmitted when the bit width is free.

4. The multi-FPGA collaborative training neural network method based on distributed optimization according to claim 1, characterized in that, In step 1.4, the method for optimizing the network is as follows: The loss function is measured using the cross-entropy function, and the calculation formula is as follows: The final goal is to collaboratively optimize the global objective function as follows, where f(x) is the local function to be optimized: Calculate the output value, error term, and gradient of the connection weights and biases for each neuron. Read the convolution weights and biases from the data buffer and update the weights and biases using the backpropagation algorithm and gradient descent. The specific function is as follows: In the formula, η is the learning rate, ω is the weight value, b is the bias value, and ω l and b l This represents the weights and biases passed from layer l-1 to layer l, O k l The k-th neuron of the l-th layer has already used the activation function. Considering the accuracy requirements of training, 16-bit fixed-point precision is used to store all weight parameters, intermediate activations, and calculated weights and loss gradients. Multiplication and division are replaced by digital shifting. Considering the data processing requirements, a pipeline is built to make full use of the FPGA on-chip resources and improve parallelism. Finally, the weight values ​​and bias terms when the loss function reaches its minimum value are obtained through the gradient descent algorithm.

5. The multi-FPGA collaborative training neural network method based on distributed optimization according to claim 1, characterized in that, In step 2, the PS end of the ZYNQ development board has an Ethernet interface. The Ethernet address is modified so that multiple FPGAs can communicate with the host computer via Ethernet. The host computer randomly allocates training data and sends it to the PS end of the ZYNQ development board, and then transmits it to the PL end for training via AXI-DMA. The intermediate values ​​of the calculated weights are passed to the cache end. Finally, after receiving the signal that the training of each development board is completed, the final weight values ​​are received and processed uniformly.

Images