Deployment method of flame detection pulse neural network on FPGA

Abstract

The invention discloses a method for deploying a flame detection spiking neural network on an FPGA (Field Programmable Gate Array). According to the method, codes of the flame detection spiking neural network are written by using a Viis HLS tool, an IP (Internet Protocol) core file is generated and burnt on the FPGA, a related digital circuit is generated, the processing speed of the flame detection spiking neural network is accelerated, and the operation power consumption is reduced. And in the design process, the technical means such as ATAN arc tangent activation function calculation and short-circuit logic calculation strategies are realized by utilizing a lookup table, so that unnecessary calculation and power consumption can be reduced, the use amount of a DSP (Digital Signal Processor) is remarkably reduced, the hardware calculation operation process is simplified, and the pulse neural network can run on an FPGA (Field Programmable Gate Array) with relatively low power consumption. Therefore, the method provided by the invention provides a set of solution for implementing effective deployment on the low-power-consumption edge equipment based on the FPGA for a flame detection application scene.

Description

Technical Field

[0001] The present invention relates to the technical fields of image recognition technology and neural network technology hardware deployment, and in particular to a method for deploying a flame detection pulse neural network on an FPGA. Background Art

[0002] Since it was proposed in the 1980s, neuromorphic computing has attracted widespread attention from industry and academia. As a new generation of computing that is expected to reduce computing energy, latency, and learning complexity in neural networks, neuromorphic computing has gradually formed a basic algorithmic paradigm with spiking neural networks (SNNs) as the core. The core of this paradigm is to replace analog values ​​with peak values ​​("1" or "0") to transmit information. At present, there is an energy efficiency gap between the traditional artificial neural network architecture and the brain. With the rapid development of smart sensors and the Internet of Things, the amount of data and the complexity of related processing tasks are increasing day by day. How to implement neural networks in edge devices with limited power, energy, and resources has become a task worth exploring.

[0003] Many industries have tried to apply pulse neural networks based on their low power consumption. For example, the medical industry has tried to build a state discrimination model by monitoring physiological signals such as electrocardiogram, electrooculogram, and electromyography for use in neuromorphic hardware. In the industrial stage, traditional visual inspection has shortcomings such as slow response speed. Belbachir uses visual perception systems to effectively handle industrial automation tasks, such as piece counting and classification of moving objects.

[0004] With the increasing emphasis on public environment and traffic safety, there is a need for fire warnings in industrial living environments and notifications of fires caused by traffic accidents. Therefore, the technology of capturing environmental or traffic image data based on cameras and implementing flame detection has been applied. As a neural network model widely used in image recognition, the research on flame recognition using flame images to train spiking neural networks has been realized, but it is usually designed and deployed on high-energy GPU platforms. However, with the increasing application of edge computing technology, the deployment of flame recognition technology based on spiking neural networks on edge devices has attracted attention. For example, flame recognition with the help of edge devices such as environmental area cameras and traffic tunnel patrol robots can help to detect fires and issue warnings more timely.

[0005] From the perspective of technical implementation, pulse neural networks can be deployed on CPUs, GPUs, and FPGAs. The factors that determine deployment and implementation conditions are mainly processing performance, energy consumption, and deployment convenience. Among them, the CPU computing power is insufficient compared to the GPU, and it is difficult to give full play to the characteristics of pulse neural networks; the GPU computing power can well meet the processing performance requirements of pulse neural networks, but its power consumption is too high, resulting in limited application scenarios, and it is difficult to apply it to edge devices that need to rely on batteries for power; while FPGA has the characteristics of low power consumption and low latency, and FPGA has rich logic resources. By writing internal structures, large-scale parallel computing can be achieved. Deploying pulse neural networks on FPGAs can give full play to the characteristics of parallel computing of pulse neural networks, and compared with deploying them on CPUs and GPUs, it has the advantages of low energy consumption and easy deployment on edge devices.

[0006] However, the main challenge faced by the current FPGA deployment solution of pulse neural network is that the operating frequency of FPGA is usually 100-300MHz, which is much lower than the operating frequency of CPU and GPU. Therefore, it is difficult to achieve high computing performance with simple platform transplantation design. If the model architecture of pulse neural network is streamlined to adapt to the deployment requirements of FPGA, it is necessary to balance the problems of excessive model calculation delay and insufficient recognition accuracy.

[0007] Therefore, how to deploy pulse neural networks on FPGA platforms with limited computing resources, storage resources, and memory access bandwidth has become a problem to be solved by technical personnel in this field. Summary of the invention

[0008] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for deploying a flame detection pulse neural network on an FPGA, so as to provide a solution for flame detection application scenarios that can be effectively deployed on a low-power edge device based on an FPGA.

[0009] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0010] A method for deploying a flame detection pulse neural network on an FPGA comprises the following steps:

[0011] S1, using flame images and non-flame images as training samples, pre-training the pulse neural network for flame detection to obtain a trained flame detection pulse neural network; the flame detection pulse neural network includes an image encoding unit, a pulse neural network unit and a classification neuron unit;

[0012] S2, performing optimization processing on the trained flame detection pulse neural network for network parameter calculation to obtain an optimized flame detection pulse neural network;

[0013] S3. Use Vitis HLS tool to write the code of image encoding unit, pulse neural network unit and classification neuron unit in flame detection pulse neural network, and perform functional simulation verification. Then use Vivado software to perform code synthesis and convert the code generated by HLS tool into IP core file that can be run and executed by FPGA.

[0014] S4, performing hardware architecture deployment design corresponding to the flame detection pulse neural network structure on FPGA;

[0015] S5. Burn the obtained IP core file to the FPGA to complete the deployment of the flame detection pulse neural network on the FPGA.

[0016] Specifically, in the flame detection pulse neural network:

[0017] The image encoding unit is used to perform Poisson encoding on the input image data, and map the input image data into corresponding pulse sequence data;

[0018] The spiking neuron network unit contains multiple layers of neurons, each of which is fully connected with each neuron in the adjacent layer with weights, and the synaptic weights of the links connecting the neurons are neural network parameters determined through training; each neuron is used to update the membrane potential according to the weighted result of the input pulse, and when the membrane potential reaches the preset membrane potential threshold, the membrane potential is reset after the pulse is transmitted to the next layer of neurons connected to it;

[0019] The classification neuron unit includes two classification neurons that are fully connected to each neuron in the last layer of the pulse neuron network unit, respectively. The two classification neurons represent the detection results of the presence of flame in the image and the absence of flame in the image, respectively. Each classification neuron is used to count the number of pulses transmitted from the pulse neuron network unit to each of them, and the detection results represented by the classification neurons with the largest number of pulses are used as the flame detection results output by the flame detection pulse neural network.

[0020] Specifically, in the spiking neuron network unit, the membrane potential value attenuation of a single neuron is determined according to the LIF neuron model; the calculation method of the membrane potential update value of a single neuron is:

[0021] V i =atan_function(spike i,in )

[0022] Among them, V i represents the updated membrane potential value of any i-th neuron, spike i,inrepresents the input pulse value of the i-th neuron; atan_function(·) represents the ATAN inverse tangent activation function operation, and the ATAN inverse tangent activation function operation method is:

[0023]

[0024] Among them, α is the function constant of the ATAN inverse tangent activation function;

[0025] The input pulse value of the i-th neuron is spike i,in for:

[0026]

[0027] Among them, spike ji It represents the weighted result of the input pulse transmitted from the jth neuron connected to the i-th neuron through the synaptic weight of the connection link to the i-th neuron, j = 1, 2, ..., n i , n i Represents the total number of neurons that are fully connected to the i-th neuron in the forward direction.

[0028] Specifically, in the pulse neuron network unit, the pulse signal value transmitted by a single neuron to the next layer of neurons connected to it is:

[0029]

[0030] Among them, spike i,out V represents the pulse signal value sent by any i-th neuron; threshold Indicates the preset membrane potential threshold.

[0031] Specifically, in step S2, the optimization process of calculating the network parameters of the trained flame detection pulse neural network includes:

[0032] The ATAN inverse tangent activation function values ​​corresponding to different numerical values ​​are calculated in advance and recorded in the ATAN inverse tangent activation function value lookup table; when calculating the updated value of the neuron's membrane potential, the corresponding ATAN inverse tangent activation function value is obtained by querying the ATAN inverse tangent activation function value lookup table.

[0033] Specifically, in step S2, the optimization process of calculating the network parameters of the trained flame detection pulse neural network also includes:

[0034] When calculating the updated value of the neuron's membrane potential, the weighted result of the input pulse transmitted by any neuron in the forward connection through the synaptic weight of the connection link is calculated using the following short-circuit logic strategy: for any i-th neuron, if the pulse signal value sent by the j-th neuron in its forward connection is greater than 0, the weighted result of the corresponding input pulse takes the value spike ji is the synaptic weight value of the corresponding connection link; if the pulse signal value sent by the jth neuron of its forward connection is less than or equal to 0, the weighted result value of the corresponding input pulse is spike ji The value is 0; that is:

[0035]

[0036] Among them, spike j,out W represents the pulse signal value sent by the jth neuron connected to the i-th neuron; ji Represents the synaptic weight of the link between the i-th neuron and the j-th neuron of the forward connection.

[0037] Specifically, the step S3 specifically includes:

[0038] S301. According to the unit division structure of the image coding unit, the pulse neural network unit and the classification neural network unit in the flame detection pulse neural network, use the C++ high-level programming language and the HLS component to write a program, design the functional modules and test modules corresponding to each unit, and create an HLS project;

[0039] S302, for each unit, use the test module to perform functional simulation on the functional module, and use the debugger integrated in the HLS tool to perform functional debugging during the simulation process until the functional simulation is correct;

[0040] S303, calculating the ATAN inverse tangent activation function values ​​corresponding to different values, recording them in an ATAN inverse tangent activation function value lookup table, converting the ATAN inverse tangent activation function value lookup table into a format suitable for use with FPGA, and writing it into the code of the spiking neural network unit through the Vitis HLS tool;

[0041] S304, using Vivado software to perform code synthesis, and using the HLS tool to save the synaptic weight values ​​of the connections between the neurons in the pulse neuron network unit of the trained flame detection pulse neural network as the header file of the code, and then converting the code generated by the HLS tool into an IP core file that can be run and executed by FPGA.

[0042] Specifically, in step S4, the specific method of performing the hardware architecture deployment design corresponding to the flame detection pulse neural network structure on the FPGA is:

[0043] The hardware architecture deployed on the FPGA includes image encoding module, pulse neuron processing module, classification statistics module, data cache module and control center module;

[0044] The image encoding module is used to perform Poisson encoding on the input image data, and map the input image data into corresponding pulse sequence data;

[0045] The pulse neuron processing module includes a pulse buffer unit and a plurality of interconnected physical neurons; wherein the pulse buffer unit is used to buffer the pulse sequence data obtained by mapping the image data transmitted by the image encoding module, and send it to the physical neurons; the physical neurons are used to update the membrane potential according to the weighted result of the input pulses, and when the membrane potential reaches a preset threshold, the membrane potential is reset after the pulse is transmitted to the next layer of neurons connected thereto;

[0046] The classification statistics module includes two pulse statistics units and a statistical data storage; the two pulse statistics units are used to respectively count the number of high-order pulse values ​​and the number of low-order pulse values ​​transmitted by the pulse neuron processing module, respectively representing the number of detection results of flames in the recognition image and flames in the image; the statistical data storage is used to store and output the statistical results of the two pulse statistics units;

[0047] The data cache module is connected to the pulse neuron processing module and the classification statistics module respectively, and is used to cache the pulse signal value output and transmitted by the pulse neuron processing module, and send it to the classification statistics module for classification statistics;

[0048] The control center module is connected to the pulse neuron processing module and the classification statistics module respectively, and is used to send control signals to control the image encoding module to store the pulse sequence data obtained by Poisson encoding into the specified address in the pulse neuron processing module of the pulse cache unit, and control each physical neuron in the pulse neuron processing module to start or stop updating the membrane potential, and the order and synaptic weight value of pulse transmission between physical neurons, and control the storage address of the pulse signal value output and transmitted by the pulse neuron processing module in the data cache module and the address order of the classification statistics module to read the pulse signal value from the data cache module for statistics.

[0049] Specifically, the physical neuron includes a neuron controller, and a membrane potential update unit, a membrane potential memory, a forward weight management unit, a forward weight memory, a threshold voltage management unit, a threshold voltage memory, and an internal pulse interaction unit respectively connected to the neuron controller;

[0050] Wherein, the forward weight management unit is connected to the forward weight memory and is used to retrieve the synaptic weight value stored in the forward weight memory;

[0051] The threshold voltage management unit is connected to the threshold voltage memory and is used to retrieve the preset membrane potential threshold stored in the threshold voltage memory;

[0052] The membrane potential updating unit is connected to the membrane potential memory, the internal pulse interaction unit, the forward weight management unit and the threshold voltage management unit respectively, and is used to update the membrane potential according to the pulse signal value received by the physical neuron and the synaptic weight value read by the forward weight management unit, store the updated membrane potential in the membrane potential memory, and compare the membrane potential with the preset membrane potential threshold read by the threshold voltage management unit, and send a pulse signal and reset the membrane potential in the membrane potential memory when the membrane potential reaches the preset membrane potential threshold;

[0053] The internal pulse interaction unit is connected to the membrane potential update unit, and is used to transmit the pulse signal value received by the physical neuron to the membrane potential update unit, and transmit the pulse signal transmitted by the membrane potential update unit to other physical neurons;

[0054] The neuron controller is used to receive control signals from the control center module, control the operation of each unit in the physical neuron, issue addresses for signal reading and writing in the physical neuron, and control the transmission path for pulse signal interaction between the internal pulse interaction unit in the physical neuron and other physical neurons.

[0055] Specifically, the specific method of the membrane potential updating unit when performing the updating calculation of the membrane potential is:

[0056] S401, first use the short-circuit logic strategy to determine the weighted result of the input pulse transmitted from each forward-connected physical neuron to the physical neuron; if the pulse signal value transmitted by the forward-connected physical neuron is greater than 0, read the synaptic weight value from the forward weight memory through the forward weight management unit as the weighted result value of the input pulse; if the pulse signal value transmitted by the forward-connected physical neuron is less than or equal to 0, directly take the value of 0 as the weighted result value of the input pulse;

[0057] S402, accumulating the weighted result values ​​of the input pulses transmitted from each forward-connected physical neuron to the physical neuron to obtain the input pulse value of the physical neuron;

[0058] S403. Obtain the ATAN inverse tangent activation function value corresponding to the input pulse value of the physical neuron by querying the ATAN inverse tangent activation function value lookup table, and store it in the membrane potential memory as the membrane potential update value of the physical neuron.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. The flame detection pulse neural network deployment method on FPGA of the present invention uses the Vitis HLS tool to write the code of the flame detection pulse neural network and generate an IP core file, which is burned to the FPGA to generate related digital circuits, thereby accelerating the processing speed of the flame detection pulse neural network and reducing the operating power consumption, providing a set of solutions for flame detection application scenarios based on FPGA for effective deployment on low-power edge devices.

[0061] 2. In the design process, the scheme of the present invention uses technical means such as lookup tables to implement ATAN inverse tangent activation function calculation and short-circuit logic calculation strategy, which can reduce unnecessary calculations and power consumption, significantly reduce the use of DSP, and is more conducive to simplifying the hardware calculation operation process, so that the pulse neural network can run on the FPGA with lower power consumption. At the same time, the parallel processing capability of the FPGA enables the neural network to process multiple input data at the same time, improving the throughput of the overall system. This high efficiency makes the FPGA-based pulse neural network have significant advantages in real-time processing and high-performance computing.

[0062] 3. The present invention uses HLS tools for high-level design, which can easily modify and optimize the functions and performance of each module. At the same time, the hardware programmability of FPGA enables us to adjust the structure and parameters of the neural network according to actual needs. This flexibility enables the FPGA-based flame detection pulse neural network deployment to adapt well to the application scenarios and needs of different edge device conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0064] Figure 1 is a flow chart of the method of the present invention;

[0065] Figure 2 This is a schematic diagram of the LIF neuron simulation model;

[0066] Figure 3 It is a schematic diagram of the topological structure of the spiking neural network;

[0067] Figure 4Schematic diagram of the high-level synthesis mapping process for the flame detection pulse neural network deployed on an FPGA;

[0068] Figure 5 This is a schematic diagram of the experimental environment framework in the embodiment;

[0069] Figure 6 An example diagram of the Fire dataset in the embodiment;

[0070] Figure 7 A process diagram of training and testing a spiking neural network using the Fire dataset in an embodiment;

[0071] Figure 8 This is a comparison chart of experimental data of different SNN processors in the embodiments;

[0072] Fig. 9 This is a statistical result diagram of the detection rate of the fire picture and the non-fire picture of the solution of the present invention at the FPGA end in the embodiment;

[0073] Fig.10 This is a comparison chart of the detection rate, processing speed and energy consumption of the solution of the present invention on the PC side and the FPGA side in the embodiment;

[0074] Fig.11 This is a statistical graph of the detection results of 127 pictures with different pixels by the solution of the present invention in the embodiment. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0076] In the present invention, in order to explore the application of low-power pulse neural networks in security and other fields, the software-level neural network is mapped on a programmable FPGA device by combining the HLS tool, and a universal software mapping method is proposed. The encoding and pulse neural network structure are realized by quickly establishing an interface at the hardware level, and a small-scale, fast-processing structure and related interfaces are generated and attempted to be applied to edge-constrained devices. While ensuring low energy efficiency and high speed, the data loss rate and processing efficiency are reduced, and an attempt is made to propose new ideas for the application of pulse neural networks for flame detection.

[0077] To this end, the present invention provides a method for deploying a flame detection pulse neural network on an FPGA, comprising the following steps:

[0078] S1, using flame images and non-flame images as training samples, pre-training the pulse neural network for flame detection to obtain a trained flame detection pulse neural network; the flame detection pulse neural network includes an image encoding unit, a pulse neural network unit and a classification neuron unit;

[0079] S2, performing optimization processing on the trained flame detection pulse neural network for network parameter calculation to obtain an optimized flame detection pulse neural network;

[0080] S3. Use Vitis HLS tool to write the code of image encoding unit, pulse neural network unit and classification neuron unit in flame detection pulse neural network, and perform functional simulation verification. Then use Vivado software to perform code synthesis and convert the code generated by HLS tool into IP core file that can be run and executed by FPGA.

[0081] S4, performing hardware architecture deployment design corresponding to the flame detection pulse neural network structure on FPGA;

[0082] S5. Burn the obtained IP core file to the FPGA to complete the deployment of the flame detection pulse neural network on the FPGA.

[0083] It can be seen that the present invention provides a method for accelerating the deployment and design of flame detection pulse neural networks on FPGA based on high-level synthesis (High-Level-Synthesis). The code of the flame detection pulse neural network is written using the Vitis HLS tool and an IP core file is generated, which is burned onto the FPGA to generate related digital circuits, thereby accelerating the processing speed of the flame detection pulse neural network and reducing the operating power consumption. Moreover, the scheme of the present invention has been tested and found to have a flame detection recognition accuracy of 98.6%, and the energy consumption is only 0.546% of the PS platform software implementation scheme. Therefore, the method of the present invention provides a solution for flame detection application scenarios to be effectively deployed on low-power edge devices based on FPGA.

[0084] The flame detection pulse neural network of the present invention and its deployment and implementation on FPGA are further elaborated and described in more detail below.

[0085] 1 Spiking Neuron Model

[0086] In a pulse neural network, neurons serve as the basic computing units that simulate the structure and function of the nervous system of an organism. Neurons receive postsynaptic pulses from other neurons, so that the membrane potential is superimposed with corresponding weights. When the membrane potential voltage exceeds the threshold, the neuron will release a pulse and transmit it to other neurons. At the same time, it enters a refractory period and does not respond to external input pulses for a period of time. In order to make the neuron model close to biological neurons while maintaining significant advantages in computational efficiency, the academic community has proposed models such as the HH model, the Izhikevichi model, and the LIF neuron model. Although complex neuron models have better bionic structures, their simulation and hardware design are more complex. The LIF neuron model has unique computational efficiency while retaining a certain degree of biological reality. Therefore, in the scheme of the present invention, the pulse neuron in the flame detection pulse neural network preferably adopts the LIF neuron model. The dynamic changes of the membrane potential V(t) of this model can be described by the differential equation of formula (1):

[0087]

[0088] In the formula, τ m is the membrane time constant, which is the product of the cell membrane capacitance and resistance, and determines the leakage decay rate of the membrane potential; R m I(t) is the voltage form of the external input stimulus; V rest is the resting potential, i.e. the membrane potential when there is no input pulse; V(t) is the membrane potential at time t. The synapse connected to it can be described by a simple synaptic transfer function:

[0089]

[0090] Where W is the synaptic weight, which determines the degree of influence of each input pulse on the target neuron. Positive values ​​represent excitatory synapses, and negative values ​​represent inhibitory synapses; τ s is the decay time constant of the synaptic current, which determines the duration of the synaptic input current; t represents the current moment, and this time variable is used to describe the state of the synaptic current at a certain moment; t spike Indicates the time when the presynaptic neuron generates a spike. When the presynaptic neuron generates a spike, the spike is transmitted to the postsynaptic neuron through the synapse and causes a change in the membrane potential of the postsynaptic neuron. spike It is to record the moment when this pulse is generated, (tt spike ) is used to represent the time from when the pulse is generated t spike The time elapsed from time t to the current time t.

[0091] Figure 2 The simulation results of the LIF neuron model are shown, in which the neuron time constant is 3.0 and the threshold is 1.0. After continuous input of constant current, the threshold is reached to emit pulses and restore the resting potential to 0, which can show the neural dynamic characteristics and hardware affinity. The study introduced the neural network topology of this classic model and hardware deployment, in which the information flow from the input layer to the output of the output layer is unidirectional. In the hardware mapping stage, the high-level synthesis tool of C++ code is used to improve the mapping effectiveness and performance of the structure.

[0092] There are related technical studies in the prior art for the transplantation and deployment of pulse neural networks on FPGAs. Li and M. Heidarpur et al. chose to use RTL-level languages ​​(such as VHDL or Verilog) to design networks. Although this helps fine control of details, traditional language design reduces the efficiency of the overall design. The high-level synthesis design method (HLS) takes the high-level description of the function, the RTL library and the self-set specific constraints as input, quickly adjusts the clock cycle, binds the description to the relevant functional units, stores the data in the storage unit to establish a suitable bus structure, and finally realizes the relevant logic synthesis through digital circuits. Javier Duarte et al. established the HLS4ML framework, which realizes the conversion of ML models into HLS-compatible high-level parameter models through a complete high-level framework. These models can be configured, synthesized and implemented to run on FPGAs. At present, there are also Vitis HLS tools that can be used for file code editing and simulation of pulse neural networks for FPGA deployment. The compiled HLS code can be converted into IP core files that can be run and executed by FPGAs through code synthesis through Vivado software.

[0093] 2 Flame recognition pulse neural network and hardware system deployment

[0094] The flame detection pulse neural network of the present invention is obtained based on pulse neural network training, and the hardware deployment is based on FPGA implementation, which will be explained in detail below.

[0095] 2.1 Spiking Neural Network Architecture

[0096] Spiking neural network (SNN) is a neural network model that transmits information by simulating the way neurons emit pulses (or spikes). The basic unit of SNN is the spiking neuron, which transmits information by emitting pulses. The output of each neuron is not a continuous signal, but a discrete pulse sequence. The update of the membrane potential of the neuron follows a differential equation involving time step, constant, membrane potential, output peak, resting potential, reset potential and the weight of the input synapse. When the membrane potential exceeds the threshold, the neuron emits a pulse, the output potential is 1, and the membrane potential returns to the reset potential; when the membrane potential is below the threshold, the neuron does not emit a pulse, and the output potential remains at 0. Spiking neurons are connected through synapses, and the weight of the synapse determines the intensity and efficiency of information transmission between neurons.

[0097] Figure 3 It is the topological structure of the spiking neural network. Each neuron is fully connected to the neurons in the next layer to capture feature information as much as possible, and the pulse emission of the neurons is output through the output layer as the result of the binary classification.

[0098] Specifically, the flame detection pulse neural network is obtained through flame detection training based on the pulse neural network, and its architecture includes an image coding unit, a pulse neuron network unit and a classification neuron unit. Among them: the image coding unit is used to perform Poisson coding on the input image data and map the input image data into corresponding pulse sequence data; the pulse neuron network unit includes multiple layers of neurons, wherein each neuron is fully connected with each neuron in the adjacent layer with weights, and the synaptic weights of the connection links between each neuron are neural network parameters determined by training; each neuron is used to update the membrane potential according to the weighted result of the input pulse, and when the membrane potential reaches the preset membrane potential threshold, the membrane potential is reset after the pulse is transmitted to the next layer of neurons connected to it; the classification neuron unit includes two classification neurons that are fully connected with each neuron in the last layer of the pulse neuron network unit, and the two classification neurons represent the detection results of flame in the image and no flame in the image respectively, and each classification neuron is used to count the number of pulses transmitted to each of them from the pulse neuron network unit, and the detection results represented by the classification neurons with the largest number of pulses are used as the flame detection results output by the flame detection pulse neural network.

[0099] The overall flame detection pulse neural network structure includes an image encoding unit with Poisson coding, a pulse neuron network unit consisting of multiple layers of neurons connected by a fully connected layer, and a classification neuron unit for binary classification. Finally, the flame detection recognition result is judged by the output of the two classification neurons of the classification neuron unit.

[0100] The overall design process of deploying the flame detection pulse neural network on FPGA can be Figure 4 To illustrate, including model training, hardware mapping and FPGA implementation. By using PyTorch to train the pulse neural network (SNN) model, distill the model, and optimize the parameters, it is suitable for hardware implementation; then, the Vitis HLS tool is used to map the optimized SNN model described in C++ and deploy it on the FPGA. During the mapping process, it is important to optimize the RAM memory usage rate. Memory usage is not only related to the size of the model structure, but also affected by the accuracy of synaptic weights. Therefore, the method of the present invention optimizes the network parameter calculation of the flame detection pulse neural network, and also optimizes and improves the hardware architecture deployment design, which will be introduced in detail later.

[0101] 2.1.1 Poisson Coding

[0102] Before inputting the image into the spiking neural network (SNN), the image needs to be pulsed. Poisson coding is a commonly used pulse method, which can convert the pixel value of the image into a pulse sequence to meet the input requirements of the SNN.

[0103] Poisson coding is a method of encoding input data into a pulse sequence whose distribution of the number of pulses conforms to the Poisson process. For a two-dimensional image, each pixel can generate an independent Poisson code. During the encoding process, the total time T needs to be set in advance, and then for each time step, the probability of each pixel generating a pulse is proportional to the pixel value. In this way, the number of pulses generated by each pixel in a period of time conforms to the Poisson distribution.

[0104] Specifically, the image encoding unit performs Poisson encoding on the input image data as follows:

[0105] According to the set Poisson coding time length T, for each pixel of the image data, a pulse is generated at each time step within the time length T according to its pixel value with the following probability P:

[0106]

[0107] Among them, k is the number of pulses excited within the time length T; λ is the average value of the Poisson distribution which is proportional to the pixel value; e is a natural constant; therefore, within the time length T, each pixel will generate a set of pulse sequences consisting of 0 and 1; thus, the pulse sequences of each pixel in the image data constitute the pulse sequence data corresponding to the image data.

[0108] 2.1.2 Optimization of network parameter calculation

[0109] In the pulse neuron network unit of the flame detection pulse neural network, the present invention adopts the LIF neuron model to construct the pulse neuron, that is, the attenuation of the membrane potential value of a single neuron is determined according to the LIF neuron model. However, in the original LIF neuron model, the transfer function for transmitting and sending pulse values ​​between neurons is usually a Sigmoid activation function that is more complicated to calculate. In order to simplify the calculation of network parameters, the present invention uses the ATAN inverse tangent activation function as the transfer function for transmitting and sending pulse values ​​between neurons. The ATAN inverse tangent activation function is sparse and will return an output close to 0 when the input is close to 0, encouraging fewer pulse neurons to "activate" pulse outputs under given input conditions, which is more conducive to achieving sparse computing, thereby reducing unnecessary pulse transfer calculations and power consumption.

[0110] Using the ATAN inverse tangent activation function as the transfer function, the updated value of the membrane potential of a single neuron is calculated as:

[0111] V i =atan_function(spike i,in )

[0112] Among them, V i represents the updated membrane potential value of any i-th neuron, spike i,in represents the input pulse value of the i-th neuron; atan_function(·) represents the ATAN inverse tangent activation function operation, and the ATAN inverse tangent activation function operation method is:

[0113]

[0114] Wherein, α is the function constant of the ATAN inverse tangent activation function; in practical applications, α=2 is usually taken.

[0115] The input pulse value of the i-th neuron is spike i,in for:

[0116]

[0117] Among them, spike jiIt represents the weighted result of the input pulse transmitted from the jth neuron connected to the i-th neuron through the synaptic weight of the connection link to the i-th neuron, j = 1, 2, ..., n i , n i Represents the total number of neurons that are fully connected to the i-th neuron in the forward direction.

[0118] The pulse signal value sent by a single neuron to the next layer of neurons connected to it is determined by whether its updated membrane potential reaches the preset membrane potential threshold. If it reaches it, it outputs a pulse signal value that transmits a high potential. If it does not reach it, it outputs a pulse signal value that transmits a low potential. If the preset membrane potential threshold value is taken as 1, it can be expressed through program code as:

[0119] return(mem_potential>=float8(1.0))? float8(1.0):float8(0.0);

[0120] Conditional judgment: mem_potential>=float8(1.0), that is, check whether the updated membrane potential mem_potential of the neuron reaches 1.0; if it reaches, return float8(1.0) as the sent pulse signal value, that is, the pulse signal value sent and transmitted by the neuron is 1; if it does not reach, return float8(0.0) as the sent pulse signal value, that is, the pulse signal value sent and transmitted by the neuron is 0.

[0121] If expressed in mathematical formula, the pulse signal value sent by a single neuron to the next layer of neurons connected to it can be expressed as:

[0122]

[0123] Among them, spike i,out V represents the pulse signal value sent by any i-th neuron; threshold Indicates the preset membrane potential threshold. In practical applications, V threshold =1.

[0124] In the parameter calculation optimization of the flame detection pulse neural network, in order to further simplify the parameter calculation process of the flame detection pulse neural network, the present invention adopts a lookup table to realize the calculation of the ATAN inverse tangent activation function. The specific method is: pre-calculate the ATAN inverse tangent activation function values ​​corresponding to different values ​​and record them in the ATAN inverse tangent activation function value lookup table; when calculating the updated value of the neuron membrane potential, query the ATAN inverse tangent activation function value lookup table to obtain the corresponding ATAN inverse tangent activation function value. In this way, the operation processing of the ATAN inverse tangent activation function is simplified to a table lookup operation, which effectively reduces the usage of DSP and is more conducive to simplifying the hardware calculation operation process.

[0125] On the other hand, when calculating the updated value of the neuron's membrane potential, the present invention also adopts a short-circuit logic strategy to further simplify the calculation of the weighted result of the input pulse transmitted by any neuron of the forward connection through the synaptic weight of the connection link. The original calculation strategy of the short-circuit logic is that in a conditional statement (such as an "if" statement or a ternary operator "?:"), the second condition will be calculated only when the first condition is true. It can be expressed as:

[0126] return(spike[ji]=(spike_output[j]>float8(0.0))? weights[j][i]:float8(0.0);

[0127] Conditional judgment: spike_output[j]>float8(0.0), that is, check whether the pulse signal value spike_output[j] sent by the j-th neuron of the forward connection is greater than 0.0; the short-circuit logic represented is: if the value of spike_output[j] is greater than 0.0, the program will return the synaptic weight weights1[i][j] of the connection link between the i-th neuron and the j-th neuron of the forward connection as the weighted result value of the input pulse (equivalent to 1 multiplied by the synaptic weight value); if it is not greater than 0.0, it will directly return float8(0.0), that is, directly return the value of 0 as the weighted result value of the input pulse.

[0128] Specifically, the weighted result of the input pulse transmitted by any neuron in the forward connection through the synaptic weight of the connection link is calculated using the following short-circuit logic strategy: for any i-th neuron, if the pulse signal value sent by the j-th neuron in its forward connection is greater than 0, the weighted result of the corresponding input pulse takes the value spike jiis the synaptic weight value of the corresponding connection link; if the pulse signal value sent by the jth neuron of its forward connection is less than or equal to 0, the weighted result value of the corresponding input pulse is spike ji The value is 0; that is:

[0129]

[0130] Among them, spike j,out W represents the pulse signal value sent by the jth neuron connected to the i-th neuron; ji Indicates the synaptic weight of the link between the ith neuron and the jth neuron of the forward connection. In this way, the relatively complex weighted multiplication calculation is converted into a relatively simple assignment operation, which is equivalent to avoiding the weighted multiplication of the synaptic weight, further reducing the usage of DSP, and being more conducive to simplifying the hardware calculation operation process.

[0131] 2.1.3 Generation of IP core files

[0132] After obtaining the optimized flame detection pulse neural network, it is necessary to use the Vitis HLS tool through step S3 to map the optimized flame detection pulse neural network described in C++ to the FPGA; this step includes high-level description, RTL library synthesis, IP core generation, and digital circuit generation, and ultimately realizes the deployment of the flame detection pulse neural network.

[0133] Specifically, the processing process is:

[0134] S301. According to the unit division structure of the image coding unit, the pulse neural network unit and the classification neural network unit in the flame detection pulse neural network, use the C++ high-level programming language and the HLS component to write a program, design the functional modules and test modules corresponding to each unit, and create an HLS project;

[0135] S302, for each unit, use the test module to perform functional simulation on the functional module, and use the debugger integrated in the HLS tool to perform functional debugging during the simulation process until the functional simulation is correct;

[0136] S303, calculating the ATAN inverse tangent activation function values ​​corresponding to different values, recording them in an ATAN inverse tangent activation function value lookup table, converting the ATAN inverse tangent activation function value lookup table into a format suitable for use with FPGA, and writing it into the code of the spiking neural network unit through the Vitis HLS tool;

[0137] S304, using Vivado software to perform code synthesis, and using the HLS tool to save the synaptic weight values ​​of the connections between the neurons in the pulse neuron network unit of the trained flame detection pulse neural network as the header file of the code, and then converting the code generated by the HLS tool into an IP core file that can be run and executed by FPGA.

[0138] The obtained IP core file can be burned into the FPGA to implement the deployment of the flame detection pulse neural network on the FPGA.

[0139] 2.1.4 Hardware Architecture Deployment

[0140] The present invention carries out corresponding FPGA hardware architecture deployment design according to the flame detection pulse neural network structure, and improves and optimizes it during the deployment process to make it easy for hardware implementation.

[0141] Specifically, the hardware architecture deployed on the FPGA includes an image encoding module, a pulse neuron processing module, a classification statistics module, a data cache module, and a control center module.

[0142] The image encoding module is used to perform Poisson encoding on the input image data, and map the input image data into corresponding pulse sequence data.

[0143] The pulse neuron processing module includes a pulse buffer unit and a plurality of interconnected physical neurons; wherein the pulse buffer unit is used to buffer the pulse sequence data obtained by mapping the image data transmitted by the image encoding module and send it to the physical neurons; the physical neurons are used to update the membrane potential according to the weighted result of the input pulses, and when the membrane potential reaches a preset threshold, the membrane potential is reset after the pulse is transmitted to the next layer of neurons connected thereto;

[0144] The classification statistics module includes two pulse statistics units and a statistical data storage device; the two pulse statistics units are used to respectively count the number of high-order pulse values ​​and the number of low-order pulse values ​​transmitted by the pulse neuron processing module, respectively representing the number of detection results of flames in the recognition image and no flames in the image; the statistical data storage device is used to store and output the statistical results of the two pulse statistics units.

[0145] The data cache module is connected to the pulse neuron processing module and the classification statistics module respectively, and is used to cache the pulse signal value output and transmitted by the pulse neuron processing module, and send it to the classification statistics module for classification statistics.

[0146] The control center module is connected to the pulse neuron processing module and the classification statistics module respectively, and is used to send control signals to control the image encoding module to store the pulse sequence data obtained by Poisson encoding into the specified address in the pulse neuron processing module of the pulse cache unit, and control each physical neuron in the pulse neuron processing module to start or stop updating the membrane potential, as well as the order and synaptic weight value of pulse transmission between physical neurons, and control the storage address of the pulse signal value output by the pulse neuron processing module in the data cache module and the address order of the classification statistics module to read the pulse signal value from the data cache module for statistics.

[0147] It can be seen that there is a clear correspondence between the FPGA hardware architecture deployment design and the architecture structure of the flame detection pulse neural network: the image coding module, the pulse neuron processing module, and the classification statistics module correspond to the image coding unit, the pulse neuron network unit, and the classification neuron unit in the flame detection pulse neural network respectively; the data cache module is used to cache the pulse signal values ​​between the pulse neuron processing module and the classification statistics module to prevent data loss when the classifier processing speed is slow; the control center module is used to perform the overall scheduling control of the flame detection pulse neural network calculation operation.

[0148] Specifically for a single physical neuron in the pulse neuron processing module, its deployment design includes a neuron controller, and a membrane potential update unit, a membrane potential memory, a forward weight management unit, a forward weight memory, a threshold voltage management unit, a threshold voltage memory and an internal pulse interaction unit respectively connected to the neuron controller.

[0149] Among them, the forward weight management unit is connected to the forward weight memory and is used to call the synaptic weight value stored in the forward weight memory.

[0150] The threshold voltage management unit is connected to the threshold voltage memory and is used to retrieve the preset membrane potential threshold stored in the threshold voltage memory.

[0151] The membrane potential update unit is respectively connected to the membrane potential memory, the internal pulse interaction unit, the forward weight management unit and the threshold voltage management unit, and is used to update the membrane potential according to the pulse signal value received by the physical neuron and the synaptic weight value read by the forward weight management unit, store the updated membrane potential in the membrane potential memory, and compare the membrane potential with the preset membrane potential threshold read by the threshold voltage management unit. When the membrane potential reaches the preset membrane potential threshold, a pulse signal is sent and the membrane potential in the membrane potential memory is reset.

[0152] The internal pulse interaction unit is connected to the membrane potential update unit, and is used to transmit the pulse signal value received by the physical neuron to the membrane potential update unit, and transmit the pulse signal transmitted by the membrane potential update unit to other physical neurons.

[0153] The neuron controller is used to receive control signals from the control center module, control the operation of each unit in the physical neuron, issue addresses for reading and writing signals in the physical neuron, and control the transmission path for pulse signal interaction between the internal pulse interaction unit in the physical neuron and other physical neurons.

[0154] Thus, a single physical neuron circuit can realize the operation and processing of a single pulse neuron in the flame detection pulse neural network.

[0155] Among them, combined with the optimization processing of network parameter calculation for flame detection pulse neural network in the method of the present invention, in the calculation of membrane potential update value, a lookup table is used to implement ATAN inverse tangent activation function calculation and short-circuit logic calculation strategy. Therefore, in the hardware deployment design, the membrane potential update unit is also targeted for the corresponding optimization design in the update calculation method of the membrane potential. The specific method is as follows:

[0156] S401, first use the short-circuit logic strategy to determine the weighted result of the input pulse transmitted from each forward-connected physical neuron to the physical neuron; if the pulse signal value transmitted by the forward-connected physical neuron is greater than 0, read the synaptic weight value from the forward weight memory through the forward weight management unit as the weighted result value of the input pulse; if the pulse signal value transmitted by the forward-connected physical neuron is less than or equal to 0, directly take the value of 0 as the weighted result value of the input pulse;

[0157] S402, accumulating the weighted result values ​​of the input pulses transmitted from each forward-connected physical neuron to the physical neuron to obtain the input pulse value of the physical neuron;

[0158] S403. Obtain the ATAN inverse tangent activation function value corresponding to the input pulse value of the physical neuron by querying the ATAN inverse tangent activation function value lookup table, and store it in the membrane potential memory as the membrane potential update value of the physical neuron.

[0159] Thus, the relatively complex inverse tangent function calculation and weighted multiplication calculation are converted into relatively simple memory data extraction and assignment operations, which is equivalent to the weighted multiplication of synaptic weights and the inverse tangent activation function operation being avoided, and the subsequent addition operation of pulse transmission to other physical neurons will not add any non-zero value to the transmitted pulse. In this way, the use of DSP is greatly reduced, which is more conducive to simplifying the hardware calculation operation process.

[0160] 3. Embodiment

[0161] In this embodiment, a data set is used to test and verify the solution of the present invention to better reflect the technical advantages of the present invention.

[0162] 3.1 Experimental Environment

[0163] This experiment uses the XCZU3EG-FFVD1760-1-I core board of the UltraScale+MPSoCs architecture in the Xilinx ZYNQ series as the verification platform. The PS side of the platform chip integrates 4 ARMCortex TM -A53 processor. The overall design frequency is 100Mhz, and the design and test software is Vitis HLS2020.1 and Vivado2020.1. As a reference and for training related parameters, the computer used in the experiment is Intel Xeon (R) Silver 4210R CPU @ 2.4GHZ × 20 and NVIDIA GeForce RTX3070 graphics card, and the environment is the Spiking Jelly framework deployed in Pytorch under Anaconda Python 3.8.

[0164] The synaptic weights with different precision configurations are implemented on this platform, and the PS and PL ends are connected through an additional AXI IP interface, such as Figure 5 As shown in the figure, multiple available high-performance ports are used for data flow input and output, and the pulse neural network acceleration block deployed on the PL side is connected to DDR4 through AXI to establish a device that can be monitored in real time. By optimizing the pipeline and network structure size, its time characteristics and resource utilization are adjusted, and the performance of the pulse neural network processing unit on the FPGA device and the high-performance graphics processing unit for the same data set is compared in terms of the final accuracy and power consumption.

[0165] In order to further explore the effect of the pulse neural network application, a Fire dataset was established through the flame videos recorded by BC Ko et al. Figure 6 As shown, the dataset contains two sub-files. The Fire class contains 5,000 pictures of flames at different angles and distances, with the surrounding environments being indoors and outdoors, and day and night. The No_Fire class contains day and night pictures of similar fire-free environments.

[0166] 3.2 SNN Training Evaluation

[0167] The SNN is directly trained and evaluated using the method proposed by FANG W. During the training process, the surrogate function is the ATAN inverse tangent activation function, which provides sufficient nonlinear characteristics and biological plausibility. The learning rate is set to 0.0001. The overall detailed structure is shown in Figure 7 As shown in the leftmost part, it includes a Poisson encoder and two fully connected layers connected to it after flattening, and the result is judged by the output of two classification neurons.

[0168] In order to establish the spiking neural network and its mapping for flame detection tasks, the Fire dataset is used in the training process, of which 7,000 images are used as the training set for training and the remaining 3,000 images are used for testing. In order to improve the versatility of the model, the images are randomly flipped and rotated during the data loading stage. Figure 7 The data in (b) and (c) show the accuracy and loss function changes of the neural network when trained on the Fire dataset. The recognition accuracy of the spiking neural network converges stably after about 170 epochs. Figure 7 (d) shows the pulse firing frequency of the last layer of classification neurons for flame detection, and the detection accuracy of the overall data set is 98.8%. When testing inference, the network maintained a processing speed of 2700 images per second on the 3070, and the Nvidia-smi tool recorded the average power of the graphics card to be 71.4W within 10 minutes.

[0169] 3.3SNN Hardware Mapping Evaluation

[0170] In the method proposed in the present invention, compared with the traditional RTL-level design process, by using a high-level language synthesis method, the trained network parameters are imported into the header file as the benchmark parameters of the macro definition. The number of layers and neurons in each layer is changed according to different goals in actual applications to adapt to tasks of different complexity.

[0171] Table 1 compares some SNN processors in recent years. Experimental results show that the proposed SNN processor maintains high accuracy and processing speed while using less resources than other tasks, and is competitive among FPGA-based accelerators.

[0172] In the method proposed in the present invention, compared with the traditional RTL-level design process, by using a high-level language synthesis method, the trained network parameters are imported into the header file as the benchmark parameters of the macro definition. The number of layers and neurons in each layer is changed according to different goals in actual applications to adapt to tasks of different complexity.

[0173] Table 1 compares some SNN processors in recent years. Experimental results show that the proposed SNN processor maintains high accuracy and processing speed while using less resources than other tasks, and is competitive among FPGA-based accelerators.

[0174] Table 1 Comparison of different SNN processors

[0175]

[0176] The power measurement of the pulse processing unit is performed in the active (network operation) and idle (network idling) states, such as Figure 8 As shown in the figure, the data of the pulse neural network processing unit are measured by Vitis HLS and Vivado. It can be seen that the dynamic total power of the processing unit is 0.39W, which is 0.546% of the 3070 when running the same work. In order to maintain real-time performance and enable the processing unit to be used in a resource-constrained environment, the input image or video stream is pre-processed by using the HLS tool to establish a high-level interface. Even a lightweight network can fully obtain the feature data in the image. Fig. 9 To measure the accuracy of the processing unit, we introduced the test file in HLS and imported 2,000 test set images as test images. In a thousand cycles of testing, 99.7% of the task neurons fired pulses normally. Through the detection of 1,000 fire images, the pulse processing unit maintained a correct pulse firing frequency of 98.6%. Fig.10 Compared with the PC-based computer used for training and inference testing, the test accuracy of the FPGA side is slightly reduced, while the processing speed and energy consumption advantages are obvious.

[0177] 3.4 Flame Detection Example Based on Pulse Neural Network

[0178] Table 2 Comparison of flame detection test results of different models and platforms

[0179]

[0180] The comparative test results of flame detection systems established using different models and platforms are shown in Table 2, where each solution attempts to optimize processing speed and energy consumption while maintaining as high an accuracy rate as possible. However, it can be seen from Table 2 that compared with the processing speed of nearly 100 FPS of similar models on the computer side, the pulse neural network using the solution of the present invention in the application of edge devices has more advantages in processing timeliness and energy consumption.

[0181] In order to explore the relevant applications of spiking neural networks in resource-constrained scenarios in practical applications, 127 different pixel images from two public datasets were introduced as test files, and the following results were obtained: Fig.11According to the results, 100 images were correctly identified as flames. A set of fast flame intelligent recognition tools that are independent of computers was established through the generated pulse processing unit and Xilinx chip design tools. Through the AXI interface and preprocessing module generated by HLS, the interfaces of different devices can be quickly connected for processing. The image processing speed of the system can reach 9,800 images per second, and the video data can also be processed in real time through the camera image transmission protocol interface. Compared with traditional flame detection equipment based on artificial neural networks, the system also maintains high accuracy and processing speed without computers, and can be used in scenarios with limited energy resources.

[0182] 4 Overview

[0183] The present invention proposes a deployment design scheme for a flame detection pulse neural network on FPGA. It is developed by using a high-level synthesis language and using Vivado advanced synthesis technology to design a more suitable neural network architecture for the programmable logic part, exploring the application of brain-like intelligent computing on edge devices, and verifying it on an FPGA platform. Experimental results show that the pulse neural network processing unit generated by the processor on the FPGA exhibits better energy efficiency and processing efficiency than traditional GPUs. The performance of the solution proposed in the present invention on the Fire dataset shows that compared with traditional second-generation artificial neural networks, it can still maintain high processing speed and low power consumption under resource-constrained conditions, and is suitable for application scenarios such as flame detection drones or flame detection centers of security systems. Therefore, the method of quickly mapping pulse neural networks on software platforms to hardware edge devices has obvious advantages in the application of low-power edge devices.

[0184] In summary, the method for deploying the flame detection pulse neural network on FPGA of the present invention uses the VitisHLS tool to write the code of the flame detection pulse neural network and generate an IP core file, burn it to the FPGA, generate related digital circuits, accelerate the processing speed of the flame detection pulse neural network and reduce the running power consumption. In the design process, the technical means such as the lookup table to realize the ATAN inverse tangent activation function calculation and the short-circuit logic calculation strategy can reduce unnecessary calculations and power consumption, significantly reduce the usage of DSP, and are more conducive to simplifying the hardware calculation operation process, so that the pulse neural network can run on the FPGA with lower power consumption. At the same time, the parallel processing capability of the FPGA enables the neural network to process multiple input data at the same time, improving the throughput of the overall system. This high efficiency makes the FPGA-based pulse neural network have significant advantages in real-time processing and high-performance computing. Using the HLS tool for high-level design, the functions and performance of each module can be easily modified and optimized. At the same time, the hardware programmability of the FPGA enables us to adjust the structure and parameters of the neural network according to actual needs. This flexibility enables the deployment of the flame detection pulse neural network based on the FPGA to be well adapted to the application scenarios and needs of different edge device conditions. After experimental testing, the scheme of the present invention can achieve 98.6% flame detection and recognition accuracy, and the energy consumption is only 0.546% of the PS platform software implementation scheme. Therefore, the method of the present invention provides a solution for flame detection application scenarios based on FPGA for effective deployment on low-power edge devices.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A method for deploying a flame detection pulse neural network on an FPGA, characterized in that: The steps include: S1, using flame images and non-flame images as training samples, pre-training the pulse neural network for flame detection to obtain a trained flame detection pulse neural network; the flame detection pulse neural network includes an image encoding unit, a pulse neural network unit and a classification neuron unit; S2, performing optimization processing on the trained flame detection pulse neural network for network parameter calculation to obtain an optimized flame detection pulse neural network; S3. Use Vitis HLS tool to write the code of image encoding unit, pulse neural network unit and classification neuron unit in flame detection pulse neural network, and perform functional simulation verification. Then use Vivado software to perform code synthesis and convert the code generated by HLS tool into IP core file that can be run and executed by FPGA. S4, performing hardware architecture deployment design corresponding to the flame detection pulse neural network structure on FPGA; S5. Burn the obtained IP core file to the FPGA to complete the deployment of the flame detection pulse neural network on the FPGA.

2. The method for deploying the flame detection pulse neural network on FPGA according to claim 1 is characterized in that: In the flame detection pulse neural network: The image encoding unit is used to perform Poisson encoding on the input image data, and map the input image data into corresponding pulse sequence data; The spiking neuron network unit contains multiple layers of neurons, each of which is fully connected with each neuron in the adjacent layer with weights, and the synaptic weights of the links connecting the neurons are neural network parameters determined through training; each neuron is used to update the membrane potential according to the weighted result of the input pulse, and when the membrane potential reaches the preset membrane potential threshold, the membrane potential is reset after the pulse is transmitted to the next layer of neurons connected to it; The classification neuron unit includes two classification neurons that are fully connected to each neuron in the last layer of the pulse neuron network unit, respectively. The two classification neurons represent the detection results of the presence of flame in the image and the absence of flame in the image, respectively. Each classification neuron is used to count the number of pulses transmitted from the pulse neuron network unit to each of them, and the detection results represented by the classification neurons with the largest number of pulses are used as the flame detection results output by the flame detection pulse neural network.

3. The method for deploying the flame detection pulse neural network on FPGA according to claim 2 is characterized in that: In the spiking neuron network unit, the membrane potential value attenuation of a single neuron is determined according to the LIF neuron model; the calculation method of the membrane potential update value of a single neuron is: V i =atan_function(spike i,in ) Among them, V i represents the updated membrane potential value of any i-th neuron, spike i,in represents the input pulse value of the i-th neuron; atan_function(·) represents the ATAN inverse tangent activation function operation, and the ATAN inverse tangent activation function operation method is: Among them, α is the function constant of the ATAN inverse tangent activation function; The input pulse value of the i-th neuron is spike i,in for: Among them, spike ji It represents the weighted result of the input pulse transmitted from the jth neuron connected to the i-th neuron through the synaptic weight of the connection link to the i-th neuron, j = 1, 2, ..., n i , n i Represents the total number of neurons fully connected to the i-th neuron.

4. The method for deploying a flame detection pulse neural network on an FPGA according to claim 3, characterized in that: In the pulse neuron network unit, the pulse signal value transmitted by a single neuron to the next layer of neurons connected to it is: Among them, spike i,out V represents the pulse signal value sent by any i-th neuron; threshold Indicates the preset membrane potential threshold.

5. The method for deploying a flame detection pulse neural network on an FPGA according to claim 3, characterized in that: In step S2, the optimization process of calculating the network parameters of the trained flame detection pulse neural network includes: The ATAN inverse tangent activation function values ​​corresponding to different numerical values ​​are calculated in advance and recorded in the ATAN inverse tangent activation function value lookup table; when calculating the updated value of the neuron's membrane potential, the corresponding ATAN inverse tangent activation function value is obtained by querying the ATAN inverse tangent activation function value lookup table.

6. The method for deploying the flame detection pulse neural network on FPGA according to claim 3 is characterized in that: In step S2, the optimization process of calculating the network parameters of the trained flame detection pulse neural network also includes: When calculating the updated value of the neuron's membrane potential, the weighted result of the input pulse transmitted by any neuron in the forward connection through the synaptic weight of the connection link is calculated using the following short-circuit logic strategy: for any i-th neuron, if the pulse signal value sent by the j-th neuron in its forward connection is greater than 0, the weighted result of the corresponding input pulse takes the value spike ji is the synaptic weight value of the corresponding connection link; if the pulse signal value sent by the jth neuron of its forward connection is less than or equal to 0, the weighted result value of the corresponding input pulse is spike ji The value is 0; that is: Among them, spike j,out W represents the pulse signal value sent by the jth neuron connected to the i-th neuron; ji Represents the synaptic weight of the link between the i-th neuron and the j-th neuron of the forward connection.

7. The method for deploying a flame detection pulse neural network on an FPGA according to claim 5, characterized in that: The step S3 specifically includes: S301. According to the unit division structure of the image coding unit, the pulse neural network unit and the classification neural network unit in the flame detection pulse neural network, use the C++ high-level programming language and the HLS component to write a program, design the functional modules and test modules corresponding to each unit, and create an HLS project; S302, for each unit, use the test module to perform functional simulation on the functional module, and use the debugger integrated in the HLS tool to perform functional debugging during the simulation process until the functional simulation is correct; S303, calculating the ATAN inverse tangent activation function values ​​corresponding to different values, recording them in an ATAN inverse tangent activation function value lookup table, converting the ATAN inverse tangent activation function value lookup table into a format suitable for use with FPGA, and writing it into the code of the pulse neural network unit through the VitisHLS tool; S304, using Vivado software to perform code synthesis, and using the HLS tool to save the synaptic weight values ​​of the connections between the neurons in the pulse neuron network unit of the trained flame detection pulse neural network as the header file of the code, and then converting the code generated by the HLS tool into an IP core file that can be run and executed by FPGA.

8. The method for deploying the flame detection pulse neural network on FPGA according to claim 2 is characterized in that: In step S4, the specific method of performing the hardware architecture deployment design corresponding to the flame detection pulse neural network structure on the FPGA is: The hardware architecture deployed on the FPGA includes image encoding module, pulse neuron processing module, classification statistics module, data cache module and control center module; The image encoding module is used to perform Poisson encoding on the input image data, and map the input image data into corresponding pulse sequence data; The pulse neuron processing module includes a pulse buffer unit and a plurality of interconnected physical neurons; wherein the pulse buffer unit is used to buffer the pulse sequence data obtained by mapping the image data transmitted by the image encoding module, and send it to the physical neurons; the physical neurons are used to update the membrane potential according to the weighted result of the input pulses, and when the membrane potential reaches a preset threshold, the membrane potential is reset after the pulse is transmitted to the next layer of neurons connected thereto; The classification statistics module includes two pulse statistics units and a statistical data storage; the two pulse statistics units are used to respectively count the number of high-order pulse values ​​and the number of low-order pulse values ​​transmitted by the pulse neuron processing module, respectively representing the number of detection results of flames in the recognition image and flames in the image; the statistical data storage is used to store and output the statistical results of the two pulse statistics units; The data cache module is connected to the pulse neuron processing module and the classification statistics module respectively, and is used to cache the pulse signal value output and transmitted by the pulse neuron processing module, and send it to the classification statistics module for classification statistics; The control center module is connected to the pulse neuron processing module and the classification statistics module respectively, and is used to send control signals to control the image encoding module to store the pulse sequence data obtained by Poisson encoding into the specified address in the pulse neuron processing module of the pulse cache unit, and control each physical neuron in the pulse neuron processing module to start or stop updating the membrane potential, and the order and synaptic weight value of pulse transmission between physical neurons, and control the storage address of the pulse signal value output and transmitted by the pulse neuron processing module in the data cache module and the address order of the classification statistics module to read the pulse signal value from the data cache module for statistics.

9. The method for deploying a flame detection pulse neural network on an FPGA according to claim 8, characterized in that: The physical neuron includes a neuron controller, and a membrane potential update unit, a membrane potential memory, a forward weight management unit, a forward weight memory, a threshold voltage management unit, a threshold voltage memory and an internal pulse interaction unit respectively connected to the neuron controller; Wherein, the forward weight management unit is connected to the forward weight memory and is used to retrieve the synaptic weight value stored in the forward weight memory; The threshold voltage management unit is connected to the threshold voltage memory and is used to retrieve the preset membrane potential threshold stored in the threshold voltage memory; The membrane potential updating unit is connected to the membrane potential memory, the internal pulse interaction unit, the forward weight management unit and the threshold voltage management unit respectively, and is used to update the membrane potential according to the pulse signal value received by the physical neuron and the synaptic weight value read by the forward weight management unit, store the updated membrane potential in the membrane potential memory, and compare the membrane potential with the preset membrane potential threshold read by the threshold voltage management unit, and send a pulse signal and reset the membrane potential in the membrane potential memory when the membrane potential reaches the preset membrane potential threshold; The internal pulse interaction unit is connected to the membrane potential update unit, and is used to transmit the pulse signal value received by the physical neuron to the membrane potential update unit, and transmit the pulse signal transmitted by the membrane potential update unit to other physical neurons; The neuron controller is used to receive control signals from the control center module, control the operation of each unit in the physical neuron, issue addresses for signal reading and writing in the physical neuron, and control the transmission path for pulse signal interaction between the internal pulse interaction unit in the physical neuron and other physical neurons.

10. The method for deploying the flame detection pulse neural network on FPGA according to claim 9, characterized in that: The specific method of the membrane potential updating unit when performing the updating calculation of the membrane potential is: S401, firstly adopting the short-circuit logic strategy to determine the weighted result of the input pulse transmitted from each forward-connected physical neuron to the physical neuron; if the pulse signal value transmitted by the forward-connected physical neuron is greater than 0, then reading the synaptic weight value from the forward weight memory through the forward weight management unit as the weighted result value of the input pulse; If the pulse signal value transmitted by the physical neuron of the forward connection is less than or equal to 0, the value of 0 is directly taken as the weighted result value of the input pulse; S402, accumulating the weighted result values ​​of the input pulses transmitted from each forward-connected physical neuron to the physical neuron to obtain the input pulse value of the physical neuron; S403. Obtain the ATAN inverse tangent activation function value corresponding to the input pulse value of the physical neuron by querying the ATAN inverse tangent activation function value lookup table, and store it in the membrane potential memory as the membrane potential update value of the physical neuron.

Images