Face mask detection hardware acceleration platform and method
By adopting an improved target detection network on the face mask detection hardware acceleration platform, combining pulse convolution module and convolution layer, the problem of high energy consumption in traditional CNN networks during edge devices is solved, and low power consumption and high efficiency are achieved facial mask detection.
Patent Information
- Application Number
- CN202411129486.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-15
AI Technical Summary
The existing face mask detection technology is difficult to effectively reduce due to the use of traditional CNN networks when deploying edge devices.
A hardware acceleration platform for face mask detection is designed, using an improved target detection network, which consists of fourteen series pulse convolution modules, convolution modules and convolution layers, encapsulated and deployed on FPGAs through IP cores. , combined with ARM processor, use multi-channel bus for data transmission, achieving efficient data processing and energy consumption reduction.
By converting the feature extraction network into SNN, the resource consumption and energy consumption of the model are reduced, the energy consumption in the face mask detection process is reduced, and the detection efficiency is improved, achieving high performance and low power consumption on edge devices Test.
Smart Images

Figure CN118964286A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent platform computing technology, and in particular to a human face mask detection hardware acceleration platform and method. Background Art
[0002] At present, in the field of face mask detection, convolutional neural networks are often used for detection and learning. Convolutional neural networks (CNN) are very suitable for processing two-dimensional data such as image information, so most detection tasks use convolutional neural networks. However, CNN networks often require a large amount of data calculations and often require a lot of energy consumption during reasoning. SNN networks (Segmental Neural Nets) have the advantages of low power consumption and sparsity, and are more suitable for deployment in some small edge devices.
[0003] In current research, most SNNs are applied to relatively simple tasks such as image recognition tasks. However, in target detection tasks, spiking neural networks that can complete such complex tasks are still a major research hotspot and challenge. Since the existing network Spiking-YOLO network (Spiking Neural Network for Object Detection) is relatively large, it still needs to be lightweight before it can be deployed on FPGA (Field Programmable Gate Array).
[0004] When existing facial mask detection technology is deployed on edge devices, it is often necessary to use traditional CNN networks to deploy the network. However, CNN networks often have the disadvantage of high power consumption, resulting in large energy consumption when performing facial mask detection. Summary of the invention
[0005] The present invention provides a hardware acceleration platform and method for face mask detection, which are used to solve the technical problem that the existing face mask detection technology causes large energy consumption when performing face mask detection.
[0006] A first aspect of the present invention provides a face mask detection hardware acceleration platform, comprising a processor and a programmable gate array; the programmable gate array is deployed with an external memory, and an input data cache unit group, a computing unit and an output data cache unit group connected in sequence;
[0007] The output data cache unit group is connected to the external memory through a multi-channel bus, and the external memory is connected to the input data cache unit group through the multi-channel bus;
[0008] The computing unit stores an improved target detection network encapsulated by an IP core, and the processor is communicatively connected with the external memory;
[0009] The processor is used to obtain a mask face image to be detected, and transmit the mask face image to be detected to the computing unit through the external memory and the input data cache unit group in sequence;
[0010] The computing unit is used to perform face mask detection on the mask face image to be detected through the improved target detection network, output the detection result, and transmit the detection result to the external memory through the output data cache unit group.
[0011] Optionally, the improved target detection network includes fourteen pulse convolution modules, convolution modules and convolution layers connected in series; the processing process of the mask face image to be detected by the improved target detection network includes:
[0012] Fourteen pulse convolution modules connected in series are used to extract features of the mask face image to be detected, and a pulse convolution feature map is output;
[0013] Performing a convolution operation on the pulse convolution feature map through a convolution module, and outputting a convolution feature map;
[0014] The convolution feature map is input into the convolution layer for convolution operation, and the detection result is output.
[0015] Optionally, the pulse convolution module includes a convolution layer, a normalization layer and an integral emission layer; the image processing process of the pulse convolution module includes:
[0016] Performing a convolution operation on the input feature map of the pulse convolution module through a convolution layer, and outputting an intermediate feature map;
[0017] Normalizing the intermediate feature map using a normalization layer to generate a normalized feature map;
[0018] The normalized feature map is subjected to data conversion through an integral emission layer to output a pulse output feature map.
[0019] Optionally, the convolution module includes a convolution layer, a normalization layer and an activation function layer; the step of performing a convolution operation on the pulse convolution feature map through the convolution module and outputting the convolution feature map includes:
[0020] Performing a convolution operation on the pulse convolution feature map through a convolution layer to generate an intermediate convolution feature map;
[0021] Normalizing the intermediate convolution feature map using a normalization layer, and outputting an intermediate normalized feature map;
[0022] The intermediate normalized feature map is input into the activation function layer for nonlinear mapping to generate a convolutional feature map.
[0023] Optionally, the processor is further used to receive and display the detection result sent by the external memory.
[0024] Optionally, the input data cache unit group is used to transmit the mask face image to be detected to the computing unit through a ping-pong operation;
[0025] The output data cache unit group is used to transmit the detection result to the external memory through a ping-pong operation.
[0026] Optionally, the model training process of the improved target detection network includes:
[0027] Acquire an image training data set, use the image training data set to train a preset initial convolutional neural network, and determine a trained target convolutional neural network;
[0028] The model weights of the improved initial detection network are updated using the model weights of the trained target convolutional neural network to determine an improved intermediate detection network;
[0029] Based on a preset direct training method, the improved intermediate detection network is trained using the image training data set to determine the improved target detection network.
[0030] A second aspect of the present invention provides a hardware acceleration method for face mask detection, comprising:
[0031] When receiving a mask face image to be detected, the mask face image to be detected is input into an improved target detection network encapsulated by an IP core; the improved target detection network includes fourteen pulse convolution modules, convolution modules and convolution layers connected in series;
[0032] Fourteen pulse convolution modules connected in series are used to extract features of the mask face image to be detected, and a pulse convolution feature map is output;
[0033] Performing a convolution operation on the pulse convolution feature map through a convolution module, and outputting a convolution feature map;
[0034] The convolution feature map is input into the convolution layer for convolution operation, and the detection result is output.
[0035] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the hardware acceleration method for face mask detection as described in any one of the above items.
[0036] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the hardware acceleration method for face mask detection as described in any one of the above items.
[0037] It can be seen from the above technical solutions that the present invention has the following advantages:
[0038] The first aspect of the above technical solution of the present invention provides a hardware acceleration platform for face mask detection, including a processor and a programmable gate array; the programmable gate array is deployed with an external memory, and an input data cache unit group, a computing unit and an output data cache unit group connected in sequence; wherein the output data cache unit group is connected to the external memory through a multi-channel bus, and the external memory is connected to the input data cache unit group through a multi-channel bus; the computing unit stores an improved target detection network encapsulated by an IP core, and the processor is communicatively connected to the external memory; the processor is used to obtain a mask face image to be detected, and transmit the mask face image to be detected to the computing unit in sequence through the external memory and the input data cache unit group; the computing unit is used to perform face mask detection on the mask face image to be detected through the improved target detection network, output the detection result and transmit the detection result to the external memory through the output data cache unit group; based on the above scheme, through the improved target detection network encapsulated by the IP core stored in the computing unit, the acquired mask face image to be detected is subjected to face mask detection, and the detection result is output, the entire improved target detection network is deployed on the edge device, which can reduce the resource consumption of the model, thereby reducing the energy consumption caused in the process of performing face mask detection.
[0039] The second aspect of the above technical solution of the present invention provides a hardware acceleration method for face mask detection. When a mask face image to be detected is received, the mask face image to be detected is input into an improved target detection network encapsulated by an IP core; the improved target detection network includes fourteen pulse convolution modules, convolution modules and convolution layers connected in series; fourteen pulse convolution modules are used to extract features of the mask face image to be detected, and a pulse convolution feature map is output; a convolution operation is performed on the pulse convolution feature map through the convolution module, and a convolution feature map is output; the convolution feature map is input into the convolution layer for convolution operation, and a detection result is output; based on the above scheme, the obtained mask face image to be detected is subjected to face mask detection through the improved target detection network encapsulated by the IP core, and the process of outputting the detection result can reduce the resource consumption of the model, thereby reducing the energy consumption caused in the process of face mask detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0041] Figure 1 A schematic diagram of the structure of a hardware acceleration platform for face mask detection provided in Embodiment 1 of the present invention;
[0042] Figure 2 A schematic diagram of the structure of an improved target detection network provided in Embodiment 1 of the present invention;
[0043] Figure 3 An overall framework diagram of the training of an improved target detection network provided in the first embodiment of the present invention;
[0044] Figure 4 A flowchart of the steps of a hardware acceleration method for face mask detection provided in Embodiment 2 of the present invention;
[0045] The meanings of the symbols in the accompanying drawings are as follows:
[0046] 1. Processor; 2. External memory; 3. Computing unit; 4. Input data cache unit group; 5. Output data cache unit group. DETAILED DESCRIPTION
[0047] The embodiments of the present invention provide a hardware acceleration platform and method for face mask detection, which are used to solve the technical problem that the existing face mask detection technology causes large energy consumption when performing face mask detection.
[0048] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] Terminology explanation:
[0050] SNN (Segmental Neural Nets): Spiking neural networks.
[0051] CNN (Convolutional Neural Network): Convolutional Neural Network.
[0052] YOLO (You Only Look Once): A one-stage object detection algorithm.
[0053] IF (Integrate-and-Fire Neuron Model): Integrate and fire neurons.
[0054] FPGA (Field Programmable Gate Array): A field programmable gate array that can be programmed by the user or designer after manufacturing to implement specific hardware functions.
[0055] BPTT(Back Propagation Through Time): Back propagation through time.
[0056] See also Figure 1 , Figure 1 This is a structural schematic diagram of a face mask detection hardware acceleration platform provided in Example 1 of the present invention.
[0057] A face mask detection hardware acceleration platform provided by the present invention comprises a processor 1 and a programmable gate array; the programmable gate array is deployed with an external memory 2, and an input data cache unit group 4, a computing unit 3 and an output data cache unit group 5 which are connected in sequence; the output data cache unit group 5 is connected to the external memory 2 through a multi-channel bus, and the external memory 2 is connected to the input data cache unit group 4 through a multi-channel bus; the computing unit 3 stores an improved target detection network encapsulated by an IP core, and the processor 1 is communicatively connected with the external memory 2;
[0058] Processor 1, used for obtaining the face image of the mask to be detected, and transmitting the face image of the mask to be detected to the calculation unit 3 through the external memory 2 and the input data buffer unit group 4 in sequence;
[0059] The computing unit 3 is used to perform face mask detection on the mask face image to be detected through an improved target detection network, output the detection result, and transmit the detection result to the external memory 2 through the output data cache unit group 5.
[0060] It should be noted that the processor 1 is an ARM processor (Advanced RISC Machine), the programmable gate array is an FPGA (Field Programmable Gate Array), and the programmable gate array is deployed with an external memory 2, and an input data cache unit group 4, a computing unit 3 and an output data cache unit group 5 connected in sequence. The present invention reuses the pulse convolution module in the improved target detection network, and the network part with large computational amount is accelerated in the FPGA to make an IP core, that is, the computing unit 3 stores the improved target detection network after IP core encapsulation, and then reuses the IP core by calling the ARM processor, and obtains the final result (detection result) after multiple calls. The transmission and storage of data in the neural network is a very important link. The present invention is designed to utilize the AXI bus (multi-channel bus, Advanced eXtensible The external memory 2 is used to store the initial weights, biases and image information to reduce the squeeze on the chip memory. The image information (the mask face image to be detected) is loaded into the input cache buffer (input data cache unit group 4) through the external memory 2, and after the pulse convolution operation, it is stored in the output cache buffer (output data cache unit group 5), and then written into the external memory 2; wherein, the improved target detection network after IP core encapsulation is composed of multiple pulse convolution modules after IP core encapsulation, convolution modules after IP core encapsulation, and convolution layers after IP core encapsulation.
[0061] Furthermore, for the construction of the basic unit, that is, for the construction of the pulse convolution module after the IP core is encapsulated, the design of the configurable pulse convolution module / convolution module divides the input data into four paths, and uses the pipeline and the three-level addition tree to implement the convolution operation; for the IF neuron model (integral emission layer) in the pulse convolution module, the input data is added to the cache data of the neuron (initial value is 0) to obtain the data input and then compare it with the threshold. If it is greater than or equal to the threshold, a pulse will be sent, and the data input will be restored to the reset potential, and the result will be stored in the cache data and added to the next input data; if the data input and is less than the threshold, no pulse will be sent, and the input data and are directly cached in the cache data. The above two calculations can be called to obtain the pulse convolution module. After parallel expansion on HLS (High-Level Synthesis), the parallel calculation of multiple pulse convolution operation modules is realized.
[0062] As a further improvement, the improved target detection network includes fourteen serially connected pulse convolution modules, convolution modules and convolution layers; the processing process of the improved target detection network for the mask face image to be detected includes:
[0063] Fourteen pulse convolution modules connected in series are used to extract features of the mask face image to be detected, and a pulse convolution feature map is output; a convolution operation is performed on the pulse convolution feature map through the convolution module, and a convolution feature map is output; the convolution feature map is input into the convolution layer for convolution operation, and the detection result is output.
[0064] Please note that Figure 2 , based on the convolutional neural network of the YOLO algorithm (You Only Look Once), the present invention converts the feature extraction network into SNN. This is because most of the energy consumption in the YOLO model is consumed in the feature extraction layer, so after the replacement, the resources and power consumption of the feature extraction layer in the network can be reduced, thereby reducing the power consumption as a whole. The entire feature extraction structure is composed of 14 layers of networks. The present invention adds IF neurons after the convolution module to form a new unit module-the pulse convolution module SConv, which can convert the convolution data into pulse data. The present invention uses the SConv layer to replace the Conv layer. In addition, since it is difficult to do pooling calculations in the pulse neural network, and in order to realize the friendliness of hardware resources, the present invention will use the SConv layer with a step size of 2 to replace the original pooling layer, thereby building a 14-layer SNN feature extraction network. As for the subsequent prediction network, the present invention still uses CNN for the final prediction. The CNN network uses real number calculations to obtain more accurate prediction values. The feature processing decoding layer uses the Conv layer to ensure the accuracy of network prediction, and finally outputs the prediction result of the target.
[0065] Specifically, the improved target detection network proposed in the present invention is an SNN-CNN hybrid target detection network, which is composed of fourteen serially connected pulse convolution modules, convolution modules and convolution layers, wherein the fourteen serially connected pulse convolution modules constitute a feature extraction network, the convolution module and the convolution layer constitute a target prediction output network, the pulse convolution module consists of a convolution layer, a normalization layer and an integral emission layer, and the integral emission layer is an IF neuron model; the convolution module consists of a convolution layer, a normalization layer and an activation function layer, and the activation function layer is a ReLU activation function layer.
[0066] As a further improvement, the pulse convolution module includes a convolution layer, a normalization layer and an integral emission layer; the image processing process of the pulse convolution module includes:
[0067] The input feature map of the input pulse convolution module is convolved through the convolution layer to output the intermediate feature map; the intermediate feature map is normalized by the normalization layer to generate a normalized feature map; the normalized feature map is converted into data by the integral emission layer to output the pulse output feature map.
[0068] It should be noted that the input feature map is the feature map of the input pulse convolution module. It can be understood that it can correspond to the feature map of the input pulse convolution module for image processing during the model training or detection process; the normalized feature map is the intermediate map generated in the pulse convolution module; the pulse output feature map is the feature map output by the pulse convolution module. It can be understood that it can correspond to any feature map output after the pulse convolution module performs image processing during the model training or detection process.
[0069] As a further improvement, the convolution module includes a convolution layer, a normalization layer, and an activation function layer; the step of performing a convolution operation on the pulse convolution feature map through the convolution module and outputting the convolution feature map includes:
[0070] The pulse convolution feature map is convolved through the convolution layer to generate an intermediate convolution feature map; the intermediate convolution feature map is normalized by the normalization layer and the intermediate normalized feature map is output; the intermediate normalized feature map is input into the activation function layer for nonlinear mapping to generate a convolution feature map.
[0071] As a further improvement, the processor 1 is further used to receive and display the detection result sent by the external memory 2.
[0072] As a further improvement, the input data cache unit group 4 is used to transmit the mask face image to be detected to the calculation unit 3 through a ping-pong operation; the output data cache unit group 5 is used to transmit the detection result to the external memory 2 through a ping-pong operation.
[0073] The input data cache unit group 4 includes an input data cache group 0 and an input data cache group 1 .
[0074] The output data cache unit group 5 includes an output data cache group 0 and an output data cache group 1 .
[0075] It should be noted that the ping-pong operation is used in conjunction with two input and output buffer groups (input data cache unit group 4, output data cache unit group 5) to cover the computing time and data transmission time. When one buffer (data cache group) is used for computing, the other buffer can be used for data transmission at the same time. Once the calculation is completed, the result in the calculation buffer can be transmitted to the target location, and at the same time, the other buffer starts to receive new data for the next round of calculation. This alternating method can minimize the idle time between computing and data transmission and improve system efficiency.
[0076] As a further improvement, the model training process of the improved target detection network includes:
[0077] Obtain an image training data set, use the image training data set to train a preset initial convolutional neural network, and determine a trained target convolutional neural network;
[0078] The model weights of the improved initial detection network are updated using the model weights of the trained target convolutional neural network to determine the improved intermediate detection network;
[0079] Based on the preset direct training method, an improved intermediate detection network is trained using an image training dataset to determine an improved object detection network.
[0080] The initial convolutional neural network is preset to be a CNN network with the same structure as the improved target detection network.
[0081] It should be noted that the commonly used SNN network weight training methods include direct training and conversion methods. Among them, the ANN-SNN conversion training algorithm often requires a longer number of time steps to obtain an accuracy close to that of the original ANN network, thereby increasing the detection time consumption. Although some articles have proposed some improvement methods for the ANN-SNN conversion network for the performance of the converted network, they still have not been able to solve the problem well. Although the direct training method can obtain better network performance in a smaller time step (for example, T=4), its training time is often longer.
[0082] For further information, see Figure 3 In order to reduce the specific time step and obtain better detection accuracy, and at the same time reduce the training time of the network, the present invention adopts a migration weight and a direct training algorithm to realize the training of the entire SNN-CNN hybrid network architecture (improved target detection network), and trains the entire hybrid network architecture by replacing the gradient method and back propagation; in order to accelerate the convergence speed of the network, the present invention constructs a CNN network with the same structure (preset initial convolutional neural network), trains it using an image training data set, and then directly migrates the weights of the trained target convolutional neural network obtained after training to the improved initial detection network (untrained improved target detection network), that is, replaces the model weights of the improved initial detection network with the model weights of the trained target convolutional neural network to obtain an improved intermediate detection network, and then uses a direct training method (preset direct training method) to train the improved intermediate detection network to obtain an improved target detection network. The above process accelerates the fitting time of the network, thereby reducing the training time of the model.
[0083] As a comparison of technical effects, it can be combined with existing technologies for reference. Spiking neural network (SNN) is a third-generation neural network with spiking neurons as computing units, which can simulate the encoding and processing of human brain information. Unlike traditional artificial neural networks (ANN, Artificial Neural Network), SNN uses discrete pulse signals to transmit information, and its neurons have memory functions, so it has high biological realism and real-time processing capabilities. At present, SNN has been widely studied and applied to various high-performance computing tasks; there are three conventional SNN training algorithms, one is an unsupervised algorithm, Spike Timing Dependent Plasticity (STDP), that is, according to the order of neuron learning, the strength of the connection between neurons is adjusted. The other two belong to supervised algorithms, one is achieved through the network migration algorithm of Convolutional Neural Network (CNN); the other is through time back propagation (BPTT), while using alternative functions to calculate discontinuous pulse mechanism problems to achieve direct training of SNN.
[0084] Furthermore, the one-stage target detection algorithms YOLO and SSD are more common in the current target detection algorithms. The main goal and method of the hybrid architecture proposed in this paper is to replace the main energy-consuming part of the detection network, the feature extraction network part, with a low-power SNN, and to use a CNN network to ensure the accuracy of the final prediction results. This paper selects YOLO as the basic network to implement the hybrid framework. In order to make it hardware-friendly, some structural modifications are made on this basis to simplify the structure of the network. Since it is a pulse neural network, there must be a basic pulse neuron model. Here, the present invention selects the simplest IF neuron model. The behavior of the IF model is as follows: the membrane voltage of the neuron is obtained, accumulated, and then compared with the threshold voltage. If the accumulated membrane voltage is greater than the threshold voltage, a pulse can be sent, and the membrane potential will also be subtracted from the threshold to obtain a new membrane potential, which is then passed to the next neuron.
[0085] Furthermore, in the field of facial mask detection, the existing technology mostly uses traditional CNN. Compared with SNN, CNN often has the disadvantage of high power consumption; the existing SNN-based target detection network often requires a larger time step to obtain better accuracy, resulting in higher delay time; when the existing technology is training the pulse neural network, if the ANN2SNN conversion algorithm is used, a larger time step is required to ensure accuracy, and if a direct algorithm is used, it takes a longer time to train the network; when the existing technology is deployed on edge devices, it often consumes more resources to deploy the network, and at the same time causes greater energy consumption.
[0086] In view of the above problems, the present invention converts the feature extraction network into SNN based on the convolutional neural network of the YOLO algorithm. This is because most of the energy consumption in the YOLO model is consumed in the feature extraction layer. Therefore, after the replacement, the resources and power consumption of the feature extraction layer in the network can be reduced, thereby reducing the power consumption as a whole. For the subsequent prediction network, the present invention still uses CNN for the final prediction to ensure the accuracy of its prediction. Specifically, the present invention builds a pulse-convolution hybrid neural network (SNN-CNN) to solve the problem of high power consumption of traditional convolutional neural networks (CNN) during the reasoning process, and at the same time solves the problem that the SNN network requires a large time step to ensure detection accuracy; the network is trained using a weight transfer + direct training algorithm, first using a CNN network with the same structure to train using a data set, then the weights are transferred to the SNN-CNN network, and then the network is trained and fine-tuned using a direct training method, which can solve the problem of long training time for the pulse neural network model; based on HLS, a configurable pulse convolution / convolution module is implemented to take into account both pulse convolution and convolution operations in the hybrid network, so the data input of the network can be directly converted into pulse data, and no additional encoding module is required for data encoding; based on ARM+FPGA, a hardware acceleration platform supporting the SNN-CNN hybrid network is built, with a configurable pulse convolution / convolution module, and a pulse convolution with a step size of 2 is used to replace the pooling operation, the three calculations are combined, the resource consumption of the model is reduced, and the network model deployment in resource-constrained edge devices is realized.
[0087] Furthermore, the present invention builds an SNN-CNN hybrid target detection network based on the YOLO target detection network. The SNN-CNN hybrid network training method combining weight migration and direct training is used to accelerate the network fitting speed, and a hardware acceleration platform supporting the SNN-CNN hybrid network is built based on ARM+FPGA, that is, based on ARM+FPGA, the SNN-CNN hybrid model is adopted and the proposed weight migration + direct training algorithm is used to accelerate the network fitting, thereby accelerating the training speed of the SNN-CNN network, thereby achieving high-performance, low-power face mask detection.
[0088] Furthermore, the present invention utilizes the YOLO target detection network, replaces the convolution layer in the feature extraction network with a pulse convolution layer, and replaces the pure CNN network structure composed entirely of convolution layers with a SNN-CNN hybrid network structure obtained by constructing a feature extraction layer and a feature processing and prediction structure composed of a convolution layer by a pulse convolution layer, and then constructs a hardware-based pulse convolution / convolution operation module to build an entire ARM+FPGA-based SNN-CNN hybrid network detection accelerator; at the same time, based on the YOLO target detection network, a SNN-CNN hybrid target detection network is constructed, and the present invention uses a pulse neural network SNN to replace the CNN structure in the original feature extraction structure to obtain a feature extraction structure based on SNN. At the same time, in order to ensure the prediction accuracy of the network, CNN is still used to construct the feature processing and prediction structure. A SNN-CNN hybrid network training method based on a combination of transfer training and direct training: First, based on a CNN with the same structure, the CNN network is trained on a data set, and the trained network weights are directly migrated to the SNN-CNN network. The SNN-CNN network is then trained and fine-tuned using a direct training method, thereby reducing the network training time. A configurable pulse convolution / convolution operation module, characterized in that: the module has convolution operations and calculation operations of IF neurons, and two different operations, pulse convolution and convolution, can be configured by whether the IF neurons are enabled. A hardware acceleration platform supporting SNN-CNN hybrid networks built on ARM+FPGA, characterized in that: the present invention calls and configures a configurable pulse convolution / convolution operation module, and deploys the entire SNN-CNN hybrid network target detection model on an edge device.
[0089] In summary, the face mask detection method based on the SNN-CNN hybrid network of the present invention has low energy consumption compared with the face mask detection algorithm using the traditional CNN. Compared with the face mask detection algorithm using the traditional SNN, it has a lower time step advantage and precision advantage. Compared with the traditional SNN training method, the transfer training adopted by the present invention is combined with direct training to speed up the training and fitting speed of the network. Compared with the existing convolution hardware operation module, the configurable pulse convolution / convolution operation module can support both pulse convolution operation and convolution operation, which is more flexible. Compared with the traditional neural network with the same structure using SNN, its power consumption performance will have obvious advantages. At the same time, a hardware acceleration platform for implementing the SNN-CNN hybrid network can be deployed on ARM+FPGA, making it possible to apply this method to wearable devices.
[0090] In an embodiment of the present invention, the present invention provides a face mask detection hardware acceleration platform, including a processor and a programmable gate array; the programmable gate array is deployed with an external memory, and an input data cache unit group, a computing unit and an output data cache unit group connected in sequence; wherein the output data cache unit group is connected to the external memory through a multi-channel bus, and the external memory is connected to the input data cache unit group through a multi-channel bus; the computing unit stores an improved target detection network encapsulated by an IP core, and the processor is communicatively connected to the external memory; the processor is used to obtain a mask face image to be detected, and transmit the mask face image to be detected to the computing unit in sequence through the external memory and the input data cache unit group; the computing unit is used to perform face mask detection on the mask face image to be detected through the improved target detection network, output the detection result and transmit the detection result to the external memory through the output data cache unit group; based on the above scheme, through the improved target detection network encapsulated by the IP core stored in the computing unit, the acquired mask face image to be detected is subjected to face mask detection, and the detection result is output, the entire improved target detection network is deployed on the edge device, which can reduce the resource consumption of the model, thereby reducing the energy consumption caused in the process of performing face mask detection.
[0091] See also Figure 4 , Figure 4 A flowchart of the steps of a hardware acceleration method for face mask detection provided in Embodiment 2 of the present invention.
[0092] The present invention provides a hardware acceleration method for face mask detection, comprising:
[0093] Step 401: When a mask face image to be detected is received, the mask face image to be detected is input into an improved target detection network encapsulated by an IP core; the improved target detection network includes fourteen serially connected pulse convolution modules, convolution modules and convolution layers.
[0094] In this embodiment, when a mask face image to be detected is received, the mask face image to be detected is input into an improved target detection network encapsulated by an IP core; the improved target detection network includes fourteen serially connected pulse convolution modules, convolution modules and convolution layers.
[0095] Step 402: Use fourteen pulse convolution modules connected in series to extract features from the mask face image to be detected, and output a pulse convolution feature map.
[0096] In this embodiment, fourteen pulse convolution modules connected in series are used to extract features of the mask face image to be detected, and a pulse convolution feature map is output.
[0097] Step 403: Perform a convolution operation on the pulse convolution feature map through a convolution module, and output the convolution feature map.
[0098] In this embodiment, a convolution operation is performed on the pulse convolution feature map through a convolution module to output the convolution feature map.
[0099] Step 404: input the convolution feature map into the convolution layer for convolution operation, and output the detection result.
[0100] In this embodiment, the convolution feature map is input into the convolution layer for convolution operation, and the detection result is output.
[0101] In an embodiment of the present invention, the present invention provides a hardware acceleration method for face mask detection. When a mask face image to be detected is received, the mask face image to be detected is input into an improved target detection network encapsulated by an IP core; the improved target detection network includes fourteen pulse convolution modules, convolution modules and convolution layers connected in series; fourteen pulse convolution modules are used to extract features of the mask face image to be detected, and a pulse convolution feature map is output; a convolution operation is performed on the pulse convolution feature map through the convolution module, and a convolution feature map is output; the convolution feature map is input into the convolution layer for convolution operation, and a detection result is output; based on the above scheme, the obtained mask face image to be detected is subjected to face mask detection through the improved target detection network encapsulated by the IP core, and the process of outputting the detection result can reduce the resource consumption of the model, thereby reducing the energy consumption caused in the process of performing face mask detection.
[0102] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the hardware acceleration method for face mask detection in any of the above embodiments.
[0103] An embodiment of the present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the hardware acceleration method for face mask detection in any of the above embodiments are implemented.
[0104] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0105] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0106] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A face mask detection hardware acceleration platform, characterized in that: It includes a processor and a programmable gate array; the programmable gate array is deployed with an external memory, and an input data cache unit group, a computing unit and an output data cache unit group connected in sequence; The output data cache unit group is connected to the external memory through a multi-channel bus, and the external memory is connected to the input data cache unit group through the multi-channel bus; The computing unit stores an improved target detection network encapsulated by an IP core, and the processor is communicatively connected with the external memory; The processor is used to obtain a mask face image to be detected, and transmit the mask face image to be detected to the computing unit through the external memory and the input data cache unit group in sequence; The computing unit is used to perform face mask detection on the mask face image to be detected through the improved target detection network, output the detection result, and transmit the detection result to the external memory through the output data cache unit group.
2. The face mask detection hardware acceleration platform according to claim 1, characterized in that: The improved target detection network includes fourteen pulse convolution modules, convolution modules and convolution layers connected in series; the processing process of the improved target detection network on the mask face image to be detected includes: Fourteen pulse convolution modules connected in series are used to extract features of the mask face image to be detected, and a pulse convolution feature map is output; Performing a convolution operation on the pulse convolution feature map through a convolution module, and outputting a convolution feature map; The convolution feature map is input into the convolution layer for convolution operation, and the detection result is output.
3. The face mask detection hardware acceleration platform according to claim 2, characterized in that: The pulse convolution module includes a convolution layer, a normalization layer and an integral emission layer; the image processing process of the pulse convolution module includes: Performing a convolution operation on the input feature map of the pulse convolution module through a convolution layer, and outputting an intermediate feature map; Normalizing the intermediate feature map using a normalization layer to generate a normalized feature map; The normalized feature map is subjected to data conversion through an integral emission layer to output a pulse output feature map.
4. The face mask detection hardware acceleration platform according to claim 2, characterized in that: The convolution module includes a convolution layer, a normalization layer and an activation function layer; the step of performing a convolution operation on the pulse convolution feature map through the convolution module and outputting the convolution feature map includes: Performing a convolution operation on the pulse convolution feature map through a convolution layer to generate an intermediate convolution feature map; Normalizing the intermediate convolutional feature map using a normalization layer, and outputting an intermediate normalized feature map; The intermediate normalized feature map is input into the activation function layer for nonlinear mapping to generate a convolutional feature map.
5. The face mask detection hardware acceleration platform according to claim 1, characterized in that: The processor is further used to receive and display the detection result sent by the external memory.
6. The face mask detection hardware acceleration platform according to claim 1, characterized in that: The input data buffer unit group is used to transmit the mask face image to be detected to the computing unit through a ping-pong operation; The output data cache unit group is used to transmit the detection result to the external memory through a ping-pong operation.
7. The face mask detection hardware acceleration platform according to claim 1, characterized in that: The model training process of the improved target detection network includes: Acquire an image training data set, use the image training data set to train a preset initial convolutional neural network, and determine a trained target convolutional neural network; The model weights of the improved initial detection network are updated using the model weights of the trained target convolutional neural network to determine an improved intermediate detection network; Based on a preset direct training method, the improved intermediate detection network is trained using the image training data set to determine the improved target detection network.
8. A hardware acceleration method for face mask detection, characterized in that: include: When receiving a mask face image to be detected, the mask face image to be detected is input into an improved target detection network encapsulated by an IP core; The improved target detection network includes fourteen pulse convolution modules, convolution modules and convolution layers connected in series; Fourteen pulse convolution modules connected in series are used to extract features of the mask face image to be detected, and a pulse convolution feature map is output; Performing a convolution operation on the pulse convolution feature map through a convolution module, and outputting a convolution feature map; The convolution feature map is input into the convolution layer for convolution operation, and the detection result is output.
9. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the hardware acceleration method for face mask detection as described in any one of claim 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the hardware acceleration method for face mask detection as described in any one of claim 8 is implemented.
Citation Information
Cited By
Training method based on biology-information collaborative spiking neural network and neuromorphic optical flow prediction method based on training method
CN119991742A
