A real-time facial emotion recognition method based on binary neural network of FPGA

Abstract

The invention discloses a binary neural network real-time facial emotion recognition method based on FPGA, the method comprising: step 1) preprocessing a collected video image; step 2) based on a Harr-like feature template, using a window sliding mechanism to detect and crop a human face in the preprocessed image, and intercepting a part containing the human face image, and using a circular LBP operator to extract local binary information from the intercepted image; step 3) inputting the extracted local binary information into a pre-established and trained facial emotion recognition model to obtain an emotion recognition result; the facial emotion recognition model is a binary neural network, and by adopting an XNOR gate design, a bit accumulation operation and a threshold comparison method in the FPGA, forward reasoning is performed to realize emotion recognition.

Description

Technical Field

[0001] The present invention belongs to the field of emotional computing and FPGA heterogeneous computing, and in particular to a binary neural network real-time facial emotion recognition method based on FPGA. Background Art

[0002] Today, computers play an extremely important role in industry and daily life, and are rapidly becoming an indispensable part of social development. Therefore, the need to study the interaction between humans and computers is increasing day by day. Of course, the premise of human-computer interaction is that a computer can accurately and quickly analyze people's emotions and intentions, and make appropriate responses accordingly. The various emotions revealed in facial expressions are a universal and effective way to express human intentions. Analyzing people's intentions through the emotions revealed by the face is called facial emotion recognition technology. This technology is applied to many fields, such as robotics, security detection, and the recently popular detection of autistic children.

[0003] Facial emotion technology can be divided into two categories according to the different feature extraction methods:

[0004] One is the traditional manual feature method to achieve emotion detection, which includes: image preprocessing, feature extraction and feature classification.

[0005] In the image preprocessing stage, irrelevant information is removed through filtering and histogram equalization, and information related to facial emotions is enhanced. In the feature extraction stage, feature extraction methods such as local binary patterns (LBP), active shape models (ASM), and Haar-like features are used to extract facial features from images. In the expression classification stage, classifiers such as support vector machine (SVM), k-nearest neighbor (KNN) or AdaBoost are used to classify the extracted features into corresponding emotions. However, the method using manual features requires the design of a suitable feature extractor and expression classifier separately. When performing systematic optimization, it is also necessary to formulate optimization plans for the feature extractor and feature classifier separately. Therefore, the manual feature extraction method cannot achieve simultaneous optimization of the two stages. In addition, this method is sensitive to external factors, which may lead to a serious decline in the final recognition accuracy (such as posture changes, occlusion, and lighting).

[0006] The other type is methods based on deep learning. With the breakthrough progress of hardware technology and big data technology in recent years, deep learning has become increasingly popular because of its excellent performance in solving complex problems. Especially in computer vision, such as image classification and object detection, methods based on convolutional neural networks (CNN) have better robustness. In addition, through end-to-end training, CNN can optimize parameters at one time, thereby simplifying the optimization method. Although in the study of facial emotion recognition, CNN-based methods have shown higher performance than conventional methods, due to the deep structure of the network with multiple layers and millions of parameters, for example, the VGG-16 network contains 140 million 32-bit floating-point parameters, the entire network needs to occupy more than 500 megabytes of storage space and perform 1.6×10^10 floating-point operations. Such a large number of parameters and calculations make CNN heavily dependent on high-performance hardware computing platforms such as GPUs. However, in practical applications, devices with limited computing resources are usually used. For example, embedded devices based on Field Programmable Gate Array (FPGA) usually have only a few thousand computing units, which is far from being able to process millions of floating-point operations in common deep models. There is a serious contradiction between complex models and limited computing resources. This directly leads to great limitations on facial emotion detection in applications that require real-time processing, such as Advanced Driving Assistance System (ADAS), which collects driver facial information through cameras and transmits it to embedded systems for processing. However, in practical applications, it is difficult for embedded computing platforms to accurately determine the driver's status in real time, and thus cannot correctly assist driving; in the baby emotion recognition and soothing system, the image recognition system recognizes the baby's emotions and then starts the voice soothing module through the main control server to soothe the crying baby, achieving the effect of remote care. In this system, real-time detection of baby emotions is particularly critical. If the baby's emotional status cannot be conveyed in real time, remote care cannot be achieved.

[0007] To solve the problem of deploying CNN, other platforms with more suitable performance, energy consumption and cost are needed. One of the most suitable choices is the System on Chip (SoC) hardware platform, because SoC itself is designed for embedded applications. Compared with GPU, SoC consumes less energy and costs less. In addition, SoC also has the comprehensive interaction capability of FPGA hardware components and ARM processors. FPGA can effectively accelerate CNN network reasoning, and ARM processor can flexibly control the system to achieve the effect of accelerating CNN network. Therefore, the method based on the combination of FPGA and ARM to achieve real-time recognition of facial emotions has become a new research direction. Summary of the invention

[0008] The existing technical defects are that there is a serious contradiction between complex models and limited computing resources. Specifically, there are two aspects:

[0009] The convolutional neural network method relies heavily on high-performance hardware computing platforms such as GPUs because of the large number of calculation parameters and large amount of calculation;

[0010] The current hardware computing platforms, such as CPU and GPU, consume a lot of energy and are costly, which makes facial emotion recognition technology unfavorable for deployment and popularization on mobile terminals.

[0011] The purpose of the present invention is to overcome the above-mentioned defects of the prior art and propose a real-time facial emotion recognition method based on FPGA binary neural network.

[0012] In order to achieve the above object, the present invention proposes a binary neural network real-time facial emotion recognition method based on FPGA, the method comprising:

[0013] Step 1) preprocessing the collected video images;

[0014] Step 2) Based on the Harr-like feature template, a window sliding mechanism is used to detect and crop the face in the preprocessed image, and the part containing the face image is intercepted, and a circular LBP operator is used to extract local binary information from the intercepted image;

[0015] Step 3) inputting the extracted local binary information into a pre-established and trained facial emotion recognition model to obtain an emotion recognition result;

[0016] The facial emotion recognition model is a binary neural network, which implements emotion recognition by performing forward reasoning using XNOR gate design, bit accumulation operation and threshold comparison method in FPGA.

[0017] As an improvement of the above method, the step 1) specifically includes:

[0018] The captured video images are divided into frames, and then the contrast and brightness of the divided images are enhanced, and the image size is scaled proportionally.

[0019] As an improvement of the above method, the step 2) specifically includes:

[0020] Based on the Haar-like feature template, the preprocessed image is described by Haar features, and the integral image is used to accelerate the extraction of Haar features.

[0021] A sliding window of a preset size is used to slide over all positions on the image after feature extraction in turn to detect faces; and the size of the sliding window is gradually increased to a certain multiple, and this step is repeated to obtain detection windows that may contain faces at various sizes;

[0022] AdaBoost and classifier cascade structure are used to screen detection windows of different sizes, eliminating the possibility that sliding windows of different sizes circle the same target, and returning the coordinate information of the face window;

[0023] Cutting out a portion of the image containing the face according to the coordinate information of the window containing the face;

[0024] The circular LBP operator is used to extract local binary information from the cropped image.

[0025] As an improvement of the above method, the input of the facial emotion recognition model is local binary information, and the output is a recognition result. The facial emotion recognition model includes a plurality of cascaded block layers, a routing layer, a reordering layer and a plurality of cascaded convolutional layers in sequence, wherein:

[0026] The multiple cascaded Block layers are used to obtain enhanced feature maps; each Block layer includes a convolution layer, a batch normalization layer and a pooling layer;

[0027] The reordering layer is used to extract the front-end input feature map and sample it by position in its neighborhood. The pixels at the same position are output as sub-images, and the length and width of the output sub-image are consistent with the length and width of the next cascaded feature map;

[0028] The routing layer is used to cascade the sub-graph output by the reordering layer with the feature graph enhanced by multiple cascaded block layer operations, and input the sub-graph into the convolution layer;

[0029] The multiple cascaded convolutional layers are used to implement the final emotion classification task, wherein each convolutional layer has the same structure and the convolution kernel size is 1×1.

[0030] As an improvement of the above method, the method further includes a training step of a facial emotion recognition model performed on a host computer; specifically including:

[0031] The images with known emotion labels in the Fer2013 facial expression dataset are augmented by flipping, translating and cropping to obtain an augmented dataset, and a certain number of images are selected from the dataset to establish a training set.

[0032] The circular LBP operator is used to extract the local binary information of the images in the training set to further enhance the useful information in the training set images;

[0033] The local binary information is input into the facial emotion recognition model, and self-supervised learning is performed by adjusting the configuration information of each layer until the training requirements are met to obtain a trained facial emotion recognition model;

[0034] Extract the trained model weights and convert them into binary weight files suitable for hardware circuit operation.

[0035] A binary neural network real-time facial emotion recognition system based on FPGA, the system comprises: an image preprocessing module deployed on a host computer and a face detection module, a hardware reasoning module and a facial emotion recognition model deployed on a hardware computing platform, wherein:

[0036] The image preprocessing module is used to preprocess the collected video images;

[0037] The face detection module is used to detect and crop the face in the preprocessed image, and intercept the part of the image containing the face, and use the circular LBP operator to extract local binary information from the intercepted image;

[0038] The hardware inference module is used to input the extracted local binary information into a pre-established and trained facial emotion recognition model to obtain an emotion recognition result;

[0039] The facial emotion recognition model is a binary neural network, which implements emotion recognition by performing forward reasoning using XNOR gate design, bit accumulation operation and threshold comparison method in FPGA.

[0040] As an improvement of the above system, the hardware computing platform is a PYNQ board integrating an ARM processor and FPGA.

[0041] As an improvement of the above system, the configuration information of the facial emotion recognition model is stored in the memory area of ​​the ARM processor, and the facial emotion recognition model is deployed on the FPGA.

[0042] As an improvement of the above system, the processing process of the hardware reasoning module specifically includes:

[0043] The configuration information of the facial emotion recognition model is read from the memory area of ​​the ARM processor and sent to the parameter cache module PCM of the FPGA through the AXI_LITE bus. The local binary information and weight parameters of the emotion recognition model are transmitted from the dynamic random access memory DRAM to the PE processing array deployed in the FPGA through the AXI_MEM bus for forward reasoning. The emotion recognition results obtained by the reasoning calculation are transmitted to the DRAM through the AXI_MEM bus.

[0044] The PE processing array includes multiple PE processing units for parallel processing. Each PE processing unit is implemented based on the dot product operation rule of the binary neural network and adopts the XNOR gate. The number of "1" bits in the bit accumulation operation statistics parameter is stored in the accumulator register and compared with the set threshold value in an unsigned manner to obtain the calculation result of the processing array.

[0045] Compared with the prior art, the advantages of the present invention are:

[0046] 1. The present invention is based on a binary neural network design system. Under the condition of binary compression of model weights, the speed of network forward reasoning is greatly promoted. In addition, the network is deployed on an FPGA development board, and the calculation amount of the network is reduced by using programmable hardware logic circuits, which further speeds up the operation rate of the entire network and maintains good accuracy.

[0047] 2. In the present invention, a method combining traditional machine learning and deep learning is used for emotion recognition. The input image is processed with a circular LBP operator to enhance the texture features of the image. More feature information can be extracted in subsequent network reasoning, thereby improving the recognition accuracy.

[0048] 3. In the present invention, the optimized design of the PE processing array not only reduces the computational complexity and the consumption of hardware resources, but also avoids the complex calculation of the BN layer during the reasoning process, thereby improving the system performance;

[0049] 4. The present invention adds a routing layer to the binary neural network, cascades the input feature maps of multiple layers as the input feature maps of the next layer, and enhances the information content of the feature maps by fusing multi-dimensional information, thereby improving the recognition accuracy. The convolutional layer is used instead of the fully connected layer, so that the convolutional network can slide on a larger input image, breaking through the previous limitation on the input image size. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of the real-time facial emotion recognition method of binary neural network based on FPGA of the present invention;

[0051] Figure 2 It is the face monitoring flow chart;

[0052] Figure 3 This is the structure diagram of the facial emotion recognition model;

[0053] Figure 4 is a comparison of the Fer2013 dataset before and after enhancement, where Figure 4(a) is before enhancement and Figure 4(b) is after enhancement;

[0054] Figure 5 It is a hardware architecture for real-time facial emotion detection.

[0055] Figure 6 It is the data path of the PE processing array. DETAILED DESCRIPTION

[0056] The present invention is a method for real-time facial emotion recognition based on binary neural network of FPGA. The FPGA hardware computing platform is PYNQ-Z1, which is equipped with FPGA components and ARM processors respectively. In the implementation process, the present invention can obtain higher computing performance through FPGA and provide perfect application functions by relying on ARM processor. The present invention uses a USB camera to collect video images in real time, and the collected images are sent to the host computer for preprocessing through the development board. The real-time facial emotion recognition method of the present invention is divided into four modules in total: image preprocessing module, face detection module, model building and training module, and hardware reasoning module. The image preprocessing module mainly enhances the collected image, such as contrast, brightness, etc., and scales the image in proportion to the input size of the binary neural network. The face detection module mainly detects and crops the face in the image, and extracts the face features and inputs them into the hardware reasoning module. The hardware reasoning module recognizes the input image by loading the weight model generated by a large amount of training data, and finally outputs the result;

[0057] Image preprocessing module: After the video images collected by the USB camera are framed, the image processing module is used to preprocess each frame of the collected image, including image contrast and brightness enhancement, image grayscale, and image bit depth conversion from 8 bits to 24 bits (true color).

[0058] Face detection module: Use the face detection module to detect and capture faces in preprocessed images. Specifically, it uses the traditional machine learning algorithm Harr-like feature template to describe the features of each window in the image, uses the window sliding mechanism to frame the face window in the image and returns the coordinate information of the face window, and then captures the part of the image containing the face based on the returned coordinate information.

[0059] Model building and training module: Build a binary neural network on the upper computer software side, load the facial emotion dataset Fer2013 that has been processed with data enhancement and circular LBP operator, train and extract the trained model weights, and finally convert the model weights into a binary weight file suitable for hardware circuit operation.

[0060] Hardware reasoning module: The above binary neural network is deployed on the hardware computing platform, and the binary weights are transferred to the on-chip memory of the board. The hardware side loads the weights to perform forward reasoning for real-time emotion recognition and feedback the real-time detection results.

[0061] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0062] Example 1

[0063] like Figure 1 As shown, Embodiment 1 of the present invention proposes a binary neural network real-time facial emotion recognition method based on FPGA, the method comprising:

[0064] Step 1) preprocessing the collected video images;

[0065] Step 2) Based on the Harr-like feature template, a window sliding mechanism is used to detect and crop the face in the preprocessed image, and the part containing the face image is intercepted, and a circular LBP operator is used to extract local binary information from the intercepted image;

[0066] Step 3) inputting the extracted local binary information into a pre-established and trained facial emotion recognition model to obtain an emotion recognition result;

[0067] The facial emotion recognition model is a binary neural network, which implements emotion recognition by performing forward reasoning using XNOR gate design, bit accumulation operation and threshold comparison method in FPGA.

[0068] The present invention uses image processing technology combining deep learning and traditional machine learning to apply to the FPGA hardware computing platform PYNQ-Z1, realizing low-power, high-precision real-time facial emotion recognition. Depending on the functions implemented, it specifically includes: image preprocessing, face detection, model building and training, and hardware reasoning.

[0069] 1. Image preprocessing

[0070] PYNQ is a board developed based on Python language and hardware library launched by Xilinx. The board provides a single-chip solution combining ARM processor and FPGA programmable logic, which is very suitable for computing-intensive and feature-rich embedded application design. The present invention uses PYNQ-Z1 development board equipped with USB camera to perform real-time video acquisition, and transmits the acquired image to the host computer, which then pre-processes each frame of the recognized image, including enhancing the contrast and brightness of the image, graying the image, and converting the image bit depth from 8 bits to 24 bits (true color).

[0071] 2. Face Detection

[0072] Using the traditional machine learning sliding window mechanism + cascade classifier method, such as Figure 2 shown.

[0073] The image pre-processed in the above steps is characterized by Haar features. The so-called Haar features are that the image window has two kinds of white and black rectangles, and the characteristic value of the window is defined as the sum of white rectangle pixels minus the sum of black rectangle pixels. The Haar feature value reflects the grayscale change of the image and can provide effective information for distinguishing between human faces and non-human faces. Since the sum of grayscale values ​​of multiple pixels in the local area needs to be calculated each time when extracting Haar features, the speed of feature extraction is greatly limited. Therefore, the integral map method is used to accelerate the feature extraction of Haar. The integral map is to store the pixel sum of the rectangular area formed by the image from the starting point to each point in the memory as an array. In the process of Haar feature extraction, when the pixel sum of a certain area is needed, the elements in the array can be directly indexed, thereby reducing the amount of calculation and improving the efficiency of feature extraction. After the feature extraction of the picture is completed, the sliding window mechanism slides over all positions on the pre-processed picture in turn with the basic detection size, that is, the 32*32 size used during training, and determines whether it is a target in all detection windows, and gradually increases the detection window scale (the present invention is set to expand 1.3 times each time) to complete the picture detection at each scale. The sliding window mechanism simplifies the face detection problem into a binary classification problem, but also brings a huge number of window recognitions, which will increase the requirements for feature classification methods. Common feature classification methods include SVM, K nearest neighbor and AdaBoost, and AdaBoost is often selected for classification tasks because of its low computational cost and high classification accuracy. Specifically, AdaBoost cascades multiple classifiers in different ways, dynamically adjusts the complexity of the subsequent classifiers according to the difficulty of classification of the front-end window, and thus gathers weak ones into strong ones to achieve higher classification accuracy at a lower computational cost. Finally, all detection windows that may contain faces are merged to eliminate the possibility of sliding windows of different scales delineating the same target, and the coordinate information of the window containing the face is returned.

[0074] 3. Model building and training

[0075] The backbone network for facial emotion recognition is a binary neural network. A binary neural network (BNN) is a neural network that uses only two values, 1 and -1, to represent weights and excitations. Compared with full-precision networks, binary neural networks significantly reduce the memory footprint of the hardware computing platform. In addition, binary neural networks can use simple combinations such as XNOR gates and bit accumulation (popcount) to replace floating-point multiplication and addition operations to implement convolution operations, thereby reducing the amount of calculation and greatly facilitating the deployment of models on resource-constrained devices.

[0076] The overall network framework of the present invention is a convolutional neural network topology inspired by VGG16 and residual networks. The network model includes four types: convolution layer, pooling layer, reordering layer and routing layer. The specific network structure is as follows Figure 3 shown.

[0077] The data set used in the present invention is the Fer2013 facial expression data set, and the original data set includes 35886 facial expression pictures, each picture is composed of a grayscale image with a fixed size of 48×48, and there are 7 kinds of expressions. Since the original data set has the problems of unbalanced data distribution and too few pictures, the present invention performs data enhancement by flipping, translating and cutting, and the final enhanced data set includes 76375 facial expression pictures, which are proportionally compressed to a size of 32×32, including 61100 training pictures and 15275 test pictures. As shown in Figure 4 (a) and Figure 4 (b), after the data set is enhanced, the number of pictures of each category of expression tends to be balanced, especially the number of pictures in the "disgust" category increases by nearly 20 times. By training the facial emotion recognition network with the enhanced data set, the generalization ability of the network is significantly improved, and the network also learns the characteristics of translation invariance, thereby improving the final recognition accuracy.

[0078] The present invention uses a circular LBP operator to extract local binary information of the input image before performing network forward reasoning. LBP (Local Binary Pattern) is an operator used to describe local texture features of an image; it has significant advantages such as rotation invariance and grayscale invariance. The circular LBP operator formula is described as follows:

[0079]

[0080] Where (x c ,y c ) represents the coordinates of the center point, I(p) and I(c) represent the grayscale values ​​of the pth sampling point and the center point, respectively, and s(·) is the threshold function. Compared with the original LBP operator, the circular LBP operator uses a circular neighborhood with a radius of R to compare the center point pixel value with the adjacent pixel values, and converts the binary number obtained from the comparison into an LBP value representing the texture information of the center point.

[0081] After being processed by the circular LBP operator, the influence of illumination on the feature extraction of the input image can be effectively reduced, while the local texture features of the image are enhanced. In this way, the convolutional neural network can extract more distinctive facial features and improve the final recognition accuracy.

[0082] The present invention adds a reordering layer and a routing layer to the network structure, wherein the combination of a convolutional layer, a batch normalization layer (Batch Normalization Layer, BN) and a pooling layer is called a "Block". The function of the reordering layer is to extract the front-end input feature map and sample it by position in its neighborhood. The pixels at the same position output a sub-map, and the length and width of the output sub-map must be consistent with the length and width of the next cascaded feature map. The function of the routing layer is to cascade the above-mentioned sub-map with the feature map after all "Block" operations as the input feature map of the next layer. Through the cascading of multi-dimensional feature maps, the amount of information in the feature map will be greatly improved, thereby improving the accuracy of recognition.

[0083] 4. Hardware Inference

[0084] The present invention designs a real-time facial emotion recognition method based on FPGA. The hardware architecture used is the PYNQ board developed based on Python language and hardware library launched by Xilinx. The hardware architecture of the board adopts the design method of ARM+FPGA, and integrates ARM processor and programmable logic FPGA at the same time. The part containing ARM processor is usually called processing system (PS), and the part containing FPGA is called programmable logic (PL). There are multiple communication modes between PS and PL, the most common of which is based on bus protocol interface (Advanced eXtensible Interface, AXI). In addition, communication can also be carried out through clock signals, interrupts and other modes. Figure 5 Demonstrated hardware architecture for real-time facial emotion detection:

[0085] The ARM processor sends the configuration information to the parameter cache module PCM of the FPGA through the AXI_LITE bus, and transmits the input image binary information and weight parameters from the dynamic random access memory DRAM to the FPGA's PE processing array for inference calculation through the AXI_MEM bus. After that, the calculated emotion recognition results are transmitted to DRAM through the AXI_MEM bus. Finally, the host computer completes the visualization of the recognition results. Each PE receives exactly the same control signal and input image data, but performs multiplication and accumulation operations on different parts of the input data and weights at the same time. Figure 6 Shows the data path of the PE processing array:

[0086] In the PE processing array, due to the unique dot product operation rules of BNN, it is implemented using the XNOR gate, and then the number of "1" bits in the popcount statistical parameter is used to replace the signed bit arithmetic accumulation, and the result is added to the accumulation register. After that, the output of the convolution layer is compared with the pre-set threshold in an unsigned manner, and the output greater than or equal to the threshold is activated as the final output. The above optimization technology not only reduces the computational complexity and consumption of hardware resources, but also avoids the complex calculation of the BN layer during the inference process, thereby improving system performance.

[0087] The results show that the real-time reasoning of facial emotions can be realized on the hardware board, which fully utilizes the advantages of FPGA's high parallel computing capability and ARM processor's flexible control system. While maintaining the original accuracy, it significantly improves the recognition speed, providing strong technical support for the transplantation of real-time facial emotion recognition on mobile devices with limited resources.

[0088] Technical innovations of the present invention:

[0089] 1. The present invention uses an image processing method that combines deep learning with traditional manual feature extraction to solve the problem of facial emotion recognition. Among them, the traditional manual feature extraction algorithm greatly makes up for the problem of insufficient local feature extraction of images in deep learning, and also significantly improves deep learning against adverse factors such as lighting;

[0090] 2. The present invention deploys the above-mentioned facial emotion recognition method on an FPGA board, which ensures high accuracy of facial emotion recognition while greatly reducing the amount of calculation and power consumption by taking advantage of FPGA parallel computing;

[0091] 3. The present invention uses a new method based on circular LBP and binary neural network to train the facial emotion recognition network. LBP feature information is added to the original image input to the network, thereby enhancing the texture information and anti-interference ability of the image.

[0092] 4. The present invention adds a routing layer to the binary neural network, cascades the input feature maps of multiple layers as the input feature maps of the next layer, and enhances the information content of the feature maps by fusing multi-dimensional information, thereby improving the recognition accuracy. The convolutional layer is used instead of the fully connected layer, so that the convolutional network can slide on a larger input image, breaking through the previous limitation on the input image size.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the embodiments, it should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention and should be included in the scope of the claims of the present invention.

Claims

1. A binary neural network real-time facial emotion recognition method based on FPGA, the method comprising: Step 1) preprocessing the collected video images; Step 2) Based on the Harr-like feature template, a window sliding mechanism is used to detect and crop the face in the preprocessed image, and the part containing the face image is intercepted. The circular LBP operator is used to extract local binary information from the intercepted image; Step 3) inputting the extracted local binary information into a pre-established and trained facial emotion recognition model to obtain emotion recognition results; The facial emotion recognition model is a binary neural network, which uses XOR gate design, bit accumulation operation and threshold comparison method in FPGA to perform forward reasoning and realize emotion recognition; The input of the facial emotion recognition model is local binary information, and the output is a recognition result. The facial emotion recognition model includes a plurality of cascaded block layers, a routing layer, a reordering layer and a plurality of cascaded convolutional layers, wherein: The multiple cascaded Block layers are used to obtain enhanced feature maps; each Block layer includes a convolution layer, a batch normalization layer and a pooling layer; The reordering layer is used to extract the front-end input feature map and sample it by position in its neighborhood. The pixels at the same position are output as sub-images, and the length and width of the output sub-image are consistent with the length and width of the next cascaded feature map; The routing layer is used to cascade the sub-graph output by the reordering layer with the feature graph enhanced by multiple cascaded block layer operations, and input the sub-graph into the convolution layer; The multiple cascaded convolutional layers are used to implement the final emotion classification task, wherein each convolutional layer has the same structure and the convolution kernel size is 1×1.

2. The real-time facial emotion recognition method based on binary neural network of FPGA according to claim 1, characterized in that: The step 1) specifically includes: The captured video images are divided into frames, and then the contrast and brightness of the divided images are enhanced, and the image size is scaled proportionally.

3. The real-time facial emotion recognition method based on binary neural network of FPGA according to claim 2 is characterized in that: The step 2) specifically includes: Based on the Haar-like feature template, the preprocessed image is described by Haar features, and the integral image is used to accelerate the extraction of Haar features. A sliding window of a preset size is used to slide over all positions on the image after feature extraction in turn to detect faces; and the size of the sliding window is gradually increased to a certain multiple, and this step is repeated to obtain detection windows that may contain faces at various sizes; AdaBoost and classifier cascade structure are used to screen detection windows of different sizes, eliminating the possibility that sliding windows of different sizes circle the same target, and returning the coordinate information of the face window; Cutting out a portion of the image containing the face according to the coordinate information of the window containing the face; The circular LBP operator is used to extract local binary information from the cropped image.

4. The real-time facial emotion recognition method based on binary neural network of FPGA according to claim 1 is characterized in that: The method also includes a facial emotion recognition model training step performed on a host computer; specifically including: The images with known emotion labels in the Fer2013 facial expression dataset are augmented by flipping, translating and cropping to obtain an augmented dataset, and a certain number of images are selected from the dataset to establish a training set. The circular LBP operator is used to extract the local binary information of the images in the training set to further enhance the useful information in the training set images; The local binary information is input into the facial emotion recognition model, and self-supervised learning is performed by adjusting the configuration information of each layer until the training requirements are met to obtain a trained facial emotion recognition model; Extract the trained model weights and convert them into binary weight files suitable for hardware circuit operation.

5. A system based on the FPGA-based binary neural network real-time facial emotion recognition method according to claim 1, characterized in that: The system includes: an image preprocessing module deployed on a host computer and a face detection module, a hardware reasoning module and a facial emotion recognition model deployed on a hardware computing platform, wherein: The image preprocessing module is used to preprocess the collected video images; The face detection module is used to detect and crop the face in the preprocessed image, and intercept the part of the image containing the face, and use the circular LBP operator to extract local binary information from the intercepted image; The hardware inference module is used to input the extracted local binary information into a pre-established and trained facial emotion recognition model to obtain an emotion recognition result; The facial emotion recognition model is a binary neural network, which implements emotion recognition by performing forward reasoning using XNOR gate design, bit accumulation operation and threshold comparison method in FPGA.

6. The system of the binary neural network real-time facial emotion recognition method based on FPGA according to claim 5, characterized in that: The hardware computing platform is a PYNQ board that integrates an ARM processor and FPGA.

7. The system of the binary neural network real-time facial emotion recognition method based on FPGA according to claim 6, characterized in that: The configuration information of the facial emotion recognition model is stored in the memory area of ​​the ARM processor, and the facial emotion recognition model is deployed on the FPGA.

8. The system of the binary neural network real-time facial emotion recognition method based on FPGA according to claim 7, characterized in that: The processing process of the hardware reasoning module specifically includes: The configuration information of the facial emotion recognition model is read from the memory area of ​​the ARM processor and sent to the parameter cache module PCM of the FPGA through the AXI_LITE bus. The local binary information and weight parameters of the emotion recognition model are transmitted from the dynamic random access memory DRAM to the PE processing array deployed in the FPGA through the AXI_MEM bus for forward reasoning. The emotion recognition results obtained by the reasoning calculation are transmitted to the DRAM through the AXI_MEM bus. The PE processing array includes multiple PE processing units for parallel processing. Each PE processing unit is implemented based on the dot product operation rule of the binary neural network and is implemented using the XNOR gate. The number of "1" bits in the statistical parameter is accumulated by bit accumulation operation, stored in the accumulator register, and compared with the set threshold value in an unsigned manner to obtain the calculation result of the processing array.

Images