A feature-enhanced lightweight network FGNet face expression recognition method

Abstract

The application claims a feature-enhanced lightweight network FGNet face expression recognition method, belonging to the technical field of pattern recognition. The method comprises the following steps: firstly, aiming at the problem that the deep convolutional neural network is too complex, a lightweight network model FGNet based on the Keras neural network framework is designed, and channel slicing and channel shuffling are added to the backbone; secondly, a lightweight and efficient attention module is designed to extract different scale features and output rich multi-scale information by using the channel and spatial feature extraction capability of the network attention module, so as to improve the performance of the network model; then, aiming at the problem of uneven sample categories of the face expression database, an adaptive class weight is proposed, and is weighted to the self-defined loss function, so as to further improve the accuracy of each expression recognition; finally, a face expression recognition system platform is designed and demonstrated, and end-to-end face expression recognition is realized.

Description

Technical Field

[0001] This invention belongs to the field of computer pattern recognition technology, specifically a method for facial expression recognition. Background Technology

[0002] Facial expressions are the richest resource and the most effective way to express emotions in nonverbal communication, playing a vital role in interpersonal interactions. Facial expressions contain rich information about human behavior and are the primary carriers of emotion. They convey subtle emotional responses and corresponding psychological states, highlighting their importance in interpersonal communication. Facial expression recognition technology has gained attention and become a research hotspot due to the increasing emphasis on facial expression information. Facial expression recognition involves using computers to acquire facial expression images, preprocess them, extract features, and classify them. It analyzes facial expression information to infer a person's psychological state, ultimately achieving intelligent human-computer interaction. Facial expression recognition technology is a key area of ​​research in affective computing, representing a highly challenging interdisciplinary field that integrates psychology, physiology, computer vision, biometrics, affective computing, and artificial psychology theory. Its research has significant implications for natural and harmonious human-computer interaction, distance education, and safe driving.

[0003] Unlike other image vision fields, facial expression recognition presents several challenges: First, facial expressions are highly nuanced, with subtle changes in facial features potentially representing drastically different emotions; second, facial expressions are short-lived, often lasting less than a second, which places high demands on the real-time performance of expression detection; and third, facial expression data varies significantly between different faces and is poorly resistant to noise from lighting and background, all of which pose considerable challenges to the development and application of facial expression recognition.

[0004] A single facial image contains a wealth of information, and the facial expressions a person makes at different times in a video sequence are not entirely the same. Therefore, facial expression recognition requires extracting effective information such as texture features and facial features. Extracting this effective information is crucial for improving recognition speed and accuracy. Facial feature extraction is the most important part of facial expression recognition research; the robustness and completeness of the extracted features have a decisive impact on the final recognition result. In recent years, a series of feature extraction methods have been proposed, such as Local Binary Pattern Recognition (LCR), Gabor wavelet transform, and Principal Component Analysis (PCA), which are widely used in various image processing applications. After feature extraction, facial expressions need to be classified based on the extracted features. A series of feature classification methods, such as Bayesian classifiers, support vector machines, and the K-nearest neighbor algorithm, have become well-known and widely used through continuous research.

[0005] However, these traditional feature extraction methods mainly rely on researchers manually extracting features from facial images, which is not only time-consuming and labor-intensive, but also limited by human factors, failing to fully describe facial expressions. Furthermore, traditional classification methods require many prior conditions, are prone to misclassification due to imbalanced sample classification, and are sensitive to missing data.

[0006] Today, with the ever-increasing computing power of computers and the continuous emergence of massive datasets, deep learning algorithms have flourished. Compared to traditional methods, deep learning algorithms integrate feature extraction and classification processes, reducing operational steps and automatically extracting the intrinsic features of sample data. They possess powerful feature extraction capabilities, extracting more abstract and essential features from images without requiring manual intervention from researchers, resulting in better recognition performance and robustness. They have performed exceptionally well in various computer vision-related competitions. Convolutional neural networks (DCNNs), as a crucial component of deep learning, have been widely applied in fields such as computer vision, demonstrating excellent model performance and strong robustness in tasks such as image recognition, image segmentation, and image classification. To continuously improve performance, the depth and complexity of deep DCNNs have been constantly increasing, and the construction of new, compact, and efficient DCNNs has attracted considerable attention from researchers.

[0007] While deep learning network models offer rapid recognition efficiency after training, the training phase is a lengthy process. Even with hardware acceleration such as GPUs, training deeper networks with massive amounts of data often requires researchers several days or longer. Therefore, obtaining a lightweight, fast, and highly accurate network model is a key focus of academic research.

[0008] The value and significance of facial expression feature extraction from an academic research perspective. Traditional hand-designed facial expression feature extraction methods rely on manual annotation to extract key features, which yields relatively accurate information. However, limited by manual design and small datasets, these methods are difficult to apply in practical engineering. Deep learning-based facial expression feature extraction methods eliminate the need for manual design and extraction, replacing it with large datasets for training and automatic learning of effective features. This significantly improves the accuracy and adaptability of the algorithm model. However, they suffer from high computational complexity, long processing time, and overfitting. Furthermore, most deep learning-based facial expression recognition methods do not consider the problem of imbalanced datasets, relying solely on traditional Softmax or cross-entropy loss, resulting in poor accuracy for classes with few samples. Therefore, constructing lightweight network models based on feature extraction enhancement from a deep learning perspective is of great research significance for facial expression recognition.

[0009] CN112613479A discloses an expression recognition method based on a lightweight streaming network and an attention mechanism, comprising the following steps: Step S1: Acquire an expression image dataset and preprocess it; Step S2: Construct a lightweight streaming network incorporating an attention mechanism, and perform classification using a cross-entropy loss function at the end of the network; Step S3: Train the lightweight streaming network incorporating an attention mechanism based on the preprocessed expression image dataset; Step S4: Input the image data to be tested into the trained lightweight streaming network incorporating an attention mechanism to obtain the recognition result. This invention can effectively classify expression images and improve the performance of expression image classification.

[0010] The differences between this patent and the present invention are as follows: 1. The patent is based on MobileNetV2 and ShuffleNet basic blocks to construct the basic building blocks of a lightweight streaming network, which are stacked to form the main body of the lightweight streaming network. The present invention, however, uses GhostNet as the basic model, combining the "phantom" feature map generation characteristics of the Ghost module with the channel "shuffling" and feature reuse characteristics of ShuffleNetV2 to construct a lightweight face recognition network framework—the Feature Enhancement Ghost Network Framework FGNet; 2. The attention mechanism module described in the patent mainly includes two parts: channel module attention and spatial module attention. The attention module designed in the present invention extracts spatial features of different scales by grouping the input and performing convolutions of different sizes, then extracts channel weights through effective channel attention ECA, and performs channel weighting on the spatial feature map, ultimately outputting rich multi-scale information, thereby improving the performance of the network model. 3. While the patent uses cross-entropy loss for classification, this invention addresses the issue of imbalanced class distribution in facial expression databases. To resolve this, this invention proposes adaptive class weights, calculating weights based on the number of expression samples in the database and weighting them with a custom loss function. This custom loss function consists of a weighted focus loss function and a center loss function, ensuring that network training gives equal attention to expression samples of different categories, thereby further improving the accuracy of expression recognition. Summary of the Invention

[0011] This invention aims to solve the problems of the prior art. It proposes a feature-enhanced lightweight network, FGNet, for facial expression recognition. The technical solution of this invention is as follows:

[0012] A lightweight, feature-enhanced network (FGNet) method for facial expression recognition includes the following steps:

[0013] 101. Using GhostNet as the base model, and combining the "phantom" feature map generation feature of the Ghost module with the channel "shuffling" and feature reuse features of ShuffleNetV2, a lightweight face recognition network framework—Feature Enhancement Ghost Network Framework FGNet—is constructed. Linear operations are used to generate "phantom" feature maps to optimize the number of network parameters, and channel "shuffling" is used to optimize the feature extraction capability of the network model.

[0014] 102. Utilizing the ability of the attention module in the network to extract channel and spatial features, design an attention module to extract features at different scales and output multi-scale information, and add the attention module to the network model in step 101;

[0015] 103. Calculate the weight for each type of expression sample based on the number of samples in the database, and apply the weight to a custom loss function; thereby achieving a balance in the network's training focus on each expression category.

[0016] 104. Import the FGNet network model into the facial expression recognition system to achieve real-time facial expression recognition;

[0017] The design of the FGNet network framework in step 101 is represented as follows:

[0018] A1. Employing the "ghost" module characteristics proposed in the "ghost" network, feature maps are generated. Based on the original feature maps, a series of linear transformations are applied to generate "phantom" feature maps containing the required information. Specifically, the original feature map Y' is generated by convolution of the input x, and then linear operations are applied to each original feature in Y'. Generate s "phantom" feature maps:

[0019] (1)

[0020] in It is the i-th original feature map in Y'. It is the j-th linear operation used to generate the j-th "phantom" feature map. ;

[0021] B1. The network module is designed as follows: The input x is processed through a 3×3 depthwise separable convolution layer to obtain the feature matrix. Then, after a 1×1 2D convolution, we get... By utilizing the properties of the hourglass block, a 1×1 2D convolution is performed to obtain... The feature matrix is ​​obtained after passing through a 3×3 depthwise separable convolution layer. By utilizing the identity mapping property of ResNet, the input x is compared with... Perform an identity mapping to obtain Then and The first layer output is obtained after feature fusion. , will get As the input x to the next depthwise separable convolution, this module implements nested loops. After passing through n layers of the network, the output is... ;

[0022] C1. At the input, channel slicing is used to divide the input feature map into two branches in the channel dimension: the number of channels is a and Aa respectively. The left branch is mapped equally, and the right branch contains the network module in step A, and the input and output channels are the same. The outputs of the two branches are concatted, and finally the channels of the concat result are shuffled to ensure that the two branches communicate with each other.

[0023] D1. Input the final output features from step C1 into the attention module designed in step 102 to obtain a multi-scale feature map;

[0024] E1. The loss function designed in step 103 is used at the end of the network to classify facial expression images and obtain the final recognition accuracy.

[0025] In step 102, an attention module is designed to extract features at different scales and output rich multi-scale information, specifically as follows:

[0026] A2. First, construct multi-scale features. Assuming the input is X, first split the input X into S parts. , Let S represent the S-th group after splitting. Then, features of different scales are extracted from different parts, and finally, the extracted multi-scale features are concatenated using Concat. The above process is shown in the following formula:

[0027] (2)

[0028] (3)

[0029] (4)

[0030] In the formula F is the feature map output for each part, Conv is the convolution operation, K is the size of the convolution kernel, and G is the grouping size of the channels.

[0031] B2. An attention vector is obtained through effective channel attention (ECA) to extract features at different scales; that is, attention weights are extracted for different parts of the features, and the output attention vector is... It can be represented as:

[0032] (5)

[0033] C2. To achieve attention information interaction and integrate cross-dimensional information, the attention vectors are concatenated:

[0034] (6)

[0035] D2. Next, Softmax is used to recalibrate the attention vector. The corrected attention vector A is defined as follows:

[0036] (7)

[0037] E2. Apply the corrected attention vector to the multi-scale feature map and use the result as the output. The output Y is defined as follows:

[0038] (8);

[0039] Step 103 proposes adaptive class weights and weights them to a custom loss function, specifically as follows:

[0040] A3. Focal Loss is a loss function that optimizes the cross-entropy loss function by adding parameters to assign weights to easy and difficult samples. First, the binary classification focal loss function is extended to multi-class tasks. The formula for the multi-class focal loss function s is as follows:

[0041] (9)

[0042] In the formula The probability of a sample belonging to a certain category is predicted by the model, representing the difficulty of classifying the sample. When, it is an easily classifiable sample, when At that time, it was an easily classifiable sample; The balancing parameter represents the adjustment weight for positive samples when calculating the loss. As the attenuation parameter, it dominates the overall trend. When setting the parameter value, it varies with... When it increases, The corresponding reduction is necessary.

[0043] B3. To address the problem of class imbalance in samples, an adaptive class weighting system was designed. , The calculation is based on the total number of samples in the emoji library and the number of samples in each category, and is performed using the following formula:

[0044] (10)

[0045] in The number of categories in the emoji database. , For the first Class weights For the first The number of samples in each class;

[0046] C3. Apply the category weights obtained from formula (10) The weighted focus loss function is obtained by weighting the multi-class focus loss function. The formula is as follows:

[0047] (11)

[0048] D3. The cascaded center loss function reduces the distance between samples within the same class and increases the distance between samples from different classes. The definition is as follows:

[0049] (12)

[0050] in, This represents the center of the y-th category. For the sample size, Let i be the feature vector of the i-th class of samples;

[0051] E3. The obtained weighted focal loss function and center loss function are used to jointly supervise the training of the deep neural network, and the final loss function is... The definition is as follows:

[0052] = = + (13)

[0053] In the formula Used to balance the loss function as well as Further optimize the recognition results.

[0054] Furthermore, the process of importing the FGNet network model into the facial expression recognition system to achieve facial expression recognition is as follows:

[0055] A. First, build a facial expression recognition system based on a deep convolutional neural network using Python. The system program is implemented using Keras, OpenCV, and PyQt5 libraries, and controls such as "Select Model", "Real-time Camera", "Open Image", "Time Taken" and "Recognition Result" are created in the UI interface.

[0056] B. Map the operation behavior of the corresponding UI control to the corresponding function to perform the operation;

[0057] C. First, click the "Select Model" button, then select the model file—the FGNet model—and perform recognition based on that model. Click "Real-time Camera" to open the camera and recognize facial expressions in the real-time image, or click the "Select Image" button and select a face image to recognize the expressions within it. After the system recognizes the expressions, it can display the time taken for this recognition and the recognition result with the highest probability, and show the recognition probability of each type of expression.

[0058] The advantages and beneficial effects of this invention are as follows:

[0059] The innovation of this invention mainly lies in steps 101, 102, and 103.

[0060] The innovation of step 101 lies in the following: Based on the "ghost" module of the basic model, this invention employs channel segmentation, channel merging, and channel "shuffling" operations to enhance the network's extraction and reuse of channel features. It also utilizes the hourglass block's characteristic of first reducing dimensionality and then increasing it to construct network modules, reducing feature loss during module extraction while enhancing the module's feature extraction capabilities. The constructed lightweight network model maintains a good balance between speed and accuracy when applied to facial expression recognition, and the lightweight network can be well deployed on mobile devices.

[0061] The innovation of step 102 lies in the fact that this invention uses grouped convolution to extract multi-scale spatial features and employs a more effective channel attention (ECA) to extract feature weights and weight them onto the multi-scale feature map. Compared with the traditional convolutional neural network attention module CBAM, it can output richer multi-scale feature information, thereby improving the performance of the network model.

[0062] The innovation of step 103 lies in the fact that this invention addresses the problem of imbalanced sample categories in current facial expression databases. Based on this, it proposes an adaptive category weighting method, which calculates weights based on the number of expression samples in the database and weights them to a custom loss function. This custom loss function consists of a weighted multi-class focus loss function and a center loss function. When applied to facial expression recognition, it ensures that different categories of expression samples receive equal attention during network training, thereby further improving the accuracy of various expression recognition methods. Attached Figure Description

[0063] Figure 1 This invention provides a preferred embodiment of the FGNet lightweight network framework diagram;

[0064] Figure 2 This is an example of the design process for the attention module;

[0065] Figure 3 This is a statistical analysis of the number of samples in a commonly used emoji library;

[0066] Figure 4 This is a demonstration image of the facial expression recognition system. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0068] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0069] This invention uses a feature-enhanced lightweight convolutional neural network as a framework, designs a lightweight network FGNet, and designs a lightweight and efficient attention module to act on the network. Secondly, it proposes adaptive class weights and weights them on a custom loss function, which are applied at the end of the network layer.

[0070] The methods in the embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein the drawings are only a part of the embodiments of the present invention.

[0071] As attached Figure 1 As shown, this invention designs a lightweight network model based on the Keras neural network framework—FGNet (Features Enhanced Ghost Network). This model builds a lightweight network on the basis of the "ghost" module of the GhostNet network, and adds channel split, channel concat, and channel shuffle to enhance the network's feature extraction capabilities. The design process of its network framework is as follows:

[0072] 1) Because mainstream CNN networks have significant redundancy when computing intermediate feature maps, the "ghost" module feature proposed in "ghost" networks is adopted to generate more feature maps through lower-cost operations. A series of linear transformations are applied to the original feature maps to generate many "phantom" feature maps containing the required information at a very low cost. Specifically, the original feature map Y' is generated by convolving the input x once, and then linear operations are applied to each original feature in Y'. Generate s "phantom" feature maps:

[0073] (1)

[0074] in It is the i-th original feature map in Y'. It is the j-th linear operation used to generate the j-th "phantom" feature map. ;

[0075] 2) The specific design of the network module is as follows: The input x is processed through a 3×3 depthwise separable convolution layer to obtain the feature matrix. (x), then after a 1×1 2D convolution, we get (x), by utilizing the properties of the hourglass block, is obtained through a 1×1 2D convolution. (x), after passing through a 3×3 depthwise separable convolution layer, yields the feature matrix. (x), utilizing the identity mapping property of ResNet, connects the input x with... Perform an identity mapping on (x) to obtain Then (x) and The first layer output is obtained after feature fusion. , will get As the input x to the next depthwise separable convolution, this module implements nested loops. After passing through n layers of the network, the output is... ;

[0076] 3) Based on the above-designed network modules, to enhance the network's ability to extract and reuse channel features, channel segmentation is used at the input to divide the input feature map into two branches in the channel dimension: the number of channels is 'a' and Aa respectively. The left branch performs the same mapping, and the right branch contains the network module from step A, with the input and output channels being the same. The outputs of the two branches are concatted, and finally, the channels of the concat result are shuffled to ensure information exchange between the two branches.

[0077] 4) Input the final output feature from step C into the appendix. Figure 2 Multi-scale feature maps are obtained from the attention module designed in the middle;

[0078] 5) A custom loss function is used at the end of the network to classify facial expression images and obtain the final recognition accuracy.

[0079] Furthermore, an efficient facial expression recognition network requires an effective attention module to enhance its ability to extract channel and spatial features. To address this, this invention designs a lightweight and efficient attention module. It first extracts features at different scales through grouped convolutions, then performs channel weighting using effective channel attention, ultimately outputting rich multi-scale information, thereby improving the network model's performance. The design process of the attention module is shown in the attached figure. Figure 2 As shown:

[0080] 1) First, construct multi-scale features. Assuming the input is X, we first split the input into S parts. Then, features at different scales are extracted from different parts, and finally, the extracted multi-scale features are concatenated using Concat. The above process is shown in the following formula:

[0081] (2)

[0082] (3)

[0083] (4)

[0084] In the formula F is the feature map output for each part, Conv is the convolution operation, K is the size of the convolution kernel, and G is the grouping size of the channels.

[0085] 2) Based on the above features, channel-level attention vectors are obtained through an effective channel attention ECA method to extract features at different scales. That is, attention weights are extracted for different parts of the features, and the output attention vectors are... It can be represented as:

[0086] (5)

[0087] 3) To better achieve attention information interaction and integrate cross-dimensional information, we concatenate the attention vectors obtained above:

[0088] (6)

[0089] 4) Next, Softmax is used to recalibrate the channel attention vectors obtained above. The corrected attention vector A is defined as follows:

[0090] (7)

[0091] 5) Apply the corrected attention vector to the multi-scale feature map and use the result as the output. The output Y is defined as follows:

[0092] (8)

[0093] Further details are attached. Figure 3 As shown, facial expression databases suffer from class imbalance. To address this issue, this invention proposes adaptive class weights, which are calculated based on the number of expression samples in the database and then weighted by a custom loss function. The custom loss function consists of a Focal Loss function and a Center Loss function. This loss ensures that network training gives equal attention to expression samples of different classes, thereby further improving the accuracy of expression recognition. The loss function is designed as follows:

[0094] 1) The focus loss function is an optimized version of the cross-entropy loss function. Its innovation lies in adding parameters to assign weights to easily distinguishable positive or negative samples. Smaller weights are assigned to easily distinguishable positive or negative samples, while larger weights are assigned to difficult-to-distinguish samples. First, the binary focus loss function is extended to multi-class tasks. The formula for the multi-class focus loss function is as follows:

[0095] (9)

[0096] In the formula The probability of a sample belonging to a certain category is predicted by the model, representing the difficulty of classifying the sample. When, it is an easily classifiable sample, when At that time, it was an easily classifiable sample; The balancing parameter represents the adjustment weight for positive samples when calculating the loss. As the attenuation parameter, it dominates the overall trend. When setting the parameter value, it varies with... When it increases, The corresponding reduction is necessary.

[0097] 2) To address the problem of class imbalance in samples, an adaptive class weight was designed. , The calculation is based on the total number of samples in the emoji library and the number of samples in each category, and is performed using the following formula:

[0098] (10)

[0099] in The number of categories in the emoji database. , For the first Class weights For the first The number of samples in each class;

[0100] 3) The category weights obtained from formula (9) The weighted focus loss function is obtained by weighting the multi-class focus loss function, as shown in the following formula:

[0101] (11)

[0102] 4) Simultaneously, to reduce intra-class variance caused by factors such as pose and lighting, the cascaded center loss function reduces the distance between samples within the same class and increases the distance between samples from different classes. The definition is as follows:

[0103] (12)

[0104] in, This represents the center of the y-th category. For the sample size, Let i be the feature vector of the i-th class of samples;

[0105] 5) The weighted focal loss function and the center loss function obtained by formula (10) are used to jointly supervise the training of the deep neural network, and the final loss function is obtained. The definition is as follows:

[0106] = = + (13)

[0107] In the formula Used to balance the loss function as well as Further optimize the recognition results.

[0108] Furthermore, to verify the effectiveness of the model, a pre-trained lightweight network model was imported into the facial expression recognition system to perform expression recognition on static facial images and videos. The recognition results are shown in the attached figure. Figure 4 As shown:

[0109] 1) First, a face expression recognition system based on a deep convolutional neural network is built using Python. The system program is implemented using Keras, OpenCV, and PyQt5 libraries, and controls such as "Select Model", "Real-time Camera", "Open Image", "Time Taken" and "Recognition Result" are created in the UI interface.

[0110] 2) Map the operation behavior of the corresponding UI control to the corresponding function to perform the operation;

[0111] 3) First, click the "Select Model" button, then select the model file - FGNet model. Then, the system will perform recognition based on this model. You can click "Real-time Camera" to open the camera and recognize facial expressions in the real-time image, or you can click the "Select Image" button and select a face image to recognize the expressions in it. After the system recognizes the expressions, it can display the time taken for this recognition and the recognition result with the highest probability, and show the recognition probability of each type of expression.

[0112] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0113] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0114] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A lightweight, feature-enhanced FGNet facial expression recognition method, characterized in that, Includes the following steps:

101. Using GhostNet as the base model, and combining the "phantom" feature map generation feature of the Ghost module with the channel "shuffling" and feature reuse features of ShuffleNetV2, a lightweight face recognition network framework—Feature Enhancement Ghost Network Framework FGNet—is constructed. Linear operations are used to generate "phantom" feature maps to optimize the number of network parameters, and channel "shuffling" is used to optimize the feature extraction capability of the network model.

102. Utilizing the ability of the attention module in the network to extract channel and spatial features, design an attention module to extract features at different scales and output multi-scale information, and add the attention module to the network model in step 101; 103. Calculate the weight for each type of expression sample based on the number of samples in the database, and apply the weight to a custom loss function; thereby achieving a balance in the network's training focus on each expression category.

104. Import the FGNet network model into the facial expression recognition system to achieve real-time facial expression recognition; The design of the FGNet network framework in step 101 is represented as follows: A1. Employing the "ghost" module characteristics proposed in the "ghost" network, a feature map is generated. Based on the original feature map, a series of linear transformations are applied to generate a "phantom" feature map containing the required information. Specifically, the original feature map Y' is generated by convolution of the input x, and then linear operations are applied to each original feature in Y'. Generate s "phantom" feature maps: (1) in It is the i-th original feature map in Y'. It is the j-th linear operation used to generate the j-th "phantom" feature map. ; B1. The network module is designed as follows: The input x is processed through a 3×3 depthwise separable convolution layer to obtain the feature matrix. Then, after a 1×1 2D convolution, we get... By utilizing the properties of the hourglass block, a 1×1 2D convolution is performed to obtain... The feature matrix is ​​obtained after passing through a 3×3 depthwise separable convolution layer. By utilizing the identity mapping property of ResNet, the input x is compared with... Perform an identity mapping to obtain Then and The first layer output is obtained after feature fusion. , will get As the input x to the next depthwise separable convolution, this module implements nested loops. After passing through n layers of the network, the output is... ; C1. At the input, channel slicing is used to divide the input feature map into two branches in the channel dimension: the number of channels is a and Aa respectively. The left branch is mapped equally, and the right branch contains the network module in step A, and the input and output channels are the same. The outputs of the two branches are concatted, and finally the channels of the concat result are shuffled to ensure that the two branches communicate with each other. D1. Input the final output features from step C1 into the attention module designed in step 102 to obtain a multi-scale feature map; E1. The loss function designed in step 103 is used at the end of the network to classify facial expression images and obtain the final recognition accuracy. In step 102, an attention module is designed to extract features at different scales and output rich multi-scale information, specifically as follows: A2. First, construct multi-scale features. Assuming the input is X, first split the input X into S parts. , This represents the Sth group after splitting. Then, features of different scales are extracted from different parts, and finally, the extracted multi-scale features are concatenated using Concat. The above process is shown in the following formula: (2) (3) (4) In the formula F is the feature map output for each part, Conv is the convolution operation, K is the size of the convolution kernel, and G is the grouping size of the channels. B2. An attention vector is obtained through effective channel attention (ECA) to extract features at different scales; that is, attention weights are extracted for different parts of the features, and the output attention vector is... It can be represented as: (5) C2. To achieve attention information interaction and integrate cross-dimensional information, the attention vectors are concatenated: (6) D2. Next, Softmax is used to recalibrate the attention vector. The corrected attention vector A is defined as follows: (7) E2. Apply the corrected attention vector to the multi-scale feature map and use the result as the output. The output Y is defined as follows: (8)； Step 103 proposes adaptive class weights and weights them to a custom loss function, specifically as follows: A3. Focal Loss is a loss function that optimizes the cross-entropy loss function by adding parameters to assign weights to easy and difficult samples. First, the binary classification focal loss function is extended to multi-class tasks. The formula for the multi-class focal loss function s is as follows: (9) In the formula The probability of a sample belonging to a certain category is predicted by the model, representing the difficulty of classifying the sample. When, it is an easily classifiable sample, when At that time, it was an easily classifiable sample; The balancing parameter represents the adjustment weight for positive samples when calculating the loss. As the attenuation parameter, it dominates the overall trend. When setting the parameter value, it varies with... When it increases, The corresponding reduction is necessary. B3. To address the problem of class imbalance in samples, an adaptive class weighting system was designed. , The calculation is based on the total number of samples in the emoji library and the number of samples in each category, and is performed using the following formula: (10) in The number of categories in the emoji database. , For the first Class weights For the first The number of samples in each class; C3. Apply the category weights obtained from formula (10) The weighted focus loss function is obtained by weighting the multi-class focus loss function. The formula is as follows: (11) D3. The cascaded center loss function reduces the distance between samples within the same class and increases the distance between samples from different classes. The definition is as follows: (12) in, This represents the center of the y-th category. For the sample size, Let i be the feature vector of the i-th class of samples; E3. The obtained weighted focal loss function and center loss function are used to jointly supervise the training of the deep neural network, and the final loss function is... The definition is as follows: = = + (13) In the formula Used to balance the loss function as well as Further optimize the recognition results.

2. The feature-enhanced lightweight network FGNet facial expression recognition method according to claim 1, characterized in that, The process of importing the FGNet network model into the facial expression recognition system to achieve facial expression recognition is as follows: A. First, build a facial expression recognition system based on a deep convolutional neural network using Python. The system program is implemented using Keras, OpenCV, and PyQt5 libraries, and controls such as "Select Model", "Real-time Camera", "Open Image", "Time Taken" and "Recognition Result" are created in the UI interface. B. Map the operation behavior of the corresponding UI control to the corresponding function to perform the operation; C. First, click the "Select Model" button, then select the model file—the FGNet model—and perform recognition based on that model. Click "Real-time Camera" to open the camera and recognize facial expressions in the real-time image, or click the "Select Image" button and select a face image to recognize the expressions within it. After the system recognizes the expressions, it can display the time taken for this recognition and the recognition result with the highest probability, and show the recognition probability of each type of expression.

Images