An expression recognition method and system based on A-LGS and convolutional neural network

Abstract

The application discloses an expression recognition method and system based on A-LGS and a convolutional neural network, and the method comprises the following steps: collecting images with facial expression features to establish a data set; marking labels on the images in the data set; constructing an average local graph structure A-LGS based on a local graph structure LGS, and using the A-LGS to describe the local features of the images in the training set; constructing an expression recognition model based on a VGG network model; training the expression recognition model, and the training process comprises the following steps: inputting the images in the training set into the expression recognition model, extracting global features of the images, concatenating and fusing the local features of the images described by the A-LGS and the global features of the images, and testing the expression recognition model by using a test set; and the system comprises a data processing module, a feature extraction module and a model training module. The application solves the problems of deviation of the classification performance of the neural network and low classification accuracy caused by uneven sample distribution in the prior art when LGS and the VGG network are used for feature extraction of images.

Description

Technical Field

[0001] This invention relates to the field of facial expression recognition technology, and in particular to a facial expression recognition method and system based on A-LGS and convolutional neural networks. Background Technology

[0002] Facial expression recognition technology is an important research direction in the fields of computer vision and artificial intelligence. For example, it can enable robots to better understand users' emotions and intentions, thereby achieving more intelligent and natural human-computer interaction. It can also be used to assist in the diagnosis and treatment of mental and neurological diseases, and can be applied to the field of security monitoring, such as video surveillance and security checkpoints, which can help improve regulatory efficiency and security.

[0003] Existing technologies often employ Local Graph Structure (LGS) and VGG networks for image feature extraction and classification. However, LGS uses a graph structure to describe the relationship between the central pixel and surrounding pixels, which leads to problems such as unbalanced utilization of pixels on the left and right sides and weighting that is more favorable to the left than the right. VGG networks are a classic convolutional neural network model that can be used for image classification and feature extraction, but their convergence speed is relatively slow. Directly using this network may result in imbalanced sample distribution, leading to classification bias. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide an expression recognition method and system using A-LGS and convolutional neural networks with high accuracy in image classification.

[0005] Technical Solution: To achieve the above objectives, the present invention provides an expression recognition method based on A-LGS and convolutional neural networks, comprising the following steps:

[0006] Step S1: Collect images with facial expression features and build a dataset;

[0007] Step S2: Label the images in the dataset;

[0008] Step S3: Construct the average local graph structure A-LGS based on the local graph structure LGS, and use A-LGS to describe the local features of the images in the training set.

[0009] Step S4: Construct an expression recognition model based on the VGG network model;

[0010] Step S5: Train the facial expression recognition model. The training process includes inputting images from the training set into the facial expression recognition model, extracting global features of the images, and concatenating and fusing the local features and global features of the images described by A-LGS.

[0011] Step S6: Test the facial expression recognition model using the test set.

[0012] In step S1, collecting images with facial expression features means collecting RGB images with facial expression features and removing backgrounds unrelated to facial expressions.

[0013] In step S2, the labeling refers to classifying facial expressions in all images into 7 categories: Surprise, Fear, Disgust, Happiness, Sadness, Anger, and Neutral, which are labeled as 0 to 6 respectively.

[0014] In step S3, constructing the average local graph structure A-LGS based on the local graph structure LGS means using two different graph structures to replace one graph structure in the local graph structure (LGS) to describe the relationship between the center pixel and the surrounding pixels of the image. Then, the binary value of the local graph structure LGS is calculated and multiplied by the corresponding weight to obtain two feature values. The average of the two feature values ​​is taken as the feature value of the center pixel to establish the average local graph structure A-LGS.

[0015] In step S3, describing the local features of an image using the Average Local Graph Structure (A-LGS) means uniformly cutting the image into A-LGS images, then statistically analyzing the feature values ​​within the A-LGS images to obtain a feature histogram, which serves as the local feature information of the image.

[0016] Among them, the construction of the expression recognition model based on the VGG network model in step S4 refers to the introduction of the loss function L-SoftMax and the focus loss Focal Loss into the VGG network model to construct the expression recognition model.

[0017] The loss function L-SoftMax is:

[0018]

[0019] In the formula,

[0020] The Focal Loss is:

[0021] L f =-α(1-p i ) γ lnp i ,

[0022] In the formula, The probability of the label corresponding to the model's predicted result; x represents the i-th sample, W yiLet θ represent the yi-th column of W. yi W yi The angle between α and xi, m is the coefficient that controls the classification boundary; α is the balance parameter, γ is the focusing parameter, which is a hyperparameter greater than or equal to 0, used to control the weights of easy-to-classify and hard-to-classify samples, and λ is the weight coefficient.

[0023] After introducing the L-SoftMax loss function and Focal Loss, the final loss function obtained by adaptively adjusting the weight coefficients of the VGG network is:

[0024] L = L i +λL f .

[0025] The step S5, which involves training the facial expression recognition model, refers to training the model using the ten-fold cross-validation method. This involves dividing the dataset into ten equal-sized subsets, taking one subset as the validation set, and using the remaining nine subsets as the training set. This process is repeated ten times, with a different subset used as the validation set each time, and finally obtaining the average value of the evaluation results of the ten facial expression recognition models.

[0026] In step S5, inputting images from the dataset into the facial expression recognition model and extracting global features of the images refers to using the adaptive learning capability of the neural network of the facial expression recognition model to extract features of the images. The convolutional layer continuously learns different features on the same image, and the fully connected layer completely stitches together the extracted features to obtain the global features of the image.

[0027] The serial splicing and fusion method described in step S5 is: X = concat[Xdeep, Xalgs], where Xdeep is the global feature vector; Xalgs is the local feature vector; and concat[*] is the serial splicing and fusion function.

[0028] The present invention also provides an expression recognition system based on a convolutional neural network of A-LGS, including a data processing module, a feature extraction module, and a model training module;

[0029] The data processing module is used to collect images with facial expression features and to label the images;

[0030] The feature extraction module is used to extract local feature information from an image;

[0031] The model training module includes building an expression recognition model based on the VGG network model and training it.

[0032] Beneficial effects: The present invention has the following advantages:

[0033] 1. This invention employs a combination of A-LGS and VGG networks for facial expression recognition. Specifically, it constructs an average local graph structure (A-LGS) based on local graph structure (LGS) and uses A-LGS to describe local features of images in the training set. Simultaneously, it builds an facial expression recognition model based on a VGG network model to extract global image features. Then, it concatenates and fuses the local and global image features, fully utilizing the advantages of both methods. This enables the facial expression recognition model to analyze and recognize images more accurately, thereby improving the model's recognition accuracy, generalization, and robustness.

[0034] 2. The VGG network model used in this invention has a simple structure and relatively small number of parameters, making it suitable for image classification tasks. By introducing the loss function L-SoftMax and the focal loss to construct the expression recognition model, the problem of uneven distribution of sample numbers is solved. This not only improves the classification accuracy of the model but also makes the model structure more stable and reliable. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0036] Figure 2 This is a schematic diagram of the average local graph structure A-LGS.

[0037] Figure 3 This is a schematic diagram illustrating the facial expression recognition process during model training.

[0038] Figure 4 This is a schematic diagram showing the results of testing images using an facial expression recognition model. Detailed Implementation

[0039] The technical solution of the present invention will be described in detail below with reference to the embodiments and accompanying drawings.

[0040] like Figure 1 As shown, the facial expression recognition method based on A-LGS and convolutional neural networks of the present invention includes the following steps:

[0041] Step S1: Collect images with facial expression features; this means collecting RGB images with facial expression features and removing backgrounds unrelated to facial expressions.

[0042] Step S2: Label the images in the dataset; the labeling refers to classifying facial expressions in all images into 7 categories: Surprise, Fear, Disgust, Happiness, Sadness, Anger, and Neutral, and labeling them as 0 to 6 respectively.

[0043] Step S3: Construct the average local graph structure A-LGS based on the local graph structure LGS, and use A-LGS to describe the local features of the images in the training set.

[0044] Among them, constructing the average local graph structure (A-LGS) based on the local graph structure (LGS) involves using two different graph structures to replace one of the graph structures in the local graph structure (LGS) to describe the relationship between the center pixel and surrounding pixels. Then, the binary values ​​of the local graph structure (LGS) are calculated separately and multiplied by their respective weights to obtain two feature values. The average of these two feature values ​​is taken as the feature value of the center pixel, thus establishing the average local graph structure (A-LGS). Figure 2 As shown.

[0045] The use of Average Local Graph Structure (A-LGS) to describe local image features involves uniformly dividing the image into A-LGS images, then statistically analyzing the feature values ​​within the A-LGS images to obtain feature histograms, which serve as local feature information of the image.

[0046] Step S4: Construct an expression recognition model based on the VGG network model; this refers to introducing the loss function L-SoftMax and the focal loss into the VGG network model to construct an expression recognition model.

[0047] The loss function L-SoftMax is:

[0048]

[0049] In the formula,

[0050] The Focal Loss is:

[0051] L f =-α(1-p i ) γ lnp i ,

[0052] In the formula, The probability of the label corresponding to the model's predicted result; x represents the i-th sample, W yi Let θ represent the yi-th column of W. yi W yi The angle between α and xi, m is the coefficient that controls the classification boundary; α is the balance parameter, γ is the focusing parameter, which is a hyperparameter greater than or equal to 0, used to control the weights of easy-to-classify and hard-to-classify samples, and λ is the weight coefficient.

[0053] After introducing the L-SoftMax loss function and the FocalLoss loss, the final loss function obtained by adaptively adjusting the weight coefficients of the VGG network is:

[0054] L = L i +λL f .

[0055] Step S5: Train the facial expression recognition model, such as... Figure 3 As shown, the training process includes inputting images from the training set into the facial expression recognition model, extracting global image features, and concatenating and fusing the local and global image features described by A-LGS.

[0056] Training the facial expression recognition model involves using a ten-fold cross-validation method. This method divides the dataset into ten equal subsets, takes one subset as the validation set, and uses the remaining nine subsets as the training set. This process is repeated ten times, with a different subset used as the validation set each time. Finally, the average value of the evaluation results of the ten facial expression recognition models is obtained.

[0057] The process of inputting images from the dataset into the facial expression recognition model and extracting global features involves using the adaptive learning capability of the neural network in the facial expression recognition model to extract image features. The convolutional layer continuously learns different features from the same image, and the fully connected layer concatenates the extracted features to obtain the global features of the image.

[0058] The concatenation and fusion method is: X = concat[Xdeep, Xalgs], where Xdeep is the global feature vector; Xalgs is the local feature vector; and concat[*] is the concatenation and fusion function.

[0059] Step S6: Test the facial expression recognition model using the test set.

[0060] like Figure 4 As shown, the image shows the test results of the facial expression recognition model. The iteration batch size was set to 120, the batch size was set to 5, the initial learning rate was 0.001, and the final accuracy was 93.939% using the 10-fold cross-validation method. This demonstrates that the method of combining A-LGS and VGG networks for facial expression recognition—that is, constructing an average local graph structure A-LGS based on the local graph structure LGS, using A-LGS to describe the local features of the images in the training set, and simultaneously building an facial expression recognition model based on the VGG network model to extract global image features—is effective. The method of concatenating and fusing the local and global image features is also effective.

[0061] The present invention also provides an expression recognition system based on a convolutional neural network of A-LGS, including a data processing module, a feature extraction module, and a model training module;

[0062] The data processing module is used to collect images with facial expression features and to label the images;

[0063] The feature extraction module is used to extract local feature information from an image;

[0064] The model training module includes building an expression recognition model based on the VGG network model and training it.

[0065] This invention employs a combination of A-LGS and VGG networks for facial expression recognition. Specifically, it constructs an average local graph structure (A-LGS) based on local graph structure (LGS) and uses A-LGS to describe local features of images in the training set. Simultaneously, it builds an facial expression recognition model based on a VGG network model to extract global image features. The local and global image features are then concatenated and fused, fully leveraging the advantages of both methods. This allows the facial expression recognition model to analyze and recognize images more accurately, thereby improving the model's recognition accuracy, generalization, and robustness.

[0066] This invention introduces the L-SoftMax loss function and Focal Loss into the VGG network model to construct an expression recognition model. The L-SoftMax loss function can better handle class imbalance and improve the model's classification accuracy. Focal Loss, by weighting difficult samples, reduces the influence of easy samples, thus focusing on learning those important but difficult-to-distinguish samples, improving the model's robustness and generalization ability. The VGG network model has a simple structure and relatively small number of parameters, making it suitable for image classification tasks. Introducing the L-SoftMax loss function and Focal Loss not only improves the model's classification accuracy but also makes the model's structure more stable and reliable.

Claims

1. A facial expression recognition method based on A-LGS and convolutional neural network, characterized in that, The method comprises the following steps: Step S1: collecting images with facial expression features to establish a data set; Step S2: labeling the images in the data set and dividing them into a training set and a test set; Step S3: constructing an average local graph structure A-LGS based on a local graph structure LGS, and using the A-LGS to describe the local features of the images in the training set, specifically, cutting the images uniformly, processing them into A-LGS images respectively, and then statistically obtaining feature histograms in the A-LGS images as the local feature information of the images, wherein the average local graph structure A-LGS is constructed by using two different graph structures to replace one graph structure in the local graph structure LGS to describe the relationship between the center pixel and the surrounding pixels of the image, then calculating two feature values by multiplying the binary values of the local graph structure LGS by the corresponding weights respectively, and taking the average of the two feature values as the feature value of the center pixel; Step S4: constructing an expression recognition model based on a VGG network model, specifically, introducing a loss function L-SoftMax and a focal loss Focal Loss into the VGG network model to construct the expression recognition model; Step S5: training the expression recognition model, which comprises inputting the images in the training set into the expression recognition model, extracting the global features of the images, and concatenating and fusing the local features of the images described by the A-LGS and the global features of the images; Step S6: testing the expression recognition model using the test set. 2.The A-LGS and convolutional neural network based expression recognition method of claim 1, wherein, The collection of images with facial expression features in step S1 refers to the collection of RGB images with facial expression features, and the removal of backgrounds irrelevant to facial expressions. 3.The A-LGS and convolutional neural network based expression recognition method of claim 1, wherein, The label marking in step S2 refers to the division of facial expressions in all images into 7 categories: surprise, fear, disgust, happiness, sadness, anger, and normal, marked as 0-6 respectively. 4.The A-LGS and convolutional neural network based expression recognition method of claim 1, wherein, The loss function L-SoftMax is: ， In the formulae, ; The focal loss Focal Loss is: ， In the formula, is the probability of the model prediction result corresponding to the label; represents the i-th sample, represents the i-th sample, represents the i-th sample, represents the i-th sample, represents the i-th sample, represents the i-th sample, represents the i-th sample, represents the i-th sample, is a coefficient for controlling the classification boundary; is a balance parameter, is a focusing parameter, which is a hyperparameter greater than or equal to 0, used to control the weights of easy-to-classify samples and difficult-to-classify samples, is a weight coefficient; After introducing the loss function L-SoftMax and the focal loss Focal Loss, the final loss function obtained by adaptive adjustment of the VGG network weight coefficient is: 。 5.The A-LGS and convolutional neural network based expression recognition method of claim 1, wherein, The training of the expression recognition model in step S5 refers to the use of ten-fold cross-validation method to train the expression recognition model, that is, dividing the data set into ten equal-sized subsets, taking one subset as the validation set and the remaining nine subsets as the training set, and repeating the process ten times, each time using a different subset as the validation set, and finally obtaining the average value of the ten expression recognition model evaluation results. 6.The A-LGS and convolutional neural network based expression recognition method of claim 1, wherein, The input of the images in the data set into the expression recognition model in step S5 refers to the use of the neural network adaptive learning capability of the expression recognition model to extract the features of the images, and the convolution layer continuously learns different features from the same image, and the fully connected layer completely splices the extracted features to obtain the global features of the image. 7.The A-LGS and convolutional neural network based expression recognition method of claim 1, wherein, The series splicing fusion mode described in step S5 is: wherein, is a global feature vector; is a local feature vector, is a series splicing fusion function.

8. A system suitable for the expression recognition method of claim 1 based on A-LGS-based convolutional neural network, characterized in that, The method comprises a data processing module, a feature extraction module, and a model training module. The data processing module is used for collecting images with facial expression features and marking the images with labels. The feature extraction module is used for extracting local feature information of the images. The model training module includes constructing an expression recognition model based on a VGG network model and training the model.

Images