Facial expression prediction method and device, medium and electronic equipment
Through the circular residual graph attention network model, facial expression recognition is performed in an undesirable environment, combined with local and global features, the problem of low facial expression recognition accuracy in the existing technology is solved, and higher recognition accuracy is achieved and robustness.
Patent Information
- Application Number
- CN202410996456.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-12-03
AI Technical Summary
The accuracy of facial expression recognition of facial images acquired in undesirable environments is low, especially in complex environments such as security monitoring and driver assistance systems, which are difficult to achieve accurate recognition.
Using the circular residual graph attention network model, the graph data set is constructed by obtaining the image nodes and their spatial connection relationships and node features of the face image, and the attention mechanism is used to allocate dynamic weights to perform feature learning to realize facial expression recognition. . The model combines local features and global context information to improve the accuracy and reliability of identification.
It significantly improves the accuracy and sensitivity of facial expression recognition, can resist noise and irrelevant information interference in complex environments, and provides stronger robustness and accuracy.
Smart Images

Figure CN119068522A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of facial expression recognition, and in particular to a facial expression prediction method, device, medium and electronic equipment. Background Art
[0002] Facial expression recognition is an important human-computer interaction technology that can be applied to many fields such as mental health analysis, security monitoring, and driver assistance systems. However, human expressions are rich and varied. The same expression may show different characteristics in different people, and even the same expression of the same person in different situations may be different. This puts higher requirements on the generalization ability of the algorithm and brings challenges to the application of facial expression recognition.
[0003] In the technology of facial expression recognition, facial images may be affected by various factors such as lighting changes, facial occlusion or image quality differences, resulting in noise and irrelevant information that greatly interferes with the accuracy of expression recognition results, making the accuracy of facial expression recognition for facial images acquired in complex and non-ideal environments low. In practical applications, such as security monitoring, driver assistance systems and other fields, it is often necessary to perform expression recognition on facial images acquired in complex and non-ideal environments.
[0004] Therefore, it is necessary to provide a facial expression prediction method that is robust and can accurately recognize facial expressions. Summary of the invention
[0005] The purpose of the present invention is to provide a facial expression prediction method, device, medium and electronic equipment to solve the problem of low accuracy of facial expression recognition for facial images obtained under non-ideal environments in the prior art.
[0006] In a first aspect, the facial expression prediction method provided by the present invention includes: obtaining m facial images, cutting the m facial images respectively to obtain image nodes of the facial images, m>1; obtaining the spatial connection relationship and node features of the image nodes, and encapsulating the spatial connection relationship and node features according to the image nodes corresponding to each image to generate a graph data set; constructing a cyclic residual graph attention network model, the cyclic residual graph attention network model is used to assign attention weights to the image nodes according to the graph data set, and perform feature learning according to the attention weight assignment results to realize facial expression recognition; inputting the test set obtained according to the division of the graph data set into the cyclic residual graph attention network model for model training and optimization to obtain a facial expression prediction model, and the facial expression prediction model is used to perform facial expression recognition on the image to be recognized to obtain an expression prediction result.
[0007] The beneficial effect of the facial expression prediction method provided by the present invention is that the constructed cyclic residual graph attention network model adopts the attention mechanism to assign dynamic weights to image nodes, so that the model can automatically emphasize more important neighbor nodes during the feature aggregation process, so that it can more accurately focus on the facial features that contribute the most to the recognition task, thereby improving the accuracy and sensitivity of expression recognition. It can effectively combine local features with global contextual information. This means that when performing facial expression recognition, the cyclic residual graph attention network model constructed by the present invention not only considers local facial movements, but also takes into account the contextual information of the overall facial expression. This comprehensive feature representation provides strong support for complex expression recognition decisions and significantly improves the accuracy and reliability of recognition.
[0008] In a possible embodiment, obtaining the spatial connection relationship of image nodes includes obtaining the connection relationship between the image nodes and the corresponding adjacent nodes in the up, down, left, right, and diagonal directions; performing spatial connection relationship and node feature encapsulation according to the image nodes corresponding to each image to generate a graph data set, including: applying a graph neural network framework to create a data set, encapsulating Data objects according to the image nodes corresponding to each image, wherein a Data object includes the image nodes of a face image and the corresponding spatial connection relationship and node features, and generating a graph data set according to the encapsulated Data objects.
[0009] In another possible embodiment, the cyclic residual graph attention network model includes a cyclic residual graph attention layer, an aggregation layer and a linear layer; wherein the cyclic residual graph attention layer is used to assign attention weights between each image node and its neighboring nodes according to the spatial connection relationship and node features of the image nodes, and obtain a new feature representation of the image node according to the attention weight allocation result, and after fusing the new feature representation with the initial input features of the corresponding image node, a nonlinear factor is introduced to obtain the target feature representation; the aggregation layer is used to aggregate all target feature representations to obtain a graph-level feature representation; the linear layer is used to map the graph-level feature representation to a facial expression classification label.
[0010] In other possible embodiments, the training set obtained according to the division of the graph data set is input into the cyclic residual graph attention network model for model training and optimization to obtain a facial expression prediction model, including: inputting the test set into the cyclic residual graph attention network model, training the cyclic residual graph attention network model according to a preset model training optimizer; calculating the model output accuracy after the cyclic residual graph attention network model completes a round of training; when the accuracy reaches a set threshold, the training is completed to obtain a trained facial expression prediction model.
[0011] In a second aspect, the present invention also provides a facial expression prediction device, which includes: an image processing unit, used to obtain m facial images, and cut the m facial images respectively to obtain image nodes of the facial images, m>1; a graph data set generation unit, used to obtain the spatial connection relationship and node features of the image nodes, and encapsulate the spatial connection relationship and node features according to the image nodes corresponding to each image to generate a graph data set; a model construction unit, used to construct a cyclic residual graph attention network model, the cyclic residual graph attention network model is used to assign attention weights to the image nodes according to the graph data set, and perform feature learning according to the attention weight assignment results to realize facial expression recognition; a prediction unit, used to input the test set obtained according to the division of the graph data set into the cyclic residual graph attention network model for model training and optimization to obtain a facial expression prediction model, and the facial expression prediction model is used to perform facial expression recognition on the image to be recognized to obtain an expression prediction result.
[0012] Among them, the graph data set generation unit obtains the spatial connection relationship of the image nodes, including obtaining the connection relationship between the image nodes and the corresponding adjacent nodes in the up, down, left, right and diagonal directions; the spatial connection relationship and node feature encapsulation are performed according to the image nodes corresponding to each image to generate a graph data set, including: applying the graph neural network framework to create a data set, encapsulating Data objects according to the image nodes corresponding to each image, a Data object includes the image nodes of a face image and the corresponding spatial connection relationship and node features, and generating a graph data set according to the encapsulated Data objects.
[0013] The cyclic residual graph attention network model constructed by the model construction unit includes a cyclic residual graph attention layer, an aggregation layer and a linear layer; wherein the cyclic residual graph attention layer is used to assign attention weights between each image node and its neighboring nodes according to the spatial connection relationship and node features of the image nodes, and obtain a new feature representation of the image node according to the attention weight allocation result, and after fusing the new feature representation with the initial input features of the corresponding image node, a nonlinear factor is introduced to obtain a target feature representation; the aggregation layer is used to aggregate all target feature representations to obtain a graph-level feature representation; the linear layer is used to map the graph-level feature representation to a facial expression classification label.
[0014] The prediction unit inputs the training set obtained according to the division of the graph data set into the cyclic residual graph attention network model for model training and optimization to obtain a facial expression prediction model, including: inputting the test set into the cyclic residual graph attention network model, and training the cyclic residual graph attention network model according to a preset model training optimizer; calculating the model output accuracy after the cyclic residual graph attention network model completes a round of training; when the accuracy reaches a set threshold, the training is completed to obtain a trained facial expression prediction model.
[0015] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the above-mentioned facial expression prediction method when executed by a processor.
[0016] In a fourth aspect, the present invention also provides an electronic device, comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-mentioned facial expression prediction method.
[0017] For the beneficial effects of the second to fourth aspects, reference may be made to the description of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 A schematic diagram of a flow chart of a facial expression prediction method provided by an embodiment of the present invention;
[0019] Figure 2 A schematic diagram of executing the facial expression prediction method provided by an embodiment of the present invention;
[0020] Figure 3 Schematic diagram of the inner workings of the graph attention convolution.
[0021] Figure 4 A schematic diagram of the architecture of the cyclic residual graph attention layer provided in an embodiment of the present invention;
[0022] Figure 5 A schematic diagram of a facial expression prediction device provided by an embodiment of the present invention;
[0023] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.
[0025] In view of the problems existing in the prior art, embodiments of the present invention provide a method, device, medium and electronic device for predicting facial expressions.
[0026] This embodiment provides a method for predicting facial expressions. Figures 1 to 4 , the method comprising:
[0027] S101: Obtain m face images, and perform segmentation processing on the m face images to obtain image nodes of the face images, where m>1.
[0028] In S101, in a possible embodiment, m face images (m>1) are obtained, and the m face images are respectively cut to obtain image nodes of the face images. The cutting process steps for each face image in the m images are the same, including: the face image is evenly cut into n×n small blocks, wherein the specific value of n is determined according to the size of the original image and the level of detail to be captured.
[0029] The size of each face image is determined based on the size of the original image and the level of detail to be captured. This segmentation method is conducive to capturing the local characteristics of facial expressions in detail, which is particularly critical for expression recognition. Such block division not only helps to better understand the subtle changes in facial parts when expressing emotions, but also provides a rich source of local information for subsequent models.
[0030] S102: Obtain the spatial connection relationship and node features of the image nodes, and encapsulate the spatial connection relationship and node features according to the image nodes corresponding to each image to generate a graph data set.
[0031] In S102, in a possible embodiment, each small block obtained after cutting a face image is regarded as an image node in the face image, wherein the feature of the image node is composed of its original pixel value. The connection method between the image nodes is based on their spatial connection method, and the method based on spatial connection between image nodes can reflect the relationship between local areas in the image.
[0032] Exemplarily, the spatial connection relationship of the image nodes includes the connection relationship between the image nodes and the corresponding adjacent nodes in the up, down, left, right, and diagonal directions. In practical applications, the spatial connection relationship of the image nodes to be acquired can be selected according to the specific situation of the image. For example, when the goal is to obtain a complex spatial relationship, the spatial connection relationship of the image nodes to be acquired is to obtain the connection relationship between the image nodes and the corresponding adjacent nodes in the up, down, left, right, and diagonal directions. When the goal is to obtain a simple spatial relationship, the spatial connection relationship of the image nodes to be acquired can be to obtain only the connection relationship between the image nodes and the corresponding adjacent nodes in the up, down, left, right, and diagonal directions; or it can be to obtain only the connection relationship between the image nodes and the corresponding adjacent nodes in the diagonal direction.
[0033] In a possible embodiment, obtaining the spatial connection relationship between image nodes includes obtaining the connection relationship between the image nodes and the corresponding adjacent nodes in the up, down, left, right, and diagonal directions. The spatial connection relationship and node feature encapsulation are performed according to the image nodes corresponding to each image to generate a graph data set, including: applying a graph neural network framework to create a data set, encapsulating a Data object according to the image nodes corresponding to each image, wherein a Data object includes the image nodes of a face image and the corresponding spatial connection relationship and node features, and generating a graph data set according to the encapsulated Data object.
[0034] In a possible embodiment, encapsulating the spatial connection relationship and node features corresponding to each image to generate a graph dataset includes: inheriting the InMemoryDataset method of PyTorch Geometric, instantiating a Data object, adding the node features and spatial connection relationship (edge relationship between nodes) corresponding to each image node of a face image to the same Data object. Then, using the process method and the collate method, multiple Data objects are saved locally to generate a graph dataset.
[0035] Exemplarily, the process of generating a graph data set is as follows: first, inherit the InMemory Dataset method in the PyTorch Geometric framework and instantiate the Data object. Among them, the PyTorch Geometric framework is a graph neural network framework based on PyTorch. According to the face image, the spatial connection relationship and node features of the corresponding image nodes are placed in the Data object. Specifically, the image nodes obtained by cutting a face image, and the spatial connection relationship and node features of these image nodes are placed in the same Data object, so that each Data object contains the data of a face image, including node features, edge indexes, etc. Then use the process method and collate method to save the Data object locally. Specifically, the process method will integrate the information of multiple Data together. This can facilitate the accelerated storage of data in memory. The collate method converts the integrated Data object into a tuple containing all data and corresponding slice information. The slice information is used to record the position of each original Data object after merging to ensure that the data can be correctly restored when loading. Through the above steps, we can efficiently store and process graph data sets, thus laying the foundation for subsequent deep learning processing.
[0036] In the overall concept of the present invention, a method based on a cyclic residual graph attention network is designed for facial expression recognition, so the traditional two-dimensional face image is converted into a data structure suitable for graph processing, that is, the generation of the graph data set in the present invention is very critical. Since the original face image is located in Euclidean space, the pixel arrangement therein is fixed and regular, which contrasts with the flexibility and irregularity of the graph data structure. In order to overcome this difference and make full use of the ability of graph neural networks to process complex patterns and relationships, it is necessary to reorganize the face image into a graph data structure.
[0037] S103: Construct a cyclic residual graph attention network model, which is used to assign attention weights to image nodes according to a graph data set, and perform feature learning based on the attention weight assignment results to achieve facial expression recognition.
[0038] In S103, in a possible embodiment, the cyclic residual graph attention network model includes a cyclic residual graph attention layer, an aggregation layer and a linear layer. Among them, the cyclic residual graph attention layer is used to assign attention weights between each image node and its neighboring nodes according to the spatial connection relationship and node features of the image nodes, and obtain a new feature representation of the image node according to the attention weight allocation result, and the new feature representation is fused with the initial input features of the corresponding image node and a nonlinear factor is introduced to obtain the target feature representation. The aggregation layer is used to aggregate all target feature representations to obtain a graph-level feature representation. The linear layer is used to map the graph-level feature representation to the facial expression classification label.
[0039] In a possible embodiment, the construction of the cyclic residual graph attention network model includes: using GATConv (graph attention convolution) in the PyTorchGeometric framework as the backbone to construct a cyclic residual graph attention layer (R-GATConv). Exemplarily, the GATConv of the PyTorch Geometric framework used is from GraphAttention Networks. See the attached manual Figure 4 The cyclic residual graph attention layer includes two graph residual attention layers (the first graph residual attention layer and the second graph residual attention layer), a Relu layer and a skip connection. The skip connection can fuse the initial input features of the input cyclic residual graph attention layer with the output of the graph residual attention layer.
[0040] Specifically, in the cyclic residual graph attention layer, the first graph residual attention layer is used to receive the node features and graph structure information of the input cyclic residual graph attention layer, and calculate the weight between each image node and its neighboring nodes through the attention mechanism to determine the importance of the neighboring nodes. Then, the features of the neighboring nodes are weighted and summed according to the attention weights to generate a new node feature representation to update the node features, and then the initial input features are fused with the updated node features using jump connections. The fused features are processed using the Relu layer activation function, and nonlinear factors are introduced to enhance the feature expression capability. Among them, the node features and graph structure information of the input cyclic residual graph attention layer are the information in the graph dataset.
[0041] The second graph residual attention layer receives the output and spatial connection relationship of the first graph residual attention layer, and performs attention calculation and feature aggregation similar to the first graph residual attention layer, and applies the Relu layer for activation. Then, the output of the first graph residual attention layer is fused with the output of the second graph residual attention layer through a jump connection to obtain further updated node features. Finally, the initial input node features (the information in the graph dataset input to the cyclic residual graph attention layer) are directly passed to the final output through a jump connection, and are fused with the features finally obtained after processing by the two graph residual attention convolutional layers to obtain the final target feature representation after processing by the cyclic residual graph attention layer.
[0042] The architectural design of the cyclic residual graph attention layer can effectively avoid the node features gradually becoming the same (over-smoothing) during graph attention network training. It can also alleviate problems such as gradient disappearance or explosion during model training, thereby improving the efficiency and stability of model training.
[0043] In a specific embodiment, in the graph residual attention layer, for each image node, it is necessary to calculate the attention coefficient between it and the neighboring nodes. This coefficient determines the importance of each neighboring node when aggregating neighboring node features. The calculation process includes: first, the attention coefficient is calculated. For image node i and each of its neighboring nodes j, their feature vectors are transformed into a common representation space through a shared weight matrix (learnable parameter). This step ensures that the node features are compared and calculated in the same space, and then an attention scoring function is used to calculate the attention coefficient. These attention coefficients reflect the importance of neighbor node j to image node i. In order to ensure that the attention coefficients are numerically comparable, the softmax function is used for each node i to normalize the attention coefficients of all its neighboring nodes to ensure that the sum of all attention coefficients of each node is 1. As a result, when aggregating neighboring node features, the contribution of each neighboring node is weighted according to the normalized attention coefficient. The expression of the normalized attention coefficient is as follows: Among them, a ij represents the normalized attention coefficient, a represents the learnable weight vector, and a T represents the transpose of the weight vector, x i Represents the feature vector of image node i, x j represents the feature vector of node j, LeakyReLU represents an activation function that has a smaller slope in the negative region instead of completely becoming zero, exp is used to map the activated value to the positive space through an exponential function to ensure the non-negativity of the attention weight, and N(i) represents the set of neighbor nodes of image node i. For each image node i, the feature vector of each of its neighbor nodes j is multiplied by the corresponding normalized attention coefficient a ij , and then summing up these weighted feature vectors, we can get the new feature representation of the image node i. Obtaining the new feature representation of the image node by convolutionally aggregating the information of neighboring nodes through the graph residual attention layer is a key step in the graph residual attention layer, which is used to dynamically aggregate the information of neighboring nodes. In this way, it is ensured that each node not only relies on its own features, but also combines the feature information of its surrounding neighboring nodes to form a richer and more representative node feature representation.
[0044] The new feature representation processed by the graph residual attention layer is represented by the nonlinear activation function Relu through the Relu layer to introduce nonlinear features. Then the two graph residual attention layers are stacked, that is, the output features of the first graph residual attention layer (the node features obtained after processing by the first graph residual attention layer) are used as the input features of the second graph residual attention layer to form a deep graph residual attention network. A jump connection is made between the initial input and the output features of the second graph residual attention layer, that is, the input features of the first graph residual attention layer and the output features of the second graph residual attention layer are fused to form a cyclic residual architecture. This cyclic residual connection allows input features to be transferred and fused between multiple layers, promotes the training of deeper networks, and improves the stability and training efficiency of the network. Increase the stability of network training and feature expression capabilities.
[0045] In one possible embodiment, the aggregation layer applies the SetTransformer Aggregation mechanism to aggregate all target feature representations obtained after processing by the cyclic residual graph attention layer. Set Transforme is a neural network model for processing sets (unordered data). It is based on the self-attention mechanism of the standard Transformer model, but is optimized to effectively handle the inherent properties of the set, that is, it does not depend on the order of elements (permutation-invariant). Aggregating the target feature representation through the SetTransformer Aggregation mechanism can not only maintain the global information of the graph structure, but also overcome the influence of the node order on the results, thereby better capturing the important features and relationships in the graph.
[0046] The graph-level feature representation is obtained by aggregating all target feature representations through the aggregation layer. It is a comprehensive global feature representation generated by combining the initial features of each node with the important features of its neighboring nodes. The graph-level feature representation not only integrates the initial features of each node, but also includes the mutual relationship and important features between it and its neighboring nodes. After this processing, the information of each node is effectively integrated into the global features, making the graph-level representation more comprehensive and discriminative.
[0047] In a specific embodiment, the aggregation operation performed by the aggregation layer is to aggregate the information of all image nodes through multi-head attention pooling. Specifically, the attention head is first calculated by the following formula: Among them, head i represents the i-th attention head, Q represents the query matrix, K represents the key matrix, and W i Q Represents the query weight matrix, W i k represents the key weight matrix, d k Indicates the dimension of the key vector, used to scale the result of the dot product.
[0048] Set Transformer uses a multi-head attention mechanism as the core of the encoder and decoder. Specifically, the post-encoder part is used to encode the input (referring to the target feature representation obtained after processing by the cyclic residual graph attention layer), and then the decoder is used to process the output of the encoder to generate the final output. The formula is as follows: Among them, ST(H) represents the graph representation obtained by processing the input H using Set Transformer, ENCODER represents the encoder, and DECODER represents the decoder.
[0049] The linear layer maps the graph-level feature representation obtained by the aggregation layer to the facial expression classification label to obtain the classification category of the facial expression. Specifically, the linear layer performs a linear transformation on the output features of the aggregation layer and maps the high-dimensional feature representation to the classification label space. For each classification label, the softmax function is used to convert it into a probability distribution. Each probability in the probability distribution represents the possibility that the input image is a certain label. Finally, the category with the largest probability is selected as the facial expression classification result of the input image.
[0050] S104: Input the test set obtained according to the division of the graph data set into the cyclic residual graph attention network model for model training and optimization to obtain a facial expression prediction model, and the facial expression prediction model is used to perform facial expression recognition on the image to be recognized to obtain an expression prediction result.
[0051] In S104, in a possible embodiment, a test set obtained according to the division of the graph data set is input into the cyclic residual graph attention network model for model training and optimization to obtain a facial expression prediction model, including: inputting the test set into the cyclic residual graph attention network model, training the cyclic residual graph attention network model according to a preset model training optimizer; calculating the model output accuracy after the cyclic residual graph attention network model completes a round of training; completing the training when the accuracy reaches a set threshold, and obtaining a trained facial expression prediction model.
[0052] In a possible embodiment, training the cyclic residual graph attention network model requires defining an Adam optimizer as a model training optimizer. Exemplarily, the learning rate of the optimizer can be defined as 1e-4, and the learning rate decay is used at the same time, and it is set to decay to half of the original learning rate every 50 rounds. Defining the learning rate adjustment method of the optimizer according to the situation described in the example can maintain a high learning rate in the early stage of model training, promote rapid convergence, and finely adjust the model parameters by reducing the learning rate in the later stage, thereby improving the generalization ability and stability of the model. Due to the imbalance of the categories of the data set used for model training, a weighted cross entropy loss function is used to optimize the loss of the model during training. Specifically, the weighted cross entropy loss function is to multiply a coefficient on the basis of the traditional cross entropy function. This coefficient is the inverse of the category in the total category, so that the category with a smaller number will obtain a larger weight, so that the model will pay more attention to those categories with a smaller number. Thereby effectively alleviating the impact of category imbalance and improving the recognition accuracy of a few categories.
[0053] In a specific embodiment, when performing model training of the cyclic residual graph attention network model, the constructed graph data set is divided to obtain a training set, and the training set is used as the input of the cyclic residual graph attention network model. The constructed cyclic residual graph attention network model is loaded and the Adam optimizer is initialized, the training set is input into the cyclic residual graph attention network model and forward propagated through the model, and the attention weight calculation is performed by the cyclic residual graph attention network model to perform feature learning to realize facial expression recognition to obtain facial image expression classification, and the weighted cross entropy loss function is used to calculate the loss value of the current batch. The gradient is calculated by the back propagation algorithm, and the model parameters are updated using the Adam optimizer. After each round of training, the accuracy (the proportion of the number of samples predicted correctly by the model to the total number of samples in the training set) and the loss value (the gap between the evaluation model prediction result and the actual value) are output, and the parameters of the model are optimized through multiple rounds of iterations. When the accuracy of the model training reaches the set threshold, the training is stopped, thereby obtaining a trained facial expression prediction model.
[0054] Exemplarily, the set threshold of the model training accuracy is greater than or equal to 98%.
[0055] In an embodiment of the present invention, after obtaining the facial expression prediction model, the facial expression prediction model is also tested by a test set obtained by dividing the graph data set to verify the performance of the trained facial expression prediction model on unseen data. The purpose of this test step is to make a final evaluation of the model performance, and it also marks the transition of the model from the training stage to the practical application stage.
[0056] In a possible embodiment, the test set and training set used in the embodiment of the present invention are obtained by dividing the graph dataset in a ratio of 8:2.
[0057] In a specific embodiment, the classic network architectures such as Vgg16 and Resnet 18 in the prior art and the facial expression recognition model obtained by this solution are used to predict expressions respectively, and the prediction results are compared. Specifically, expression data from the AffectNet database and the MMAFEDB database are selected. The AffectNet database targets a challenging problem in the field of computer vision - automatic emotion calculation in the wild. At present, there are relatively few annotated databases that can be used for facial expression recognition in the wild, and most of them only cover discrete emotion models based on categories. For the needs of emotion calculation based on continuous dimensional models (such as the value and arousal of emotions), facial expression databases with detailed annotations are even more scarce. The AffectNet database collects more than one million facial images obtained through Internet searches. These searches use 1,250 emotion-related keywords in six different languages and are conducted through three major search engines. Of these collected images, about half have been manually annotated, identifying seven discrete facial expressions and their corresponding emotional values and arousal intensities. So far, AffectNet represents the largest database of facial expressions in the wild. MMAFEDB is a facial expression recognition dataset, mainly including facial expression data of European and American people. The dataset is divided into three subsets: training set, validation set and test set. The total data volume exceeds 120,000 pictures, which are classified into seven basic expression categories.
[0058] Specifically, the Affectnet database consists of 80,276 normal expressions, 146,198 happy expressions, 29,487 sad expressions, 16,288 surprised expressions, 8,191 fearful expressions, 5,264 disgusted expressions, and 28,130 angry expressions. The MMAFEDB database consists of 8,624 angry expressions, 4,542 disgusted expressions, 6,209 fearful expressions, 39,526 happy expressions, 41,081 normal expressions, 11,062 surprised expressions, and 16,636 sad expressions.
[0059] The experimental results of various methods in terms of recognition accuracy obtained through testing, where the recognition accuracy is calculated based on the ratio of the number of correct predictions to the total number (expressed as a percentage) are shown in the following table:
[0060]
[0061]
[0062] It is obvious from the data in the table that the method proposed in the present invention surpasses the existing facial expression recognition method using classic network architectures such as Vgg16 and Resnet18 in terms of performance, showing significant advantages.
[0063] After the method of the present invention is used for a facial expression recognition test, compared with the prior art, it can be concluded that the accuracy of expression prediction using the method of the present invention is higher than that of most existing expression prediction methods.
[0064] A cyclic residual graph attention network model is constructed in the facial expression prediction method of the present invention, in which the cyclic residual graph attention layer adopts an attention mechanism to assign dynamic weights to image nodes, so that the model can automatically emphasize more important neighbor nodes during the feature aggregation process, thereby being able to more accurately focus on the facial features that contribute most to the recognition task, thereby improving the accuracy and sensitivity of expression recognition.
[0065] The design of the cyclic residual graph attention layer that can focus on the most critical information enables the facial expression prediction method provided by the present invention to resist noise and irrelevant information in the input data to a certain extent. By focusing on key facial features, the adverse effects on the accuracy of facial expression recognition caused by various factors such as illumination changes, facial occlusion or image quality differences are reduced, so that facial expressions can still be accurately recognized under undesirable conditions, showing excellent robustness.
[0066] Aggregating the target feature representation through the aggregation layer can aggregate the information of all image nodes and effectively combine local features with global context information. This means that when performing facial expression recognition, the cyclic residual graph attention network model constructed by the present invention not only considers local facial movements, but also takes into account the context information of the overall facial expression. This comprehensive feature representation provides strong support for complex expression recognition decisions and significantly improves the accuracy and reliability of recognition.
[0067] In summary, the facial expression prediction method provided by the present invention can effectively interpret the facial expressions and emotions of the target person. Compared with the classic expression recognition method in the prior art, the method provided by the present invention can effectively improve the recognition rate.
[0068] See the instruction manual Figure 5 This embodiment also provides a facial expression prediction device, which is used to implement the above method embodiment. The device includes:
[0069] The image processing unit 201 is used to obtain m face images, and perform segmentation processing on the m face images to obtain image nodes of the face images, where m>1.
[0070] The graph data set generation unit 202 is used to obtain the spatial connection relationship and node features of the image nodes, and encapsulate the spatial connection relationship and node features according to the image nodes corresponding to each image to generate a graph data set.
[0071] The model construction unit 203 is used to construct a cyclic residual graph attention network model, which is used to assign attention weights to the image nodes according to the graph data set, and perform feature learning based on the attention weight assignment results to achieve facial expression recognition.
[0072] The prediction unit 204 is used to input the test set obtained according to the division of the graph data set into the cyclic residual graph attention network model for model training and optimization to obtain a facial expression prediction model, and the facial expression prediction model is used to perform facial expression recognition on the image to be recognized to obtain an expression prediction result.
[0073] The graph data set generation unit 202 obtains the spatial connection relationship of the image nodes, including obtaining the connection relationship between the image nodes and the corresponding adjacent nodes in the up, down, left, right and diagonal directions. The spatial connection relationship and node feature encapsulation are performed according to the image nodes corresponding to each image to generate a graph data set, including: applying a graph neural network framework to create a data set, encapsulating a Data object according to the image nodes corresponding to each image, wherein one of the Data objects includes an image node of a face image and the corresponding spatial connection relationship and node features, and generating a graph data set according to the encapsulated Data object.
[0074] The cyclic residual graph attention network model constructed by the model construction unit 203 includes a cyclic residual graph attention layer, an aggregation layer and a linear layer; wherein the cyclic residual graph attention layer is used to assign attention weights between each image node and its neighboring nodes according to the spatial connection relationship and node features of the image nodes, and obtain a new feature representation of the image node according to the attention weight allocation result, and after fusing the new feature representation with the initial input features of the corresponding image node, a nonlinear factor is introduced to obtain the target feature representation; the aggregation layer is used to aggregate all target feature representations to obtain a graph-level feature representation; the linear layer is used to map the graph-level feature representation to the facial expression classification label.
[0075] The prediction unit 204 inputs the training set obtained according to the division of the graph data set into the cyclic residual graph attention network model to perform model training and optimization to obtain a facial expression prediction model, including: inputting the test set into the cyclic residual graph attention network model, and training the cyclic residual graph attention network model according to a preset model training optimizer; calculating the model output accuracy after the cyclic residual graph attention network model completes a round of training; when the accuracy reaches a set threshold, the training is completed to obtain a trained facial expression prediction model.
[0076] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.
[0077] In other embodiments of the present application, the present application discloses an electronic device, such as Figure 6 As shown, the electronic device 300 may include: one or more processors 301; a memory 302; a display 303; one or more applications (not shown); and one or more computer programs 304. The above components may be connected via one or more communication buses 305. The one or more computer programs 304 are stored in the above memory and configured to be executed by the one or more processors 301. The one or more computer programs 304 include instructions, which may be used to execute the following: Figure 1 , Figure 5 And each step in the corresponding embodiment.
[0078] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0079] Each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0080] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk.
[0081] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.
Claims
1. A facial expression prediction method, It is characterized in that include: Obtain m face images, and perform segmentation processing on the m face images to obtain image nodes of the face images, where m>1; Obtain the spatial connection relationship and node features of the image nodes, and encapsulate the spatial connection relationship and node features according to the image nodes corresponding to each image to generate a graph dataset; Constructing a cyclic residual graph attention network model, wherein the cyclic residual graph attention network model is used to assign attention weights to the image nodes according to the graph data set, and perform feature learning according to the attention weight assignment result to realize facial expression recognition; The test set obtained by dividing the graph data set is input into the cyclic residual graph attention network model for model training and optimization to obtain a facial expression prediction model, and the facial expression prediction model is used to perform facial expression recognition on the image to be recognized to obtain an expression prediction result.
2. The method according to claim 1, It is characterized in that Acquiring the spatial connection relationship of the image node includes acquiring the connection relationship between the image node and the corresponding adjacent nodes in the up, down, left, right, and diagonal directions; According to the image nodes corresponding to each image, the spatial connection relationship and node feature encapsulation are performed to generate a graph dataset, including: A data set is created using a graph neural network framework. Data objects are encapsulated according to the image nodes corresponding to each image. Each Data object includes an image node of a face image and the corresponding spatial connection relationship and node features. A graph data set is generated based on the encapsulated Data objects.
3. The method according to claim 1, It is characterized in that The cyclic residual graph attention network model includes a cyclic residual graph attention layer, an aggregation layer and a linear layer; The cyclic residual graph attention layer is used to allocate attention weights between each image node and its neighboring nodes according to the spatial connection relationship and node features of the image nodes, and obtain a new feature representation of the image node according to the attention weight allocation result, and the new feature representation is fused with the initial input features of the corresponding image node and a nonlinear factor is introduced to obtain the target feature representation; The aggregation layer is used to aggregate all target feature representations to obtain a graph-level feature representation; The linear layer is used to map the graph-level feature representation to a facial expression classification label.
4. The method according to claim 1, It is characterized in that The training set obtained by dividing the graph data set is input into the cyclic residual graph attention network model for model training and optimization to obtain a facial expression prediction model, including: Inputting the test set into the cyclic residual graph attention network model, and training the cyclic residual graph attention network model according to a preset model training optimizer; After the cyclic residual graph attention network model completes a round of training, the model output accuracy is calculated; When the accuracy reaches the set threshold, the training is completed and a trained facial expression prediction model is obtained.
5. A facial expression prediction device, It is characterized in that The device comprises: An image processing unit, used for acquiring m face images, and performing segmentation processing on the m face images to obtain image nodes of the face images, where m>1; A graph data set generation unit, used to obtain the spatial connection relationship and node features of the image nodes, and encapsulate the spatial connection relationship and node features according to the image nodes corresponding to each image to generate a graph data set; A model building unit, used to build a cyclic residual graph attention network model, wherein the cyclic residual graph attention network model is used to assign attention weights to the image nodes according to the graph data set, and perform feature learning according to the attention weight assignment result to realize facial expression recognition; The prediction unit is used to input the test set obtained by dividing the graph data set into the cyclic residual graph attention network model for model training and optimization to obtain a facial expression prediction model, and the facial expression prediction model is used to perform facial expression recognition on the image to be recognized to obtain an expression prediction result.
6. The device according to claim 5, It is characterized in that The graph data set generation unit obtains the spatial connection relationship of the image node, including obtaining the connection relationship between the image node and the corresponding adjacent nodes in the up, down, left, right and diagonal directions; According to the image nodes corresponding to each image, the spatial connection relationship and node feature encapsulation are performed to generate a graph dataset, including: A data set is created using a graph neural network framework. Data objects are encapsulated according to the image nodes corresponding to each image. Each Data object includes an image node of a face image and the corresponding spatial connection relationship and node features. A graph data set is generated based on the encapsulated Data objects.
7. The device according to claim 5, It is characterized in that The cyclic residual graph attention network model constructed by the model construction unit includes a cyclic residual graph attention layer, an aggregation layer and a linear layer; The cyclic residual graph attention layer is used to allocate attention weights between each image node and its neighboring nodes according to the spatial connection relationship and node features of the image nodes, and obtain a new feature representation of the image node according to the attention weight allocation result, and the new feature representation is fused with the initial input features of the corresponding image node and a nonlinear factor is introduced to obtain the target feature representation; The aggregation layer is used to aggregate all target feature representations to obtain a graph-level feature representation; The linear layer is used to map the graph-level feature representation to a facial expression classification label.
8. The device according to claim 5, It is characterized in that The prediction unit inputs the training set obtained by dividing the graph data set into the cyclic residual graph attention network model for model training and optimization to obtain a facial expression prediction model, including: Inputting the test set into the cyclic residual graph attention network model, and training the cyclic residual graph attention network model according to a preset model training optimizer; After the cyclic residual graph attention network model completes a round of training, the model output accuracy is calculated; When the accuracy reaches the set threshold, the training is completed and a trained facial expression prediction model is obtained.
9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the facial expression prediction method according to any one of claims 1 to 4 is implemented.
10. An electronic device, It is characterized in that include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so that the electronic device executes the facial expression prediction method according to any one of claims 1 to 4.
