Face emotion recognition method, device, equipment and storage medium
By introducing a selective convolutional kernel network and a classifier into the basic neural network, the problem of low accuracy in facial emotion recognition in existing technologies is solved, and more efficient facial emotion recognition in images is achieved.
Patent Information
- Application Number
- CN202310237387.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-10
- Publication Date
- 2025-11-25
- Estimated Expiration
- 2043-03-10
Smart Images
Figure CN116363732B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, device, and storage medium for facial emotion recognition. Background Technology
[0002] Facial expression recognition is the technology that studies how computers can extract and identify facial expressions from static images or video sequences. If computers can accurately understand facial expressions and identify which category they belong to, it will significantly change the relationship between humans and computers, thereby achieving better human-computer interaction.
[0003] Emotion recognition is of great importance in natural human-computer interaction. However, due to the large number of attribute categories and complex patterns of facial expressions, each recognition model in existing facial expression recognition methods needs to memorize a lot of information, which leads to complex computation in the facial expression recognition process and relatively low accuracy and efficiency.
[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention
[0005] The main objective of this invention is to provide a facial emotion recognition method, which aims to solve the technical problem of inaccurate facial emotion recognition in existing technologies.
[0006] To achieve the above objectives, the present invention provides a facial emotion recognition method, the method comprising the following steps:
[0007] The image to be recognized is input into the facial emotion recognition model, which includes a basic neural network, a selective convolutional kernel network, and a classifier.
[0008] Based on the basic neural network, multiple basic features are obtained by feature extraction from the image to be identified;
[0009] The independent weights of each basic feature are calculated based on the selective convolutional kernel network, and the basic features are fused based on the independent weights to obtain fused features;
[0010] Based on the independent weights corresponding to each reference feature in the fused features, the target receptive field is determined by the classifier, and the facial emotion in the image to be identified is obtained based on the target receptive field.
[0011] Optionally, the step of calculating independent weights for each basic feature based on the selective convolutional kernel network, and fusing the basic features based on the independent weights to obtain fused features, includes:
[0012] The selective convolutional kernel network calculates the independent weights of each basic feature through adaptive receptive field;
[0013] Multiple reference features are obtained by filtering the multiple basic features based on the independent weights;
[0014] The multiple reference features are fused to obtain the fused feature.
[0015] Optionally, fusing the plurality of reference features to obtain the fused feature includes:
[0016] The multiple reference features are fused to obtain an initial fused feature;
[0017] The initial fusion features are downsampled to obtain sampled features;
[0018] The sampled features are subjected to feature dimensionality reduction to obtain fused features.
[0019] Optionally, the step of determining the target receptive field based on the independent weights corresponding to each reference feature in the fused features through the classifier, and obtaining the facial emotion in the image to be identified based on the target receptive field, includes:
[0020] The classifier obtains the weights of multiple receptive fields in the fusion feature based on the independent weights corresponding to each reference feature in the fusion feature;
[0021] By comparing the weights of each receptive field, the target receptive field is obtained;
[0022] The target receptive field is classified to obtain the facial emotions in the image to be identified.
[0023] Optionally, before inputting the image to be recognized into the facial emotion recognition model, the following steps are included:
[0024] Acquire initial training images and initial verification images, and preprocess the initial training images and initial verification images to obtain training tensors and verification tensors;
[0025] The training tensor is input into the initial model to train the initial model until the recognition accuracy of the initial model meets the preset condition, thus obtaining the initial emotion recognition model.
[0026] The initial emotion recognition model is validated by a validation tensor. If the validation result does not meet the preset requirements, the initial emotion recognition model is retrained. If the validation result meets the preset requirements, a facial emotion recognition model is obtained.
[0027] Optionally, the preprocessing of the initial training image and the initial verification image to obtain the training tensor and the verification tensor includes:
[0028] The initial training image and the initial verification image are resized to obtain training images and verification images of uniform size;
[0029] Data augmentation is performed on the training image and the verification image to obtain augmented training image and augmented verification image;
[0030] The enhanced training image and the enhanced verification image are vector-transformed to obtain the training tensor and the verification tensor.
[0031] Optionally, the step of inputting the training tensor into the initial model to train the initial model until the recognition accuracy of the initial model meets a preset condition, thereby obtaining an initial emotion recognition model, includes:
[0032] Obtain training parameters, including learning rate and learning rate decay;
[0033] The initial model with set parameters is obtained by setting the initial model according to the learning rate and the learning rate decay.
[0034] The initial model after setting the input parameters of the training tensor is used to obtain the pre-identified emotion, and the pre-identified emotion is compared with the reference emotion corresponding to the training tensor.
[0035] When the number of consecutive correct comparison results meets the preset condition, the training is completed and the initial emotion recognition model is obtained.
[0036] Furthermore, to achieve the above objectives, the present invention also proposes a facial emotion recognition device, the facial emotion recognition device comprising:
[0037] The data input module is used to input the image to be recognized into the facial emotion recognition model, which includes a basic neural network, a selective convolutional kernel network, and a classifier.
[0038] The feature extraction module is used to extract multiple basic features from the image to be identified based on the basic neural network.
[0039] The feature fusion module is used to calculate the independent weights of each basic feature based on the selective convolutional kernel network, and to fuse the basic features based on the independent weights to obtain fused features;
[0040] The classification and recognition module is used to determine the target receptive field through the classifier based on the independent weights corresponding to each reference feature in the fused features, and to obtain the facial emotion in the image to be recognized based on the target receptive field.
[0041] Furthermore, to achieve the above objectives, the present invention also proposes a facial emotion recognition device, which includes: a memory, a processor, and a facial emotion recognition program stored in the memory and executable on the processor, wherein the facial emotion recognition program is configured to implement the steps of the facial emotion recognition method described above.
[0042] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a facial emotion recognition program, wherein when the facial emotion recognition program is executed by a processor, it implements the steps of the facial emotion recognition method described above.
[0043] This invention adds a selective convolutional kernel network after the first set of residual structures in the residual artificial neural network to adaptively filter the features extracted from the entire range of the residual artificial neural network and assign independent weights to each feature. Based on the independent weights, it finds reference features that are more representative of emotions in the image to be identified, and performs facial emotion recognition based on the reference features to complete the classification of facial emotions in the features to be identified. This allows for faster and more accurate identification of facial emotions in the image to be identified. Attached Figure Description
[0044] Figure 1 This is a schematic diagram of the structure of a facial emotion recognition device in the hardware operating environment involved in the embodiments of the present invention;
[0045] Figure 2 This is a flowchart illustrating the first embodiment of the facial emotion recognition method of the present invention;
[0046] Figure 3 This is a flowchart illustrating the second embodiment of the facial emotion recognition method of the present invention;
[0047] Figure 4 This is a structural block diagram of the first embodiment of the facial emotion recognition device of the present invention.
[0048] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0049] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.
[0050] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a facial emotion recognition device in the hardware operating environment of an embodiment of the present invention.
[0051] like Figure 1As shown, the facial emotion recognition device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.
[0052] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on facial emotion recognition devices and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.
[0053] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a facial emotion recognition program.
[0054] exist Figure 1 In the facial emotion recognition device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the facial emotion recognition device of the present invention can be set in the facial emotion recognition device, and the facial emotion recognition device calls the facial emotion recognition program stored in the memory 1005 through the processor 1001 and executes the facial emotion recognition method provided in the embodiment of the present invention.
[0055] This invention provides a facial emotion recognition method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a facial emotion recognition method according to the present invention.
[0056] In this embodiment, the facial emotion recognition method includes the following steps:
[0057] Step S10: Input the image to be recognized into the facial emotion recognition model, which includes a basic neural network, a selective convolutional kernel network, and a classifier.
[0058] Understandably, the image to be recognized can be a series of frames extracted by the processor from a video, or it can be an image input by the user.
[0059] It should be understood that a facial emotion recognition model can be a model that has been trained and tested, and whose testing requirements have been met. This facial emotion recognition model can more accurately identify the emotions of people in the image to be recognized.
[0060] It should be noted that the facial emotion recognition model includes a basic neural network, a selective convolutional kernel network, and a classifier. The basic neural network can be a ResNet50 network (also called a residual artificial neural network). The basic neural network has multiple sets of residual structures. The selective convolutional kernel network is placed after the first set of residual structures in the basic neural network. The classifier is placed after the basic neural network to form a complete facial emotion recognition model.
[0061] It's worth noting that current emotion recognition models extract features from the entire content of a single frame of a video or an image. These extracted features are then used to identify facial emotions. However, the entire image contains not only facial expressions but also background, clothing, lighting, hair, glasses, and other distracting elements. These elements are also used for feature extraction by the deep neural network, but the extracted features from these elements do not contribute to facial emotion recognition. The facial emotion recognition model in this invention incorporates a selective convolutional kernel network, allowing the deep neural network to focus on extracting features from specific regions of the face, thus solving the problem of background and other factors interfering with feature information extraction.
[0062] It is important to emphasize that in practical use of this facial emotion recognition model, it can consist of two parts: hardware and software. The hardware part comprises a smart terminal, network transmission equipment, and image acquisition equipment, providing support for the operation of the facial emotion recognition model and enabling image data processing and interconnection. The software part includes two modules: Python (Python is a computer programming language; it is an object-oriented, dynamically typed language) and HTML (Hypertext Markup Language), which are responsible for model training, image detection, and visualization design functions, respectively. This enables the acquisition and storage of facial image data, providing the raw data source for intelligent recognition.
[0063] It should be emphasized that the raw data collected by the face detection module is subjected to feature extraction and classification calculations, and the results are fed back to the visualization system interface to achieve real-time recognition of facial expressions. The two software programs, Python and HTML, are used to implement different functions. Specifically, Python analyzes the video captured by the camera of the image acquisition device frame by frame to obtain the image to be tested, and then inputs the image into the face detection module, which is responsible for the data acquisition and storage tasks of the entire system.
[0064] In practical implementation, Python is used as the system's development language. A ResNet residual network is used to build the network model, and the parameters are trained on the FER2013 dataset (which consists of 35,886 images of different facial expressions, with 28,708 images for training, 3,589 for validation, and 3,589 for testing), the RAF-DB dataset (a real-world emotional faces database used for facial expressions), and the KDEF dataset (KDEF is an expression recognition dataset, short for Karolinska Directed Emotional Faces, which includes 70 models, 7 different expressions, and 4,900 color images). The trained model recognizes the test images, and the results are then input into the visualization module. The HTML interface is the visualization module of the expression recognition system; the left side is the control interface, and the right side is the results interface, displaying the recognition time, recognition result, and expression intensity, facilitating a comprehensive and accurate description of facial expressions.
[0065] Step S20: Extract multiple basic features from the image to be identified based on the basic neural network.
[0066] Understandably, a basic neural network can extract features from the image to be recognized, but the extracted features include a lot of useless information, such as background features, clothing features, and building features.
[0067] It should be understood that the basic features are a general description of all the features extracted by the basic neural network, including useful features that help identify facial emotions and useless features that interfere with the accuracy of the model's emotion recognition.
[0068] Step S30: Calculate the independent weights of each basic feature according to the selective convolutional kernel network, and fuse the basic features according to the independent weights to obtain the fused features.
[0069] Understandably, selective convolutional kernel networks adaptively select different convolutional kernels when performing convolution operations. Different convolutional kernels divide multiple basic features into multiple branches, which are then fused together.
[0070] It should be noted that the weights obtained after fusion are independent weights of each branch in the selective convolutional kernel network.
[0071] In practice, one branch can be understood as collecting background features at multiple scales. After fusing the background features at multiple scales, the complete background information is obtained. The weight of the fused features of this branch is then obtained based on the fused background information.
[0072] It should be emphasized that, based on the independent weights of each basic feature calculated using the selective convolutional kernel network, and by fusing the basic features according to these independent weights, the fused features can be:
[0073] The selective convolutional kernel network calculates independent weights for each basic feature through an adaptive receptive field; it then filters the multiple basic features based on these independent weights to obtain multiple reference features; and finally, it fuses the multiple reference features to obtain a fused feature.
[0074] The adaptive receptive field can be a type of adaptive convolutional kernel in a selective convolutional kernel network. The independent weights of each basic feature can be calculated based on the adaptive receptive field, which is the independent weight of each branch in the adaptive receptive field. The magnitude of the independent weights of all branches is compared, and the basic features corresponding to the branches with weights less than the weight threshold are removed. The weight threshold can be 0.3, 0.4, etc., and can be set according to the actual situation.
[0075] Understandably, the reference feature can be a fusion feature whose weight is greater than the weight threshold after the basic features are fused, and a fusion feature is obtained by fusing all the remaining features after removing the features whose weight is lower than the weight threshold.
[0076] It should be further noted that the fusion of the multiple reference features to obtain the fused feature may result in a large amount of data or a large size of the fused feature, which may be detrimental to subsequent recognition.
[0077] It is worth noting that the initial fused features are obtained by fusing the multiple reference features; the initial fused features are downsampled to obtain sampled features; and the sampled features are dimensionality reduced to obtain fused features.
[0078] All of the above features can be represented as multi-dimensional vectors, including basic features, fusion features, and other features; dimensionality reduction of sampled features can be achieved by converting all feature vectors into one-dimensional vector information.
[0079] Step S40: Based on the independent weights corresponding to each reference feature in the fused features, the target receptive field is determined by the classifier, and the facial emotion in the image to be identified is obtained according to the target receptive field.
[0080] Understandably, the classifier is pre-trained to classify different features, with each category representing a facial emotion.
[0081] It should be noted that in convolutional neural networks, the receptive field is defined as the size of the region on the input image that is mapped by the pixels on the feature map output by each layer of the convolutional neural network.
[0082] It should be understood that the weights of each branch in the selective convolutional kernel network are input into the classifier. The softmax function (normalized exponential function) in the classifier calculates the weights of each receptive field, compares the weights of each receptive field, and selects the receptive field with the largest weight as the best receptive field.
[0083] It should be emphasized that the target receptive field is determined by the classifier based on the independent weights corresponding to each reference feature in the fusion feature. The facial emotion in the image to be identified is obtained by the classifier based on the independent weights corresponding to each reference feature in the fusion feature. The weights of each receptive field are compared to obtain the target receptive field. Feature classification is performed on the target receptive field to obtain the facial emotion in the image to be identified.
[0084] This embodiment adds a selective convolutional kernel network after the first set of residual structures in the residual artificial neural network to adaptively filter the features extracted from the entire range of the residual artificial neural network and assign independent weights to each feature. Based on the independent weights, it finds reference features that are more representative of emotions in the image to be identified, and performs facial emotion recognition based on the reference features to complete the classification of facial emotions in the features to be identified. This enables faster and more accurate identification of facial emotions in the image to be identified.
[0085] refer to Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of a facial emotion recognition method according to the present invention.
[0086] Based on the first embodiment described above, the facial emotion recognition method in this embodiment further includes the following steps before step S10:
[0087] Step S101: Obtain the initial training image and the initial verification image, and preprocess the initial training image and the initial verification image to obtain the training tensor and the verification tensor.
[0088] Understandably, the initial training images can be the training set of the RAF-DB, FER2013, KDEF, or ImageNet public datasets, and the initial validation images can be the validation set of the RAF-DB or FER2013 public datasets, where facial expressions include seven categories: surprise, fear, disgust, happiness, sadness, anger, and neutral.
[0089] Understandably, the training tensor and validation tensor can be vectors obtained after preprocessing the initial training image and the initial validation image. The vector information after preprocessing the initial training image is called the training tensor; the vector information after preprocessing the initial validation image is called the validation tensor.
[0090] It should be understood that the initial verification image may be obtained by using transfer learning to transfer knowledge from the source domain ImageNet dataset to the face emotion recognition model, or by using a ResNet50 model pre-trained on the ImageNet dataset to transfer to the Fer2013 and Raf-Db datasets.
[0091] It should be noted that preprocessing the initial training image and the initial verification image to obtain the training tensor and the verification tensor can be done by converting the size of the initial training image and the initial verification image to obtain training images and verification images with uniform size.
[0092] Data augmentation is performed on the training images and the validation images to obtain augmented training images and augmented validation images (data augmentation can be performed by randomly flipping, randomly occluding, or randomly cropping images of uniform size to augment the dataset, which can prevent overfitting during training).
[0093] The enhanced training image and the enhanced verification image are vector-transformed to obtain the training tensor and the verification tensor.
[0094] Step S102: Input the training tensor into the initial model to train the initial model until the recognition accuracy of the initial model meets the preset conditions, and obtain the initial emotion recognition model.
[0095] Understandably, after the basic architecture of the model is completed, in order to enable the model to perform specific functions more accurately, similar to enabling the facial emotion recognition model in this application to more accurately and effectively recognize facial emotions, the model needs to be trained, and after training, the trained model is validated using additional image data.
[0096] It should be understood that training and validating with a single set of images is prone to overfitting, which is similar to having a high recognition rate during training but being prone to errors in actual use.
[0097] Understandably, the parameters of the model need to be set during the training process, and the continuous optimization of the parameters through training is also a way to improve the model's recognition accuracy.
[0098] It should be understood that the initial model's recognition accuracy is considered to be satisfied when the initial model correctly recognizes 10-15 facial emotions with a change of less than 1% during the training process.
[0099] It should be noted that the training tensor is input into the initial model to train the initial model until the recognition accuracy of the initial model meets the preset condition. The resulting initial emotion recognition model can be:
[0100] Obtain training parameters, including learning rate and learning rate decay;
[0101] The initial model with set parameters is obtained by setting the learning rate and the learning rate decay. The training parameters can be referred to in the following table:
[0102] parameter numerical values Input_shape [224,224,3] Optimizer SGD Initial learning rate 0.01 Minimum learning rate 0.0001 Momentum 0.9 Descent Algorithm COS Batch size 32
[0103] The initial model after setting the input parameters of the training tensor is used to obtain the pre-identified emotion. The pre-identified emotion is compared with the reference emotion corresponding to the training tensor. When the number of consecutive correct comparison results meets the preset condition, the training is completed and the initial emotion recognition model is obtained.
[0104] Step S103: Validate the initial emotion recognition model using a validation tensor. If the validation result does not meet the preset requirements, retrain the initial emotion recognition model. If the validation result meets the preset requirements, obtain the facial emotion recognition model.
[0105] Understandably, the preset requirement could be to input the verification tensor into the initial emotion recognition model and determine whether the initial emotion recognition model has a correct recognition rate of more than 90% for the verification tensor. If the initial emotion recognition model has a correct recognition rate of more than 90% for the verification tensor, then the initial emotion recognition model is considered to meet the preset requirement and is used as the face emotion recognition model.
[0106] It should be noted that when humans acquire image information, they quickly scan and focus on areas rich in information, ignoring other areas with little or no information. This is a good method for filtering out the most valuable information from a large amount of data. For example, when judging facial expressions, humans focus on areas rich in expressive information, such as the eyes and mouth, while ignoring areas with little reference value, such as the nose and forehead.
[0107] Region-level attention mechanisms require manual specification of region division rules, such as the size, location, and number of regions for each region. Due to a lack of prior experience, it is difficult for humans to find the most suitable division method. In addition, multiple regions will cause the model to consume more computing resources during training and inference.
[0108] This embodiment trains a facial emotion recognition model incorporating the SKNet selective convolutional kernel network using transfer learning. During training, the model receives more training classification samples, further improving recognition accuracy. When using the facial emotion recognition model for emotion recognition in subsequent images, it can more accurately and effectively identify facial emotions.
[0109] Furthermore, this embodiment of the invention also proposes a storage medium storing a facial emotion recognition program, which, when executed by a processor, implements the steps of the facial emotion recognition method described above.
[0110] Reference Figure 4 , Figure 4 This is a structural block diagram of the first embodiment of the facial emotion recognition device of the present invention.
[0111] like Figure 4 As shown, the facial emotion recognition device proposed in this embodiment of the invention includes:
[0112] Data input module 10 is used to input the image to be recognized into the facial emotion recognition model, which includes a basic neural network, a selective convolutional kernel network, and a classifier;
[0113] Feature extraction module 20 is used to extract multiple basic features from the image to be identified based on the basic neural network;
[0114] The feature fusion module 30 is used to calculate the independent weights of each basic feature according to the selective convolutional kernel network, and fuse the basic features according to the independent weights to obtain fused features;
[0115] The classification and recognition module 40 is used to determine the target receptive field through the classifier based on the independent weights corresponding to each reference feature in the fused features, and to obtain the facial emotion in the image to be recognized based on the target receptive field.
[0116] This embodiment adds a selective convolutional kernel network after the first set of residual structures in the residual artificial neural network to adaptively filter the features extracted from the entire range of the residual artificial neural network and assign independent weights to each feature. Based on the independent weights, it finds reference features that are more representative of emotions in the image to be identified, and performs facial emotion recognition based on the reference features to complete the classification of facial emotions in the features to be identified. This enables faster and more accurate identification of facial emotions in the image to be identified.
[0117] In one embodiment, the feature fusion module 30 is further configured to allow the selective convolutional kernel network to calculate the independent weights of each basic feature through an adaptive receptive field;
[0118] Multiple reference features are obtained by filtering the multiple basic features based on the independent weights;
[0119] The multiple reference features are fused to obtain the fused feature.
[0120] In one embodiment, the feature fusion module 30 is further configured to fuse the plurality of reference features to obtain an initial fused feature;
[0121] The initial fusion features are downsampled to obtain sampled features;
[0122] The sampled features are subjected to feature dimensionality reduction to obtain fused features.
[0123] In one embodiment, the classification and recognition module 40 is further configured to allow the classifier to obtain the weights of multiple receptive fields in the fusion feature based on the independent weights corresponding to each reference feature in the fusion feature;
[0124] By comparing the weights of each receptive field, the target receptive field is obtained;
[0125] The target receptive field is classified to obtain the facial emotions in the image to be identified.
[0126] In one embodiment, the data input module 10 is further configured to acquire an initial training image and an initial verification image, and preprocess the initial training image and the initial verification image to obtain a training tensor and a verification tensor;
[0127] The training tensor is input into the initial model to train the initial model until the recognition accuracy of the initial model meets the preset condition, thus obtaining the initial emotion recognition model.
[0128] The initial emotion recognition model is validated by a validation tensor. If the validation result does not meet the preset requirements, the initial emotion recognition model is retrained. If the validation result meets the preset requirements, a facial emotion recognition model is obtained.
[0129] In one embodiment, the data input module 10 is further configured to convert the size of the initial training image and the initial verification image to obtain training images and verification images with uniform size;
[0130] Data augmentation is performed on the training image and the verification image to obtain augmented training image and augmented verification image;
[0131] The enhanced training image and the enhanced verification image are vector-transformed to obtain the training tensor and the verification tensor.
[0132] In one embodiment, the data input module 10 is further configured to acquire training parameters, including a learning rate and a learning rate decay.
[0133] The initial model with set parameters is obtained by setting the initial model according to the learning rate and the learning rate decay.
[0134] The initial model after setting the input parameters of the training tensor is used to obtain the pre-identified emotion, and the pre-identified emotion is compared with the reference emotion corresponding to the training tensor.
[0135] When the number of consecutive correct comparison results meets the preset condition, the training is completed and the initial emotion recognition model is obtained.
[0136] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.
[0137] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.
[0138] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0139] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0141] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A facial emotion recognition method, characterized in that, The facial emotion recognition method includes: The image to be recognized is input into the facial emotion recognition model, which includes a basic neural network, a selective convolutional kernel network, and a classifier. Based on the basic neural network, multiple basic features are obtained by feature extraction from the image to be identified; The independent weights of each basic feature are calculated based on the selective convolutional kernel network, and the basic features are fused based on the independent weights to obtain fused features; Based on the independent weights corresponding to each reference feature in the fusion feature, the target receptive field is determined by the classifier, and the facial emotion in the image to be identified is obtained according to the target receptive field. The process of calculating independent weights for each basic feature based on the selective convolutional kernel network, and fusing the basic features based on the independent weights to obtain fused features includes: The selective convolutional kernel network calculates the independent weights of each basic feature through adaptive receptive field; Multiple reference features are obtained by filtering the multiple basic features based on the independent weights; The multiple reference features are fused to obtain the fused feature.
2. The facial emotion recognition method as described in claim 1, characterized in that, The process of fusing the multiple reference features to obtain the fused feature includes: The multiple reference features are fused to obtain an initial fused feature; The initial fusion features are downsampled to obtain sampled features; The sampled features are subjected to feature dimensionality reduction to obtain fused features.
3. The facial emotion recognition method as described in claim 1, characterized in that, The independent weights corresponding to each reference feature in the fused features are used by the classifier to determine the target receptive field, and the facial emotion in the image to be identified is obtained based on the target receptive field, including: The classifier obtains the weights of multiple receptive fields in the fusion feature based on the independent weights corresponding to each reference feature in the fusion feature; By comparing the weights of each receptive field, the target receptive field is obtained; The target receptive field is classified to obtain the facial emotions in the image to be identified.
4. The facial emotion recognition method as described in claim 1, characterized in that, Before inputting the image to be recognized into the facial emotion recognition model, the following steps are included: Acquire initial training images and initial verification images, and preprocess the initial training images and initial verification images to obtain training tensors and verification tensors; The training tensor is input into the initial model to train the initial model until the recognition accuracy of the initial model meets the preset condition, thus obtaining the initial emotion recognition model. The initial emotion recognition model is validated by a validation tensor. If the validation result does not meet the preset requirements, the initial emotion recognition model is retrained. If the validation result meets the preset requirements, a facial emotion recognition model is obtained.
5. The facial emotion recognition method as described in claim 4, characterized in that, The preprocessing of the initial training image and the initial verification image to obtain the training tensor and the verification tensor includes: The initial training image and the initial verification image are resized to obtain training images and verification images of uniform size; Data augmentation is performed on the training image and the verification image to obtain augmented training image and augmented verification image; The enhanced training image and the enhanced verification image are vector-transformed to obtain the training tensor and the verification tensor.
6. The facial emotion recognition method as described in claim 4, characterized in that, The step of inputting the training tensor into the initial model to train the initial model until the recognition accuracy of the initial model meets a preset condition, thereby obtaining an initial emotion recognition model, includes: Obtain training parameters, including learning rate and learning rate decay; The initial model with set parameters is obtained by setting the initial model according to the learning rate and the learning rate decay. The initial model after setting the input parameters of the training tensor is used to obtain the pre-identified emotion, and the pre-identified emotion is compared with the reference emotion corresponding to the training tensor. When the number of consecutive correct comparison results meets the preset condition, the training is completed and the initial emotion recognition model is obtained.
7. A facial emotion recognition device, characterized in that, The facial emotion recognition device includes: The data input module is used to input the image to be recognized into the facial emotion recognition model, which includes a basic neural network, a selective convolutional kernel network, and a classifier. The feature extraction module is used to extract multiple basic features from the image to be identified based on the basic neural network. The feature fusion module is used to calculate the independent weights of each basic feature based on the selective convolutional kernel network, and to fuse the basic features based on the independent weights to obtain fused features; The classification and recognition module is used to determine the target receptive field through the classifier based on the independent weights corresponding to each reference feature in the fused features, and to obtain the facial emotion in the image to be recognized based on the target receptive field. The feature fusion module is further configured to: calculate the independent weights of each basic feature through the adaptive receptive field of the selective convolutional kernel network; filter the multiple basic features according to the independent weights to obtain multiple reference features; and fuse the multiple reference features to obtain fused features.
8. A facial emotion recognition device, characterized in that, The device includes: a memory, a processor, and a facial emotion recognition program stored in the memory and executable on the processor, the facial emotion recognition program being configured to implement the facial emotion recognition method as described in any one of claims 1 to 6.
9. A storage medium, characterized in that, The storage medium stores a facial emotion recognition program, which, when executed by a processor, implements the facial emotion recognition method as described in any one of claims 1 to 6.