Methods and devices for facial expression recognition

Abstract

This application relates to the field of image expression recognition technology, and provides a method and apparatus for expression recognition. The method includes: acquiring multiple training samples and inputting them into an expression recognition model; extracting features from each expression training image; determining the target feature vector of each expression training image; classifying and recognizing each target feature vector; determining the initial loss of the expression recognition model; performing dimensionality reduction on each initial feature vector matrix and performing a real-number transformation on the dimensionality reduction result; performing activation mapping on the real-number vectors of each expression training image to obtain the weight values ​​corresponding to each expression training image, performing scaling transformation to obtain a weight sequence; correcting the initial loss of the expression recognition model based on the weight sequence, and updating the parameters in the expression recognition model based on the corrected loss, thereby improving the recognition accuracy of the expression recognition model.

Description

Technical Field

[0001] This application relates to the field of image expression recognition technology, and in particular to a method and apparatus for expression recognition. Background Technology

[0002] Conventional facial expression recognition is achieved by analyzing facial images. First, face detection is performed on the input image to determine the location and size of the face. Feature extraction is then performed on the facial image to extract vectors representing the facial features. Using a dataset of known facial expression labels, a classifier is trained to determine facial expressions based on these feature vectors. The same feature extraction and classifier prediction are then applied to the test data's facial images to obtain the test data's facial expression labels. However, existing facial expression recognition datasets often contain noisy data. If the model fits this noisy data during training, its accuracy will be severely affected.

[0003] In existing technologies, algorithms perform data cleaning and data augmentation, but this method does not significantly improve recognition accuracy.

[0004] Therefore, existing technologies suffer from noise data caused by labeling uncertainties, which reduces the accuracy of facial expression recognition. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method, apparatus, electronic device, and readable storage medium for facial expression recognition, in order to solve the problem that noise data caused by annotation uncertainty in the prior art reduces the accuracy of facial expression recognition.

[0006] A first aspect of this application provides a method for facial expression recognition, comprising: acquiring multiple training samples and inputting the multiple training samples into an facial expression recognition model, wherein each training sample contains a facial expression training image and a label for the facial expression training image; extracting features from each facial expression training image to obtain an initial feature vector matrix for each facial expression training image; determining a target feature vector for each facial expression training image based on the initial feature vector matrix of each facial expression training image; classifying and recognizing each target feature vector to obtain a recognition result for each facial expression training image; determining an initial loss for the facial expression recognition model based on the recognition result and the label of each facial expression training image; performing dimensionality reduction processing on each initial feature vector matrix and performing feature transformation on the dimensionality reduction result to obtain a pre-mapped vector for each facial expression training image; performing activation mapping processing on the pre-mapped vector of each facial expression training image to obtain a weight value corresponding to each facial expression training image; performing scaling transformation on the weight value corresponding to each facial expression training image to obtain a weight sequence; correcting the initial loss of the facial expression recognition model based on the weight sequence, and updating the parameters in the facial expression recognition model based on the corrected loss.

[0007] A second aspect of this application provides an expression recognition apparatus, comprising: an acquisition module for acquiring multiple training samples and inputting the multiple training samples into an expression recognition model, wherein each training sample includes an expression training image and a label for the expression training image; a first processing module for extracting features from each expression training image to obtain an initial feature vector matrix for each expression training image; a second processing module for determining a target feature vector for each expression training image based on the initial feature vector matrix of each expression training image; a third processing module for classifying and recognizing each target feature vector to obtain a recognition result for each expression training image; and a fourth processing module for classifying and recognizing each expression training image based on the initial feature vector matrix of each expression training image. The first module uses the recognition results of the training images and the labels of each training image to determine the initial loss of the expression recognition model. The second module performs dimensionality reduction on each initial feature vector matrix and performs feature transformation on the dimensionality reduction result to obtain the pre-mapped vector of each training image. The third module performs activation mapping on the pre-mapped vector of each training image to obtain the weight value corresponding to each training image. The fourth module performs scaling transformation on the weight value corresponding to each training image to obtain the weight sequence. The fifth module updates the parameters in the expression recognition model based on the weight sequence and the initial loss of the expression recognition model based on the corrected loss.

[0008] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0009] A fourth aspect of this application provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0010] The beneficial effects of this application embodiment compared with the prior art are as follows: By acquiring multiple training samples and inputting them into the expression recognition model, the corresponding recognition results are obtained. Based on the recognition results and the real labels of the expression training images, the initial loss of the expression recognition model is obtained. On the other hand, the expression training images are input into the backbone network to obtain the corresponding initial feature vector matrix. The matrix is ​​then subjected to dimensionality reduction and feature transformation to obtain the corresponding pre-mapped vector. The pre-mapped vector is input into the activation function for activation mapping to obtain the corresponding weight values. The weight values ​​are then scaled and transformed to obtain a weight sequence. The initial loss of the expression recognition model is corrected based on the weight sequence. The parameters of the expression recognition model are updated based on the corrected loss. This achieves the adjustment of sample loss through network adaptive methods, thereby improving recognition accuracy, reducing the impact of noisy data on model accuracy, optimizing the data processing flow, and significantly improving the accuracy of occluded expression recognition. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram illustrating an application scenario of this application embodiment;

[0013] Figure 2 This is a flowchart illustrating a facial expression recognition method provided in an embodiment of this application;

[0014] Figure 3 This is a schematic diagram of the structure of the facial expression recognition model provided in the embodiments of this application;

[0015] Figure 4 This is a schematic diagram of the structure of the sample importance adaptive weighting module provided in the embodiments of this application;

[0016] Figure 5 This is a schematic diagram of the structure of an expression recognition device provided in an embodiment of this application;

[0017] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0019] A method and apparatus for facial expression recognition according to embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0020] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application. The application scenario may include terminal devices 1, 2, and 3, server 4, and network 5.

[0021] Terminal devices 1, 2, and 3 can be hardware or software. When terminal devices 1, 2, and 3 are hardware, they can be various electronic devices with displays that support communication with server 4, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 1, 2, and 3 are software, they can be installed on the aforementioned electronic devices. Terminal devices 1, 2, and 3 can be implemented as multiple software programs or software modules, or as a single software program or software module; this embodiment does not impose any limitations on this. Furthermore, various applications can be installed on terminal devices 1, 2, and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0022] Server 4 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 4 can be a single server, a server cluster consisting of several servers, or a cloud computing service center; this embodiment of the application does not impose any limitations on these aspects.

[0023] It should be noted that server 4 can be either hardware or software. When server 4 is hardware, it can be various electronic devices that provide various services to terminal devices 1, 2, and 3. When server 4 is software, it can be multiple software programs or software modules that provide various services to terminal devices 1, 2, and 3, or it can be a single software program or software module that provides various services to terminal devices 1, 2, and 3. This application embodiment does not impose any limitations on this.

[0024] Network 5 can be a wired network using coaxial cable, twisted pair, and fiber optic connection, or it can be a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), and Infrared. This application embodiment does not limit this.

[0025] Users can establish a communication connection with server 4 via network 5 through terminal devices 1, 2, and 3 to receive or send information. Specifically, server 4 can obtain multiple training samples from terminal devices 1, 2, or 3, each training sample containing facial expression training images and their labels; and input these samples into the facial expression recognition model to obtain the corresponding recognition results. Based on the recognition results and the true labels of the facial expression training images, the initial loss of the facial expression recognition model is obtained. On the other hand, the facial expression training images are input into the backbone network to obtain the corresponding initial feature vector matrix, which is then subjected to dimensionality reduction and feature transformation to obtain the corresponding pre-mapped vector. The pre-mapped vector is input into the activation function for activation mapping to obtain the corresponding weight values. The weight values ​​are then scaled to obtain a weight sequence. The initial loss of the facial expression recognition model is corrected based on the weight sequence, and the parameters of the facial expression recognition model are updated based on the corrected loss.

[0026] It should be noted that the specific types, quantities, and combinations of terminal devices 1, 2, and 3, server 4, and network 5 can be adjusted according to the actual needs of the application scenario, and this application embodiment does not impose any restrictions on this.

[0027] Figure 2 This is a flowchart illustrating a facial expression recognition method provided in an embodiment of this application. Figure 2 The training method for the facial expression recognition model can be derived from... Figure 1 The server executes this. For example... Figure 2 As shown, the facial expression recognition method includes:

[0028] Step 201: Obtain multiple training samples and input them into the facial expression recognition model. Each training sample contains a facial expression training image and its label.

[0029] Specifically, training samples can be images, but they need to undergo appropriate preprocessing and feature extraction so that the model can extract useful information from them.

[0030] The training samples in the facial expression recognition model include facial expression training images and labels for the facial expression training images. The labels of the facial expression training images are used to indicate the facial expression category of the objects in the facial expression training images, including but not limited to anger, disgust, fear, happiness, sadness, surprise, and neutral. For example, as an example, the facial expression category of the object in the facial expression training image 001 can be category one, the facial expression category of the object in the training image 002 can be category two, and the facial expression category of the object in the training image 003 can be category three. Here, category one can be "happiness", category two can be "surprise", and category three can be "sadness".

[0031] The labels of the facial expression training images were obtained through data annotation, which may contain annotation noise. Multiple training samples provided a large number of different categories of facial expression training images for the training process of the facial expression recognition model, providing a data foundation for the learning and training of the facial expression recognition model and facilitating the feature extraction process of the facial expression recognition model.

[0032] Throughout the training process, the facial expression recognition model improves the accuracy of facial expression recognition by understanding and recognizing facial expressions and associating the facial expressions of objects in the training images with facial expression categories. The feature extraction network is the backbone network, which can be a ResNet50.

[0033] Step 202: Extract features from each expression training image to obtain the initial feature vector matrix of each expression training image.

[0034] Specifically, feature extraction for each facial expression training image can be performed by inputting each facial expression training image into the backbone network, and then extracting features through the backbone network to obtain the corresponding initial feature vector matrices.

[0035] The backbone network is the core structure of a deep learning model, serving as the backbone network structure. It typically comprises multiple layers of convolutional neural networks and plays a crucial role in tasks such as image recognition and object detection. Its primary function is to convert raw images into feature vectors for use by subsequent branch networks. The design and optimization of the backbone network directly determine the performance of the deep learning algorithm. The initialization parameters of the backbone network are those of the face quality model.

[0036] For example, the input to the backbone network is the raw information of the facial expression training image. As an example, it can be a matrix of 10006003. The backbone network extracts the features of the facial expression training image through convolutional layers, pooling layers, etc., and finally outputs an initial feature vector matrix divided into 4 layers (250150256, 12575512, 63381024, 34192048). Taking 250150256 as an example, 250*150 is the feature size and 256 is the number of channels. This is how the initial feature vector matrix corresponding to the facial expression training image is obtained.

[0037] Step 203: Determine the target feature vector of each expression training image based on the initial feature vector matrix of each expression training image.

[0038] Specifically, the target feature vector can be a feature representation of an image, which can be used for tasks such as image classification and object detection. It can be obtained by inputting the initial feature vector matrix of each expression training image into a global average pooling layer, and then obtaining the initial feature vector matrix of each expression training image through dimensionality reduction pooling. The pooled feature vectors corresponding to each expression training image are then input into a fully connected layer for feature transformation. This fully connected layer is a feature transformation matrix initialized based on a random function, which yields the target feature vector of each expression training image.

[0039] For example, the input facial expression training image is an RGB image of size 224*224*3. As an example, firstly, the global average pooling layer will perform average pooling on this facial expression training image. The pooling window size is 8*8. Then the pooling layer will divide the facial expression training image into 28*28 regions, each region of size 8*8, and average the pixels in each region to obtain the average value of each region. The original 224*224*3 image is then transformed into a 28*28*3 output image.

[0040] The output image then enters a fully connected layer. This layer uses a feature transformation matrix initialized with a random function. For example, a fully connected layer with 128 nodes will, as an example, connect to every pixel of the input image. The fully connected layer will perform feature transformation on the input image based on the connection weights and bias terms, converting it into a 128-dimensional feature vector. This feature vector is the output of the fully connected layer, representing the feature representation of the input image, i.e., the target feature vector.

[0041] The global average pooling layer retains the spatial information extracted in the previous steps, while the fully connected layer can integrate the features to form a more abstract and compact feature representation, thus improving the operating efficiency of the facial expression recognition model.

[0042] Step 204: Classify and recognize each target feature vector to obtain the recognition results of each expression training image.

[0043] Specifically, classification and recognition are achieved through a classification layer. By linearly transforming the input target feature vector and adjusting the weights and biases, the target feature vector is converted into another form of feature representation. The features are then classified, that is, the features are mapped to different expression categories, including but not limited to anger, disgust, fear, happiness, sadness, surprise, and neutral. The classification layer classifies the input target feature vector, calculates the probability corresponding to each expression category label, and outputs the expression category label with the highest probability as the recognition result.

[0044] The recognition result can be a label corresponding to anger, disgust, fear, happiness, sadness, surprise, and neutral, etc., without limitation here, and is used to indicate the expression category corresponding to the expression training image judged by the expression recognition model.

[0045] For example, as a case study, the predicted expression category of the object in the expression training image 001 is category one, with a prediction probability of 0.729. The probability that the expression category of the object in the expression training image 001 is category two is 0.016, and the probability that the expression category of the object in the expression training image 001 is category three is 0.255.

[0046] Step 205: Determine the initial loss of the expression recognition model based on the recognition results and labels of each expression training image.

[0047] Specifically, the initial loss can be obtained through the cross-entropy loss function. During the training of the facial expression recognition model, the cross-entropy loss function measures the difference between the image recognition result and the corresponding label for each training image. Based on the image recognition result and the corresponding label for each training image (which can be the true facial expression category of the object in the image), the cross-entropy loss values ​​are calculated. These cross-entropy loss values ​​are then used to constrain the facial expression recognition model, aiming to minimize the cross-entropy loss value during training. This reduces the error rate and improves the prediction accuracy of the facial expression recognition model.

[0048] Step 206: Dimensionality reduction is performed on each initial feature vector matrix, and feature transformation is performed on the dimensionality reduction result to obtain the pre-mapped vector of each expression training image.

[0049] Specifically, the initial feature vector matrix of each expression training image is input into a global average pooling layer. Through dimensionality reduction pooling, the initial feature vector matrix of each expression training image is obtained, and the pooled feature vector of each expression training image is obtained. The pooled feature vector is then input into a fully connected layer for feature transformation. This fully connected layer is a feature transformation matrix initialized based on a random function, which is different from the random initialization parameters of the fully connected layer in step 203. This yields a pre-mapped vector for each expression training image. The pre-mapped vector serves as the input vector for subsequent activation mapping, facilitating the subsequent activation mapping to obtain weight values.

[0050] Step 207: Perform activation mapping processing on the pre-mapped vectors of each expression training image to obtain the weight values ​​corresponding to each expression training image.

[0051] Specifically, activation mapping can be performed by mapping the pre-mapped vector between 0 and 1, so that each expression training image has a corresponding weight value in the range of 0 to 1. A pre-mapped vector can be converted into a probability distribution between 0 and 1. When the output value is 1, it means there is a positive correlation, and when the output value is 0, it means there is no correlation. This can be processed by activation functions, including but not limited to the Sigmoid function, ReLU function, or Tanh function.

[0052] This method yields the weight values ​​corresponding to the pre-mapped vectors of each facial expression training image. The weight values ​​represent the importance of the corresponding facial expression training image, thus expressing the facial expression training image in the form of weight values, which facilitates subsequent weight partitioning and normalization processing.

[0053] Step 208: Scale and transform the weight values ​​corresponding to each expression training image to obtain a weight sequence.

[0054] Specifically, a weight sequence can be a dataset with multiple weight values, each weight value corresponding to a facial expression training image. The weight sequence is used to represent the set of weight values ​​corresponding to all facial expression training images.

[0055] Two different label values ​​are selected from the weight values ​​and scaled and normalized. The weight values ​​are scaled according to the two different selected values ​​to normalize all weight values ​​into a weight sequence with a mean of 0 and a variance of 1.

[0056] For example, as a case study, find the maximum value of 0.8 and the minimum value of 0.1 in the weight sequence.

[0057] For each weight value, normalization is performed: the maximum weight value of 0.8 is mapped to 1, the minimum weight value of 0.1 is mapped to 0, and other weight values ​​are mapped to a value between 0 and 1 based on their relative magnitude to the maximum and minimum values. Thus, the processed weight sequence has a mean of 0 and a variance of 1.

[0058] Step 209: Correct the initial loss of the facial expression recognition model based on the weight sequence, and update the parameters in the facial expression recognition model based on the corrected loss.

[0059] Specifically, the weight sequence representing the importance of the training images of facial expressions is multiplied by the initial loss to output the corrected loss, and the parameters in the facial expression recognition model are updated based on the corrected loss.

[0060] First, the gradient is calculated using the backpropagation algorithm. Then, using the chain rule, the gradient of the corrected loss with respect to the parameters of the facial expression recognition model is passed back to each layer of the model, thus calculating the gradient of the parameters at each layer. Next, the parameters of the facial expression recognition model are updated using the gradient descent algorithm. By iteratively adjusting the parameters, the value of the corrected loss gradually decreases. The learning rate is typically used to control the step size of the parameter updates. The above steps are repeated until the loss of the facial expression recognition model is less than or equal to a preset standard value or a preset number of iterations is reached. In each iteration, the entire training set is used to calculate the loss and gradient, and then all parameters are updated.

[0061] According to the technical solution provided in this application embodiment, multiple training samples are acquired and input into the expression recognition model to obtain corresponding recognition results. Based on the recognition results and the real labels of the expression training images, the initial loss of the expression recognition model is obtained. On the other hand, the expression training images are input into the backbone network to obtain the corresponding initial feature vector matrix. The matrix is ​​then subjected to dimensionality reduction and feature transformation to obtain the corresponding pre-mapped vector. The pre-mapped vector is input into the activation function for activation mapping to obtain the corresponding weight values. The weight values ​​are then scaled and transformed to obtain a weight sequence. The initial loss of the expression recognition model is corrected based on the weight sequence. The parameters of the expression recognition model are updated based on the corrected loss. This achieves the adjustment of sample loss through network adaptive methods, thereby improving recognition accuracy, reducing the impact of noisy data on model accuracy, optimizing the data processing flow, and significantly improving expression recognition accuracy.

[0062] In some embodiments, before extracting features from each expression training image to obtain an initial feature vector matrix for each expression training image, the method further includes: determining a random value, generating occlusion pixels based on the random value, and generating an occlusion matrix based on the occlusion pixels; determining the occlusion position of each expression training image based on a random function; and occluding the occlusion position of each expression training image using the occlusion matrix to obtain each occluded image.

[0063] Specifically, a random number of facial expression training images from the input training samples are randomly occluded, with a region being occluded. Due to the random occlusion, the size and shape of the occluded rectangular region may differ across different facial expression training images, generating various facial expression training images with different occlusion levels. An occlusion matrix of random size is also generated, with pixel values ​​in this matrix being random. This occlusion matrix is ​​then used to occlude each facial expression training image, resulting in occluded images. The occluded position in each facial expression training image can be any location on the image. The occluded images obtained after occlusion enhancement differ in features from the corresponding facial expression training images, but the labels of the facial expression training images and their corresponding occluded images remain consistent. This reduces the probability of overfitting and improves the robustness of the facial expression recognition model to occlusion.

[0064] According to the technical solution provided in the embodiments of this application, by generating a random-sized occlusion matrix at random positions of the expression training image, and occluding the random positions of the expression training image through the occlusion matrix, an occluded image is obtained. This reduces the probability of overfitting, improves the robustness of the expression recognition model to occlusion, enriches the types of training samples, and improves the training level of the expression recognition model.

[0065] In some embodiments, the expression recognition model includes a global average pooling layer and a fully connected layer; dimensionality reduction is performed on each initial feature vector matrix, and feature transformation is performed on the dimensionality reduction result to obtain a pre-mapped vector for each expression training image, including: inputting the initial feature vector matrix of each expression training image into the global average pooling layer, and performing dimensionality reduction on the initial feature vector matrix of each expression training image through the global average pooling layer to obtain the dimensionality reduction result of each expression training image; inputting the dimensionality reduction result into the fully connected layer, and performing feature transformation on the dimensionality reduction result through the fully connected layer to obtain the pre-mapped vector of each expression training image.

[0066] Specifically, global average pooling is a commonly used pooling method. It calculates the average value of all pixels in the initial input feature vector matrix for each channel, resulting in a new 1x1 channel map. This pooling method is frequently used in convolutional neural networks, especially in classification tasks.

[0067] In typical convolutional neural networks, convolutional layers are used to extract features from two-dimensional data, including but not limited to images or videos. These features are then fed into different types of networks for tasks such as classification, regression, and image segmentation. When using a global average pooling layer, the average value of all pixels in the initial feature vector matrix for each channel is calculated; this average value represents the feature of that channel. This effectively reduces the number of parameters in the model and improves its generalization ability. Furthermore, because the global average pooling layer considers every pixel in the initial input feature vector matrix, it can retain more detailed information to a certain extent.

[0068] Fully connected layers can be used to map the dimensionality reduction result to the sample label space, that is, to map the original data to the hidden feature space. Each node in a fully connected layer is connected to all nodes in the previous global average pooling layer, so it can synthesize the features extracted by the previous global average pooling layer, i.e., the dimensionality reduction result. Due to the fully connected nature of fully connected layers, they usually have the largest number of parameters. For example, as an example, in VGG16, the first fully connected layer FC1 has 4096 nodes, and the previous layer POOL2 has 77512 = 25088 nodes. This transfer requires 4096 * 25088 weights, which will consume a lot of memory. Therefore, in practical applications, fully connected layers usually appear in the last few layers to weight the features designed earlier. This reduces the number of parameters and the amount of computation.

[0069] According to the technical solution provided in the embodiments of this application, the initial feature vector matrix of each expression training image is input into a global average pooling layer, and the initial feature vector matrix of each expression training image is reduced in dimensionality by the global average pooling layer to obtain the dimensionality reduction result of each expression training image. Then, the dimensionality reduction result is transformed by a fully connected layer to obtain the pre-mapped vector of each expression training image. The pre-mapped vector is used to activate the mapping in subsequent steps to obtain the weight value of the expression training image. This reduces the number of parameters of the model, improves the generalization ability of the model, weakens the number of parameters, reduces the amount of computation, and improves the recognition accuracy of the expression recognition model.

[0070] In some embodiments, scaling transformation is performed on the weight values ​​corresponding to each expression training image to obtain a weight sequence, including: determining a first selected value and a second selected value; scaling and normalizing the weight values ​​corresponding to each expression training image according to the first selected value and the second selected value to obtain a weight sequence.

[0071] Specifically, a weight sequence can be a dataset with multiple weight values, each weight value corresponding to a facial expression training image. The weight sequence is used to represent the set of weight values ​​corresponding to all facial expression training images.

[0072] The first and second selected values ​​can be used as marker values ​​for calculating other weight values. By selecting the first and second selected values ​​from the weight values, scaling and normalization processing is performed. The weight values ​​are scaled according to the first and second selected values ​​to normalize all weight values ​​into a weight sequence with a mean of 0 and a variance of 1.

[0073] For example, as a case study, we iterate through the weight sequence and obtain the first selected value 0.1 and the second selected value 0.9. We then normalize the weight value 0.5: (0.5-0.1) / (0.9-0.1)=0.5. 0.5 is the new weight value in the weight sequence after scaling and normalization.

[0074] Thus, the second selected value in the original dataset is mapped to 1, the first selected value is mapped to 0, and other weight values ​​are mapped to a value between 0 and 1 based on their relative magnitude to the first and second selected values.

[0075] According to the technical solution provided in the embodiments of this application, by selecting two different values ​​as two selected values ​​from the weight values, and scaling and normalizing the other weight values ​​based on the two selected values, a new weight sequence containing multiple new weight values ​​is obtained, thereby reducing the influence of noisy data on the expression recognition model and improving the accuracy of the expression recognition model.

[0076] In some embodiments, before scaling and transforming the weight values ​​corresponding to each expression training image to obtain a weight sequence, the method further includes: dividing the weight values ​​into a high-weight interval and a low-weight interval according to a preset weight threshold; sorting the weight values ​​in the high-weight interval and the low-weight interval respectively to determine the maximum weight value and the minimum weight value.

[0077] Specifically, the preset weight threshold can be a manually set threshold used to divide the weight values ​​into high weight ranges and low weight ranges. Each weight value is compared with the preset weight threshold. Values ​​greater than the preset weight threshold are weight values ​​in the high weight range, and values ​​less than the preset weight threshold are weight values ​​in the low weight range.

[0078] The weight values ​​of the high-weight interval and the low-weight interval are sorted separately, and the maximum and minimum weight values ​​of the two weight intervals are selected respectively. This reduces the computational cost of subsequent scaling and normalization, simplifies the subsequent steps, and improves the training efficiency of the facial expression recognition model.

[0079] For example, the preset weight threshold is 0.5, with 0.5 as the threshold limit. As an example, 0.6 is the weight value of the high weight range, and 0.4 is the weight value of the low weight range.

[0080] In the high-weighted interval and the low-weighted interval, the maximum and minimum values ​​are selected respectively. For example, in the high-weighted interval, the maximum value is 0.9 and the minimum value is 0.3, while in the low-weighted interval, the maximum value is 0.8 and the minimum value is 0.2.

[0081] According to the technical solution provided in the embodiments of this application, by presetting a weight threshold, all weight values ​​are divided into two partitions: a high-weight interval and a low-weight interval. The values ​​are then sorted in the two partitions to select the maximum and minimum values ​​of the high-weight interval and the low-weight interval. This reduces the computational load of subsequent scaling and normalization, simplifies subsequent steps, improves the training efficiency of the facial expression recognition model, and enhances the recognition accuracy of the facial expression recognition model.

[0082] In some embodiments, the initial loss of the expression recognition model is corrected based on the weight sequence, and the parameters in the expression recognition model are updated based on the corrected loss. This includes: multiplying the weight sequence and the initial loss of the expression recognition model to obtain the corrected loss; calculating the gradient corresponding to each expression training image through the backpropagation algorithm to obtain the corresponding gradient value; updating the expression training images through the gradient descent algorithm, and stopping when the corrected loss is less than or equal to a preset value to obtain the trained expression recognition model.

[0083] Specifically, the corrected loss is obtained by multiplying the weight sequence and the initial loss of the expression recognition model. Then, the gradient values ​​corresponding to the expression training images are calculated, and the expression training images are updated based on the gradient descent algorithm.

[0084] The process involves calculating the error between the recognition result and the labeled facial expression training images, and then calculating the gradient of the loss function with respect to each parameter. This is achieved through the backpropagation algorithm, specifically by calculating the gradient of each intermediate variable sequentially, starting from the output layer, according to the chain rule, until the input layer is reached.

[0085] Backpropagation is an algorithm used to train artificial neural networks. It involves inputting multiple training samples into the input layer of the artificial neural network, through the hidden layer, and finally reaching the output layer to output the result. This is the forward propagation process of the artificial neural network. Since there is an error between the output result of the artificial neural network and the actual result, the error between the estimated value and the actual value is calculated and propagated backward from the output layer to the hidden layer until it reaches the input layer. During the backpropagation process, the values ​​of various parameters are adjusted according to the error. The above process is iterated until convergence.

[0086] After obtaining the gradient of each parameter, the value of each parameter is updated using the gradient descent algorithm. Based on the gradient of each parameter and the learning rate, the value of the parameter is adjusted in the direction of reducing the loss function.

[0087] Then, repeat the above steps until the value of the loss function is less than or equal to this preset value. At this point, we consider the facial expression recognition model to have been successfully trained.

[0088] According to the technical solution provided in this application, the corrected loss is obtained by multiplying the weight sequence and the initial loss of the expression recognition model. Then, the gradient value corresponding to the expression training image is calculated, and the expression training image is updated based on the gradient descent algorithm. This reduces the influence of noisy data on the expression recognition model, increases the influence of correct recognition results on the expression recognition model, and improves the accuracy of the expression recognition model's prediction results.

[0089] In some embodiments, after updating the parameters in the expression recognition model based on the corrected loss, the method further includes: acquiring an image to be recognized, which includes an occluded image to be recognized or an unoccluded image to be recognized; extracting features from the image to be recognized to obtain a feature vector matrix to be recognized; performing dimensionality reduction on the feature vector matrix to be recognized to obtain a dimensionality reduction result of the image to be recognized; performing feature transformation on the dimensionality reduction result of the image to be recognized to obtain a feature vector to be recognized of the image to be recognized; and performing classification and recognition on the feature vector to be recognized of the image to be recognized to obtain an expression recognition result corresponding to the image to be recognized, wherein the expression recognition result is used to indicate the expression category of the image to be recognized.

[0090] Specifically, the above process describes the application of an expression recognition model. After acquiring the image to be recognized, it is input into the expression recognition model. The backbone network of the model extracts features from the image to obtain a feature vector matrix. Key feature information is extracted and represented as a vector matrix. This feature vector matrix is ​​then input into the global average pooling layer of the expression recognition model for dimensionality reduction. The dimensionality reduction result is then input into the fully connected layer of the model for feature transformation. The dimensionality reduction results are integrated to obtain the target feature vector of the image to be recognized. This allows the expression recognition model to better understand and learn the features of each training image, improving the model's classification accuracy and generalization ability. Finally, the target feature vector of the image to be recognized is input into the classification layer for classification, yielding the expression recognition result. This result indicates the expression category of the object in the image.

[0091] According to the technical solution provided in the embodiments of this application, the application process of this facial expression recognition model reduces the complexity of the image recognition model, prevents overfitting, and improves the classification accuracy and generalization ability of the model.

[0092] Figure 3This is a schematic diagram of the structure of the facial expression recognition model provided in the embodiments of this application, as shown below. Figure 3 As shown, the structure of this facial expression recognition model includes:

[0093] Backbone network 301 is used to extract the initial feature vector matrix of each expression training image from each expression training image;

[0094] The first global average pooling layer 302 is used to reduce the dimensionality of the initial feature vector matrix of each expression training image to obtain the dimensionality reduction result.

[0095] The first fully connected layer 303 is used to perform feature transformation on the dimensionality reduction result to obtain the target feature vector of each expression training image;

[0096] Classification layer 304 is used to classify and recognize each target feature vector to obtain the recognition results of each expression training image;

[0097] The sample importance adaptive weighting module 305 is used to determine the weight sequence formed by the set of weight values ​​corresponding to each expression training image.

[0098] The structure of this facial expression recognition model involves obtaining the corresponding initial feature vector matrix from each training image through the backbone network 301, inputting it into the first global average pooling layer 302 for dimensionality reduction, and then inputting the dimensionality reduction result into the first fully connected layer 303 to obtain the corresponding target feature vector. The target feature vector is then classified and recognized by the classification layer 304 to obtain the recognition result. On the other hand, the initial feature vector matrix is ​​input into the sample importance adaptive weighting module 305 to obtain the weight sequence. The initial loss is corrected based on the initial loss and the weight sequence, and the facial expression recognition model is updated in reverse, thereby improving the accuracy of the facial expression recognition model and reducing the impact of noise on the facial expression recognition model.

[0099] Figure 4 This is a schematic diagram of the structure of the adaptive weighting module for sample importance provided in an embodiment of this application, as shown below. Figure 4 As shown, the structure of this sample importance adaptive weighting module includes:

[0100] The second global average pooling layer 401 is used to reduce the dimensionality of the initial feature vector matrix of each expression training image to obtain the dimensionality reduction result.

[0101] The second fully connected layer 402 is used to perform feature transformation on the dimensionality reduction result to obtain the pre-mapped feature vectors of each expression training image.

[0102] The sample importance adaptive weighting module can perform dimensionality reduction through the second global average pooling layer 401 to obtain the dimensionality reduction result. The dimensionality reduction result is input into the pre-mapped feature vector corresponding to the second fully connected layer 402, and activation mapping is performed through the activation function to obtain the corresponding weight values. The corresponding weight values ​​are divided into high weight intervals and low weight intervals through a preset weight threshold. The weight values ​​are sorted in the two intervals respectively, and the maximum and minimum values ​​of the two intervals are selected. Other weight values ​​are scaled and normalized, and the two sets of normalized results are output. This improves the accuracy of the expression recognition model and increases the influence of the correct label on the model.

[0103] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0104] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0105] Figure 5 This is a schematic diagram of the structure of an expression recognition device provided in an embodiment of this application. Figure 5 As shown, the facial expression recognition device includes:

[0106] The acquisition module 501 is used to acquire multiple training samples and input the multiple training samples into the expression recognition model. Each training sample contains an expression training image and a label for the expression training image.

[0107] The first processing module 502 is used to extract features from each expression training image to obtain the initial feature vector matrix of each expression training image.

[0108] The second processing module 503 is used to determine the target feature vector of each expression training image based on the initial feature vector matrix of each expression training image.

[0109] The third processing module 504 is used to classify and recognize each target feature vector to obtain the recognition results of each expression training image.

[0110] The fourth processing module 505 is used to determine the initial loss of the expression recognition model based on the recognition results of each expression training image and the labels of each expression training image.

[0111] The fifth processing module 506 is used to perform dimensionality reduction on each initial feature vector matrix and to perform feature transformation on the dimensionality reduction result to obtain the pre-mapped vector of each expression training image.

[0112] The sixth processing module 507 is used to perform activation mapping processing on the pre-mapped vectors of each expression training image to obtain the weight values ​​corresponding to each expression training image.

[0113] The seventh processing module 508 is used to scale and transform the weight values ​​corresponding to each expression training image to obtain a weight sequence;

[0114] The update module 509 is used to correct the initial loss of the expression recognition model based on the weight sequence, and to update the parameters in the expression recognition model based on the corrected loss.

[0115] This facial expression recognition device acquires multiple training samples and inputs them into an facial expression recognition model to obtain corresponding recognition results. Based on the recognition results and the real labels of the facial expression training images, the initial loss of the facial expression recognition model is obtained. On the other hand, the facial expression training images are input into the backbone network to obtain the corresponding initial feature vector matrix. This matrix is ​​then subjected to dimensionality reduction and feature transformation to obtain the corresponding pre-mapped vector. The pre-mapped vector is input into an activation function for activation mapping to obtain the corresponding weight values. The weight values ​​are then scaled to obtain a weight sequence. The initial loss of the facial expression recognition model is corrected based on the weight sequence. The parameters of the facial expression recognition model are then updated based on the corrected loss. This achieves the adjustment of sample loss through network adaptive methods, thereby improving recognition accuracy, reducing the impact of noisy data on model accuracy, optimizing the data processing flow, and significantly improving the accuracy of occluded facial expression recognition.

[0116] In some embodiments, before extracting features from each expression training image to obtain an initial feature vector matrix for each expression training image, the expression recognition device is further configured to: determine a random value; generate occlusion pixels based on the random value and generate an occlusion matrix based on the occlusion pixels; determine the occlusion position of each expression training image based on a random function; and occlude the occlusion position of each expression training image using the occlusion matrix to obtain each occluded image.

[0117] In some embodiments, the fifth processing module 506 is specifically used to input the initial feature vector matrix of each expression training image into a global average pooling layer, and perform dimensionality reduction processing on the initial feature vector matrix of each expression training image through the global average pooling layer to obtain the dimensionality reduction result of each expression training image; input the dimensionality reduction result into a fully connected layer, and perform feature transformation on the dimensionality reduction result through the fully connected layer to obtain the pre-mapped vector of each expression training image.

[0118] In some embodiments, the seventh processing module 508 is specifically used to determine a first selected value and a second selected value; and to scale and normalize the weight values ​​corresponding to each expression training image according to the first selected value and the second selected value to obtain a weight sequence.

[0119] In some embodiments, before scaling and transforming the weight values ​​corresponding to each expression training image to obtain a weight sequence, the expression recognition device is further configured to divide the weight values ​​into a high-weight interval and a low-weight interval according to a preset weight threshold; sort the weight values ​​in the high-weight interval and the low-weight interval respectively to determine the maximum weight value and the minimum weight value.

[0120] In some embodiments, the update module 509 is specifically used to multiply the weight sequence and the initial loss of the expression recognition model to obtain the corrected loss; calculate the gradient corresponding to each expression training image through the backpropagation algorithm to obtain the corresponding gradient value; update the expression training images through the gradient descent algorithm, and stop when the corrected loss is less than or equal to a preset value to obtain the trained expression recognition model.

[0121] In some embodiments, after updating the parameters in the expression recognition model based on the corrected loss, the expression recognition device is further configured to: acquire an image to be recognized, the image to be recognized including an occluded image to be recognized or an unoccluded image to be recognized; extract features from the image to be recognized to obtain a feature vector matrix to be recognized; perform dimensionality reduction processing on the feature vector matrix to be recognized to obtain a dimensionality reduction result of the image to be recognized; perform feature transformation on the dimensionality reduction result of the image to be recognized to obtain a feature vector to be recognized of the image to be recognized; and perform classification recognition on the feature vector to be recognized of the image to be recognized to obtain an expression recognition result corresponding to the image to be recognized, the expression recognition result being used to indicate the expression category of the image to be recognized.

[0122] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0123] Figure 6 This is a schematic diagram of the electronic device 6 provided in an embodiment of this application. Figure 6 As shown, the electronic device 6 of this embodiment includes a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, it implements the steps in the various method embodiments described above. Alternatively, when the processor 601 executes the computer program 603, it implements the functions of each module / unit in the various device embodiments described above.

[0124] Electronic device 6 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 6 may include, but is not limited to, processor 601 and memory 602. Those skilled in the art will understand that... Figure 6This is merely an example of electronic device 6 and does not constitute a limitation on electronic device 6. It may include more or fewer components than shown, or different components.

[0125] The processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0126] The memory 602 can be an internal storage unit of the electronic device 6, such as a hard disk or RAM of the electronic device 6. The memory 602 can also be an external storage device of the electronic device 6, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 6. The memory 602 can also include both internal and external storage units of the electronic device 6. The memory 602 is used to store computer programs and other programs and data required by the electronic device.

[0127] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0128] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which may be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0129] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for facial expression recognition, characterized in that, include: Multiple training samples are acquired and input into the facial expression recognition model. Each training sample contains a facial expression training image and a label for the facial expression training image. Feature extraction is performed on each of the aforementioned facial expression training images to obtain an initial feature vector matrix for each of the aforementioned facial expression training images; Based on the initial feature vector matrix of each of the expression training images, the target feature vector of each of the expression training images is determined; The target feature vectors are classified and identified to obtain the recognition results of each of the expression training images; The initial loss of the expression recognition model is determined based on the recognition results of each of the expression training images and the labels of each of the expression training images; The initial feature vector matrices are dimensionality reduced, and the dimensionality reduction results are transformed to obtain the pre-mapped vectors of the expression training images. Activation mapping processing is performed on the pre-mapped vectors of each of the expression training images to obtain the weight values ​​corresponding to each of the expression training images; The weight values ​​corresponding to each of the aforementioned facial expression training images are scaled and transformed to obtain a weight sequence; The initial loss of the facial expression recognition model is corrected based on the weight sequence, and the parameters in the facial expression recognition model are updated based on the corrected loss. Obtain an image to be identified, which includes an occluded image to be identified or an unoccluded image to be identified; Feature extraction is performed on the image to be identified to obtain the feature vector matrix to be identified; The dimensionality reduction process is performed on the feature vector matrix to be identified to obtain the dimensionality reduction result of the image to be identified; The dimensionality reduction result of the image to be identified is subjected to feature transformation to obtain the feature vector to be identified of the image to be identified; The feature vector of the image to be identified is classified and identified to obtain the expression recognition result corresponding to the image to be identified. The expression recognition result is used to indicate the expression category of the image to be identified.

2. The facial expression recognition method according to claim 1, characterized in that, Before performing feature extraction on each of the expression training images to obtain the initial feature vector matrix of each of the expression training images, the method further includes: Determine a random value, generate occlusion pixels based on the random value, and generate an occlusion matrix based on the occlusion pixels; The occlusion position of each of the facial expression training images is determined based on a random function; The occlusion matrix is ​​used to occlude the occlusion positions of each of the expression training images to obtain each occluded image.

3. The facial expression recognition method according to claim 1, characterized in that, The facial expression recognition model includes a global average pooling layer and a fully connected layer; The step of reducing the dimensionality of each of the initial feature vector matrices and performing feature transformation on the dimensionality reduction result to obtain the pre-mapped vectors of each of the expression training images includes: The initial feature vector matrix of each of the expression training images is input into the global average pooling layer, and the initial feature vector matrix of each of the expression training images is subjected to the dimensionality reduction processing through the global average pooling layer to obtain the dimensionality reduction result of each of the expression training images. The dimensionality reduction result is input into the fully connected layer, and the dimensionality reduction result is transformed by the fully connected layer to obtain the pre-mapped vector of each of the expression training images.

4. The facial expression recognition method according to claim 1, characterized in that, The scaling and transformation of the weight values ​​corresponding to each of the expression training images to obtain a weight sequence includes: Determine the first selected value and the second selected value; Based on the first selected value and the second selected value, the weight values ​​corresponding to each of the expression training images are scaled and normalized to obtain the weight sequence.

5. The facial expression recognition method according to claim 1, characterized in that, Before scaling and transforming the weight values ​​corresponding to each of the expression training images to obtain the weight sequence, the method further includes: According to the preset weight threshold, the weight values ​​are divided into a high-weight interval and a low-weight interval; The weight values ​​are sorted in the high-weight interval and the low-weight interval respectively to determine the maximum weight value and the minimum weight value.

6. The facial expression recognition method according to claim 1, characterized in that, The step of correcting the initial loss of the expression recognition model based on the weight sequence, and updating the parameters in the expression recognition model based on the corrected loss, includes: The corrected loss is obtained by multiplying the weight sequence and the initial loss of the facial expression recognition model. The gradient corresponding to each of the aforementioned facial expression training images is calculated using the backpropagation algorithm to obtain the corresponding gradient value; The facial expression training images are updated using a gradient descent algorithm, and the process stops when the corrected loss is less than or equal to a preset value, thus obtaining the trained facial expression recognition model.

7. A facial expression recognition device, characterized in that, include: An acquisition module is used to acquire multiple training samples and input the multiple training samples into an expression recognition model, wherein each training sample contains an expression training image and a label for the expression training image; The first processing module is used to extract features from each of the expression training images to obtain an initial feature vector matrix for each of the expression training images. The second processing module is used to determine the target feature vector of each of the expression training images based on the initial feature vector matrix of each expression training image. The third processing module is used to classify and recognize each of the target feature vectors to obtain the recognition results of each of the expression training images; The fourth processing module is used to determine the initial loss of the expression recognition model based on the recognition results of each of the expression training images and the labels of each of the expression training images. The fifth processing module is used to perform dimensionality reduction processing on each of the initial feature vector matrices, and to perform feature transformation on the dimensionality reduction processing results to obtain the pre-mapped vectors of each of the expression training images. The sixth processing module is used to perform activation mapping processing on the pre-mapped vectors of each of the expression training images to obtain the weight values ​​corresponding to each of the expression training images. The seventh processing module is used to scale and transform the weight values ​​corresponding to each of the expression training images to obtain a weight sequence; An update module is used to correct the initial loss of the expression recognition model based on the weight sequence, and to update the parameters in the expression recognition model based on the corrected loss. The facial expression recognition device is further configured to: acquire an image to be recognized, the image to be recognized including an occluded image to be recognized or an unoccluded image to be recognized; and extract features from the image to be recognized to obtain a feature vector matrix to be recognized. The dimensionality reduction process is performed on the feature vector matrix to be identified to obtain the dimensionality reduction result of the image to be identified; the dimensionality reduction result of the image to be identified is subjected to feature transformation to obtain the feature vector to be identified of the image to be identified; the feature vector to be identified of the image to be identified is classified and identified to obtain the expression recognition result corresponding to the image to be identified, and the expression recognition result is used to indicate the expression category of the image to be identified.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images