Multi-task learning facial expression recognition method based on voting mechanism

Abstract

The invention provides a multi-task learning facial expression recognition method based on a voting mechanism, and relates to the field of computer vision detection, and the method comprises the steps: S1, constructing a sample set based on obtained historical facial expression data; s2, performing statistics on the training set data to obtain quantity data of each expression, and processing the training set data based on the quantity data and data enhancement to obtain an enhanced training set; s3, performing feature point marking and face cutting on the images in the enhanced training set; S4, constructing a multi-task convolutional neural network model based on a voting mechanism; s5, obtaining a final facial expression recognition model; and S6, obtaining a corresponding facial expression recognition result based on the facial expression recognition model. According to the method, the final facial expression recognition result is selected through a voting mechanism by constructing the multi-task convolutional neural network model, and the problem that a single feature extraction mode is easily interfered by irrelevant factors is solved.

Description

Technical Field

[0001] This application relates to the field of computer vision detection, and in particular to a facial expression recognition method based on multi-task learning with a voting mechanism. Background Technology

[0002] With the development of artificial intelligence, facial expression recognition technology has been widely applied in fields such as affective computing and human-computer interaction. With the rapid development of deep learning technology, the performance of Facial Expression Recognition (FER) systems in automatically detecting and classifying facial expressions has been significantly improved. Typical FER systems rely on convolutional neural networks (CNNs) to analyze static facial images to identify facial features and predict expression categories. However, single feature extraction methods still face certain challenges in complex real-world scenarios, such as the diversity of expressions, background interference, lighting variations, and occlusion.

[0003] Facial key feature point detection and facial alignment techniques have been introduced into expression recognition systems to enhance the accuracy and robustness of recognition. By detecting key points on the face, such as the eyes, nose, and corners of the mouth, geometric information of the face can be obtained, further helping the model understand and distinguish subtle changes in facial expressions.

[0004] The application of facial cropping technology can reduce the negative impact of image noise on expression recognition during the recognition process. By removing unnecessary background information from the expression image and cropping out the facial expression for expression recognition, the accuracy and robustness of the recognition can be improved. Summary of the Invention

[0005] This invention aims to solve the problems existing in the current facial expression recognition technology and designs a facial expression recognition method that can utilize multiple information sources and tasks. This method improves the recognition accuracy by introducing multiple input sources and combining feature point labeled images and cropped face images to perform multi-task learning.

[0006] The facial expression recognition method based on a voting mechanism and multi-task learning provided by this invention includes the following steps: S1. Construct a sample set based on the acquired historical facial expression data, and divide the sample set into a training set, a validation set, and a test set; S2, Statistically obtain the quantity data of each expression from the training set data, and process the training set data based on the quantity data and data augmentation to obtain the augmented training set; S3, perform feature point labeling and face cropping on the images in the enhanced training set to obtain the first input image and the second input image, respectively; S4, Construct a multi-task convolutional neural network model based on a voting mechanism; S5, the final facial expression recognition model is obtained by optimizing the multi-task convolutional neural network model based on the sample set; S6, based on the facial expression recognition model and the acquired real-time facial expression images, obtains the corresponding facial expression recognition results.

[0007] Furthermore, the enhanced training set in S2 also includes: Count the number of data points for each expression in the training set; The number of data points for each expression is compared with a preset value. If the number of data points for the current expression is less than the preset value, data augmentation is performed on the current expression. The data augmentation operation includes randomly rotating the current expression image, horizontally translating, vertically translating, randomly scaling, and shearing to obtain an augmented training set.

[0008] Furthermore, S3, which obtains the first input image and the second input image respectively, also includes: S31. Perform feature point detection on the original expression image, extract the positions of 68 facial feature points, perform face alignment based on the coordinates of the eyes, perform face alignment on the expression images in the training set, perform facial feature point detection on the face-aligned image, mark the 68 facial feature points, generate an image marked with feature points, and perform a resize operation on the image marked with feature points to obtain the first input image. S32. Remove noise from the enhanced training set facial expression images using Gaussian blur; Perform face region detection and crop out non-face regions to obtain the cropped face image; Facial feature point detection was performed on the cropped face image, resulting in the locations of 68 feature points; The face is aligned based on the coordinates of the eyes to generate a cropped image. The image after facial cropping is resized to ensure consistent image size, resulting in the second input image.

[0009] Furthermore, step S4, which involves constructing a multi-task convolutional neural network model based on a voting mechanism, also includes: S41. Construct the convolutional layer and pooling layer of the first branch of the convolutional neural network model for the first input image, to extract the image features X1 of the first input image; S42. The convolutional and pooling layers of the second branch of the convolutional neural network model for the second input image are used to extract the image features X2 of the second input image. S43. Perform feature fusion on the corresponding image features extracted in S41 and S42 to obtain fused feature X3; S44. Construct a shared fully connected layer, input X1, X2 and X3, generate their respective results, and input the three results generated above into the output layer to generate three output results, labeled as output_1, output_2 and output_3 respectively; S45. Build a custom layer with voting functionality.

[0010] Furthermore, step S4, which involves building a custom layer with voting functionality, also includes: S451. Input output_1, output_2 and output_3 into the custom layer; for the three inputs, according to their respective weights, use the reduce_sum function in TensorFlow to perform tensor summation operation to obtain the cumulative number of votes for each emoji category; ; Let R be the cumulative number of votes for the j-th emoji, and R be the set of all dimensions of the current emoji; let the input tensor be... The set of dimensions to be summed is r is the current dimension of the current expression; for all Based on index Summing within the corresponding dimension, for all ,index Keep it fixed and unchanged; S452. Use the argmax function in TensorFlow to calculate the category with the most votes. Along the dimension of emoji types, get the index of the category with the most votes and output it.

[0011] ； For a function f(x) or a mapping When x takes values ​​in the range S (i.e. ), and the result of argmax is the set of x points that make f(x) reach its maximum value.

[0012] Furthermore, step S5 includes the following sub-steps: S51, Training a multi-task convolutional neural network model using the Adam gradient descent optimization algorithm:

[0013] Where t is the number of iterations. Here, α is the weight parameter, and α is the learning rate. for The correction, yes The correction, To use a very small number, to prevent the denominator from being 0; S52, set a cross-entropy loss function for each subtask and optimize it using custom loss weights; ; Where L is the loss value and n is the number of categories. These are the labels of the true categories of the input samples. This represents the output value of the softmax function; S53, Set the number of training iterations (epochs); S54, in each iteration, 64 samples are drawn from the training set and one sample is randomly drawn from the validation set; S55. Set an early stopping mechanism to monitor the loss function of the output result of the validation set in each iteration. If the loss function does not decrease further after several iterations, the iteration is terminated early and the model parameters are restored to the model parameters of the iteration with the lowest loss function, thus obtaining the trained multi-task convolutional neural network model. S56. Randomly select 64 samples from a pre-divided test set to test the trained multi-task convolutional neural network model and obtain the facial expression recognition accuracy.

[0014] Compared with existing technologies, the facial expression recognition method based on voting mechanism multi-task learning provided by this invention has the following beneficial effects: This invention provides a facial expression recognition method based on a voting mechanism using a multi-task learning convolutional neural network. By utilizing multiple information sources and tasks, the convolutional neural network extracts features from different information sources, fuses different image features, and combines multi-task learning with a voting mechanism, significantly improving the accuracy and robustness of facial expression recognition, especially in complex real-world scenarios where it has stronger adaptability. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the facial expression recognition method based on voting mechanism multi-task learning according to the present invention; Figure 2This is a detailed operation flowchart of S3 of the facial expression recognition method based on voting mechanism of the present invention. Figure 3 This is a schematic diagram of the structure of the multi-task convolutional neural network model of the present invention; Figure 4 This is the structure of convolutional and pooling layers used to process the input image in the multi-task convolutional neural network model of the present invention; Figure 5 This refers to the convolutional and pooling layer structures used to process the input image 2 in the multi-task convolutional neural network model of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described embodiments are merely some embodiments of the invention, and not all embodiments.

[0020] Figure 1 This is a flowchart of a facial expression recognition method based on a voting mechanism using multi-task learning, provided in an embodiment of the present invention; as shown below. Figure 1 As shown, it includes the following steps: S1. Construct a sample set based on the acquired historical facial expression data, and divide the sample set into a training set, a validation set, and a test set; S2, Statistically obtain the quantity data of each expression from the training set data, and process the training set data based on the quantity data and data augmentation to obtain the augmented training set; S3, perform feature point labeling and face cropping on the images in the enhanced training set to obtain the first input image and the second input image, respectively; S4, Construct a multi-task convolutional neural network model based on a voting mechanism; S5, the final facial expression recognition model is obtained by optimizing the multi-task convolutional neural network model based on the sample set; S6, based on the facial expression recognition model and the acquired real-time facial expression images, obtains the corresponding facial expression recognition results.

[0021] Furthermore, the enhanced training set in S2 also includes: Count the number of data points for each expression in the training set; The number of data points for each expression is compared with a preset value. If the number of data points for the current expression is less than the preset value, data augmentation is performed on the current expression. The data augmentation operation includes randomly rotating the current expression image, horizontally translating, vertically translating, randomly scaling, and shearing to obtain an augmented training set.

[0022] Furthermore, in S3, the first input image and the second input image are obtained respectively. Specifically, Figure 2 This is a detailed flowchart of step S3 of the facial expression recognition method based on a voting mechanism and multi-task learning convolutional neural network of the present invention. Figure 2 As shown: The first input image of the multi-task convolutional neural network model is processed as follows: Feature point detection was performed on the original facial expression image to extract the positions of 68 facial feature points; Based on the coordinates of the eye feature points, the face image in the image is corrected so that the face image in the image is aligned with the Cartesian coordinate system. Facial feature point detection is performed on the face-aligned image; Sixty-eight feature points on the face are labeled to generate an image with labeled feature points; Perform a resize operation on the image with marked feature points to ensure that the image size is consistent.

[0023] The second input image of the multi-task convolutional neural network model is processed as follows: Gaussian blur is applied to the original facial expression image to remove noise; The image is subjected to face detection, and the non-face areas of the image are cropped out, leaving only the faces in the image. Feature point detection was performed on the cropped face image to extract the positions of 68 facial feature points; Based on the coordinates of the eye feature points, the face image in the image is corrected so that the face image in the image is aligned with the Cartesian coordinate system. Perform a resize operation on the cropped face image to ensure that the image size remains consistent; Furthermore, in step S4, a multi-task convolutional neural network model based on a voting mechanism is constructed, such as... Figure 3 The diagram shown is a schematic representation of the multi-task convolutional neural network model of this invention: Construct a convolutional neural network model for the first input image, including convolutional and pooling layers in the first branch, to extract image features X1 from the first input image; The second branch of the convolutional neural network model for the second input image contains convolutional and pooling layers for extracting image features X2 from the second input image. Image features X1 and X2 are fused using concatenate to form fused feature X3. Concatenate can join two tensors of the same size along a specific axis. The formula is: Concatenate ); in, Let k represent two tensors, where k represents the k-th axis.

[0024] Using a shared fully connected layer, image feature one, image feature two, and fused features are input into the fully connected layer, and then passed through a softmax layer to generate their respective output results.

[0025] The mathematical expression for the softmax function is: = ; in, Let represent the probability value of class (i) output by the softmax function. This represents the (i)th element of the input. This represents the exponent of the (i)th input value. The sum of the exponents of all input values.

[0026] Use Lambda to build a custom layer with voting functionality. Input the three output results mentioned above into the custom layer, and use reduce_sum to perform a tensor summation operation on the three inputs. The result is a tensor of shape representing the number of emoji types, which represents the cumulative number of "votes" for each emoji type. The argmax algorithm is used to calculate the category with the most votes. The index of the category with the most votes is found along the dimension of expression types. The result is the index of the expression type with the most votes, which is used to generate the final classification result of the expression image.

[0027] S44. Construct a shared fully connected layer, input X1, X2 and X3, generate their respective results, and input the three results generated above into the output layer to generate three output results, labeled as output_1, output_2 and output_3 respectively; S45. Build a custom layer with voting functionality.

[0028] Furthermore, step S4, which involves building a custom layer with voting functionality, also includes: S451. Input output_1, output_2 and output_3 into the custom layer; for the three inputs, according to their respective weights, use the reduce_sum function in TensorFlow to perform tensor summation operation to obtain the cumulative number of votes for each emoji category; ; Let R be the cumulative number of votes for the j-th emoji, and R be the set of all dimensions of the current emoji; let the input tensor be... The set of dimensions to be summed is r is the current dimension of the current expression; for all Based on index Summing within the corresponding dimension, for all ,index Keep it fixed and unchanged; S452. Use the argmax function in TensorFlow to calculate the category with the most votes. Along the dimension of emoji types, get the index of the category with the most votes and output it.

[0029] ； For a function f(x) or a mapping When x takes values ​​in the range S (i.e. ), and the result of argmax is the set of x points that make f(x) reach its maximum value.

[0030] Furthermore, step S5 includes the following sub-steps: S51, the training model uses the Adam gradient descent optimization algorithm to ensure training stability while allowing the learning rate to automatically adjust as the number of training iterations increases. The Adam update formula is:

[0031] Where t is the number of iterations. Here, α is the weight parameter, and α is the learning rate. for The correction, yes The correction, To use a very small number, to prevent the denominator from being 0; S52, the loss function for each subtask during training is the categorical cross-entropy loss function, optimized using custom loss weights. The mathematical formula for this function is: ; Where L is the loss value and n is the number of categories. These are the labels of the true categories of the input samples. This represents the output value of the softmax function; S53, Set the number of training iterations (epochs); S54, in each iteration, 64 samples are drawn from the training set and one sample is randomly drawn from the validation set; S55 sets an early stopping mechanism, which monitors the loss function of the validation set output in each iteration. If the loss function does not decrease further after several iterations, the iteration ends early, and the model parameters are restored to the model parameters of the iteration with the lowest loss function.

[0032] S56. Randomly select 64 samples from a pre-divided test set to test the trained multi-task convolutional neural network model and obtain the facial expression recognition accuracy.

[0033] Figure 4 The convolutional and pooling layer structures used for processing the first input image in the multi-task convolutional neural network model of this invention are as follows: Figure 4 As shown: Concv2D has 32 kernels, a kernel size of 3x3, and a stride of 1x1. Concv2D has 32 kernels, a kernel size of 3x3, and a stride of 1x1. Batch Normalization; MaxPooling2D, with a step size of 2x2; Concv2D has 64 kernels, a kernel size of 3x3, and a stride of 1x1. Concv2D has 64 kernels, a kernel size of 3x3, and a stride of 1x1. Batch Normalization; MaxPooling2D, with a step size of 2x2; Concv2D has 128 kernels, a kernel size of 3x3, and a stride of 1x1. Batch Normalization; MaxPooling2D, with a step size of 2x2; Concv2D has 256 kernels, a kernel size of 3x3, and a stride of 1x1. Batch Normalization; MaxPooling2D, with a step size of 2x2; Concv2D has 512 kernels, a kernel size of 3x3, and a stride of 1x1. Batch Normalization; MaxPooling2D, with a step size of 2x2; Flatten layer.

[0034] Figure 5 The convolutional and pooling layer structures used for processing the second input image in the multi-task convolutional neural network model of this invention are as follows: Figure 5 As shown: Concv2D has 32 kernels, a kernel size of 3x3, and a stride of 1x1. Concv2D has 32 kernels, a kernel size of 3x3, and a stride of 1x1. Batch Normalization; MaxPooling2D, with a step size of 2x2; Concv2D has 64 kernels, a kernel size of 3x3, and a stride of 1x1. Concv2D has 64 kernels, a kernel size of 3x3, and a stride of 1x1. Batch Normalization; MaxPooling2D, with a step size of 2x2; Concv2D has 128 kernels, a kernel size of 3x3, and a stride of 1x1. Concv2D has 128 kernels, a kernel size of 3x3, and a stride of 1x1. Batch Normalization; MaxPooling2D, with a step size of 2x2; Concv2D has 256 kernels, a kernel size of 3x3, and a stride of 1x1. Concv2D has 256 kernels, a kernel size of 3x3, and a stride of 1x1. Batch Normalization; MaxPooling2D, with a step size of 2x2; Concv2D has 512 kernels, a kernel size of 3x3, and a stride of 1x1. Batch Normalization; MaxPooling2D, with a step size of 2x2; Flatten layer.

[0035] The convolutional and pooling layer structures for processing the first and second input images progressively increase the number of convolutional kernels, starting at 32 and increasing to 64, 128, 256, and 512. This design allows the model to capture basic facial features such as edges and textures (e.g., facial contours and expression lines) at shallow layers and extract more complex abstract expression features (e.g., the curve of the corners of the mouth and subtle changes in the corners of the eyes) at deeper layers. For the first input image (labeled feature points) and the second input image (facial cropped image), it can accurately mine the geometric correlation features of feature points and the texture and expression features of facial regions, providing a diverse feature base for subsequent feature fusion and voting mechanisms. 2. Continuous use of 3×3 small convolutional kernels: Stacking multiple 3×3 convolutional kernels instead of large convolutional kernels reduces parameter computation while increasing the network's non-linear expressive power. For example, the receptive field of two 3×3 convolutions is equivalent to that of one 5×5 convolution, but with fewer parameters. This allows for more detailed capture of local facial expression details and avoids the loss of feature information caused by large convolution kernels, making it particularly suitable for recognition tasks driven by subtle features such as facial expressions. Batch normalization, by adding a batch normalization layer after each convolutional layer, pulls the feature data distribution of the convolutional output back to a standard normal distribution, solving the internal covariate shift problem in deep network training. This allows for a higher learning rate during model training, faster convergence, and avoids gradient vanishing or exploding caused by changes in data distribution. It also ensures stable training of feature extraction branches for two different input images, reducing the risk of overfitting.

[0036] The MaxPooling2D pooling layer uses a 2×2 max pooling step to downsample the feature map while preserving key features, reducing feature dimensionality and computational complexity. Simultaneously, pooling improves the translation invariance of features, allowing the model to accurately extract facial features even with slight positional shifts in the facial image (such as small movements of the face within the frame), enhancing its adaptability to real-world facial images.

[0037] Structural adaptation for the first input image: The first input image is an image labeled with 68 facial feature points. Figure 4The structure uses fewer stacked convolutional layers (with only a single convolutional layer in some stages) to focus more on preserving the geometric location association information of feature points, avoiding the destruction of feature point location features due to excessive convolution, and ensuring that the spatial relationship features of feature point labeled images can be effectively extracted. For the structural adaptation of the second input image, which is the image after facial cropping, some stages in Figure 5 use two consecutive convolutions (such as the 32, 64, 128, and 256 convolution kernel stages), which can more fully extract the texture, color, and other expression-related texture features of the facial region, adapt to the information density of the cropped image, and make up for the possible feature loss after background removal.

[0038] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0039] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (Read-Only Memory). Memory includes ROM, magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0040] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0041] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0042] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A facial expression recognition method based on multi-task learning with a voting mechanism, characterized in that, include: S1. Construct a sample set based on the acquired historical facial expression data, and divide the sample set into a training set, a validation set, and a test set; S2, Statistically obtain the quantity data of each expression from the training set data, and process the training set data based on the quantity data and data augmentation to obtain the augmented training set; S3, perform feature point labeling and face cropping on the images in the enhanced training set to obtain the first input image and the second input image, respectively; S4, Construct a multi-task convolutional neural network model based on a voting mechanism; S5, the final facial expression recognition model is obtained by optimizing the multi-task convolutional neural network model based on the sample set; S6, based on the facial expression recognition model and the acquired real-time facial expression images, obtains the corresponding facial expression recognition results.

2. The facial expression recognition method based on voting mechanism multi-task learning according to claim 1, characterized in that, The enhanced training set in S2 also includes: Count the number of data points for each expression in the training set; The number of data points for each expression is compared with a preset value. If the number of data points for the current expression is less than the preset value, data augmentation is performed on the current expression. The data augmentation operation includes randomly rotating the current expression image, horizontally translating, vertically translating, randomly scaling, and shearing to obtain an augmented training set.

3. The facial expression recognition method based on voting mechanism multi-task learning according to claim 2, characterized in that, S3 obtains the first input image and the second input image respectively, and also includes: S31. Perform feature point detection on the original expression image, extract the positions of 68 facial feature points, perform face alignment based on the coordinates of the eyes, perform face alignment on the expression images in the training set, perform facial feature point detection on the face-aligned image, mark the 68 facial feature points, generate an image marked with feature points, and perform a resize operation on the image marked with feature points to obtain the first input image. S32. Remove noise from the enhanced training set facial expression images using Gaussian blur; Perform face region detection and crop out non-face regions to obtain the cropped face image; Facial feature point detection was performed on the cropped face image, resulting in the locations of 68 feature points; The face is aligned based on the coordinates of the eyes to generate a cropped image. The image after facial cropping is resized to obtain the second input image.

4. The facial expression recognition method based on voting mechanism multi-task learning according to claim 3, characterized in that, Step S4, which involves constructing a multi-task convolutional neural network model based on a voting mechanism, also includes: S41. Construct the convolutional layer and pooling layer of the first branch of the convolutional neural network model for the first input image, to extract the image features X1 of the first input image; S42. The convolutional and pooling layers of the second branch of the convolutional neural network model for the second input image are used to extract the image features X2 of the second input image. S43. Perform feature fusion on the corresponding image features extracted in S41 and S42 to obtain fused feature X3; S44. Construct a shared fully connected layer, input X1, X2 and X3, generate their respective results, and input the three results generated above into the output layer to generate three output results, labeled as output_1, output_2 and output_3 respectively; S45. Build a custom layer with voting functionality.

5. The facial expression recognition method based on voting mechanism multi-task learning according to claim 4, characterized in that, Step S4, which involves building a custom layer with voting functionality, also includes: S451. Input output_1, output_2 and output_3 into the custom layer; for the three inputs, according to their respective weights, use the reduce_sum function in TensorFlow to perform tensor summation operation to obtain the cumulative number of votes for each emoji category; ； Let R be the cumulative number of votes for the j-th emoji, and R be the set of all dimensions of the current emoji; let the input tensor be... The set of dimensions to be summed is ; r is the current dimension of the current expression; for all Based on index Summing within the corresponding dimension, for all ,index Keep it fixed and unchanged; S452. Use the argmax function in TensorFlow to calculate the category with the most votes, and along the dimension of emoji types, get the index of the category with the most votes and output it; ； When x takes values ​​in the range S, the set of x points where f(x) reaches its maximum value is obtained based on argmax.

6. The facial expression recognition method based on voting mechanism multi-task learning according to claim 5, characterized in that, Step S5 includes the following sub-steps: S51, Training a multi-task convolutional neural network model using the Adam gradient descent optimization algorithm: ； Where t is the number of iterations. Here, α is the weight parameter, and α is the learning rate. for The correction, yes The correction, To use a very small number, to prevent the denominator from being 0; S52, set a cross-entropy loss function for each subtask and optimize it using custom loss weights; ； Where L is the loss value and n is the number of categories. These are the labels of the true categories of the input samples. This represents the output value of the softmax function; S53, Set the number of training iterations (epochs); S54, in each iteration, 64 samples are drawn from the training set and one sample is randomly drawn from the validation set; S55, set an early stopping mechanism, monitor the loss function of the output result of the validation set in each iteration. If the loss function does not decrease further after several iterations, the iteration ends and the model parameters are restored to the parameters of the multi-task convolutional neural network model when the loss function is the lowest, and the trained multi-task convolutional neural network model is obtained. S56. Randomly select 64 samples from a pre-divided test set to test the trained multi-task convolutional neural network model and obtain the facial expression recognition accuracy.

Images