A facial expression recognition method based on a joint loss function

Abstract

The application discloses a face expression recognition method based on a joint loss function. By introducing a confidence-aware collaborative scaling Gaussian mixture loss term, a joint loss function is formed to train and optimize a convolutional neural network: during the training process, class-level statistical modeling is performed on deep features, local scaling factors of each class are calculated to reflect the feature dispersion degree within the batch, and the confidence weight of the center estimation of each class is comprehensively evaluated based on the historical cumulative sample number and the trace of the global covariance matrix; a collaborative scaling term is constructed using the two, the Mahalanobis distance between the sample and the class center is adaptively modulated, so that the feature constraint strength can be differentially adjusted according to the distribution state and center reliability of each class. Under the premise of maintaining zero additional parameters and computational overhead in the inference stage, the intra-class compactness of similar expression features and the inter-class separability of different expression features are enhanced, and the face expression recognition precision and robustness in a complex scene are improved.

Description

Technical Field

[0001] This invention relates to the fields of facial expression recognition and deep learning technology, and in particular to a facial expression recognition method based on a joint loss function. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, facial expression recognition has shown broad application prospects in fields such as intelligent human-computer interaction, driver status monitoring, assisted medical diagnosis, and emotion analysis in online education. Deep learning-based facial expression recognition methods automatically extract facial features through convolutional neural networks and combine them with classifiers to determine various basic expression categories such as happiness, sadness, surprise, and fear, and have become the mainstream technical approach in this field.

[0003] Existing facial expression recognition models generally employ cross-entropy loss as the core optimization objective during training. Cross-entropy loss directly measures the difference between the model's predicted class probability and the true label of the sample. By minimizing this loss value, the network can gradually learn the classification decision boundary to assign different expression samples to the correct category. This type of method has advantages such as simple mathematical form, fast convergence speed, and clear probabilistic interpretation, and is therefore widely used in various facial expression recognition tasks.

[0004] However, in complex real-world scenarios, factors such as lighting variations, facial occlusion, individual differences in facial expressions, and diverse shooting angles often lead to significant intra-class differences in the deep feature space for samples of the same type of expression, while the feature distributions of different types of expressions may overlap considerably. When training solely based on cross-entropy loss, the optimization objective focuses only on determining the classification boundary, lacking explicit structural constraints on the distribution of the deep feature space. This results in features of similar expression samples not being sufficiently clustered, and the feature intervals between dissimilar expression samples not being significant enough. This loose feature distribution limits the model's ability to accurately characterize fine-grained facial differences, and recognition accuracy and robustness are often difficult to guarantee when facing challenging conditions such as extreme viewing angles and low lighting.

[0005] To address the aforementioned issues, some research has attempted to introduce feature distribution constraint mechanisms into the loss function. One representative approach combines Gaussian mixture modeling with loss function design, guiding the network to actively compress intra-class distances and expand inter-class margins during training by measuring the distance between sample features and class centers. However, traditional Gaussian mixture loss functions typically use a fixed distance scale, making it difficult to adapt to the dynamic changes in feature distribution at different stages and batches during training. When the overall feature distribution of samples in a batch is relatively dispersed, a fixed-scale distance metric can lead to an overall larger loss value, reducing the relative discriminatory power between the true class and other classes, resulting in training instability and significantly diminishing the constraint effect. Existing solutions have attempted to introduce a global adaptive scaling mechanism based on the overall feature standard deviation of the batch to alleviate the training instability caused by a fixed scale. However, a single global scaling factor implicitly assumes that the feature distribution of all expression classes within the same batch is similarly discrete, failing to capture the distribution heterogeneity between different classes. Furthermore, existing solutions lack assessment of the reliability of statistical estimates for class centers. In the early stages of training or when samples are sparsely distributed, the inaccuracy of the class centers themselves may lead to misleading feature constraints, ultimately harming model performance.

[0006] In summary, there is an urgent need to propose a loss function design scheme that can maintain the stable classification ability of cross-entropy loss, and can also accurately perceive the feature distribution status of each category and the reliability of center estimation, thereby achieving differentiated and robust feature distribution constraints, so as to further improve the accuracy and generalization ability of facial expression recognition in complex scenarios. Summary of the Invention

[0007] The purpose of this invention is to provide a facial expression recognition method based on a joint loss function, in order to solve at least some of the problems existing in the prior art.

[0008] Technical solution: A facial expression recognition method based on a joint loss function, comprising the following steps:

[0009] The deep learning network is used to extract features from the input facial expression image to obtain deep feature representation, and the predicted probability of each expression category is obtained through classification mapping.

[0010] We perform category-level statistical modeling on deep feature representations, calculate local scaling factors and center confidence weights, and perform co-scaling to obtain adaptive corrected distances.

[0011] The cross-entropy loss based on predicted probability and the confidence-aware co-scaling Gaussian mixture loss based on adaptive corrected distance are weighted and combined to form a joint loss function.

[0012] The deep learning network is iteratively trained using a joint loss function to obtain a trained facial expression recognition model.

[0013] The image of the face to be recognized is input into the trained facial expression recognition model, and the final expression recognition result is output.

[0014] Beneficial effects: By introducing a collaborative scaling Gaussian mixture loss constructed from a category-level local scaling factor and a center confidence weight, this invention achieves refined and differentiated constraints on the feature distribution of different expression categories while maintaining lightweight inference, thereby improving the accuracy and robustness of facial expression recognition in complex scenarios. Attached Figure Description

[0015] Figure 1 This is a flowchart of the overall solution of the present invention.

[0016] Figure 2 This is a flowchart illustrating how the present invention obtains the predicted probability of each expression category through classification mapping.

[0017] Figure 3 This is a flowchart of the calculation of the local scaling factor and the center confidence weight in this invention.

[0018] Figure 4 This is a flowchart of the weighted combination to form a joint loss function according to the present invention.

[0019] Figure 5 This is a flowchart illustrating the iterative training of a deep learning network using a joint loss function, as described in this invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] like Figures 1-5 As shown, this embodiment describes in detail the data processing flow of a facial expression recognition method based on a joint loss function, specifically including:

[0022] S1: Based on a deep learning network, feature extraction is performed on the input facial expression image to obtain a deep feature representation, and the predicted probability of each expression category is obtained through classification mapping.

[0023] Specifically, firstly, a raw image dataset containing multiple categories of facial expressions is acquired. The dataset is then randomly divided into a training set and a test set according to a preset ratio (e.g., 8:2). The training set is used for model parameter learning, and the test set is used for periodic performance evaluation. For each input image, a grayscale conversion operation is preferably performed, converting a three-channel color image into a single-channel grayscale image to eliminate the interference of color information on expression feature learning. Next, the grayscale image is normalized by using bilinear interpolation to uniformly scale the image to a preset fixed resolution, such as 112×112 pixels, thus obtaining a uniformly sized image. To enhance the local contrast and detail discernibility of facial expression textures in the image, according to one aspect of this application, image enhancement is performed by combining Contrast-Limited Adaptive Histogram Equalization (CLAHE) with Gamma correction: firstly, the CLAHE algorithm is used to limit excessive contrast amplification in small areas of the image, and then Gamma correction is applied to non-linearly adjust the overall brightness, ultimately obtaining a standardized image with prominent local contrast and rich detail information. During the training phase, to improve the model's adaptability to changes in facial expression, lighting, and viewing angle, data augmentation is further performed on the standardized images. For example, random cropping is performed from the standardized images, preserving the core facial region and simulating different framing ranges; simultaneously, the images are horizontally flipped with a certain probability to generate mirrored samples. After these augmentation operations, enhanced training samples are obtained, which will serve as the actual input for subsequent deep learning networks.

[0024] After obtaining training samples of consistent quality and rich patterns, they are input into a pre-constructed facial expression recognition network. This network is designed based on a convolutional neural network architecture, with its backbone consisting of stacked convolutional layers, batch normalization layers, activation function layers, and pooling layers. Enhanced training samples are passed into the network as input data streams, undergoing filtering, non-linear activation, and downsampling operations layer by layer, gradually converging from low-level edge and texture details to high-level semantic information. According to one aspect of this embodiment, after a specific feature output layer of the network, such as the last pooling layer or a global average pooling layer, a fixed-dimensional high-dimensional feature vector is extracted. This vector is defined as a deep feature representation. This deep feature representation vector contains discriminative semantic information closely related to the expression category in the input facial image and is the core object of subsequent loss function constraints and classification boundary optimization. Further, to predict the expression category, the obtained deep feature representation vector is input into a fully connected classification layer. This classification layer contains an equal number of output nodes as the total number of expression categories K, and calculates the original output score for each category through linear transformation. Then, these raw output scores are normalized exponentially using the Softmax function, mapping any real-valued score to the open interval (0,1) and making the sum of the output values ​​of all categories equal to 1. This yields the predicted probability that the sample belongs to the k-th expression category. Symbolically, for a given sample, its deep feature representation is denoted as x, and the predicted probability vector obtained after the classification layer and Softmax is p, where the k-th element p... k This represents the confidence level of the model in determining that the sample represents the k-th type of expression. Through the above process, end-to-end forward inference from the original image to deep feature representation and then to class prediction probability is completed, providing the necessary data foundation for subsequent joint loss calculation.

[0025] S2: Perform class-level statistical modeling on the deep feature representation, calculate the local scaling factor and center confidence weight, perform co-scaling processing, and obtain the adaptive corrected distance.

[0026] Based on the deep feature representations and corresponding class labels of each sample in the current training batch, this embodiment imposes additional constraints on the model from the perspective of feature space distribution by introducing a collaborative scaling mechanism that can perceive the heterogeneity of class distribution. First, for a given training mini-batch, let the batch contain N samples, and the deep feature representation x of each sample... i (i from 1 to N) is a D-dimensional vector with its true class label z. i , z i The value is taken from {1, 2, …, K}. The first step in categorical statistical modeling is to group the deep feature representations within a batch according to the category label. For any category k, extract all values ​​from that batch that satisfy z iThe sample features of class k constitute a feature subset X belonging to class k. k The number of samples in this subset is denoted as N. k For each category k, its local scaling factor is calculated individually to reflect the feature distribution dispersion of that category in the current batch. In a preferred embodiment, the local scaling factor is calculated as follows: for the feature subset X... k The standard deviation of the feature values ​​of all samples is calculated dimension by dimension to obtain a D-dimensional standard deviation vector. Then, the average of all elements of this vector is taken to obtain a scalar value, denoted as the local scaling factor σ for this class. local(k) The smaller the value, the more compact the feature distribution of category k in the current batch; conversely, the larger the value, the more dispersed the distribution. If no samples of a certain category appear in the current batch, a preset default value can be assigned to it or a moving average can be used to fill in the gaps to avoid interruption of the calculation.

[0027] In addition to characterizing the instantaneous dispersion of each category within a batch, this scheme also evaluates the reliability of the mean vector of each category center by statistically analyzing historical information throughout the training process, i.e., calculating the confidence weight of the category centers. According to one aspect of this application, two global statistics are continuously maintained during training: one is the total number of samples participating in training for each category k since the start of training, denoted as N. global(k) This reflects the training sufficiency of the category; secondly, it reflects the global feature covariance matrix Σ of the category. global(k) This method utilizes an exponential moving average, updating the data based on new samples in each iteration to capture the long-term compactness of the categorical feature distribution. For example, this is achieved by calculating the trace of the covariance matrix, i.e., the sum of the diagonal elements tr(Σ). global(k) This yields a quantitative indicator of the global dispersion of category features. Next, relative training sufficiency and global compactness are incorporated into a comprehensive calculation formula to generate unnormalized confidence values. Specifically, the relative training sufficiency, i.e., N, is first calculated. global(k) The ratio to the total cumulative sample size across all categories; introducing a hyperparameter α, calculating the exponential term exp(-α·tr(Σ) global(k) The index term approaches 1 when the class distribution is compact and approaches 0 when the distribution is diffuse, thus attenuating the reliability of the center. Then, the relative training sufficiency is multiplied by this index term to obtain the initial confidence value for each class. Finally, the initial confidence values ​​of all K classes in the batch are normalized by dividing by the sum of all initial confidence values ​​to obtain the normalized class center confidence weight c that satisfies the probability distribution property. hat(k)Thus, when there are sufficient training samples for a certain category and the feature distribution remains compact over a long period, the confidence weight of its center is high, indicating that the mean vector of that category is a reliable reference point; conversely, a low confidence weight means that forcing features to converge towards this center may be misleading.

[0028] After obtaining the local scaling factor and center confidence weights, the process proceeds to the co-scaling stage, which aims to differentially modulate the Mahalanobis distance that measures the distance between sample features and the center of each class. For each sample i in the batch, its deep feature representation x is first calculated. i With the mean vector of the kth category center μ k The Mahalanobis distance between features. The Mahalanobis distance calculation takes into account the correlation between the various dimensions of the features, specifically by weighting using the inverse of the covariance matrix: d k = (x i -μ k ) T Σ k -1 (x i -μ k For ease of numerical optimization, half of this distance is usually taken and denoted as the original Mahalanobis distance value. Next, the collaborative scaling term is constructed. First, the local scaling factor σ for class k is calculated. local(k) The category dispersion ratio γ is obtained by comparing it with the average local scaling factor of all categories within the current batch. k A category dispersion ratio greater than 1 indicates that category k is more dispersed relative to other categories, while a ratio less than 1 indicates a more compact distribution. Then, γ... k With (1 - c) hat(k) Combining these, 1 minus the confidence weight represents the uncertainty of the center, and the product of the two forms a comprehensive modulation factor. In a preferred embodiment, the comprehensive modulation factor is input into a preset nonlinear modulation function, such as f(ξ) = 1 / (1 + β·ξ), where β is an adjustable parameter, ξ is the modulation factor, and the output of the function is the collaborative scaling coefficient. Finally, the collaborative scaling coefficient is applied to the original Mahalanobis distance d. k The adaptive correction distance d is obtained from the above. tilde_k When the categories are compact and the center confidence is high, γ k Small and C hat(k) When the scale factor is large, close to 0, and close to 1, the distance is hardly compressed, thus imposing a strong cohesion constraint. When the classes are scattered or the centers are unreliable, the scale factor increases, the scaling factor decreases, the distance is significantly compressed, and the cohesion constraint is relaxed accordingly. In this way, an adaptively corrected distance relative to each class is generated for each sample, which is used to construct the subsequent loss function.

[0029] S3: The cross-entropy loss based on predicted probability and the confidence-aware co-scaling Gaussian mixture loss based on adaptively corrected distance are weighted and combined to form a joint loss function.

[0030] This embodiment constructs a total loss function that balances classification performance and feature space structure based on a joint calculation of predicted probability and adaptively corrected distance. According to one aspect of this application, a cross-entropy loss term is first constructed to supervise network learning from the perspective of class prediction accuracy. For sample i within a batch, let its true class label be z. i After one-hot encoding, at the z-th i One bit is 1, and the rest are 0; the network's predicted probability for this sample output is p. i,k Then the cross-entropy loss value L CE The calculation method is as follows: for all samples within a batch, take the negative logarithm of the predicted probability of the category corresponding to the true label, and then calculate the average, i.e., L. CE = (-1 / N)·Σ i log(p i ,z i The smaller the value of this loss term, the higher the confidence of the model in classifying the sample into its true category, which is the main driving force for clarifying the boundary of the classification decision.

[0031] Furthermore, a confidence-aware co-scaling Gaussian mixture loss term is constructed. This loss term, based on the posterior probability form of the Gaussian mixture model, maximizes the probability that a sample belongs to its true class while forcing features away from other class centers. Specifically, for sample i, it belongs to class z. i The posterior probability can be expressed by Bayes' theorem as: the product of the prior probability and the class-conditional Gaussian probability density, divided by the sum of the products of all classes. For stable computation, the exponential term in the class-conditional probability density uses an adaptively adjusted distance d. tilde_k To replace the original Mahalanobis distance, the loss term L is defined. CCS-GML Let p(k) be the mean of the negative logarithm of the posterior probability over the batch. Assuming the prior probabilities p(k) of each class are uniform or proportional to the sample size, the covariance determinant is |Σ k | By estimating and fixing L using a moving average, then CCS-GML = (-1 / N)Σ i log [ (p(z i )·|Σz i | -1 / 2 ·exp(-d tilde_zi ) ) / (Σ k p(k)·|Σk| -1 / 2 ·exp(-d tilde_k The d used in the formula tilde_kThe generated adaptive corrected distance integrates the local scaling factor and the center confidence weight. When the deep features of a sample are close to its true class center and that center is reliable, d... tilde_zi Smaller, exp(-dt) ilde_zi When the center confidence is high, the loss value decreases; conversely, when the sample deviates from the true center or is close to the center of other classes, the loss value increases accordingly. In particular, due to the introduction of the collaborative scaling mechanism, for classes with low center confidence, even if the sample is far from its center, this distance has been compressed, and the loss value will not be too large, thus avoiding misleading penalties to the model.

[0032] Finally, the cross-entropy loss value and the confidence-aware co-scaling Gaussian mixture loss value are weighted and combined to form the joint loss function L. total In an alternative implementation, a balance coefficient λ (λ>0) is introduced, according to L... total =L CE + λ·L CCS-GML The model employs a linear weighting method. The coefficient λ adjusts the strength of the feature distribution constraint term relative to the classification task: a larger λ indicates a model that focuses more on intra-class compactness and inter-class separation; a smaller λ indicates a model that prioritizes classification accuracy. By appropriately setting this parameter, synergistic optimization of classification discriminative ability and feature distribution constraint ability can be achieved without introducing additional inference parameters and computational overhead. This allows the network to simultaneously obtain accurate classification decision boundaries and a superior feature space topology during training.

[0033] S4: Use the joint loss function to iteratively train the deep learning network to obtain the trained facial expression recognition model.

[0034] After constructing the joint loss function, it is used as the optimization objective to adjust all learnable parameters of the deep learning network through iterative training. Specifically, for each training iteration, a small batch of samples is first randomly sampled from the training set, and after preprocessing pipeline and forward propagation, the deep feature representation and predicted probability of each sample in the batch are obtained. At the same time, using the deep feature representation of the current batch and the historical moving average statistics, the local scaling factor, center confidence weight, and adaptive correction distance of each sample are calculated. Subsequently, the joint loss value composed of the cross-entropy term and the confidence-aware co-scaling Gaussian mixture term is calculated.

[0035] Following the forward propagation, the backpropagation phase begins. The joint loss value is used as the scalar objective function, and an automatic differentiation engine calculates the gradient of the loss value with respect to the weights and biases of each layer in the network. The gradient reflects the direction and magnitude of the adjustment of each parameter in reducing the overall loss. Then, a gradient descent optimizer, such as stochastic gradient descent with momentum or an adaptive moment estimation optimizer, is selected to update the network parameters based on the calculated gradients and a pre-set learning rate. During this process, in addition to the network parameters, the global covariance matrix Σ for each category is also considered. global(k) and cumulative sample size N global(k) It also updates the data using an exponential moving average based on the information from the current batch, in order to continuously reflect the long-term characteristic distribution of the category.

[0036] The forward propagation, joint loss calculation, gradient backpropagation, and parameter update processes are executed cyclically, with the network traversing the training set once in each epoch. During training, the model's expression recognition accuracy and feature distribution metrics are periodically evaluated and validated using a test set. Training terminates when the model's performance on the test set no longer shows significant improvement, or when the preset maximum number of iterations is reached. At this point, the model's network structure and optimal parameters are saved as a model file, resulting in the trained facial expression recognition model. This model can map input facial expression images to a feature space with strong inter-class discriminative power and good intra-class compactness, thus possessing high-precision recognition capabilities for complex expression patterns.

[0037] S5: Input the face image to be recognized into the trained facial expression recognition model and output the final expression recognition result.

[0038] In the actual deployment and inference phases, the trained facial expression recognition model is used to predict the expression category of newly acquired facial images. First, an image containing a face is acquired using an image acquisition device. To ensure the model's input distribution is consistent with that used during training, the image to be recognized undergoes the same standardized preprocessing operations as during training: grayscale conversion is performed sequentially to obtain a single-channel grayscale image; its size is adjusted to the fixed resolution used during training to obtain a uniform-sized image; then, an enhancement strategy combining CLAHE and Gamma correction is applied to obtain a standardized image with clear details and appropriate contrast. The preprocessing steps only include the standardization steps necessary for inference and do not involve data augmentation operations specific to training, such as random cropping or horizontal flipping.

[0039] Subsequently, the standardized image to be identified is loaded into the model input layer, along with the previously saved optimal model parameters, and a one-way forward propagation is performed. During this process, the image data flows through each layer of the network, obtaining the deep feature representation vector of the sample at the feature extraction layer, and then continues through a fully connected classification layer and a softmax function, ultimately outputting a predicted probability vector. Each element of the predicted probability vector corresponds to a predefined expression category (such as anger, disgust, fear, happiness, sadness, surprise, neutral, etc.), and its numerical value represents the probability that the model considers the image to be identified to belong to that expression category.

[0040] Finally, the decision output is performed. In a preferred embodiment, all elements of the predicted probability vector are traversed, and the index with the largest probability value is found. The largest index is directly mapped to the corresponding expression category label string, which is the expression recognition result of the model for the face image. For example, if the highest probability appears in the "happy" category corresponding to index 3, then "happy" is finally output as the recognition conclusion. In some application scenarios, the probability distribution of all categories can also be output simultaneously to provide a confidence reference for the recognition result. Through an end-to-end process, this solution completes the entire process from the original face image to the accurate expression recognition result, improving the robustness of expression recognition under complex conditions such as changes in lighting, occlusion, and individual differences while ensuring lightweight inference.

[0041] According to another aspect of this application, the predicted probability of each expression category is obtained through classification mapping, including:

[0042] Construct a facial expression recognition network, extract features layer by layer from the input standardized image or enhanced training sample, and output a deep feature representation vector of fixed dimension from the specified network layer.

[0043] After image preprocessing and data augmentation, this embodiment feeds standardized images or augmented training samples into a carefully designed deep learning network. Through layer-by-layer abstraction, deep feature vectors capable of representing the essential discriminative information of facial expressions are extracted. In this embodiment, the constructed facial expression recognition network employs a lightweight convolutional neural network architecture to balance the richness of feature representation with the efficiency of inference computation. This ensures that the final model possesses both high discriminative power for complex facial expression patterns and smooth operation in resource-constrained deployment environments.

[0044] Specifically, the input to the deep learning network is a single-channel, fixed-size image, such as a 112-pixel by 112-pixel grayscale face image. The image data first enters an initial convolutional layer, which is configured with several 3x3 convolutional filters. These filters scan the neighborhood of each pixel in the input image with a small local receptive field, extracting the most basic low-level visual features such as directional edges, corners, and texture primitives. The output of the initial convolutional layer undergoes batch normalization to stabilize the distribution of activation values, and non-linear transformation capabilities are introduced by modifying the linear unit activation function, enabling the network to fit feature extraction processes far more complex than simple linear mappings.

[0045] Furthermore, following the initial convolutional layer, the deep learning network stacks multiple feature extraction stages. Each stage consists of several convolutional blocks sequentially connected. Each convolutional block can employ standard convolution operations or a depthwise separable convolutional structure to reduce computational complexity and the total number of parameters. In a preferred embodiment, an inverse residual design is introduced within the convolutional block. This involves first expanding the channel dimension through pointwise convolution, then performing depthwise convolution to extract spatial features in the expanded high-dimensional space, and finally compressing the number of channels back to the original level through pointwise convolution, thereby achieving efficient feature aggregation. Each convolutional block is equipped with a batch normalization layer and an activation function layer to ensure smooth gradient flow during forward propagation and numerical stability during training.

[0046] Between adjacent extraction stages, downsampling operations are performed using convolutional or pooling layers with a stride of 2, halving the spatial resolution of the feature map stage by stage. As the spatial size shrinks, the effective receptive field corresponding to the convolutional kernel gradually expands, enabling the network to transition from capturing local texture details to perceiving a wider range of facial component shapes, relative positional relationships, and overall configuration patterns. Correspondingly, the number of channels in the convolutional layers is moderately increased after each downsampling, gradually expanding from 64 channels in the initial stage to 128, 256, and finally 512 channels, to give the network sufficient capacity to encode rich semantic information.

[0047] After processing by all convolutional modules, a global average pooling layer is introduced at the end of the deep learning network. This layer performs a global average of the spatial dimensions of the feature map for each channel output from the final stage, compressing a feature tensor of size H x W x C into a vector of fixed length C, where C equals the number of output channels in the final stage, exemplarily set to 512. This global average pooling operation not only eliminates the impact of changes in feature map spatial dimensions on subsequent fully connected layers but also reduces the number of parameters, enhancing the network's invariance to spatial translations of the input image. Finally, the global average pooling layer outputs a fixed-dimensional deep feature representation vector, denoted as x, where x belongs to a D-dimensional space in the real number domain, and D equals C. This deep feature representation vector encapsulates abstract semantic features closely related to expression discrimination from the input face image, such as eyebrow curvature, eyelid opening and closing, mouth corner upward angle, and nasal wing expansion—fine-grained facial dynamic cues are encoded within it, providing a unified and high-quality data representation for subsequent class probability mapping and feature space distribution constraints.

[0048] The deep feature representation vector is input into the fully connected classification layer to calculate the original output score for each category.

[0049] After obtaining the fixed-dimensional deep feature representation vector x corresponding to each input sample, this feature vector is mapped to a specific expression category space to generate a raw score that can be used for classification decisions. This mapping is performed by a fully connected classification layer, which is essentially a linear transformation operation, supplemented by optional bias terms for translation adjustment.

[0050] In one specific implementation, the fully connected classification layer contains a weight matrix W of shape D rows and K columns, where D equals the dimension of the deep feature vector, i.e., the number of channels output by the previous global average pooling layer, for example, 512; K equals the total number of predefined expression categories, which can be set to 7 according to actual needs, corresponding to the seven basic expressions: anger, disgust, fear, happiness, sadness, surprise, and neutral. In addition, the fully connected classification layer also contains a bias vector b of dimension K, used to independently offset the output of each category. When the forward propagation reaches the fully connected classification layer, the deep feature representation vector x is multiplied by the weight matrix W, linearly projecting the points in the D-dimensional feature space onto the K-dimensional category score space, and then element-wise added to the bias vector b, finally obtaining K real numbers, i.e., the original output scores of each category, which can be denoted as s. k , where the index k iterates through the set of integers from 1 to K.

[0051] For example, if the deep feature representation vector of a sample, after being multiplied by the weight matrix and added with the bias, produces a high positive score on the output neuron corresponding to the "happy" category, but a low or even negative score on the "fear" category, it indicates that, under the current network parameters, the evidence for classifying the sample as "happy" is stronger than that for "fear". The original output scores are essentially not constrained by any numerical range and can be any real number; their relative magnitudes reflect the model's preference ranking for each category. A higher score for a category means the network is more likely to consider the input image to belong to that category; conversely, a lower or even negative score indicates that the network tends to exclude that category.

[0052] During training, the weight matrix W and bias vector b of the fully connected classification layer are treated as learnable network parameters. Their initial values ​​can be randomly initialized according to a preset probability distribution (such as a Xavier uniform distribution or a He normal distribution), and iteratively updated in each backpropagation based on the gradient calculated by the loss function. As training progresses, each column of the weight matrix gradually converges to a discriminative template corresponding to the appearance pattern of each class, while the bias vector compensates for the imbalance of prior probabilities among classes. Through this linear mapping, the deep feature representations originally distributed in the high-dimensional feature space are transformed into intuitively comparable class scores, providing an intermediate bridge for subsequent Softmax probability transformation and cross-entropy loss calculation.

[0053] The original output scores are normalized using the Softmax function, and the scores are mapped to the predicted probabilities of each expression category.

[0054] The original output scores s for each category are calculated by the fully connected classification layer. k While these scores can reflect the model's relative preference for different categories, their numerical range and distribution lack probabilistic interpretation, making them unsuitable for direct loss calculation or providing confidence references for the final decision. Therefore, it is necessary to normalize the K original output scores, mapping them to a predicted probability distribution that satisfies the axiomatic definition of probability.

[0055] According to one aspect of this application, the Softmax function is used to implement normalization. The specific calculation process is as follows: for category k, first calculate its natural exponent exp(s) of score. k Then, iterate through all K categories, calculate their respective exponent values, and sum them to obtain the total value Z, which is the sum of all exp(s). j ) Accumulate j from 1 to K; finally, exp(s k Dividing by Z yields the predicted probability p of belonging to category k. k , can be represented as p k = exp(s k) / Σ j exp(s j ), where Σ j This represents the summation over all category indices j.

[0056] In engineering implementation, to ensure the stability of numerical calculations, the maximum value is usually subtracted from all original output scores before taking the exponent; that is, the corrected score s is set. prime_k = s k - max(s1, s2, …, s K Then, the corrected score is exponentially calculated and normalized. This operation does not change the relative magnitude of the final probability distribution, but it can effectively avoid numerical overflow caused by exponential operations, ensuring accurate results even in a floating-point arithmetic environment with limited precision.

[0057] After Softmax normalization, the resulting probability vector p has two core properties: First, each component p k The values ​​of all K components are strictly between 0 and 1; secondly, the sum of all K components is strictly equal to 1. These two properties allow the vector to be strictly interpreted as a probability distribution, that is, the model's confidence estimate of the current input image belonging to each expression category. Preferably, depending on the final application requirements, the model can output only the category with the highest probability value as a hard decision result, or it can output the entire probability vector for the upper-layer system to perform soft decision or multimodal fusion.

[0058] According to another aspect of this application, the calculation of the local scaling factor and the center confidence weight includes:

[0059] Deep feature representation and category labeling of collected samples.

[0060] In each training iteration, given a mini-batch of data containing N samples, after forward propagation, the deep feature representation of each sample can be obtained from a specified layer of the network, denoted as x. i Where i ranges from 1 to N, and each x i It is a D-dimensional real-valued vector. Simultaneously, the true class labels z corresponding to this batch of samples can be obtained synchronously from the data loader. i , z iThe values ​​are taken from the integer set {1, 2, …, K}, corresponding to K expression categories. These deep feature representations and category labels constitute the raw input for subsequent category-level statistical modeling and distance metric calculation. During the acquisition process, it is necessary to ensure that the deep feature representations come from the same network and the same iteration step to avoid inconsistencies introduced by asynchronous updates of network parameters. The deep feature representations are usually batch-normalized and standardized before being fed into the classification layer, but they have not been truncated by any activation function, preserving the complete information in the feature space, which is suitable for calculating statistical features within each category. The labels are one-hot encoded or in scalar form, used to group feature samples by category. This basic data acquisition step provides the data premise for subsequent dynamic scaling factor calculation and confidence evaluation, ensuring that all statistics are strictly based on the actual features of the current batch and their true category.

[0061] Extract a subset of sample features from the deep feature representation, calculate the average of the standard deviations of the feature values, and obtain the local scaling factor.

[0062] To more precisely characterize the differences in the dispersion of feature distributions across different categories within the same batch, it is necessary to group the deep feature representations according to their category labels. For any category k, within the batch, select all categories that satisfy z i For samples equal to k, their deep feature vectors are aggregated into a set X. k The number of samples contained is denoted as N. k If N k If the value is zero, meaning no samples of that type appear in the current batch, then the local statistics for that type can be skipped, or its local scaling factor can be assigned a default value. For N k For the case where it is greater than zero, for set X k The standard deviation of the feature vectors in the set is calculated along each of the D dimensions. Specifically, for each feature dimension j, the standard deviation of all x within that set is calculated. i The j-th component x i,j The average of the values ​​is calculated, followed by the sum of squared deviations of each sample from the average. The average of these deviations is then taken, and the square root is taken to obtain the standard deviation of the j-th dimension. This standard deviation measures the clustering or dispersion of samples within a given feature dimension. Subsequently, the arithmetic mean of the standard deviations across all D dimensions is taken to obtain a single scalar value. This scalar value is defined as the local scaling factor for category k, denoted as σ. local(k) The larger the value, the more dispersed the feature vectors belonging to category k are across all dimensions within the current batch; conversely, a smaller value indicates that the samples are tightly clustered in the feature space. In this way, each category obtains a dispersion index independent of other categories, providing a high-granularity basis for subsequent differential scaling distance, effectively overcoming the limitation of using a single global scaling factor in adapting to the heterogeneity of distribution between categories.

[0063] The confidence weights of the category centers are calculated by comprehensively evaluating the historical cumulative sample count and the trace of the global feature covariance matrix.

[0064] Besides the intra-batch instantaneous dispersion, the reliability of the class center estimation itself should also be considered in the feature constraints. During training, as the network parameters are continuously updated, the deep feature distributions of each class also continuously evolve. Therefore, it is necessary to dynamically maintain the global cumulative sample size N for each class k. global(k) and the global feature covariance matrix Σ global(k) N global(k) The cumulative number of samples labeled as class k since the start of training reflects the number of training exposures received by that class. A larger value indicates a more stable statistical center for that class. Σ global(k) The update is performed using an exponential moving average: whenever a new batch containing samples of class k completes its forward propagation, the covariance matrix of that batch is calculated based on the feature vectors of class k in that batch. Then, the historical covariance matrix is ​​updated using a weighted average based on the momentum coefficient, allowing it to slowly track changes in the network's feature representation. To quantify the global compactness of the class feature distribution, the trace of the covariance matrix is ​​calculated, which is the sum of the diagonal elements of the matrix, tr(Σ). global(k) The trace reflects the sum of the variances of the category features across all dimensions, serving as an indicator of the overall dispersion of the category. Next, these two global statistics are used to calculate the center confidence weight for each category. In an exemplary calculation, the relative training sufficiency of N is first calculated. global(k) Divide by the sum of the cumulative sample sizes of all K categories to obtain a value between 0 and 1; simultaneously, calculate the compactness factor exp(-α·tr(Σ) global(k) In this formula, α is a positive hyperparameter. When the class features are compact, the α factor is small, close to 1; when the class features are extremely dispersed, the α factor approaches 0, thus attenuating the confidence level. The relative training sufficiency is multiplied by the compactness factor to obtain the unnormalized confidence value. Finally, the unnormalized confidence values ​​of all classes within the batch are summed and normalized, i.e., the confidence value of each class is divided by the sum of the confidence values ​​of the entire batch, to obtain the normalized class center confidence weight, denoted as c. hat(k) A higher confidence weight for the category center indicates that the mean vector of the category center is statistically more reliable and should be more strongly preserved in subsequent distance scaling; conversely, if the weight is low, the constraint on features to converge to the center should be relaxed to avoid misleading learning.

[0065] According to another aspect of this application, a weighted combination is performed to form a joint loss function, including:

[0066] The cross-entropy loss value is calculated based on the sample prediction probability.

[0067] After the forward propagation is completed, the predicted probability vector p of N samples in the batch has been obtained. i Meanwhile, the true class label z of each sample is known. i Cross-entropy loss is one of the most widely used loss functions in supervised classification tasks. Its calculation intuitively reflects the model's confidence in predicting the true class. For each sample i, the corresponding true label z in its predicted probability vector is taken. i The probability value of that position (p) i ,z i Then, the natural logarithm of this probability value is taken and the result is negative to obtain the loss contribution of a single sample. Subsequently, the arithmetic mean of the loss contributions of all N samples in the entire batch is calculated to obtain the cross-entropy loss value, denoted as L. CE :L CE = - (1 / N) ∑ i log(pi,zi). The closer the model predicts the probability of the true class to 1, the closer the log value is to 0, and the loss value approaches 0; conversely, if the predicted probability is very small, the negative log will increase rapidly, imposing a large penalty on the model. This loss term is guided by the pure classification decision boundary, forcing the network to classify the sample to its true expression category as accurately as possible, which is the basic optimization objective to ensure the model's basic recognition ability.

[0068] Based on the adaptively corrected distance, the confidence-aware co-scaling Gaussian mixture loss value is calculated.

[0069] In addition to the classification loss, the structured constraints on the distribution of deep features are achieved through Gaussian mixture loss, incorporating an adaptively adjusted distance. In the standard Gaussian mixture model, each class k is represented by a mean vector μ. k The covariance matrix Σ k By definition, the exponential term in the class-conditional probability density function is typically represented using Mahalanobis distance. This scheme replaces the standard Mahalanobis distance with an adaptively corrected distance d. tilde_k This allows us to construct a confidence-aware co-scaling Gaussian mixture loss. For the i-th sample in a batch, it belongs to the true class z. i The posterior probability is written as: p(z) i )·|Σz i | -1 / 2 ·exp(-d tilde_zi Divide by the sum of the corresponding items for all categories. For numerical stability, the loss function takes the negative logarithm of this posterior probability. The average of the negative logarithmic posterior probabilities of all samples within the batch is obtained as L. CCS-GML = (-1 / N)Σ i log [ (p(z i )·|Σz i | -1 / 2 ·exp(-dtilde_zi ) ) / (Σ k p(k)·|Σk| -1 / 2 ·exp(-d tilde_k In the formula, the prior probability p(k) can be set to a uniform distribution or dynamically adjusted according to the sample proportion; the covariance determinant |Σ k The moving average estimation reflects the distribution volume of each category. This is because the adaptively corrected distance d... tilde_k The loss term has already incorporated the moderating effects of local dispersion and center confidence. Therefore, during training, it can not only promote the clustering of similar samples and the separation of dissimilar samples, but also intelligently avoid over-constraining inaccurate class centers, making the feature learning process more stable and robust.

[0070] A balance coefficient is introduced to perform a weighted summation of the cross-entropy loss value and the confidence-aware co-scaling Gaussian mixture loss value, resulting in a joint loss function, which is used to achieve synergistic optimization of classification and discrimination capabilities and feature distribution constraint capabilities.

[0071] To integrate the two learning objectives, the two loss terms need to be linearly combined using a balancing coefficient. Introducing the hyperparameter λ, where λ is greater than 0, the joint loss function is defined as L... total = L CE + λ·L CCS-GML The value of λ determines the relative weight of the feature distribution constraint term in the total loss. When λ is small, model training focuses more on minimizing classification error, and the constraint of the feature space has a weaker impact; when λ is large, the model focuses more on optimizing the intra-class compactness and inter-class separability of the feature distribution. In practice, λ can be set empirically (e.g., 0.1, 0.5, or 1.0) or dynamically adjusted, gradually increasing or decreasing with each training epoch. The weighted summation method does not introduce any new learnable parameters, does not require modification of the network structure, and does not incur any additional computational overhead during forward inference. Through this joint optimization design, the model receives dual gradient signals from the classification boundary and the feature distribution during backpropagation, which can coordinate the improvement of class discrimination ability and feature embedding quality, thereby achieving a comprehensive enhancement of facial expression recognition accuracy and robustness in complex scenarios.

[0072] According to another aspect of this application, iterative training of a deep learning network using a joint loss function includes:

[0073] The enhanced training samples are input into the facial expression recognition network, and the deep feature representation and predicted probability of each sample are calculated through forward propagation.

[0074] In each iteration, a batch of augmented training samples is first sampled from the training set, and forward propagation is performed. Specifically, the preprocessed and augmented image data is fed into the facial expression recognition network, and calculations are performed layer by layer to obtain the deep feature representation vector and predicted probability vector corresponding to each sample. In this process, the forward operations of all network layers are completely deterministic, requiring the same computational resources as a conventional classification model, without adding any extra modules. The deep feature representation vector and predicted probability are stored for subsequent loss calculations and parameter updates.

[0075] Based on deep feature representation, class-level statistical modeling is performed to calculate the class local scaling factor and class center confidence weight of the current batch, and an adaptive corrected distance is generated.

[0076] After obtaining the deep feature representations, categorical statistical modeling is performed. First, the samples are grouped by label, and the local scaling factor for each category is calculated. Simultaneously, based on the global cumulative sample count and global covariance matrix maintained during training history, the normalized center confidence weights for each category are updated and calculated. Then, the original Mahalanobis distance for each sample is calculated, and co-scaling is performed using the local scaling factor and center confidence weights to generate an adaptively corrected distance. In this process, the updates to the global covariance matrix and cumulative sample count only occur during the training phase and do not participate in gradient calculation; they are only used as maintenance variables.

[0077] The loss value for the current training batch is calculated using the joint loss function based on the predicted probability and the adaptively corrected distance.

[0078] Then, calculate the cross-entropy loss value L. CE And confidence-aware co-scaling Gaussian mixture loss L based on adaptively corrected distance. CCS-GML Then, the weighted sums are performed according to the set balance coefficient λ to obtain the joint loss value L of the current training batch. total The joint loss value is a scalar that comprehensively reflects the model's classification error and the degree of optimization of the feature distribution in the current batch. It is the sole objective function guiding parameter updates.

[0079] The gradient of the loss value with respect to the network parameters is calculated using the backpropagation algorithm, and the network parameters are updated using the gradient descent optimizer.

[0080] After obtaining the joint loss value, it is used as the optimization objective, and backpropagation is performed using the automatic differentiation engine of the deep learning framework. Starting from the loss node, the partial derivatives of the loss with respect to each trainable network parameter (including convolutional kernel weights, fully connected layer weights, bias terms, etc.) are calculated layer by layer along the computation graph; these are the gradient vectors. The gradient indicates the direction and magnitude of adjustment required to reduce the loss for each parameter in the current state. Subsequently, a gradient descent optimizer is selected, such as stochastic gradient descent with momentum or the Adam optimizer, and the network parameters are updated layer by layer based on the calculated gradients and a pre-set learning rate. During the parameter update process, the learning rate can be adjusted using a cosine annealing strategy or piecewise constant decay to ensure smooth convergence in the later stages of training. After each update, the network's representation and classification capabilities slowly evolve towards a better direction.

[0081] Repeat the above steps until the model converges, and you will obtain a trained facial expression recognition model.

[0082] After each training round, the model's facial expression recognition accuracy and feature distribution metrics, such as the trace ratio of the intra-class and inter-class scatter matrices, are evaluated using a test set. Convergence is determined when the model's performance on the test set no longer shows significant improvement, or when the preset maximum number of training rounds has been reached. At this point, the final network weights and related global statistics are saved as a model file, resulting in the trained facial expression recognition model. This model can be directly used in the inference process without relying on any form of joint loss calculation or co-scaling of intermediate variables, maintaining the lightweight and efficient nature of forward inference.

[0083] This invention, while retaining the stable classification optimization capability of the cross-entropy loss function, introduces a category-aware local scaling factor and a category center confidence weight based on historical statistics. Through a non-linear scaling mechanism formed by the collaboration of these two factors, it achieves refined and differentiated constraints on the feature distribution of different expression categories. Specifically, the local scaling factor independently measures the feature dispersion of each expression category in the current training batch, subjecting compact and dispersed categories to appropriate cohesion constraints. This effectively overcomes the shortcomings of a single global scaling factor, which is coarse-grained and unable to adapt to the heterogeneity of distribution among categories. The center confidence weight comprehensively evaluates the training sufficiency and long-term distribution compactness of each category center, enabling the model to dynamically perceive the reliability of each category center estimate. When the center is unreliable, it automatically relaxes the constraint strength, avoiding misleading feature learning caused by center shift in the early stages of training or for small sample categories. This mechanism makes similar expression samples more tightly clustered in the deep feature space, while maintaining clear intervals between dissimilar expression samples, improving the model's recognition accuracy and robustness in complex scenarios such as lighting changes, occlusion, and pose differences. Meanwhile, all improvements in this invention are focused on the loss function level, without increasing the number of parameters or computational overhead in the network inference stage. This allows the model to maintain its lightweight characteristics while achieving performance improvements, making it particularly suitable for real-world deployment scenarios that are sensitive to resource consumption.

[0084] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A facial expression recognition method based on a joint loss function, characterized in that, include: The deep learning network is used to extract features from the input facial expression image to obtain deep feature representation, and the predicted probability of each expression category is obtained through classification mapping. We perform category-level statistical modeling on deep feature representations, calculate local scaling factors and center confidence weights, and perform co-scaling to obtain adaptive corrected distances. The cross-entropy loss based on predicted probability and the confidence-aware co-scaling Gaussian mixture loss based on adaptive corrected distance are weighted and combined to form a joint loss function. The deep learning network is iteratively trained using a joint loss function to obtain a trained facial expression recognition model. The image of the face to be recognized is input into the trained facial expression recognition model, and the final expression recognition result is output.

2. The method according to claim 1, characterized in that, Calculate the local scaling factor and center confidence weight, including: Deep feature representation and category labels of collected samples; Extract a subset of sample features from the deep feature representation, calculate the average of the standard deviations of the feature values, and obtain the local scaling factor; The confidence weights of the category centers are calculated by comprehensively evaluating the historical cumulative sample count and the trace of the global feature covariance matrix.

3. The method according to claim 2, characterized in that, The confidence weights for the category centers are calculated, specifically including: Maintain the global cumulative sample count and global feature covariance matrix for each category during training; Based on the global cumulative sample count, the relative training sufficiency of each category is calculated; based on the global feature covariance matrix, the matrix trace reflecting the compactness of the category feature distribution is calculated. The relative training sufficiency is combined with the matrix trace to obtain the unnormalized confidence value. The obtained unnormalized confidence values ​​for each category are normalized so that the sum of all weights is 1, thus obtaining the final category center confidence weights.

4. The method according to claim 1, characterized in that, The adaptive correction distance is obtained, including: For each sample in the current training batch, calculate the Mahalanobis distance between the deep feature representation and all K class centers to obtain the original Mahalanobis distance value; Calculate the ratio of the category local scaling factor of each category to the average local scaling factor of all categories in the batch to obtain the category dispersion ratio, which reflects the relative dispersion of each category. The category dispersion ratio is combined with the category center confidence weight to form a collaborative scaling term; The collaborative scaling term is nested in a preset nonlinear modulation function to generate collaborative scaling coefficients, which are then applied to the original Mahalanobis distance value to generate an adaptive corrected distance.

5. The method according to claim 1, characterized in that, The weighted combination forms a joint loss function, including: The cross-entropy loss value is calculated based on the sample prediction probability; Based on the adaptively corrected distance, the confidence-aware co-scaling Gaussian mixture loss value is calculated; A balance coefficient is introduced to perform a weighted summation of the cross-entropy loss value and the confidence-aware co-scaling Gaussian mixture loss value, resulting in a joint loss function, which is used to achieve synergistic optimization of classification and discrimination capabilities and feature distribution constraint capabilities.

6. The method according to claim 1, characterized in that, It also includes preprocessing the input facial expression images, including: The dataset of facial expression images was divided into a training set and a test set. The input image is sequentially converted to grayscale to obtain a single-channel grayscale image; size normalization is performed to obtain a uniformly sized image; contrast-limited adaptive histogram equalization combined with Gamma correction is performed to obtain a standardized image with enhanced local contrast and detail information. During the training phase, standardized images are randomly cropped and horizontally flipped to generate enhanced training samples, thereby improving the model's ability to adapt to changes in facial expressions, lighting, and viewing angles.

7. The method according to claim 1, characterized in that, The predicted probabilities of each expression category are obtained through classification mapping, including: Construct a facial expression recognition network, extract features layer by layer from the input standardized image or enhanced training sample, and output a deep feature representation vector of fixed dimension from the specified network layer; The deep feature representation vector is input into the fully connected classification layer to calculate the original output score for each category. The original output scores are normalized using the Softmax function, and the scores are mapped to the predicted probabilities of each expression category.

8. The method according to claim 1, characterized in that, Iterative training of deep learning networks using a joint loss function includes: The enhanced training samples are input into the facial expression recognition network, and the deep feature representation and predicted probability of each sample are calculated through forward propagation. Based on deep feature representation, class-level statistical modeling is performed to calculate the class local scaling factor and class center confidence weight of the current batch, and an adaptive corrected distance is generated. Based on the predicted probability and adaptively corrected distance, the loss value of the current training batch is calculated using the joint loss function; The gradient of the loss value with respect to the network parameters is calculated using the backpropagation algorithm, and the network parameters are updated using the gradient descent optimizer. Repeat the above steps until the model converges, and you will obtain a trained facial expression recognition model.

9. The method according to claim 1, characterized in that, Output the final facial expression recognition results, including: Acquire the face image to be identified, perform standardized preprocessing, and obtain the standardized image to be identified; The standardized image to be identified is input into the trained facial expression recognition model, and deep feature representations are extracted and prediction probability vectors are calculated through a single forward propagation. Traverse the predicted probability vector, map the index corresponding to the maximum search value to the corresponding expression category label, and output it as the final expression recognition result.

Images