Facial expression recognition model feature space orientation stability training method and system

Abstract

The present application belongs to the technical field of image recognition and artificial intelligence, and specifically relates to a feature space orientation stability training method and system for a facial expression recognition model, which constructs an expression recognition model comprising a feature extraction unit and an expression classification unit; performs feature space decomposition on a hidden representation vector to obtain multiple feature directions, and divides the feature directions into expression subspaces and irrelevant subspaces according to the correlation degree of each feature direction and an expression category label; applies perturbation along the feature directions to obtain a prediction distribution response difference, and applies a differential constraint stronger than the expression subspaces to the response difference of the irrelevant subspaces; adjusts the application strength of the differential constraint according to the entropy value of a standard prediction distribution, and jointly trains the classification task. The present application realizes differential stability optimization for different feature directions, suppresses the prediction fluctuation caused by irrelevant direction perturbation while maintaining the expression ability, and improves the prediction stability of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition and artificial intelligence, in particular to a feature space directional stability training method and system for a facial expression recognition model. BACKGROUND

[0002] In the prior art, the mainstream paradigm for enhancing the prediction stability of a model is consistency regularization, the core idea of which is to add a consistency penalty of the prediction distribution of the original input and the perturbed input to the standard cross-entropy loss. Since the introduction of the Π-model, this paradigm has derived various variants such as Mean Teacher, Virtual Adversarial Training, and UDA (Unsupervised Data Augmentation). However, consistency regularization has three systematic defects: (1) the perturbation space is limited, the perturbation always acts on the input space, and the relationship with the internal representation of the model is indirect and uncontrollable; (2) the penalty strategy is blind, all prediction changes are punished equally, without distinguishing between expression semantic changes and illumination pose changes; (3) the regularization strength is rigid, with a globally fixed coefficient, without considering the differences in the inherent ambiguity of different samples.

[0003] In Chinese patent documents, there are several technical solutions related to expression recognition model training and regularization. Some solutions generate adversarial samples by superimposing adversarial perturbations on expression images to improve recognition robustness; some solutions use convolutional neural networks to extract multi-scale features of expression images for classification; and some solutions use consistency regularization and image data augmentation techniques to extract feature information from supervised data and unsupervised data. Although the above solutions use different technical routes such as adversarial perturbation, visual backbone network enhancement training, and semi-supervised consistency learning, their stability optimization mechanisms mainly focus on input perturbation construction, prediction consistency constraints, or model knowledge transfer, usually taking the overall prediction change as the constraint object, lacking a mechanism for directional analysis of the internal feature representation space of deep models, and unable to identify the contribution differences of different feature directions to the expression recognition task, nor to implement differential stability constraints on different feature subspaces. In addition, some solutions involve multi-modal expression recognition methods based on sub-space feature fusion and cross-domain expression image recognition methods based on domain-adversarial self-adaptation, but none of them involve directional analysis of the geometric structure of the hidden representation space of a pre-trained visual model, nor selective allocation of regularization penalties based on the expression information content differences of feature directions. SUMMARY

[0004] The application provides a feature space directional stability training method and system for a facial expression recognition model, aiming to solve the technical problem that the consistency regularization technology in the prior art uniformly constrains each direction of the feature space without distinguishing the functional differences of different feature directions in expression discrimination, resulting in degradation of expression discrimination ability in the stability training process.

[0005] In a first aspect, the application provides a feature space directional stability training method for a facial expression recognition model, comprising: constructing an expression recognition model for facial expression recognition, the expression recognition model comprising a feature extraction unit and an expression classification unit; performing feature space decomposition on a hidden representation vector output by the feature extraction unit for a training sample to obtain a plurality of feature directions, and obtaining a standard prediction distribution output by the expression classification unit for the training sample; dividing each feature direction into an expression subspace and an irrelevant subspace according to the degree of association between each feature direction and an expression category label; applying perturbation to the hidden representation vector along each feature direction, inputting the perturbed hidden representation vector into the expression classification unit to obtain a perturbed prediction distribution, and obtaining a response difference according to the prediction distributions before and after perturbation; applying differentiated constraints to the response difference according to the subspace to which each feature direction belongs, wherein the constraint strength applied to the response difference of a feature direction belonging to the irrelevant subspace is greater than the constraint strength applied to the response difference of a feature direction belonging to the expression subspace; adjusting the application strength of the differentiated constraints according to the entropy value of the standard prediction distribution, and performing training of the expression recognition model by combining the differentiated constraints with the classification task.

[0006] Further, the feature space decomposition step comprises: constructing a feature covariance matrix based on the hidden representation vector, performing eigenvalue decomposition on the feature covariance matrix to obtain a set of orthogonal feature direction vectors and feature values corresponding to each feature direction, the feature values representing the variance in that feature direction.

[0007] Further, dividing each feature direction into an expression subspace and an irrelevant subspace specifically comprises: using a forward selection algorithm to iteratively select a set of directions from the feature directions that have the maximum joint explanatory power for the expression category label based on the maximum increment of the joint determination coefficient as the criterion, to form the expression subspace; identifying directions in the feature directions that have a variance proportion exceeding a first threshold and a correlation with the expression category label lower than a second threshold as the irrelevant subspace.

[0008] Furthermore, the forward selection algorithm specifically includes: In the first round, the linear regression determination coefficient of each feature direction with respect to the expression category label is calculated, and the direction with the largest determination coefficient is selected and added to the expression subspace. In subsequent rounds, for each remaining direction, a multiple linear regression is performed on it in combination with the selected directions, and the direction that maximizes the increase in the joint determination coefficient is added to the expression subspace; the selection stops when the increase is lower than a preset threshold.

[0009] Furthermore, before filtering the expression subspace, the following steps are also included: The weight vector of the expression classification unit is projected into the space formed by the feature directions to obtain the expression relevance prior value corresponding to each feature direction. Based on the prior value, the first few feature directions are selected as candidate search spaces, and the forward selection algorithm is executed in the candidate search spaces.

[0010] Furthermore, the response difference is measured using the Fisher-Rao distance metric, which is defined as: ; in, and These are the predicted distributions before and after the disturbance, respectively. The total number of emoji categories. and They are respectively and In the The probability values ​​for each category.

[0011] Furthermore, the differentiated constraint is achieved through directional stability loss:

[0012]

[0013] ; in, As training samples, The hidden representation vector, For the first One characteristic direction, For the disturbance amplitude, This is the output function of the expression classification unit. The Fisher-Rao distance, For the aforementioned irrelevant subspace, For the expression subspace, , , respectively are penalty weights given to the irrelevant subspace, the expression subspace and the intermediate subspace, and .

[0014] Further, the entropy value of the standard prediction distribution adopts a Shannon entropy measurement: ; Wherein is the standard prediction distribution, is the total number of expression categories, is the probability value of the distribution on the first category; Adjust the application strength of the differential constraint according to the entropy value of the standard prediction distribution, specifically: ; Wherein is the sample-by-sample adaptive regularization strength coefficient, is the basic regularization strength, is the standard prediction distribution of the sample ; When decreases increases, and the applied constraint is strengthened; when increases decreases, and the applied constraint is weakened.

[0015] Further, before applying perturbations to the hidden representation vector along each of the feature directions, the method further comprises the step of applying a semantic equivalent perturbation operation to the training sample to generate a plurality of perturbed versions, the semantic equivalent perturbation operation specifically comprising: Selecting an image block position to be transformed with a preset region selection probability, applying a candidate visual transformation to the selected image block, the candidate visual transformation including at least one of random erasing, color jittering, brightness perturbation, and contrast perturbation; calculating the feature similarity between the candidate perturbed image and the original image at the output end of the feature extraction unit, filtering out candidate images with a similarity lower than a preset threshold, and randomly sampling from the candidate images that meet the conditions to generate perturbed versions.

[0016] In a second aspect, the present application provides a feature space directional stability training system for a facial expression recognition model, which is used to execute the method and comprises: A model construction module configured to construct an expression recognition model for facial expression recognition, the expression recognition model comprising a feature extraction unit and an expression classification unit, the feature extraction unit being configured to convert an input facial expression image into a hidden representation vector, and the expression classification unit being configured to output a prediction distribution according to the hidden representation vector; a feature space analysis module configured to perform feature space decomposition on the hidden representation vector output by the feature extraction unit for the training sample to obtain a plurality of feature directions, and divide each of the feature directions into an expression subspace and an irrelevant subspace according to the degree of association between each of the feature directions and the expression category label; a disturbance generation module configured to obtain a standard prediction distribution output by the expression classification unit for the training sample, apply a disturbance to the hidden representation vector along each of the feature directions, input the disturbed hidden representation vector into the expression classification unit to obtain a disturbed prediction distribution, and obtain a response difference according to the prediction distributions before and after the disturbance; a directional stability module configured to apply differentiated constraints to the response difference according to the subspace to which each of the feature directions belongs, wherein the constraint strength applied to the response difference of the feature direction belonging to the irrelevant subspace is greater than the constraint strength applied to the response difference of the feature direction belonging to the expression subspace; a joint training module configured to adjust the application strength of the differentiated constraints according to the entropy value of the standard prediction distribution, and combine the differentiated constraints with the classification task to train the expression recognition model.

[0017] The technical effects of the present application are: First, the hidden representation vector output by the feature extraction unit for the training sample is decomposed in the feature space to obtain a plurality of feature directions, and each of the feature directions is divided into an expression subspace and an irrelevant subspace according to the degree of association between each of the feature directions and the expression category label. This step realizes explicit decoupling of the expression feature space, provides an accurate spatial positioning basis for subsequent differentiated constraints, and solves the defect that the traditional method cannot distinguish the functional attributes of the feature directions.

[0018] Second, a disturbance is applied to the hidden representation vector along each of the feature directions, the disturbed hidden representation vector is input into the expression classification unit to obtain a disturbed prediction distribution, a response difference is obtained according to the prediction distributions before and after the disturbance, differentiated constraints are applied to the response difference according to the subspace to which each of the feature directions belongs, and the constraint strength applied to the response difference of the feature direction belonging to the irrelevant subspace is greater than the constraint strength applied to the response difference of the feature direction belonging to the expression subspace. This step breaks the uniform constraint mode of the traditional consistency regularization, realizes directional and accurate allocation of the punishment budget (i.e., the actual amount of punishment allocated to the stability regularization term in each of the feature directions), strongly suppresses the prediction fluctuation in the irrelevant direction, while deliberately preserving the response sensitivity in the expression-related direction, thereby improving the robustness while avoiding degradation of the expression discrimination ability.

[0019] Third, the difference between the prediction distributions is measured by Fisher-Rao distance. Fisher-Rao distance is the geodesic distance on the manifold of probability distributions, which is more sensitive to the change of probability in the high-confidence region than Euclidean distance, and can capture the small shift of high-confidence prediction distribution more accurately.

[0020] Fourth, the strength of the differential constraint is adjusted according to the entropy value of the standard prediction distribution, and the constraint is tightened when the prediction confidence is high and automatically reduced when the prediction is ambiguous. This step realizes the self-adaptation of the regularization strength to the inherent ambiguity of the sample, avoiding the forced fitting of noise or abnormal labels by the model, and this adjustment does not harm the classification performance of the main task.

[0021] Fifth, the projection value of the weight vector of the expression classification unit in the feature direction space is used as the prior score of the expression correlation of each direction, and the search space of the forward selection is reduced accordingly. This step makes full use of the discriminative information contained in the trained model, without relying on additional labeled data to recalculate the correlation, significantly reducing the computational overhead of the subspace discovery process.

[0022] Sixth, the feature direction is recalculated based on the current model parameters at the beginning of each training period. This step adapts to the dynamic drift of the hidden representation space during training, always maintaining the timeliness and accuracy of subspace division, and ensuring the effectiveness of directional constraints. BRIEF DESCRIPTION OF DRAWINGS

[0023] Fig. 1 The flowchart of the feature space directional stability training method of the facial expression recognition model proposed in the embodiment of the present application; Fig. 2 The correspondence scatter plot of the expression correlation index based on the prior score of the classification head weight and based on the label calculation provided in the embodiment of the present application, wherein the four types of labels represent the directions selected by both methods, the directions selected only by forward selection, the directions selected only by LASSO, and the directions not selected by both. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0025] Embodiment one Loss function of traditional consistency regularization method (such as Π-model and its variants) The indiscriminate perturbation penalty is applied in the input space, which fails to distinguish the difference in the amount of expression information in different directions in the feature space. In an embodiment, using the PCA decomposition of the class label hidden representation space (768 dimensions) of ViT-B / 16, the direction with the largest variance , accounting for 20.7% of the total variance, is the main encoding direction of the expression category, and its expression correlation reaches 0.832; while The total of the other 7 directions accounts for only 4.3% of the total variance, and the expression correlation is less than 0.10, mainly encoding non-expression information such as illumination and posture. Since the consistency regularization does not consider the semantic difference of information in the feature direction, the predicted response on is actually stronger than that on , the penalty budget is wrongly allocated to the direction without expression information, and this penalty inversion phenomenon is stable when the PCA is compressed to This embodiment provides a feature space directional stability training method for a facial expression recognition model, as shown in Figs. 1-2 , which specifically includes the following steps: S0: Model construction, first construct an expression recognition model for facial expression recognition. The expression recognition model includes a feature extraction unit and an expression classification unit.

[0026] The feature extraction unit is used to convert the input data into a fixed-dimensional hidden representation vector. In this embodiment, the input data is a facial expression image, and the feature extraction unit uses a ViT pre-trained visual model as an encoder. The processing process of ViT on the input image is as follows: divide the input image into fixed-size image blocks and flatten them; perform linear projection on each image block to obtain an image block embedding vector; add a class label at the beginning of the sequence and superimpose the position encoding; perform global self-attention coding through a multi-layer Transformer encoder to output a hidden representation vector corresponding to each label position. For a given input image, take the hidden state corresponding to the class label position as the visual representation vector of the whole image, denoted as . Taking ViT-B / 16 as an example, the vector is 768-dimensional, denoted as , where D=768.

[0027] The expression classification unit is used to output a predicted distribution according to the hidden representation vector. The expression classification unit adopts the structure of a linear fully connected layer plus a Softmax activation function. The hidden representation vector output by the feature extraction unit is input into the linear layer, and after linear transformation, a logit vector of dimensions is obtained, which is then converted into a probability distribution by the Softmax function: ; where is the weight matrix of the linear layer, with a dimension of ; The bias term has the following dimensions: ; This represents all trainable parameters of the facial expression recognition model, including the parameters of the ViT feature extraction unit and the parameters of the facial expression classification unit; Let be the total number of expression categories. The th probability distribution... Each component This indicates that the facial expression recognition model predicts the input image belongs to the first... The probability values ​​of each expression category satisfy the following conditions: and In this embodiment, the facial expression recognition task is a seven-class classification task, i.e., C=7, with the categories corresponding to anger, disgust, fear, happiness, neutrality, sadness, and surprise. However, this method is also applicable to facial expression recognition tasks with two or more categories, requiring only corresponding adjustments. The value of .

[0028] Throughout the training process, the parameters of the feature extraction unit and the expression classification unit both participate in gradient updates and are jointly optimized through backpropagation.

[0029] S1: Feature Space Analysis and Subspace Recognition. At the beginning of each training cycle, a geometric analysis of the feature space is performed. As the model parameters are continuously updated during training, the structure of the hidden representation space also changes dynamically. Therefore, it is necessary to periodically update the feature orientation partitioning to adapt to the current model state and ensure the timeliness and accuracy of the subspace partitioning.

[0030] At the beginning of each training cycle, the current model parameters are frozen, and the class label hidden representation vector for each sample is collected through one forward propagation. (768 dimensions). Construct the covariance matrix. ,in It is the mean vector. Using eigenvalue decomposition... 768 orthogonal characteristic directions were obtained. (by eigenvalue) (Descending order). The eigenvectors corresponding to the top 128 largest eigenvalues ​​are retained as the effective search space.

[0031] A forward selection algorithm is performed in 128 directions. First, the expression relevance in each direction is calculated. As a reference indicator. Forward selection process: (1) First round: Each of the 128 directions is individually matched with a label. Perform linear regression calculation Select The largest direction (in this embodiment) , )join in (2) Second round: The projection values ​​are combined with the other directions to perform a binary linear regression, and the joint values ​​are selected. The direction of maximum gain ( Gain 0.169, combined (3) Repeat until the gain is <0.01 and then stop.

[0032] irrelevant subspace The identification criteria are: the variance percentage exceeds 0.5% ( And facial expression relevance And does not belong to This embodiment The seven directions accounted for 4.3% of the total variance, and the correlation of facial expressions was less than 0.10, mainly encoding non-facial information.

[0033] S2: Semantic equivalence perturbation and response acquisition, including the following sub-steps: S21: Generation of semantically equivalent perturbation versions. For each training sample x, K perturbation versions are generated using a semantically equivalent perturbation method based on image patch visual transformation. In this embodiment, K=5.

[0034] The specific disturbance generation process is as follows.

[0035] The first step is to use a preset replacement probability. The image block to be transformed is selected independently. In this embodiment, This means that each image patch in the input image has a 15% probability of being selected as the location to be transformed. This probability value is based on the commonly used proportion of random erasure in facial expression recognition training, which can generate sufficient local visual changes while maintaining the basic semantics of the expression.

[0036] The second step involves applying candidate visual transformations to the selected image block locations. These candidate visual transformations include random erasure, color jitter, brightness perturbation, and contrast perturbation, resulting in multiple candidate perturbed images. For each selected location, the perturbation result under each candidate visual transformation is obtained.

[0037] The third step is to select the top M candidate visual transformations from the candidate visual transformations for each selected position. In this embodiment, M=5.

[0038] The fourth step involves semantic consistency filtering for each candidate perturbation image. The cosine similarity between the feature representations of the candidate perturbation image and the original image at the output of the feature extraction unit is calculated. The cosine similarity between two vectors a and b is defined as: ; Its value range is The closer the value is to 1, the more consistent the directions of the two vectors, meaning they are more semantically similar. Only vectors with a similarity value not lower than a preset threshold are retained. The candidate perturbation image. In this embodiment, The threshold is set based on the fact that perturbation images with a feature similarity of less than 0.8 have a large visual difference from the original image, and replacing them may change the core semantics of facial expressions.

[0039] Fifth, randomly sample one image from the candidate perturbation images filtered by semantic consistency as the perturbation version. If the candidate set for a selected location is empty (i.e., no candidate image satisfies the semantic consistency condition), the original image patch is retained unchanged.

[0040] Through the above process, K perturbation versions that are semantically equivalent but visually different are generated, denoted as K. .

[0041] Original sample Input the perturbation version and the model respectively to obtain the hidden representation. and and predicted distribution and .for and Each direction in Calculate the difference in predicted responses: ,in The amplitude of the disturbance.

[0042] S22: A measure of the difference in predicted response. The difference between the predicted distributions before and after the perturbation is the response difference in this characteristic direction. The difference in predicted response is measured using the Fisher-Rao distance. The Fisher-Rao distance is a geodesic distance defined on the probability distribution manifold in information geometry, which can truly reflect the intrinsic differences between probability distributions.

[0043] For two discrete probability distributions and They are all A dimensional vector, where the sum of its components is 1, i.e. Fisher-Rao distance is defined as: ; in, and They are respectively and In the The probability values ​​for each category satisfy the following condition. , .

[0044] The range of this distance is [0, π]: when p = q, =1, arccos(1)=0, the distance is 0; when and When taking values ​​in completely different regions on the probability simplex =0, arccos(0)=π / 2, the distance is π.

[0045] The mathematical properties of the Fisher-Rao distance guarantee that it is more sensitive to probability changes than the Euclidean distance in the high-confidence interval (i.e., when the predicted probability is close to 0 or 1). Specifically, when there are small changes between two high-confidence predicted distributions, the Fisher-Rao distance can give a larger metric than the Euclidean distance, thus capturing small shifts in the predicted distribution more sensitively.

[0046] Based on the above definition, the difference in predicted response along the j-th characteristic direction of the perturbation is: ; in, For the standard prediction distribution, The predicted distribution after perturbation along the j-th characteristic direction .

[0047] S3: Construction of directional stability loss, based on the subspace partitioning determined in the preceding steps (expression subspace). and irrelevant subspaces Based on the above-mentioned methods for measuring the difference in predicted response, a directional stability loss is constructed.

[0048] For a single training sample x, its standard hidden representation vector is h, and its standard prediction distribution is... The squared difference in predicted response after applying perturbations along each characteristic direction is: ; The orientational stability loss is defined as a weighted sum of the squared differences in predicted responses across all feature directions, where the weights are determined based on the subspace to which the feature directions belong:

[0049]

[0050] ; in, As training samples, The hidden representation vector, For the first One characteristic direction, For the disturbance amplitude, This is the output function of the expression classification unit. The Fisher-Rao distance, For the aforementioned irrelevant subspace, For the expression subspace, , , The penalty weights are respectively assigned to the irrelevant subspace, the expression subspace, and the intermediate subspace, and .

[0051] In this embodiment, , , .

[0052] The selection of the aforementioned penalty weighting ratios has a clear technical basis. Central direction of facial expression correlation All values ​​were below 0.10, indicating that these directions contained almost no facial expression signals; while Central direction of facial expression correlation A value of at least 0.20 indicates a clear facial expression. The signal strength difference between the two is approximately 5 to 10 times. and The 10:1 ratio between them matches the signal strength difference precisely, penalizing the direction with extremely low signal-to-noise ratio with a force close to the upper limit, and applying a weak penalty to the direction with high signal-to-noise ratio with a force close to the lower limit. Between the two ends, the monotonically decreasing nature of the three-stage penalty is maintained.

[0053] It is important to emphasize that the selective inhibition effect is determined by... The inequality direction and its order-of-magnitude matching relationship with the difference in facial expression signal intensity in the characteristic direction, rather than each The specific absolute value. If we take and The interchange of assignments and the mismatch between the constraint direction and the distribution of facial expression information in the feature space will lead to a significant degradation in selective inhibition ability.

[0054] S4: Entropy-guided joint training.

[0055] For each training sample Calculate the information entropy of the predicted distribution. Construct a sample-by-sample adaptive regularization strength: ,in When the model prediction is highly certain ( )hour Apply near-full strength constraints; when the model hesitates ( )hour The level of constraint will be halved.

[0056] Final training objective function: The AdamW optimizer is used, with a learning rate of... Batch size The training process takes 10 training cycles. Online inference requires only a single forward propagation, with zero additional overhead.

[0057] Example 2 The forward selection algorithm described in Example 1 requires traversing the training samples at the beginning of each training cycle to calculate the Pearson correlation between each PCA direction and the expression category label, adding extra computational overhead. This example provides an alternative based on prior classification head weights, utilizing the discriminative information already included in the current model's classification head parameters to replace the label correlation calculation process, thus completing direction selection without additional traversal of labeled data.

[0058] Specifically, this embodiment first uses the same facial expression recognition model structure as in Embodiment 1 for pre-training, and then performs conventional supervised training using training samples to obtain a basic facial expression recognition model containing a feature extraction unit and an facial expression classification unit. This pre-training refers to performing several rounds of standard supervised training (e.g., 5 training cycles) to enable the classification head weights W to initially possess facial expression discrimination capabilities, and then using W for directional prior scoring. The facial expression classification unit consists of a linear classification layer, and its weight matrix is ​​denoted as W. Since the classification head establishes a mapping relationship between the hidden representation space and facial expression categories through supervised training, the responsiveness of the classification head weights on different feature dimensions can reflect the contribution of the corresponding hidden representation dimension to the facial expression recognition result.

[0059] After obtaining the basic model, the hidden representation vectors output by the feature extraction unit of the training samples are extracted, and the covariance matrix is ​​constructed and feature decomposition is performed according to the method in Example 1 to obtain the first 128 principal feature directions. Then, the category head weights are... Projecting onto the PCA feature space, calculate the weighted prior scores for each feature direction:

[0060] in, Indicates the first The PCA feature direction in the first... Each hidden representation represents a component in the dimension. Indicates the class head weight in the th order. Each hidden representation corresponds to a weight on a dimension. Used to measure the The importance of each feature direction to the expression recognition task is indicated by its numerical value. The larger the value, the more the classification head relies on the information carried by that direction to make category judgments.

[0061] Based on the weighted prior scores for each direction, the 128 candidate directions were ranked, and the direction with the highest score was selected. Each direction serves as a candidate subspace for expressions, where... The preset number of candidate directions is set in this embodiment. The original 128-dimensional direction search space is reduced to a 10-dimensional candidate space. Based on this, LASSO regression is further employed to refine the direction selection, thus obtaining the final expression subspace.

[0062] Specifically, the hidden representation projection values ​​in the 10 candidate directions are constructed as the input matrix. The corresponding emoji category labels are encoded to construct the output vector. The importance coefficients for each direction are determined by the following optimization objectives:

[0063] in, This indicates the number of samples participating in the direction selection. This represents the feature matrix composed of the projection values ​​of the candidate directions. Represents the vector of emoji category labels. This represents the regression coefficients corresponding to each candidate direction. express The regularization strength parameter controls the sparsity of direction selection; in this embodiment, α = 0.2. After LASSO optimization, directions with regression coefficients close to zero are considered low-contribution directions and are discarded. The directions corresponding to non-zero coefficients form the expression subspace. .

[0064] After completing the above directional filtering, the resulting expression subspace will be... Irrelevant candidate directions determined based on variance proportion and weighted scoring The directional stability loss construction process in Example 1 is used as a common input, and the same semantic perturbation generation, subspace-level differential penalty, and entropy-guided adaptive regularization strategy are employed to complete the subsequent training.

[0065] Compared with Example 1, this example does not require repeated calculation of the correlation between feature direction and manual label in each round of direction search. Instead, it uses the discrimination information in the trained classification head for fast filtering. Therefore, it is suitable for application scenarios with high annotation costs or those that need to reduce the computational overhead of direction search.

[0066] To verify the effectiveness of the prior scores for classifier head weights, this embodiment trains a ViT model on a facial expression image dataset and compares the prior scores for weights. Relevance index of facial expressions calculated based on labels Consistency among them. The Spearman rank correlation coefficients for all 128 PCA directions were calculated experimentally, and the results are as follows: This indicates that the two methods are consistent in ranking the importance of feature orientations.

[0067] Further analysis revealed that in the weighted prior score ranking, all seven expression directions obtained in this embodiment were among the top seven. These results demonstrate that the discriminative information contained in the classifier head weights can effectively reflect expression-related directions in the hidden representation space and can serve as a fast approximation scheme based on label search methods.

[0068] Regarding computational overhead, the forward selection method in Example 1 requires traversing candidate directions round by round and calculating the joint determination coefficient gain after adding different directions, thus exhibiting serial search characteristics. This example first completes direction scoring through a single matrix projection, and then uses LASSO to optimize and filter candidate directions. Under the same experimental conditions, the weight prior calculation takes approximately 0.35 milliseconds, the LASSO filtering takes approximately 0.8 milliseconds, and the total computation time is approximately 1.1 milliseconds; Example 1 requires approximately 2.79 seconds to complete the filtering of 7 directions.

[0069] The above results demonstrate that this embodiment can significantly reduce the computational overhead in the direction selection process, making it suitable for applications with limited labeled data or those requiring rapid adjustment of the feature space structure. It should be noted that Embodiment 1 and this embodiment address different application needs: Embodiment 1 utilizes label-supervised information to obtain more refined expression direction selection results, while this embodiment utilizes existing discriminative information from the classification head to reduce additional search costs. Both can provide an expression-related subspace for subsequent orientation stability loss construction. Furthermore, the weighted prior score of this embodiment can also be used in conjunction with the forward selection process of Embodiment 1. That is, a candidate search space is first selected based on the weighted prior score, and then forward selection is performed within that candidate search space, thereby narrowing the direction search range while maintaining the accuracy of label-supervised direction selection.

[0070] refer to Fig. 2 The horizontal axis represents the expression relevance score based on the prior weight of the classifier head, and the vertical axis represents the expression relevance index calculated based on the label. The four types of labels in the figure (directions selected by both methods, directions selected by forward selection only, directions selected by LASSO only, and directions not selected by either method) represent different meanings. Fig. 2 Demonstrates prior scoring based on classifier head weights Relevance index of facial expressions calculated based on labels A scatter plot showing the correspondences across all 128 candidate directions. The horizontal axis in the plot is... The normalized value, with the ordinate as The original values. The scatter plot contains four types of labels: both selected indicates directions selected into the expression subspace by both the forward selection algorithm and the weighted prior pre-screening (combined with LASSO), a total of 5; forward selection only indicates directions selected only by the forward selection algorithm and not by the weighted prior method, a total of 2; LASSO only indicates directions selected only by the weighted prior method combined with LASSO and not by forward selection, a total of 1; neither selected indicates directions not selected by either method, a total of 120. From Fig. 2 It can be seen that the two methods show significant consistency in ranking the importance of directions. The directions selected by both methods are distributed in the region with higher prior weight scores in the scatter plot, further validating the Spearman rank correlation coefficient. The statistical results.

[0071] This figure illustrates that the two methods are consistent in ranking the importance of direction (Spearman rank correlation coefficient reaches 0.506), verifying the effectiveness of the weighted prior method.

[0072] Example 3 To verify the technical effectiveness of the feature space orientation stability training method for the facial expression recognition model described in this invention, this embodiment uses a facial expression image recognition task as the application scenario, and tests the classification performance, feature space stability, and orientation selective suppression ability of the method through comparative experiments.

[0073] The experimental environment for this embodiment is as follows: hardware configuration includes an NVIDIA Tesla V100 GPU (16GB VRAM), an Intel Core i7-12700 processor, and 32GB of DDR4 memory; the operating system is Windows 11; the software environment includes the PyTorch 2.0.1 deep learning framework, the Transformers 4.30.2 library, Python 3.9, and CUDA 11.8. The random seed was fixed at 42 throughout the experiment.

[0074] The experiment uses the publicly available facial expression image dataset RAF-DB for testing to verify the applicability of the proposed method in the image expression feature space. This embodiment employs a fixed-ratio partitioning method to construct training, validation, and test sets, ensuring that different methods are compared under the same data conditions. The dataset contains 20,471 samples, each a 100×100 facial expression image, with seven expression categories (anger, disgust, fear, happiness, neutral, sadness, and surprise). The experiment uses a 2:1:7 hierarchical partitioning ratio, resulting in 4,092 training samples, 2,044 validation samples, and 14,335 test samples. The training set is used for model parameter optimization, the validation set for hyperparameter selection, and the test set for final performance evaluation. The experiment uses a ViT-B / 16 pre-trained visual model as the feature extraction unit, and an expression recognition model is constructed by adding a classification layer. The pre-training weights use publicly available ViT-B / 16 pre-training parameters. In the implementation of the semantically equivalent perturbation operator, image patch visual transformation is used as the candidate perturbation generator, and a feature consistency threshold is used. Set to 0.8 (this threshold is used to control the degree of feature consistency between the replaced region and the original region, and is determined through preliminary experiments), replacement ratio Set it to 0.15.

[0075] To comprehensively evaluate the performance of the method of this invention, the following three comparative training methods are set up.

[0076] Method A (CE Baseline): Supervised training is performed using only the standard cross-entropy loss function, without introducing stability regularization or directional penalty terms.

[0077] Method B (Consistency Regularization): The consistency regularization training method is used, and its training objective function is... , Among the perturbation samples The above-mentioned image patch visual transformation method applies a uniform constraint to the perturbation changes in the input space.

[0078] Method C (Invention): Employing the directional penalty method proposed in this invention, including the forward selection subspace discovery process (expression subspace) described in Example 1. Contains 7 directions, independent subspace Includes 7 directions), subspace-level directional penalty mechanism ( , , ) and entropy-guided adaptive regularization mechanism ( ).

[0079] All three methods use the same training parameter settings, with a learning rate of [missing value]. The batch size is 32, the number of training rounds is 10, the optimizer is AdamW, the weight decay coefficient is set to 0.01, the learning rate scheduling strategy adopts linear warm-up and linear decay, and the warm-up steps account for 10% of the total training steps.

[0080] The experiment used four evaluation metrics to evaluate different methods. First, classification accuracy and macro-average F1 score (F1) were used to evaluate the basic facial expression recognition performance of the models. Second, prediction stability metrics based on feature space perturbations were used. The calculation method involves applying the following along the directions of the 128 PCA principal components: The structured perturbation is calculated, and the average change in the prediction distribution before and after the perturbation is used to reflect the overall response of the model in different directions of the feature space. Third, the Prediction Suppression Ratio (PSR) is defined as:

[0081] in, express The change in predicted distribution caused by subspace perturbation express The change in predicted distribution caused by subspace perturbation. The PSR metric is used to evaluate the model's ability to impose differentiated constraints on different semantic function directions, comparing only two subspaces with different penalty weights during training. ( )and ( The remaining directions use... Moderate constraints, lacking significant discriminative power, are not included in the PSR calculation. A lower PSR value indicates that the model is more effective at reducing predicted responses in irrelevant directions while preserving effective responses in expression-related directions. The PSR evaluation uses the values ​​determined at the end of training. and Subspace partitioning means that subspace members will not be recalculated or updated during the evaluation phase.

[0082] Fourth, the confidence gap (ConfGap) is calculated by first calculating the average prediction confidence of all correctly predicted samples and the average prediction confidence of all incorrectly predicted samples. The difference between the two is ConfGap. The larger this value is, the more confident the model is when it makes a correct prediction than when it makes an incorrect prediction. In other words, the model has a stronger ability to distinguish between correct and incorrect predictions.

[0083] Table 1. Performance comparison of three training methods (ViT-B / 16 + RAF-DB facial expression image dataset)

[0084] As shown in Table 1, the accuracy of the directional penalty method of this invention reaches 83.6693%, which is superior to the CE baseline method (82.8950%) and the consistency regularization method (83.2089%), respectively. This indicates that the proposed subspace-level directional constraint mechanism can achieve directional selectivity suppression while maintaining classification performance. In terms of metrics, the method of this invention achieves the lowest value, indicating that the directional penalty, while suppressing responses in irrelevant directions, makes the overall response amplitude in the feature space more convergent. Simultaneously, this invention, by reducing the constraint strength in expression-related directions, enables the model to achieve... The predicted response capability retained in the directional direction is significantly stronger than that in the irrelevant direction, with a relative reduction of about three-fifths that of the irrelevant direction. This indicates that the degree of weakening imposed by the present invention varies significantly across different functional directions. In contrast, although the consistency regularization method achieves lower... The value is fixed, but it applies uniform constraints to input perturbations, thus limiting the effective response in expression-related directions. Therefore, The metrics reflect the overall degree of change in the model's directional response, while the PSR metric is used to evaluate the effectiveness of selective constraints between different functional subspaces. Combining the two allows for a more comprehensive evaluation of the performance of the directional stability training method. Regarding the ConfGap metric, the proposed method achieves 0.1635, lower than the 0.2257 of the CE Baseline method and the 0.2030 of the consistency regularization method. This result indicates that while directional constraints improve directional selectivity, they also reduce the model's confidence and discriminative power, representing a trade-off. These results demonstrate that the proposed method, through feature space directional constraint optimization, can effectively adjust the differences in predicted responses across different feature directions, achieving directional suppression of predicted responses from irrelevant directions.

[0085] Table 2 PSR Comparison Results

[0086] The inhibition rate is calculated based on the percentage decrease in PSR relative to the CE baseline.

[0087] As shown in Table 2, the PSR of the method of this invention is 0.418, which is 44.0% lower than that of the CE Baseline method, indicating that the targeted penalty mechanism proposed in this invention can effectively reduce irrelevant subspace. The predicted response in the data, while preserving the expression-related subspace. Effective response capability.

[0088] In contrast, the consistent regularization method uses a uniform constraint approach, increasing the PSR from 0.745 at the CE baseline to 1.787. This is because the method... The predicted change in the direction decreased from 0.00289 at the CE Baseline to 0.00138, a decrease of 52.2%; while the predicted change in the irrelevant subspace direction increased from 0.00215 to 0.00247, indicating that while the method weakens the response capability in expression-related directions, it does not suppress the predicted response in irrelevant directions.

[0089] The method of this invention applies differential constraints to the subspace, thereby... The change in direction prediction was reduced to 0.33 times that of CEBaseline, while also making The change in directional prediction was reduced to 0.59 times that of the CE Baseline, and the ratio of the two reductions was 1.6 times, which reflects the model's ability to apply differentiated stability constraints to different functional directions.

[0090] To further verify the inequality direction of the aforementioned differential constraints, a reverse penalty control experiment was conducted: keeping all other settings completely unchanged, only the penalty weights of the irrelevant subspace and the expression subspace were interchanged. The base model used was ViT-B / 16, extracting class-labeled hidden representations from the training data and constructing a PCA feature space, with the first 128 directions retaining a cumulative variance of 81.9%. The forward selection method from Example 1 was used to obtain the expression subspace on the training set. and irrelevant subspaces ,in It contains 7 characteristic directions and joint determination coefficients. , Includes direction Except for the direction of the penalty weight assignment, all other training parameters are consistent with those in the previous experiments.

[0091] Table 3 Results of the reverse penalty control experiment (ViT-B / 16 + RAF-DB facial expression image dataset)

[0092] As shown in Table 3, the direction of weight assignment plays a decisive role in the selective suppression effect: when using the inequality direction of this invention, the PSR is 0.418, a decrease of 44.0% compared to 0.745 of the CE Baseline; however, after swapping the weights at both ends, the PSR increases to 0.538, indicating a significant degradation in selective suppression capability. The results of the reverse control experiment show that the selective suppression effect is determined by the inequality direction of the penalty weights and its order-of-magnitude matching relationship with the difference in facial signal intensity between the feature directions, rather than the specific absolute values ​​of each penalty weight.

[0093] It should be noted that the facial expression image experiment in this embodiment was validated under a seven-class classification condition. For expression recognition tasks with more or fewer classes, the number of classes parameter in the Fisher-Rao distance... The number of categories can be adjusted according to the actual number of categories. In this embodiment, the forward selection process uses a multi-output pseudo-determination coefficient as the direction selection criterion, which is obtained by averaging the determination coefficients of each output dimension after one-hot encoding of the category labels. The above... The same applies to binary classification tasks; the direction selection criteria in the forward selection process do not need to be changed.

[0094] Based on the above experimental results, this invention verifies that there is a difference between the variance information distribution in the hidden representation space of the expression recognition model and the expression discrimination information distribution, and obtains expression-related and irrelevant subspaces through feature direction filtering. On a facial expression image dataset, the method of this invention can reduce the prediction response caused by irrelevant direction perturbations while maintaining the information representation ability of expression-related directions. Specifically, the joint determination coefficient of the expression subspace obtained by forward selection reaches 0.716; compared with the CE Baseline method, the PSR index of this invention decreases from 0.745 to 0.418, a reduction of 44.0%; in the reverse control experiment of exchanging the direction of the penalty weight inequality, the PSR rises back to 0.538, and the suppression amplitude is reduced to 27.8%, indicating that the inequality assignment direction of the penalty weight has a clear impact on the selective suppression effect. The above results demonstrate that the subspace-level directional stability training method proposed in this invention can effectively improve the directional selectivity of the model's feature space constraints.

[0095] Based on the same inventive concept, this embodiment also provides a feature space orientation stability training system for a facial expression recognition model, which is used to execute the above method. This system includes the following functional modules: The model building module is configured to build an expression recognition model for facial expression recognition. The expression recognition model includes a feature extraction unit and an expression classification unit. The feature extraction unit is used to convert the input facial expression image into a fixed-dimensional hidden representation vector, and the expression classification unit is used to output a predicted distribution based on the hidden representation vector.

[0096] The feature space analysis module is configured to perform feature space decomposition on the hidden representation vectors output by the feature extraction unit of the training samples to obtain multiple feature directions, and divide the multiple feature directions into an expression subspace and an irrelevant subspace according to the degree of correlation between each feature direction and the expression category label. Specifically, this module constructs a feature covariance matrix and performs feature decomposition to obtain a set of orthogonal feature direction vectors; it uses a forward selection algorithm to iteratively select a set of directions with the maximum joint determination coefficient as the criterion to form an expression subspace; and identifies the directions whose variance ratio exceeds a first threshold and whose correlation with the expression category label is lower than a second threshold as irrelevant subspaces. This module can also optionally implement a fast pre-screening method based on the prior weight of the classification head.

[0097] The perturbation generation module is configured to apply semantically equivalent perturbation operations to the training samples to generate multiple perturbation versions, and obtain the prediction distributions corresponding to the training samples and their perturbation versions; apply structured perturbations to the hidden representation vectors along each of the feature directions, and obtain the difference between the prediction distributions before and after the perturbation as the response difference. The difference between the prediction distributions is measured using the Fisher-Rao distance metric.

[0098] The directional stability module is configured to apply differentiated constraints to the response differences based on the subspace to which each feature direction belongs. Specifically, the constraint strength applied to response differences belonging to feature directions in the irrelevant subspace is greater than the constraint strength applied to response differences belonging to feature directions in the expression subspace. This module constructs a directional stability loss, assigning differentiated penalty weights to the irrelevant subspace, expression subspace, and intermediate subspace, with the penalty weight being the largest for the irrelevant subspace and the smallest for the expression subspace.

[0099] The joint training module is configured to adjust the intensity of the differential constraints based on the entropy value of the standard prediction distribution, and to jointly train the expression recognition model with the differential constraints and the classification task. Specifically, this module constructs a sample-by-sample adaptive regularization intensity coefficient based on the Shannon entropy of the standard prediction distribution, combines the directional stability loss and the cross-entropy loss to form the training objective function, and uses an optimizer to update the parameters of the expression recognition model through gradient backpropagation.

[0100] The specific implementation methods of the above modules correspond exactly to the technical details described in each step of the aforementioned method, and will not be repeated here.

[0101] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A feature space orientation stability training method for a facial expression recognition model, characterized in that, include: Construct an expression recognition model for facial expression recognition, the expression recognition model including a feature extraction unit and an expression classification unit; The hidden representation vector output by the feature extraction unit of the training samples is decomposed into feature space to obtain multiple feature directions, and the standard prediction distribution output by the expression classification unit of the training samples is obtained. Based on the degree of correlation between each feature direction and the expression category label, each feature direction is divided into an expression subspace and an irrelevant subspace; The hidden representation vector is perturbed along each of the aforementioned feature directions, and the perturbed hidden representation vector is input into the expression classification unit to obtain the perturbed prediction distribution. The response difference is obtained based on the prediction distribution before and after the perturbed distribution. Differential constraints are applied to the response differences based on the subspace to which each of the aforementioned feature directions belongs, wherein the constraint strength applied to the response differences of feature directions belonging to the unrelated subspaces is greater than the constraint strength applied to the response differences of feature directions belonging to the expression subspaces. The intensity of the differential constraint is adjusted according to the entropy value of the standard prediction distribution, and the differential constraint is combined with the classification task to train the facial expression recognition model.

2. The method according to claim 1, characterized in that, The steps of the feature space decomposition include: constructing a feature covariance matrix based on the hidden representation vector, performing feature decomposition on the feature covariance matrix to obtain a set of orthogonal feature direction vectors and eigenvalues ​​corresponding to each feature direction, wherein the eigenvalues ​​represent the variance magnitude in that feature direction.

3. The method according to claim 1, characterized in that, Each of the aforementioned feature directions is divided into an expression subspace and an irrelevant subspace, specifically including: A forward selection algorithm is used to iteratively select a set of directions that have the greatest joint explanatory power for the expression category label from the feature directions, based on the maximum increment of the joint determination coefficient, and thus constitute the expression subspace. The direction in which the variance ratio exceeds the first threshold and the correlation with the expression category label is lower than the second threshold is identified as the irrelevant subspace.

4. The method according to claim 3, characterized in that, The forward selection algorithm specifically includes: In the first round, the linear regression determination coefficient of each feature direction with respect to the expression category label is calculated, and the direction with the largest determination coefficient is selected and added to the expression subspace. In subsequent rounds, for each remaining direction, a multiple linear regression is performed on it in combination with the selected directions, and the direction that maximizes the increase in the joint determination coefficient is added to the expression subspace; the selection stops when the increase is lower than a preset threshold.

5. The method according to claim 3, characterized in that, Before filtering the emoji subspace, the following is also included: The weight vector of the expression classification unit is projected into the space formed by the feature directions to obtain the expression relevance prior value corresponding to each feature direction. Based on the prior value, the first few feature directions are selected as candidate search spaces, and the forward selection algorithm is executed in the candidate search spaces.

6. The method according to claim 1, characterized in that, The response difference is measured using the Fisher-Rao distance, which is defined as: ； in, and These are the predicted distributions before and after the disturbance, respectively. The total number of emoji categories. and They are respectively and In the The probability values ​​for each category.

7. The method according to claim 1, characterized in that, The differential constraint is achieved through directional stability loss: ； in, As training samples, The hidden representation vector, For the first One characteristic direction, For the amplitude of the disturbance, The output function of the expression classification unit. The Fisher-Rao distance, For the aforementioned irrelevant subspace, For the expression subspace, , , The penalty weights are respectively assigned to the irrelevant subspace, the expression subspace, and the intermediate subspace, and .

8. The method according to claim 1, characterized in that, The entropy value of the standard prediction distribution is measured using the Shannon entropy measure: ； in For the standard prediction distribution, The total number of emoji categories. For distribution In the Probability values ​​for each category; The intensity of the differential constraint is adjusted based on the entropy value of the standard predicted distribution, specifically as follows: ； in To adapt the regularization intensity coefficient for each sample, Based on the strength of regularization, For the sample The standard predicted distribution; when When decreasing As the value increases, the applied constraint strengthens; when When it increases As the constraint decreases, the imposed constraint weakens.

9. The method according to claim 1, characterized in that, Before applying perturbation to the hidden representation vector along each of the aforementioned feature directions, the method further includes applying a semantically equivalent perturbation operation to the training samples to generate multiple perturbation versions. The semantically equivalent perturbation operation specifically includes: The image block to be transformed is selected with a preset region selection probability. A candidate visual transformation is applied to the selected image block. The candidate visual transformation includes at least one of random erasure, color jitter, brightness perturbation, and contrast perturbation. The feature similarity between the candidate perturbation image and the original image is calculated at the output of the feature extraction unit. Candidate images with similarity lower than a preset threshold are filtered out. Random samples are taken from the candidate images that meet the conditions to generate a perturbation version.

10. A feature space orientation stability training system for a facial expression recognition model, characterized in that, The system is used to perform the method according to any one of claims 1-9, comprising: The model building module is configured to build an expression recognition model for facial expression recognition. The expression recognition model includes a feature extraction unit and an expression classification unit. The feature extraction unit is used to convert the input facial expression image into a hidden representation vector, and the expression classification unit is used to output a predicted distribution based on the hidden representation vector. The feature space analysis module is configured to perform feature space decomposition on the hidden representation vector output by the feature extraction unit of the training samples to obtain multiple feature directions, and divide each feature direction into an expression subspace and an irrelevant subspace according to the degree of correlation between each feature direction and the expression category label. The perturbation generation module is configured to obtain the standard prediction distribution of the training samples output by the expression classification unit, apply perturbation to the hidden representation vector along each of the feature directions, input the perturbated hidden representation vector into the expression classification unit to obtain the perturbation prediction distribution, and obtain the response difference based on the prediction distribution before and after the perturbation. The orientation stability module is configured to apply differential constraints to the response differences based on the subspace to which each of the feature directions belongs, wherein the constraint strength applied to the response differences of the feature directions belonging to the unrelated subspace is greater than the constraint strength applied to the response differences of the feature directions belonging to the expression subspace. The joint training module is configured to adjust the intensity of the differential constraint based on the entropy value of the standard prediction distribution, and to combine the differential constraint with the classification task to train the facial expression recognition model.

Images