A method for recognizing concealed facial expressions based on multi-task decoupling

Abstract

The application provides a face expression recognition method based on multi-task decoupling, which comprises the following steps: acquiring a face video segment, inputting the preprocessed face video segment into a multi-task decoupling recognition network; converting the face video segment into a space-time patch token sequence through a three-dimensional patch embedding layer, and dividing the space-time patch token sequence into multiple local face regions; constructing shared messenger tokens, real emotion dedicated messenger tokens and explicit expression dedicated messenger tokens in each local face region; inputting the patch tokens and the three types of messenger tokens of each local face region into a multi-layer decoupling local-global attention module to obtain shared representations, real emotion dedicated representations and explicit expression dedicated representations, respectively; and obtaining explicit required expression prediction results based on the shared representations and the explicit expression dedicated representations. The application improves the real emotion recognition and mixed expression recognition performance while maintaining the explicit expression recognition ability.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically a method for facial expression recognition based on multi-task decoupling. Background Technology

[0002] Facial expressions are important nonverbal cues for understanding an individual's emotional state; however, in real-world social interactions, the facial expressions an individual displays do not always align with their true emotions. Influenced by psychological, social, or situational factors, individuals may intentionally present an outward expression that conforms to the requirements of the context, while simultaneously masking or suppressing their true emotions. The goal of masked facial expression recognition is to simultaneously determine the internal true emotion and the externally required expression from the same video of facial behavior.

[0003] Compared to conventional facial expression recognition, faked facial expression recognition exhibits significant asymmetric visibility. Explicit expressions are typically actively presented by the subject, resulting in higher visual intensity and greater stability, making them easier for the model to capture. In contrast, genuine emotions are often revealed only through subtle, brief, and locally distributed facial movements, making them highly susceptible to being masked by strong explicit facial cues during feature learning.

[0004] Existing methods often treat genuine emotion recognition and overt facial expression recognition as separate tasks, or employ a multi-task learning framework using a shared backbone network and task-specific classification heads. The former struggles to leverage the inherent correlation between the two types of labels originating from the same facial video; the latter, while capable of joint supervision, typically relies on hard parameter sharing, assuming that shared features are equally beneficial to both tasks. In spoofed facial expression recognition, this assumption is insufficient because the more visually salient overt facial expression task dominates the shared feature space, thus suppressing weak discriminative evidence of genuine emotion.

[0005] Local-to-global interactive Transformers can first extract representations within local facial regions and then exchange global information through region-representation tokens, making them suitable for spatiotemporal modeling of facial videos. However, existing region-representation tokens are often task-independent. When directly applied to spoofed facial expression recognition, they easily form a hybrid representation dominated by overt facial expressions, failing to explicitly distinguish between shared facial context, genuine emotional cues, and overt facial expression cues. Therefore, it is necessary to design a spoofed facial expression recognition method that can decouple multi-task representations at the token level and reduce task interference during feature fusion and parameter optimization stages. Summary of the Invention

[0006] This invention addresses the problems of representation mixing, suppression of real emotion cues by explicit facial expression cues, and conflicts in shared parameter optimization that arise in existing occluded facial expression recognition methods during multi-task joint learning. It proposes an occluded facial expression recognition method, device, and electronic device based on multi-task decoupling, which improves the performance of real emotion recognition and mixed expression recognition while maintaining the ability to recognize explicit expressions.

[0007] To achieve the above objectives, the present invention provides a method for facial expression recognition based on multi-task decoupling, comprising the following steps: Obtain a video segment of the face to be identified, and perform face region cropping, scale normalization, and temporal sampling on the video segment; An end-to-end multi-task decoupled occlusion facial expression recognition network is constructed. The network takes the face video segment as input, converts it into a spatiotemporal patch token sequence through a three-dimensional patch embedding layer, and then divides it according to local face regions. In each local face region, a shared messenger token, a real emotion-specific messenger token, and an overt expression-specific messenger token are constructed, which respectively carry the shared facial context, evidence of real emotion, and overt expression cues. During the feature extraction stage, several decoupled local-global attention modules are stacked. Each decoupled local-global attention module sequentially performs masked local attention, communication of the same type of global token, and shared guided local-global interaction. During the prediction phase, the updated three types of messenger tokens in all local face regions are aggregated into a shared representation, a real emotion-specific representation, and an explicit expression-specific representation, respectively. The real emotion recognition branch fuses the shared representation and the real emotion-specific representation through learnable gating. The explicit expression recognition branch fuses the shared representation and the explicit expression-specific representation through learnable gating, and performs stopping gradient processing on the shared representation. During the training phase, the sum of the cross-entropy loss of real emotion and the cross-entropy loss of explicit required expression is used as the overall objective, and gradient protection is only performed on the shared parameters of the 3D patch embedding layer and the shared Transformer backbone network. When the gradient directions of the two tasks conflict on the shared parameters, the component in the gradient of the explicit required expression task that conflicts with the gradient of the real emotion task is projected out and then merged with the gradient of the real emotion task to update the shared parameters. During the inference phase, the true emotion prediction category and the explicit required expression prediction category are directly combined to obtain a mixed expression category.

[0008] Optionally, the three-dimensional patch embedding layer divides the face video segment into multiple spatiotemporal cubes and maps each spatiotemporal cube to a corresponding visual token, forming the spatiotemporal patch token sequence.

[0009] Optionally, in each local face region, the shared messenger token is used to aggregate public face context that is useful for both the real emotion recognition task and the explicit expression recognition task, the real emotion-specific messenger token is used to aggregate subtle facial movement cues related to real emotions, and the explicit expression-specific messenger token is used to aggregate salient facial cues related to explicit expression requirements.

[0010] Optionally, the decoupled local-global attention module includes a masked local attention step; in the masked local attention step, direct attention interaction between shared messenger tokens, real emotion-specific messenger tokens, and explicit expression-specific messenger tokens is prohibited, while attention interaction between various messenger tokens and patch tokens within the same local face region is preserved.

[0011] Optionally, the decoupled local-global attention module further includes a global token communication step of the same type; in the global token communication step of the same type, shared messenger tokens, real emotion-specific messenger tokens, and explicit expression-specific messenger tokens are collected from different local face regions respectively, and the same type of messenger tokens are used to perform global information interaction between different local face regions.

[0012] Optionally, the decoupled local-global attention module further includes a shared-guided local-global interaction step; in the shared-guided local-global interaction step, only the global shared representation formed by the shared messenger token is used to update the patch token in the local face region, so as to restrict the direct backflow of real emotion-specific information and overt expression-specific information to the patch token stream.

[0013] Optionally, obtaining the true emotion prediction results and the explicit required expression prediction results includes: In the real emotion recognition branch, the shared representation and the real emotion-specific representation are fused by gating to obtain the real emotion fusion feature; In the explicit expression recognition branch, the shared representation and the explicit expression-specific representation are fused by gating to obtain the explicit expression fusion feature; Specifically, during backpropagation training, stopping gradient processing is performed on the shared representation of the input explicit expression recognition branch.

[0014] Optionally, the training of the multi-task decoupling recognition network includes: The sum of the loss for classifying true emotions and the loss for classifying explicit desired expressions is used as the overall training objective. Gradient protection is performed on the shared parameters of the 3D patch embedding layer and the shared Transformer backbone network; When the inner product of the gradient of the real emotion recognition task on the shared parameters and the gradient of the explicit expression recognition task on the shared parameters is less than zero, the components in the gradient of the explicit expression recognition task that conflict with the gradient of the real emotion recognition task are removed, and the processed gradient of the explicit expression recognition task and the gradient of the real emotion recognition task are merged and used to update the shared parameters. Among them, the task-specific gating, task adapter, and classifier are independently optimized by the corresponding task loss.

[0015] Accordingly, the present invention also provides a facial expression recognition device based on multi-task decoupling, comprising: The preprocessing module is used to acquire face video segments and perform face region cropping, scale normalization, and temporal sampling; The embedding and region segmentation module is used to convert face video segments into spatiotemporal patch token sequences through a three-dimensional patch embedding layer, and to divide the spatiotemporal patch token sequences into multiple local face regions; The messenger token construction module is used to construct a shared messenger token, a real emotion-specific messenger token, and an explicit expression-specific messenger token for each local face region. The decoupled attention module is used to obtain shared representations, dedicated representations of real emotions, and dedicated representations of explicit expressions through masked local attention, communication of global tokens of the same type, and local-global interaction guided by sharing. The prediction module is used to obtain the prediction result of the true emotion based on the shared representation and the real emotion-specific representation, and to obtain the prediction result of the explicit required expression based on the shared representation and the explicit expression-specific representation, and to combine them to obtain the mixed expression category; The training optimization module is used to train the multi-task decoupled occluded facial expression recognition network based on the real emotion classification loss and the explicit required expression classification loss, and to perform gradient protection on the shared parameters.

[0016] Accordingly, the present invention also provides an electronic device, including a processor, a communication interface, a memory, and a bus, wherein the processor, the communication interface, and the memory communicate with each other through the bus, and the processor can call logical instructions in the memory to execute the above-mentioned masked facial expression recognition method based on multi-task decoupling.

[0017] The beneficial effects of the occluded facial expression recognition method based on multi-task decoupling of the present invention are as follows: This invention sets up a shared messenger token, a real emotion-specific messenger token, and an overt expression-specific messenger token in each local facial region. This enables explicit differentiation of shared cues, real emotion cues, and overt expression cues at the token level, reducing the mixing of multi-task representations.

[0018] This invention restricts direct interaction between different semantic messenger tokens through masked local attention, enables messenger tokens of the same semantic type to exchange long-range information across local face regions through communication of global tokens of the same type, and only sends the shared global context back to the local patch token through shared guided local-global interaction, thereby suppressing premature mixing of task-specific information while introducing global context.

[0019] This invention employs asymmetric gating fusion in the real emotion recognition branch and the explicit expression recognition branch, enabling the explicit expression recognition branch to make predictions using shared context, while preventing it from directly updating the shared representation through the fusion path, thereby reducing the interference of the explicit expression task on subtle information related to real emotions.

[0020] This invention performs gradient protection during the shared parameter optimization stage. When the gradient of the real emotion task conflicts with the gradient of the explicit expression task, the component in the explicit expression task gradient that conflicts with the gradient of the real emotion task is removed, thereby alleviating the shared parameter optimization conflict and improving the performance of real emotion recognition and mixed expression recognition. Attached Figure Description

[0021] Figure 1 A flowchart of a facial expression recognition method based on multi-task decoupling provided in an embodiment of the present invention; Figure 2 This is a model structure diagram of the occluded facial expression recognition method based on multi-task decoupling provided in an embodiment of the present invention; Figure 3 This is a visualization diagram of the attention of different messenger tokens provided in an embodiment of the present invention; Figure 4 This is a structural block diagram of a facial expression recognition device based on multi-task decoupling provided in an embodiment of the present invention. Figure 5 This is a structural block diagram of the electronic device in an embodiment of the present invention. Detailed Implementation

[0022] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0023] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0024] It should be noted that, where there is no conflict, the embodiments of the present invention and the technical features thereof can be combined with each other. In this invention, "real emotion" refers to the category of emotion actually experienced internally by an individual; "explicitly requested expression" refers to the category of expression actively presented or required to be presented externally by an individual; and "mixed expression category" refers to an ordered combination of the real emotion prediction category and the explicitly requested expression prediction category.

[0025] like Figures 1 to 3 As shown, this embodiment provides a method for facial expression recognition based on multi-task decoupling. This method can be deployed on electronic devices with image processing and neural network inference capabilities, such as servers, workstations, desktop computers, edge computing devices, or dedicated video analysis devices. The input to this method is a facial video segment, and the output includes a true emotion prediction result, an explicit desired expression prediction result, and a hybrid expression category formed by combining the two.

[0026] S1. Acquire a face video segment, preprocess the face video segment, and then input it into a multi-task decoupled recognition network; Specifically, the process begins by acquiring video segments of the faces to be identified. For each video segment, face detection or face region localization is performed, and the video frames are cropped based on the detected face locations. Subsequently, the cropped face regions are scale-normalized to ensure consistent spatial dimensions across different video samples; and the video sequences are temporally sampled to maintain a consistent number of frames input to the model.

[0027] Each video segment corresponds to a genuine emotion tag and an outward required expression tag. Both the genuine emotion tag and the outward required expression tag can be one of the six basic emotions: anger, disgust, fear, happiness, sadness, and surprise. An ordered combination of the two forms a mixed expression category. For example, when the genuine emotion is happiness and the outward required expression is sadness, the mixed expression category is different from the combination of genuine sadness and outward required expression happiness.

[0028] The sampled face video clip is represented as follows: ; In the formula, B Indicates batch size. Indicates the number of input channels. T , H , W These represent the time length, height, and width, respectively. In one embodiment, the face region of each frame is scaled to 160×160 pixels, and 16 frames are sampled from each video segment at a time step of 4. This example is only for illustrating possible implementations of the invention and does not constitute a limitation on the input size, number of sampled frames, or sampling step.

[0029] S2. The face video segment is converted into a spatiotemporal patch token sequence through a three-dimensional patch embedding layer, and the spatiotemporal patch token sequence is divided into multiple local face regions; Specifically, in obtaining the aforementioned facial video clips Then, it is input into the 3D patch embedding layer. The 3D patch embedding layer divides the face video segment into multiple spatiotemporal cubes and maps each spatiotemporal cube to a visual token, thereby obtaining a spatiotemporal patch token sequence: ; in, Indicates the token channel dimension. This represents the number of spatiotemporal patch tokens. Furthermore, the number of spatiotemporal patch tokens is determined by the partitioning results in the temporal and spatial directions: ; in, Indicates the size of the time-tube patch. This represents the spatial patch size. In one example, the video is divided into spatiotemporal cubes of size 2×16×16, resulting in 8×10×10 visual tokens. Subsequently, the spatiotemporal patch token sequence is rearranged into a 3D grid and divided into M non-overlapping local face regions.

[0030] Local facial region segmentation enables the model to capture fine-grained facial motion cues in local areas such as the eyes, eyebrows, nasolabial folds, mouth, and chin. At the same time, it establishes long-range dependencies between different local regions through subsequent global communication mechanisms.

[0031] S3. Construct a shared messenger token, a real emotion-specific messenger token, and an explicit expression-specific messenger token in each local face region; S4. Input the patch token of each local face region and the three types of messenger tokens into the multi-layer decoupled local-global attention module to obtain the shared representation, the real emotion-specific representation and the explicit expression-specific representation, respectively. S5. Obtain the real emotion prediction result based on the shared representation and the real emotion-specific representation, and obtain the explicit required expression prediction result based on the shared representation and the explicit expression-specific representation. S6. Combine the predicted result of the true emotion with the predicted result of the explicit required expression to obtain a mixed expression category.

[0032] Furthermore, three types of messenger tokens are constructed in each local region. For example, for the first... Let there be a local face region, and let the patch token within it be: ; in, This indicates the number of patch tokens within the local face region. Unlike existing local-to-global interactive Transformers that use a single task-independent region to represent the token, this embodiment constructs three semantically distinct messenger tokens within each local face region: a shared messenger token, a real emotion-specific messenger token, and an overt expression-specific messenger token. The initial enhancement sequence for this local face region is represented as follows: ; in, Indicates the first A shared messenger token for a local facial region. Indicates the first A dedicated messenger token for authenticating emotions in a localized facial region. Indicates the first A dedicated messenger token for displaying facial expressions in a specific facial region.

[0033] Shared messenger tokens are used to aggregate common facial contexts that are helpful for both real emotion recognition and explicit expression recognition tasks, such as basic facial structures and dynamic changes in the eyes and mouth that both tasks rely on. Real emotion-specific messenger tokens are used to aggregate subtle, transient, and locally distributed evidence of real emotions, such as subtle changes in the corners of the mouth, eyelid changes, or short-term movements of local muscles. Explicit expression-specific messenger tokens are used to aggregate more visually significant and stable external expression cues, such as actively presented changes in the brow area, nasolabial folds, chin texture, or mouth opening.

[0034] By constructing the three types of messenger tokens mentioned above, this embodiment can establish a clear task semantic channel between the model input and the feature extraction stage, providing a foundation for subsequent masked local attention, communication of global tokens of the same type, and asymmetric gating fusion.

[0035] The enhanced sequences of each local face region are input into the L-layer decoupled local-global attention module. Each layer of the decoupled local-global attention module sequentially performs masked local attention, communication of global tokens of the same type, shared guided local-global interaction, and feedforward network refinement.

[0036] For the l Layer Input of a local facial region First, perform masked local attention: ; in, This indicates the bulls' self-attention. Representation layer normalization, This represents a local attention mask. This local attention mask prohibits direct attentional interactions between shared messenger tokens, messenger tokens specifically for real emotions, and messenger tokens specifically for explicit facial expressions, while preserving attentional interactions between various messenger tokens and local patch tokens within the same region.

[0037] Specifically, within the same local facial region, all three types of messenger tokens can absorb information from the local patch token, but they cannot directly exchange information through local self-attention. Therefore, the shared messenger token, the real emotion-specific messenger token, and the explicit expression-specific messenger token form relatively independent semantic paths during the local feature extraction stage, preventing the more visually prominent explicit expression cues from prematurely overshadowing the real emotion cues.

[0038] After completing the masked local attention, collect messenger tokens of the same type from all local face regions, and perform global token communication of the same type for each region. For messenger token types... ,have: ; in, Indicates a shared messenger token type. A dedicated messenger token type representing genuine emotions. This indicates a special messenger token type for displaying emojis. This indicates the number of local face regions.

[0039] Then, multi-head self-attention is performed on each type of global token sequence: ; This phase enables messenger tokens of the same semantic type to exchange long-range information across local facial regions, while maintaining the separation between the three semantic paths: shared, genuine emotion, and overt expression. For example, a dedicated messenger token for genuine emotion can convey subtle information related to genuine emotion between the eye, mouth, and cheek regions; a dedicated messenger token for overt expression can convey strong visual cues related to overt expression between regions such as the brow, nasolabial folds, and chin; and a shared messenger token is used to establish a holistic facial context that is helpful for both tasks.

[0040] After completing global token communication of the same type, perform local-global interaction via shared bootstrapping. Let... The local patch token subset is Then, the local patch token is updated with cross-attention using a globally shared representation: ; in, This indicates multiple points of cross-attention. This represents a globally shared representation formed by a shared messenger token. This operation only returns the globally shared face context to the local patch token, avoiding direct backflow of global information specific to real emotions and global information specific to explicit facial expressions into the patch token stream.

[0041] By using a shared guided design, local patch tokens can obtain global face context to improve spatiotemporal modeling capabilities. At the same time, information specific to real emotions and information specific to explicit facial expressions will not be directly mixed into the patch token stream, reducing the risk of mutual contamination of task-specific information in subsequent layers.

[0042] After completing the local-to-global interaction of the shared bootstrapping mechanism, the updated messenger token and the local patch token will be reassembled: ； Then, it is refined using a feedforward network: ; in, This represents a feedforward network. After... L After decoupling the local-global attention module, the model obtains spatiotemporal facial representations with multiple layers, regions, and semantic paths.

[0043] go through L After decoupling the local-global attention module, the updated three types of messenger tokens in all local face regions are aggregated into a shared representation. Special expression of true emotions and special expressions .

[0044] in, It is obtained by aggregating the shared messenger tokens of all local face regions and is used to represent the global face context that both tasks depend on. It is obtained by aggregating real emotion-specific messenger tokens from all local face regions and is used to represent real emotion-related features; It is obtained by aggregating the explicit expression-specific messenger tokens of all local facial regions, and is used to represent the explicit expression-related features. The above aggregation can be implemented using average pooling, attention pooling, or other learnable aggregation methods.

[0045] Perform asymmetric gating fusion in the real emotion recognition branch and the explicit expression recognition branch.

[0046] For the true emotion recognition branch, first calculate the true emotion gating weights: ; Subsequently, the shared representation and the dedicated representation of true emotion are fused based on the true emotion gating weights: ; in, This represents the Sigmoid function. and This represents the learnable gating parameters in the real emotion recognition branch. This represents the fusion features of real emotion. Through the above equation, the real emotion recognition branch can adaptively select the contribution ratio between the dedicated and shared representations of real emotion, thereby preserving subtle information about real emotion while utilizing the public face context.

[0047] For the explicit facial expression recognition branch, a similar gated fusion mechanism is used, but a stopping gradient processing is performed on the shared representation before fusion: ; ; in, and This indicates the explicit requirement for learnable gating parameters in the facial expression recognition branch. This indicates the fusion features of overt facial expressions. This indicates that gradient operations are stopped. This asymmetric design allows the explicit expression recognition branch to make predictions using shared context, but prevents it from directly updating the shared representation through this fusion path, thereby limiting the excessive pull of the explicit expression task on the shared representation.

[0048] Furthermore, the fused features are input into a lightweight task adapter and an independent classifier to obtain the true emotion prediction results and the explicit required expression prediction results: ; in, Represents a realistic emotion task adapter. This indicates that the display requires an emoji task adapter. This represents a true emotion classifier. This indicates that an explicit requirement is placed on the facial expression classifier. This represents the true sentiment prediction result. This indicates an explicit requirement for predicting facial expressions.

[0049] During inference, the predicted categories of real emotion and explicit required expression are directly combined to obtain a hybrid expression category. This hybrid expression category is used for evaluation or business output and does not participate in the training objective or gradient backpropagation. In this way, the model can obtain a hybrid category jointly determined by real emotion and explicit required expression without setting an additional hybrid expression classifier.

[0050] During the model training phase, the real emotion classification loss and the explicit required expression classification loss are calculated based on the real emotion label and the explicit required expression label, respectively.

[0051] The true emotion classification loss uses cross-entropy loss, and its calculation formula is as follows: ; In the formula, This indicates the number of face video clips in a training batch. Indicates the number of true emotion categories. Indicates the first The personal face video clip belongs to the first Real labels for real emotion categories This indicates that the true emotion recognition branch predicts that the face video clip belongs to the first... The probability of each true emotion category The explicit requirement for facial expression classification loss uses cross-entropy loss, and its calculation formula is as follows: ; In the formula, This indicates the number of explicit expression categories required. Indicates the first The personal face video clip belongs to the first The explicit requirement is a realistic label for the expression category. This indicates that the explicit requirement is for the facial expression recognition branch to predict that the video clip belongs to the first... Each explicit requirement is the probability of an expression category. The overall training objective is expressed as: ; in, This represents the loss in true sentiment classification. This represents the explicit required expression classification loss. Both the true emotion classification loss and the explicit required expression classification loss can be achieved using cross-entropy loss.

[0052] While the aforementioned attention routing and gating fusion can mitigate feature-level interference, conflicting gradients may still arise between the two tasks on shared parameters. Therefore, this embodiment only applies gradient protection to shared parameters such as the 3D patch embedding layer and the shared Transformer backbone network. For shared parameters... Calculate the gradients for the two tasks respectively: ; Represents true sentiment classification loss Relative to shared parameters The gradient; This indicates that explicit expression classification loss is required. Relative to shared parameters The gradient. When and When the inner product is less than zero, it indicates that the optimization directions of the two tasks conflict. In this case, the component in the gradient of the explicit expression task that conflicts with the gradient of the real emotion task is projected and removed. ; in, Let represent the processed gradient for the explicit expression recognition task, where ε is a minimal constant to prevent the denominator from being zero. The gradients for the real emotion task and the processed gradients for the explicit expression recognition task are then merged to update the shared parameters. ; when and When the inner product of the two tasks is not less than zero, it can be considered that the optimization directions of the two tasks on the shared parameters do not conflict. In this case, the gradients of the two tasks can be directly merged to update the shared parameters. Task-specific gating, task adapters, and classifiers do not perform the above gradient protection, but are optimized independently by the corresponding task losses. This strategy can limit the excessive pull of the more visually significant expressive task on the shared representation, and protect the subtle discriminative information related to real emotions.

[0053] To visually demonstrate the recognition performance of the above-mentioned scheme of this invention, experiments were conducted on the MFED masked facial expression dataset. This dataset contains 778 video clips from 22 participants, each labeled with both a true emotion label and a required explicit expression label. One-out-of-participant cross-validation was used in the experiments. Evaluation tasks included true emotion recognition (6E), required explicit expression recognition (6R), and mixed expression recognition (36C). Evaluation metrics included accuracy (ACC) and F1 score. The experimental results are shown in Table 1. Table 1 Comparison of spoofed facial expression recognition results on the MFED dataset

[0054] It should be noted that the method of Liu et al. in Table 1 refers to Liu, Y., Zhao, K., Fu, X.: Recognition of masked facial expressions based on transfer learning and data augmentation. In: 2023 International Annual Conference on Complex Systems and Intelligent Science (CSIS-IAC). pp. 80-86. IEEE (2023); Table 1 shows the method used by Zhang et al.: Zhang, Z., Liu, B., Zhou, J., Wang, H., Liu, X., Lin, B., Chen, T.: Masked facial expression recognition based on temporaloverlap module and action unit graphconvolutional network. Journal of Visual Communication and Image Representation 107, 104398 (2025).

[0055] As shown in Table 1, this invention achieves the best results in all three tasks: real emotion recognition, explicit facial expression recognition, and mixed facial expression recognition. Compared to the suboptimal method, this invention improves the accuracy and F1 score by 0.64 and 0.17 percentage points, respectively, in the real emotion recognition task; by 4.11 and 4.06 percentage points, respectively, in the explicit facial expression recognition task; and by 2.83 and 2.44 percentage points, respectively, in the mixed facial expression recognition task.

[0056] Furthermore, ablation experiments were conducted on the main components of this invention, and the results are shown in Table 2. The decoupled local-global structure corresponds to three types of messenger tokens and decoupled attention routing. Table 2 Ablation Experiment Results of the Main Components of the Invention

[0057] Table 2 shows that multi-task learning improves real emotion recognition compared to the single-stream model. Furthermore, asymmetric gating fusion further enhances the fusion effect between real emotion features and shared context, while shared parameter gradient protection reduces optimization conflicts between tasks. The complete method achieves the best results in both real emotion recognition and mixed expression recognition, indicating that the three modules have complementary effects.

[0058] Furthermore, by visualizing the attention areas of different messenger tokens, it can be found that in samples where the required facial expressions are the same but the actual emotions are different, the attention areas of the messenger tokens dedicated to facial expressions are relatively consistent, mainly concentrated in areas related to facial expressions such as the brow area, nasolabial folds, and chin texture; the messenger tokens dedicated to actual emotions will shift their attention positions according to changes in actual emotions. For example, happy samples pay more attention to the corners of the mouth and cheeks, while surprised samples pay more attention to the raised eyebrows, widened eyes, and open mouth areas; the shared messenger tokens consistently pay attention to common areas such as the eyes and mouth, which are useful for both tasks. This phenomenon indicates that the present invention can form a clear division of token roles.

[0059] This embodiment provides a facial expression recognition device based on multi-task decoupling. This device can be implemented through software, hardware, or a combination of both, and can be deployed in electronic devices. Figure 4 As shown, Figure 4 This is a structural block diagram of the occluded facial expression recognition device based on multi-task decoupling in this invention. The device includes a preprocessing module 101, an embedding and region segmentation module 102, a messenger token construction module 103, a decoupling attention module 104, a prediction module 105, and a training optimization module 106.

[0060] The preprocessing module 101 is used to acquire the face video segment to be identified, and to perform face region cropping, scale normalization and temporal sampling on the face video segment to form the data structure shown in the embodiment of the input video segment method.

[0061] The embedding and region segmentation module 102 is used to convert face video segments into spatiotemporal patch token sequences through a three-dimensional patch embedding layer, determine the token grid size according to the temporal and spatial patch segmentation relationship shown in the method embodiment, and then divide the spatiotemporal patch token sequence into multiple local face regions.

[0062] The messenger token construction module 103 is used to construct a shared messenger token, a real emotion-specific messenger token, and an overt expression-specific messenger token in each local face region, so that the shared facial context, real emotion evidence, and overt expression cues are propagated along different semantic paths.

[0063] The decoupled attention module 104 is used to perform masked local attention, communication of global tokens of the same type, local-global interaction with shared guidance, and sequence recombination and feedforward network refinement.

[0064] The prediction module 105 is used to aggregate the updated three types of messenger tokens into a shared representation, a representation specifically for true emotions, and a representation specifically for explicit expressions, and to obtain the true emotion prediction result and the explicit expression prediction result based on these. The prediction module 105 is also used to combine the true emotion prediction category and the explicit expression prediction category during the inference phase to obtain a mixed expression category.

[0065] The training optimization module 106 is used to compute the overall training objective and perform gradient protection on shared parameters. The training optimization module only performs gradient protection on shared parameters such as the 3D patch embedding layer and the shared Transformer backbone network, while the task-specific gating, task adapter, and classifier are independently optimized by the corresponding task loss.

[0066] The specific implementation process of each module in this embodiment can be referred to the corresponding steps in the above method embodiment, and will not be repeated here.

[0067] like Figure 5 As shown in the figure, an electronic device provided by an embodiment of the present invention includes: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the following method: S1. Acquire a face video segment, preprocess the face video segment, and then input it into a multi-task decoupled recognition network; S2. The face video segment is converted into a spatiotemporal patch token sequence through a three-dimensional patch embedding layer, and the spatiotemporal patch token sequence is divided into multiple local face regions; S3. Construct a shared messenger token, a real emotion-specific messenger token, and an explicit expression-specific messenger token in each local face region; S4. Input the patch token of each local face region and the three types of messenger tokens into the multi-layer decoupled local-global attention module to obtain the shared representation, the real emotion-specific representation and the explicit expression-specific representation, respectively. S5. Obtain the real emotion prediction result based on the shared representation and the real emotion-specific representation, and obtain the explicit required expression prediction result based on the shared representation and the explicit expression-specific representation. S6. Combine the predicted result of the true emotion with the predicted result of the explicit required expression to obtain a mixed expression category.

[0068] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0069] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0070] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for facial expression recognition under occlusion based on multi-task decoupling, characterized in that, include: A face video segment is acquired, and after preprocessing, the face video segment is input into a multi-task decoupled recognition network; The face video clips are converted into a spatiotemporal patch token sequence through a 3D patch embedding layer, and the spatiotemporal patch token sequence is divided into multiple local face regions. Construct a shared messenger token, a real emotion-specific messenger token, and an explicit expression-specific messenger token in each local face region; The patch tokens of each local face region and the three types of messenger tokens are input into the multi-layer decoupled local-global attention module to obtain shared representations, real emotion-specific representations, and overt expression-specific representations, respectively. Based on the shared representation and the dedicated representation of true emotion, a prediction result of true emotion is obtained; based on the shared representation and the dedicated representation of explicit expression, a prediction result of explicit required expression is obtained. The predicted results of the true emotions are combined with the predicted results of the explicit required expressions to obtain a mixed expression category.

2. The method according to claim 1, characterized in that, The preprocessing includes cropping the face region, normalizing the scale, and sampling the temporal sequence of the face video segments, so that each face video segment corresponds to a real emotion label and an outward expression label.

3. The method according to claim 1, characterized in that, The three-dimensional patch embedding layer divides the face video segment into multiple spatiotemporal cubes and maps each spatiotemporal cube to a corresponding visual token, forming the spatiotemporal patch token sequence.

4. The method according to claim 1, characterized in that, In each local face region, the shared messenger token is used to aggregate public face context that is useful for both real emotion recognition and explicit expression recognition tasks, the real emotion-specific messenger token is used to aggregate subtle facial movement cues related to real emotions, and the explicit expression-specific messenger token is used to aggregate salient facial cues related to explicit expression requirements.

5. The method according to claim 1, characterized in that, The decoupled local-global attention module includes a masked local attention step; in the masked local attention step, direct attention interaction between shared messenger tokens, real emotion-specific messenger tokens, and explicit expression-specific messenger tokens is prohibited, while attention interaction between various messenger tokens and patch tokens within the same local face region is preserved.

6. The method according to claim 5, characterized in that, The decoupled local-global attention module also includes a global token communication step of the same type; in the global token communication step of the same type, shared messenger tokens, real emotion-specific messenger tokens and explicit expression-specific messenger tokens are collected from different local face regions respectively, and the same type of messenger tokens are used to perform global information interaction between different local face regions.

7. The method according to claim 6, characterized in that, The decoupled local-global attention module also includes a shared-guided local-global interaction step; in the shared-guided local-global interaction step, only the global shared representation formed by the shared messenger token is used to update the patch token in the local face region, so as to restrict the direct backflow of real emotion-specific information and overt expression-specific information to the patch token stream.

8. The method according to claim 1, characterized in that, Obtaining true emotion prediction results and explicit demand expression prediction results includes: In the real emotion recognition branch, the shared representation and the real emotion-specific representation are fused by gating to obtain the real emotion fusion feature; In the explicit expression recognition branch, the shared representation and the explicit expression-specific representation are fused by gating to obtain the explicit expression fusion feature; Specifically, during backpropagation training, stopping gradient processing is performed on the shared representation of the input explicit expression recognition branch.

9. The method according to any one of claims 1 to 8, characterized in that, The training of the multi-task decoupling recognition network includes: The sum of the loss for classifying true emotions and the loss for classifying explicit desired expressions is used as the overall training objective. Gradient protection is performed on the shared parameters of the 3D patch embedding layer and the shared Transformer backbone network; When the inner product of the gradient of the real emotion recognition task on the shared parameters and the gradient of the explicit expression recognition task on the shared parameters is less than zero, the components in the gradient of the explicit expression recognition task that conflict with the gradient of the real emotion recognition task are removed, and the processed gradient of the explicit expression recognition task and the gradient of the real emotion recognition task are merged and used to update the shared parameters. Among them, the task-specific gating, task adapter, and classifier are independently optimized by the corresponding task loss.

10. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the occluded facial expression recognition method based on multi-task decoupling as described in any one of claims 1 to 9.

Images